From 9de343b3fe2e2d4dc84fc8386d9ce04d553f39d6 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 16 Nov 2026 09:51:09 -0600 Subject: [PATCH] perf(recovery): keep a repeatedly deferring task's saved row; stop agent settlement from re-checking its backlog (#21905) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The task recovery queues check assigned tasks for work that has no live run and no queued wake. Since #21872 they load only changed rows, and they save pending deferrals so a replaced server keeps their deadlines > - Two costs remained. A running task's check defers to now + 30 s, and each new deadline rewrote its saved deferral row: about two writes a minute per running task per queue > - Also, two writers that settle an agent after its runs end set the agent's status and `updated_at` but not `last_heartbeat_at`. The changed-rows agent rule then reads the write as a pause or resume and re-checks all of that agent's tasks for about 5 minutes > - This pull request keeps a repeatedly deferring task's saved row until its deadline is well past, and makes both writers record a heartbeat as `finalizeAgentStatus` does > - The benefit is near-zero deferral writes for running tasks and no whole-backlog re-check after an abandoned or stranded run settles an agent ## Linked Issues or Issue Description Refs #21872, #21861 **What happened?** 1. In `recovery/work.ts`, a deferring check upserted its `issue_recovery_deferrals` row whenever the deadline changed. A deferral to now + 30 s changes on every check, so a task deferring for 20 minutes wrote its row 41 times. At 1,000 concurrent agents that is roughly 30 to 70 writes a second. 2. `execution-control-reconciliation.ts` (abandoned run past its execution-control deadline) and `heartbeat/agent-status.ts` `reconcileStrandedAgentStatuses` (agent left "running" with no live run) set the agent to idle with `updated_at` but without `last_heartbeat_at`. The agent clause of `changedTaskCondition` matches an agent that is not running and whose `updated_at` is newer than its last heartbeat. So each such write made the backstop re-check every task of that agent for about 5 minutes. **Expected behavior** A task that keeps deferring writes its row a small fixed number of times. Settling an agent after its runs end does not re-check the agent's other tasks; the run's own task is still checked promptly. **Steps to reproduce** 1. Defer one task's check to now + 30 s on every check for 20 simulated minutes and count inserts into `issue_recovery_deferrals`: 41. 2. Seed an agent left "running" with 140 settled reviews, settle it with `reconcileAbandonedExecutionControl` or `reconcileStrandedAgentStatuses`, and count stranded checks of those reviews over the next backstop: 280 per agent. **Paperclip version or commit** `master` at b4fa1c7fb1. ## What Changed - `recovery/work.ts`: a deferring check skips the upsert while this server knows a saved row for the task whose deadline is less than `DEFERRAL_REFRESH_MS` (5 minutes) past. It refreshes the row after that. The refresh comes before the 10-minute stale prune margin, so a still-deferring task's row never looks stale. A boot re-checks every task with a row whatever its deadline, so the exact deadline only matters to the conditional deletes and the stale prune, which still match the value this server read or wrote. A cleared row, a failed write or a reload still drop the server's memory of the row, so the next deferral writes it again. - `execution-control-reconciliation.ts` and `heartbeat/agent-status.ts`: settling an agent after its runs end sets `lastHeartbeatAt` with `updatedAt`, as `finalizeAgentStatus` already does, through a new `settledHeartbeat(now)`. It records the settlement time only when the agent already has a heartbeat; an agent with none keeps none, so the first real heartbeat stays first (the existing "repairs a shutdown flag without inventing a heartbeat" test still passes). - `recovery/changed-tasks.ts`: the comment now states the contract: every write that settles an agent's status after its runs end sets `last_heartbeat_at`. - Tests: a write-rate census in `issue-recovery-saved-deferrals.test.ts`, and an agent-settle census and guard in `issue-recovery-company-reload-census.test.ts`. Why fix the writers, not the rule: the agent rule's discriminator already is `last_heartbeat_at` against `updated_at` ("a run's finish records a heartbeat"). Pauses, resumes, holds and configuration writes leave `last_heartbeat_at` alone; run settlement records one. No other column tells a settle-to-idle apart from a resume to idle, because both end with status idle and a new `updated_at`. Since #21872, lifecycle transitions also send their own agent hint, but configuration writes to an idle agent still rely on this rule (the #21861 guard). So the sound fix is for the two run-settling writers to follow the same contract as `finalizeAgentStatus`. Recording the heartbeat at settlement has the same effect on agent timers as a normal run finish. ## Verification Red/green: | Census | Before (master code) | After | | --- | --- | --- | | One task deferring every 30 s for 20 minutes: deferral writes | 41 (41 checks) | 4 (41 checks) | | Same task: saved deadline behind now after 20 minutes | 0 (rewritten each check) | 3 min (refreshed; the test fails above 6 min, and fails with the refresh removed: 1 write, 19.5 min) | | Two agents (140 settled reviews each) settled by an abandoned-run reconcile and a stranded-status reconcile: backlog checks over the next backstops | 560 | 0 | | Same census: statements | 12,570 | 2,507 | Guards: - The abandoned run's task is checked within 1 minute of the reconcile (passes before and after). - The three deferral tests from #21872 still pass: replacement mid-wait checks at the original deadline, the two-server interleaving keeps server B's deadline, and the stale row of a task that left the queue is removed. Focused suites, each file run alone in the foreground: `issue-recovery-company-reload-census.test.ts` 8 passed (38 s), `issue-recovery-saved-deferrals.test.ts` 6 passed (14 s), `heartbeat/agent-status.test.ts` and `execution-control-reconciliation.test.ts` 60 passed, `issue-recovery-idle-backstop.test.ts` 4 passed. An 18-file run earlier: 516 passed; the 4 failures were 1 test that this PR then fixed (heartbeat for agents with none) and 3 in `issue-recovery-idle-backstop.test.ts`, which pass alone (see Risks). Server `tsc --noEmit`: passed. `pnpm verify`: not run; required CI covers the broad suites. ## Risks - A task that keeps deferring may have a saved deadline up to 5 minutes old. Only the stale prune reads it, and it deletes a row only after the task left the queue and the deadline is 10 minutes past, so a still-deferring task's row is refreshed first. - If another server deletes the row (its check stopped deferring and it matched the same deadline) while this server keeps deferring, this server rewrites the row at its next refresh (at most about 5 minutes) instead of at its next check. Within that window a crash of this server would leave the task to its successor's next changed-rows or full pass. This needs the two servers to disagree about the same task, which only happens through facts held in one server's memory. - `last_heartbeat_at` now also moves when an abandoned or stranded run settles the agent. Agent timers then count from that time, as they do after a normal run finish. - An agent that never recorded a heartbeat still has its whole backlog re-checked once when such a writer settles it. This is rare: that agent has not finished a run yet. - Not from this PR: `issue-recovery-idle-backstop.test.ts` > "picks up a finished run, a user message, a raised budget and a resumed agent within a minute" hung once and failed 3 tests when run with other recovery files in one process. It passes alone. Its waits have no time limits, and its fixture agents can take on-demand wakes, which is the hang pattern that #21867 removed from the agent-row census. It needs the same treatment in a separate change. - No schema change. Nothing on the run-start path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- ...sue-recovery-company-reload-census.test.ts | 62 +++++++++++++++++++ .../issue-recovery-saved-deferrals.test.ts | 33 +++++++++- .../execution-control-reconciliation.ts | 6 +- server/src/services/heartbeat/agent-status.ts | 12 +++- server/src/services/recovery/changed-tasks.ts | 6 +- server/src/services/recovery/work.ts | 9 ++- 6 files changed, 123 insertions(+), 5 deletions(-) diff --git a/server/src/__tests__/issue-recovery-company-reload-census.test.ts b/server/src/__tests__/issue-recovery-company-reload-census.test.ts index 2137bd0fab9..f4888725852 100644 --- a/server/src/__tests__/issue-recovery-company-reload-census.test.ts +++ b/server/src/__tests__/issue-recovery-company-reload-census.test.ts @@ -11,6 +11,7 @@ import { documentService } from "../services/documents.js"; import { issueService } from "../services/issues.js"; import { createHeartbeatAgentStatus } from "../services/heartbeat/agent-status.js"; import { notifyExecutionRunWork } from "../services/execution-control-notifications.js"; +import { reconcileAbandonedExecutionControl } from "../services/execution-control-reconciliation.js"; import { describeIssueRecovery, issueRecoveryFixture } from "./issue-recovery.test-fixture.js"; vi.hoisted(() => { process.env.PAPERCLIP_DEBUG_WORK_SIGNALS = "stranded-assignments:1,assigned-work-invariant:1"; }); @@ -158,6 +159,67 @@ describeIssueRecovery("company hints and notification reloads (census)", () => { const checksOf = (events: Recorded[], issueId: string, after: number) => events.filter(event => event.topic === "stranded-assignments" && event.name === "check" && event.fields.issueId === issueId && event.at >= after); + /** An agent left "running" with a large settled backlog: one abandoned + * run past its execution-control deadline, or (stranded) no run at all. */ + async function settlingAgent(f: Awaited>, name: string, abandoned: boolean) { + const old = new Date(Date.now() - 2 * HOUR); + const [agent] = await state.db.insert(agents).values({ companyId: f.companyId, name, role: "engineer", status: "running", + adapterType: "process", adapterConfig: { command: process.execPath, args: ["-e", "0"], cwd: process.cwd() }, + runtimeConfig: { heartbeat: { enabled: false, intervalSec: 0, wakeOnDemand: false } }, + createdAt: old, updatedAt: old, lastHeartbeatAt: old }).returning({ id: agents.id }); + const tasks = await state.db.insert(issues).values(Array.from({ length: 140 }, (_, index) => ({ companyId: f.companyId, + title: `Settled ${index}`, status: "in_review", assigneeAgentId: agent!.id, createdAt: old, updatedAt: old }))).returning({ id: issues.id }); + let runTask: string | null = null; + if (abandoned) { + const [task] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Abandoned run's task", status: "in_progress", + assigneeAgentId: agent!.id, createdAt: old, updatedAt: old }).returning({ id: issues.id }); + runTask = task!.id; + const [run] = await state.db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: agent!.id, status: "running", + startedAt: old, updatedAt: old, contextSnapshot: { issueId: runTask }, executionControlDeadlineAt: new Date(Date.now() - MINUTE) }) + .returning({ id: heartbeatRuns.id }); + await state.db.update(issues).set({ executionRunId: run!.id, updatedAt: old }).where(eq(issues.id, runTask)); + } + return { agentId: agent!.id, tasks: tasks.map(row => row.id), runTask }; + } + + it("does not re-check an agent's settled backlog when an abandoned or stranded run settles the agent", async () => { + const f = await boot(); + const abandoned = await bounded(settlingAgent(f, "Abandoned worker", true), () => "the abandoned agent to be seeded"); + const stranded = await bounded(settlingAgent(f, "Stranded worker", false), () => "the stranded agent to be seeded"); + await advance(MINUTE, f.tick); + f.events.length = 0; + const before = state.meter.read(); + await bounded(reconcileAbandonedExecutionControl(state.db, new Date()), () => "the abandoned run to be reconciled"); + await advance(ISSUE_RECOVERY_BACKSTOP_MS + 2 * MINUTE, f.tick); + await bounded(createHeartbeatAgentStatus(state.db).reconcileStrandedAgentStatuses(() => new Set(), + { companyId: f.companyId, agentId: stranded.agentId }), () => "the stranded agent status to be reconciled"); + await advance(ISSUE_RECOVERY_BACKSTOP_MS + 2 * MINUTE, f.tick); + const [abandonedAgent, strandedAgent] = await Promise.all([abandoned.agentId, stranded.agentId].map(id => + state.db.select({ status: agents.status }).from(agents).where(eq(agents.id, id)).then(rows => rows[0]?.status))); + const backlog = new Set([...abandoned.tasks, ...stranded.tasks]); + const checks = f.events.filter(event => event.name === "check"); + const result = { backlog: backlog.size, abandonedAgent, strandedAgent, + backlogChecks: checks.filter(event => backlog.has(String(event.fields.issueId))).length, + runTaskChecks: checks.filter(event => event.fields.issueId === abandoned.runTask).length, + byWake: byWake(f.events), statements: state.meter.read().statements - before.statements }; + console.info(`agent-settle census: ${JSON.stringify(result)}`); + expect(abandonedAgent).not.toBe("running"); + expect(strandedAgent).not.toBe("running"); + expect(result.backlogChecks).toBe(0); + }, TEST_TIMEOUT_MS); + + it("still checks an abandoned run's task promptly when the run is reconciled", async () => { + const f = await boot(); + const abandoned = await bounded(settlingAgent(f, "Abandoned worker", true), () => "the abandoned agent to be seeded"); + await advance(MINUTE, f.tick); + const reconciledAt = Date.now(); + await bounded(reconcileAbandonedExecutionControl(state.db, new Date()), () => "the abandoned run to be reconciled"); + await advance(2 * MINUTE, f.tick); + const [checked] = checksOf(f.events, abandoned.runTask!, reconciledAt); + expect(checked, "the abandoned run's task was not checked after reconciliation").toBeDefined(); + expect(checked!.at - reconciledAt).toBeLessThanOrEqual(MINUTE); + }, TEST_TIMEOUT_MS); + it("does not re-check settled tasks for company hints, reloads or server replacements", async () => { const f = await boot(); const before = state.meter.read(); diff --git a/server/src/__tests__/issue-recovery-saved-deferrals.test.ts b/server/src/__tests__/issue-recovery-saved-deferrals.test.ts index 15453e20fd2..d6343c09ac1 100644 --- a/server/src/__tests__/issue-recovery-saved-deferrals.test.ts +++ b/server/src/__tests__/issue-recovery-saved-deferrals.test.ts @@ -51,7 +51,8 @@ describeIssueRecovery("saved task recovery passes and deferrals", () => { } /** One server: a coordinator and the recovery queues, with recorded checks. */ - async function server(behavior: { deferUntil: Map; suppressed: () => boolean; checks: Array<{ id: string; at: number }> }) { + async function server(behavior: { deferUntil: Map; suppressed: () => boolean; checks: Array<{ id: string; at: number }>; + rolling?: Set }) { const coordinator = createDeliveryWorkCoordinator({ owner: state.db, canRun: () => true, canReconcile: () => true, onError: error => { throw error; } }); running.add(coordinator); @@ -65,6 +66,7 @@ describeIssueRecovery("saved task recovery passes and deferrals", () => { behavior.checks.push({ id: scope.issueId, at: Date.now() }); const until = behavior.deferUntil.get(scope.issueId); if (until !== undefined && until > Date.now()) scope.onDeferredDeadline(until); + if (behavior.rolling?.has(scope.issueId)) scope.onDeferredDeadline(Date.now() + 30_000); }; const work = registerIssueRecoveryWork(state.db, coordinator, { stranded, orphanBlockers: async () => {}, recoveryActions: async () => {}, @@ -180,6 +182,35 @@ describeIssueRecovery("saved task recovery passes and deferrals", () => { expect(await bounded(saved(), () => "the deferral to be read")).toHaveLength(0); }, TEST_TIMEOUT_MS); + it("writes a task's deferral a few times, not on every check, while it defers every 30 seconds", async () => { + const f = await bounded(seed(10), () => "the fixture to be seeded"); + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] }); + const target = f.tasks[2]!; + const behavior = { deferUntil: new Map(), rolling: new Set([target]), suppressed: () => false, + checks: [] as Array<{ id: string; at: number }> }; + const statements: string[] = []; + const priorDebug = state.db.$client.options.debug; + state.db.$client.options.debug = (_connection, query) => { statements.push(query); }; + try { + await server(behavior); + await advance(20 * MINUTE); + } finally { state.db.$client.options.debug = priorDebug; } + const checks = behavior.checks.filter(check => check.id === target).length; + const writes = statements.filter(query => /insert into issue_recovery_deferrals/i.test(query)).length; + // The saved deadline still moves forward, so the row of a task that keeps + // deferring never looks stale to the prune (10 minutes past its deadline). + const [row] = await bounded(state.db.select({ dueAt: issueRecoveryDeferrals.dueAt }).from(issueRecoveryDeferrals) + .where(and(eq(issueRecoveryDeferrals.queue, "stranded-assignments"), eq(issueRecoveryDeferrals.issueId, target))), + () => "the deferral to be read"); + const savedBehind = row ? Date.now() - row.dueAt.getTime() : null; + console.info(`rolling-deferral census: ${JSON.stringify({ minutes: 20, checks, deferralWrites: writes, savedBehindMs: savedBehind })}`); + expect(checks).toBeGreaterThanOrEqual(30); + expect(writes).toBeLessThanOrEqual(5); + expect(writes).toBeGreaterThan(1); + expect(savedBehind, "the saved deferral was not kept").not.toBeNull(); + expect(savedBehind!).toBeLessThan(6 * MINUTE); + }, TEST_TIMEOUT_MS); + it("checks every task again after scheduling suppression, even with a recent saved full pass", async () => { const f = await bounded(seed(30), () => "the fixture to be seeded"); vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] }); diff --git a/server/src/services/execution-control-reconciliation.ts b/server/src/services/execution-control-reconciliation.ts index 3bfc73d77c9..87ca219cfa8 100644 --- a/server/src/services/execution-control-reconciliation.ts +++ b/server/src/services/execution-control-reconciliation.ts @@ -9,6 +9,7 @@ import { notifyExecutionRunWork } from "./execution-control-notifications.js"; import { randomUUID } from "node:crypto"; import { logger } from "../middleware/logger.js"; import { and, asc, eq, isNotNull, lte, sql } from "drizzle-orm"; +import { settledHeartbeat } from "./heartbeat/agent-status.js"; import { agents, heartbeatRuns, @@ -292,7 +293,10 @@ export async function reconcileAbandonedExecutionControl( } await tx .update(agents) - .set({ status: "idle", updatedAt: now }) + // A run settling records a heartbeat, as finalizeAgentStatus does, so + // the changed-rows agent rule (recovery/changed-tasks.ts) does not + // read it as a pause or resume and re-check the agent's whole backlog. + .set({ status: "idle", lastHeartbeatAt: settledHeartbeat(now), updatedAt: now }) .where( and( eq(agents.id, run.agentId), diff --git a/server/src/services/heartbeat/agent-status.ts b/server/src/services/heartbeat/agent-status.ts index 0d03f6ac828..511f1da9909 100644 --- a/server/src/services/heartbeat/agent-status.ts +++ b/server/src/services/heartbeat/agent-status.ts @@ -75,6 +75,14 @@ export function settledAgentWorkCondition(liveRunIds: ReadonlySet, now: OR c.receipt ? 'baseline' OR c.receipt->>'cleanupPending' = 'true'))`; } +/** The heartbeat a write records when it settles an agent after its runs end + * without finalizeAgentStatus: the settlement time, so the changed-rows agent + * rule does not re-check the agent's backlog, but never a first heartbeat for + * an agent that has none (that stays the first real one). */ +export function settledHeartbeat(now: Date) { + return sql`case when ${agents.lastHeartbeatAt} is null then null else ${now.toISOString()}::timestamptz end`; +} + export function createHeartbeatAgentStatus(db: Db) { async function finalizeAgentStatus( agentId: string, @@ -153,8 +161,10 @@ export function createHeartbeatAgentStatus(db: Db) { .orderBy(desc(heartbeatRuns.finishedAt), desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1); const nextStatus = settledStatus(lastRun?.status); await notifyAttentionRetentionWork(tx, { companyId: candidate.companyId, sourceKind: "agent_error_alert" }); + // Settling the agent's runs records a heartbeat, as finalizeAgentStatus + // does; see the agent rule in recovery/changed-tasks.ts. const [agent] = await tx.update(agents).set({ status: nextStatus, - errorReason: nextStatus === "error" ? errorReason(lastRun?.error) : null, updatedAt: now, + errorReason: nextStatus === "error" ? errorReason(lastRun?.error) : null, lastHeartbeatAt: settledHeartbeat(now), updatedAt: now, }).where(and(eq(agents.id, candidate.id), eq(agents.companyId, candidate.companyId), eq(agents.status, "running"), eq(agents.lifecycleState, "ready"))).returning(); await logActivity(tx as unknown as Db, { diff --git a/server/src/services/recovery/changed-tasks.ts b/server/src/services/recovery/changed-tasks.ts index df6cdbb9a66..0cbdd1e090f 100644 --- a/server/src/services/recovery/changed-tasks.ts +++ b/server/src/services/recovery/changed-tasks.ts @@ -6,7 +6,11 @@ import { issues } from "@paperclipai/db"; * tasks need recovery (a pause, resume, hold or configuration change). Run * lifecycle writes are left out: a run's finish records a heartbeat, and a run's * start flips the agent to running, which makes none of its other tasks - * stranded. Every run start would otherwise re-check all of the agent's tasks + * stranded. Every write that settles an agent's status after its runs end + * (finalizeAgentStatus, abandoned-run and stranded-status reconciliation) + * must set last_heartbeat_at with updated_at (see settledHeartbeat), or this + * clause re-checks all of the agent's tasks. An agent with no heartbeat yet + * keeps none, so its settlement still re-checks its (new) backlog once. Every run start would otherwise re-check all of the agent's tasks * for two backstop windows. */ export function changedTaskCondition(since: Date | SQL) { const at = since instanceof Date ? sql`${since.toISOString()}::timestamptz` : since; diff --git a/server/src/services/recovery/work.ts b/server/src/services/recovery/work.ts index 9fc274f2b70..5cff1348a9b 100644 --- a/server/src/services/recovery/work.ts +++ b/server/src/services/recovery/work.ts @@ -19,6 +19,11 @@ type Scope = { companyId: string; issueId: string; onDeferredDeadline: (at: numb const PAGE_SIZE = 100; /** How long past its deadline a saved deferral of a task that left its queue is kept. */ const STALE_DEFERRAL_MS = 10 * 60_000; +/** How long past its saved deadline a task that keeps deferring leaves its row as is. + * A boot re-checks every task with a row whatever its deadline, so only the + * stale prune reads it; refreshing well before STALE_DEFERRAL_MS keeps the + * row of a task that is still deferring from ever looking stale. */ +const DEFERRAL_REFRESH_MS = STALE_DEFERRAL_MS / 2; export const resolvedDependencyCandidate = sql`exists (select 1 from ${issueRelations} r where r.company_id = ${issues.companyId} and r.related_issue_id = ${issues.id} and r.type = 'blocks' offset 0) @@ -181,7 +186,9 @@ export function registerIssueRecoveryWork( async deferral(id: string, companyId: string, dueAt: number | null): Promise { const known = deferred.get(id); if (dueAt !== null) { - if (known === dueAt) return true; + // A task that defers again and again (now + 30 s while a run lives) + // keeps its saved row until that row's deadline is well past. + if (known !== undefined && (known === dueAt || Date.now() < known + DEFERRAL_REFRESH_MS)) return true; deferred.delete(id); const saved = await db.execute(sql`insert into issue_recovery_deferrals (queue, issue_id, company_id, due_at) values (${queue}, ${id}::uuid, ${companyId}::uuid, ${at(dueAt)})