From f4e025a1cf749f4416f6646bbe4f83c3e92df653 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 14 Nov 2026 19:41:35 -0600 Subject: [PATCH] perf(recovery): wait for an assignee that cannot run instead of retrying its continuation every pass (#21687) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run stops and recovery decides to continue the task in a fresh run, it records a pending continuation. The execution-continuation worker delivers it with `heartbeat.wakeup`. > - When the task's assignee cannot take work (paused, terminated, waiting for hire approval, or an invalid reporting chain), the wake refuses. The worker recorded only an "attempted at" time and tried again on the next 15-second pass, forever. > - Each refused wake also saved a new skipped wake row. An idle server with one such task did about 13,000 statements and 390 wake attempts every 2 hours. > - This pull request makes delivery check the assignee with the same rule the wake uses. When the assignee cannot run, the continuation waits on a stored retry time and is re-admitted at once when the assignee can run again. > - The benefit is that an idle server does almost no work for these tasks, and a resumed agent still gets its continuation within seconds. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** `deliverReconciledExecutions` (`server/src/services/execution-recovery-resolution.ts`) offers each pending continuation (`continuationDelivery: "pending"`) to `heartbeat.wakeup`. When the assignee is not invokable, the wake throws "Agent is not invokable in its current state" and saves a skipped `agent_wakeup_requests` row. The worker catches the refusal, records `continuationDeliveryAttemptedAt`, and the continuation schedule (`executionControlSchedule.continuation`) asks for another pass in 15 seconds because a pending row exists. Nothing ever changes, so the loop never ends. Measured on real PostgreSQL with the real coordinator and the real wake, on a fake clock, for one pending continuation over 2 idle hours: | Assignee status | Wake attempts (master) | Statements (master) | | --- | --- | --- | | paused | 380 | about 12,980 | | terminated | 376 | about 12,820 | | pending_approval | 395 | about 13,000 | An agent in `error` is not affected: it is invokable, and the wake delivers the continuation on the first pass. Only the statuses and conditions that make an agent not invokable cause the loop. **Expected behavior** A continuation that cannot be delivered because its assignee cannot run does not try again every pass. It waits for a real event (the agent can run again) or a spaced backstop time. **Steps to reproduce** 1. Create a task in `in_progress` assigned to an agent, with a failed run. 2. Record a pending continuation for that run, as the fresh-continuation fallback does. 3. Pause, terminate or un-approve the agent. 4. Run the execution-continuation worker. It calls the wake about every 18 seconds and saves a skipped wake row each time. **Paperclip version or commit** master at dcba9628c7. ## What Changed - `server/src/services/execution-recovery-resolution.ts`: before the wake, delivery calls `deferForUnwakeableAssignee`. When the assignee cannot run, delivery skips the wake. The pending-page query skips continuations whose stored retry time is in the future. The file stays under 1,000 lines (997). - `server/src/services/recovery/continuation-delivery-attempt.ts`: `deferForUnwakeableAssignee` reads the assignee `FOR SHARE` and evaluates it with `evaluateAgentInvokabilityFromDb`, the rule `heartbeat.wakeup` uses (including the setup exception). In the same transaction it stores `continuationDeliveryRetryAt` and a wait count on the recovery action. The wait is 1 minute and doubles per refusal up to 1 hour. The row lock orders the wait against a concurrent status change, so a resume cannot be missed. `continuationDeliveryDue` is the shared "due now" condition. `readmitAgentContinuations` notifies the continuation worker, then clears the stored time for one agent's waiting continuations. - `server/src/services/execution-control-work.ts`: the continuation schedule asks for a pass in 15 seconds only when a continuation is due. Otherwise it returns the earliest stored retry time. - `server/src/services/agent-lifecycle.ts`: when an agent's status changes from not invokable to invokable (resume, hire approval), the existing hook also calls `readmitAgentContinuations`. - `server/src/__tests__/continuation-delivery-unwakeable-agent-idle.test.ts`: five new tests on real PostgreSQL. What the user sees does not change. The continuation stays pending and the task stays assigned. The stranded-assignment check and the assigned-work invariant still own the visible outcome for a task whose assignee cannot run (the existing "can't run right now" escalation, or the "agent paused" wait). When the agent can run again, the continuation is delivered as before. ## Verification - `pnpm exec vitest run src/__tests__/continuation-delivery-unwakeable-agent-idle.test.ts` (in `server/`): - `waits instead of retrying every pass while the assignee is paused / terminated / pending_approval`: 2 idle hours after the first minute. - master: 380, 376 and 395 wake attempts (red). - this branch: 0 wake attempts, 68 statements, no skipped wake rows, the task stays `in_progress` and assigned (green). - `delivers the continuation within seconds once the paused assignee is resumed`: after 3 idle hours, the agent is resumed through the real agent lifecycle. Within 20 seconds the worker calls the wake once and the record says `continuationDelivery: "delivered"` (green). On master this test fails because the wake was already called during the idle hours. - `does not store a wait when the assignee is resumed while delivery is checking it`: a resume transaction holds the agent row while delivery checks. With the row lock, delivery sees the resumed agent and stores no wait (green). Without the lock, it stores a wait (red). - Neighbouring suites pass locally: `automatic-run-brake-recovery-passes`, `recovery-wake-herd`, `execution-control-work`, `fresh-continuation-move-forward-offer`, `fresh-continuation-retry-delivery`, `fresh-continuation-fallback`, `issue-recovery-actions`, `resource-lifecycle-events` (345 tests). - `npx tsc --noEmit -p server/tsconfig.json` passes. ## Risks - Low risk. Only a continuation whose assignee is not invokable by the wake's own rule waits. Every other refusal keeps the current behavior. - A repair that makes the agent invokable without a status change (for example a fixed reporting chain) does not fire the re-admit hook. The backstop picks it up within at most 1 hour. - A task reassigned while its continuation waits is invalidated on the next backstop attempt instead of the next 15-second pass. The new assignee gets its own assignment wake, so nothing waits on that. - No schema or API change. The new fields live in the existing `evidence` JSON. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, local PostgreSQL test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Related: #21621, #21674, #21675, #21676, #21617. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- ...ion-delivery-unwakeable-agent-idle.test.ts | 139 ++++++++++++++++++ server/src/services/agent-lifecycle.ts | 2 + server/src/services/execution-control-work.ts | 12 +- .../services/execution-recovery-resolution.ts | 4 +- .../recovery/continuation-delivery-attempt.ts | 70 ++++++++- 5 files changed, 215 insertions(+), 12 deletions(-) create mode 100644 server/src/__tests__/continuation-delivery-unwakeable-agent-idle.test.ts diff --git a/server/src/__tests__/continuation-delivery-unwakeable-agent-idle.test.ts b/server/src/__tests__/continuation-delivery-unwakeable-agent-idle.test.ts new file mode 100644 index 00000000000..e6b41276ca4 --- /dev/null +++ b/server/src/__tests__/continuation-delivery-unwakeable-agent-idle.test.ts @@ -0,0 +1,139 @@ +import { setImmediate as immediate } from "node:timers/promises"; +import { eq } from "drizzle-orm"; +import { afterEach, expect, it, vi } from "vitest"; +import { agentWakeupRequests, agents, heartbeatRuns, issueRecoveryActions, issues } from "@paperclipai/db"; +import { configureAgentLifecycle, createAgentLifecycle } from "../services/agent-lifecycle.js"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; +import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js"; +import { executionControlSchedule, registerExecutionControlWork } from "../services/execution-control-work.js"; +import { deliverReconciledExecutions } from "../services/execution-recovery-resolution.js"; +import { deferForUnwakeableAssignee, readmitAgentContinuations } from "../services/recovery/continuation-delivery-attempt.js"; +import { describeIssueRecovery, issueRecoveryFixture } from "./issue-recovery.test-fixture.js"; + +const MINUTE = 60_000; +const HOUR = 60 * MINUTE; + +describeIssueRecovery("continuation delivery to an agent that cannot be woken", () => { + const state = issueRecoveryFixture(); + const passes = new Set>(); + let lifecycleWorker: ReturnType | undefined; + const lifecycle = () => lifecycleWorker ??= configureAgentLifecycle(state.db, + { requiredPluginIds: async () => [], runHost: async () => "complete", runPlugin: async () => "complete" }); + afterEach(async () => { await lifecycleWorker?.stop(); lifecycleWorker = undefined; }); + + async function seed(status: "paused" | "terminated" | "pending_approval") { + const f = await state.fixture(); + const db = state.db; + if (status === "paused") { await createAgentLifecycle(db).pauseAgent(f.agentId); await lifecycle().process(f.agentId); } + else await db.update(agents).set({ status }).where(eq(agents.id, f.agentId)); + expect((await db.select().from(agents).where(eq(agents.id, f.agentId)))[0]!.status).toBe(status); + const [task] = await db.insert(issues).values({ companyId: f.companyId, title: "Continue the work", status: "in_progress", + assigneeAgentId: f.agentId }).returning(); + const [source] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "failed", + errorCode: "adapter_failed", startedAt: new Date(Date.now() - 3 * MINUTE), finishedAt: new Date(Date.now() - 2 * MINUTE), + contextSnapshot: { issueId: task!.id } }).returning(); + const recordedAt = new Date().toISOString(); + const [action] = await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: task!.id, + kind: "active_run_watchdog", status: "resolved", outcome: "restored", resolvedAt: new Date(), ownerType: "system", + returnOwnerAgentId: f.agentId, cause: "legacy_execution_requires_reconciliation", fingerprint: `legacy-execution:${source!.id}`, + nextAction: "Continue the task.", attemptCount: 1, + evidence: { runId: source!.id, originalFailureCode: "adapter_failed", continuationDelivery: "pending", + freshContinuation: { schema: "paperclip.fresh-continuation.v1", sourceRunId: source!.id, trigger: "recovery_refused", + delivery: "recorded", recordedAt, isolation: "writer_ended", stopProof: "writer_ended" }, + freshContinuationSettlement: { recordedAt }, + executionReconciliation: { runId: source!.id, providerStopped: true, providerStopProof: "writer_ended", + actionOutcome: "mixed", outcomeEvidence: "The run stopped; the continuation starts from the last saved work.", + actorId: "fresh-continuation", recordedAt } } }).returning(); + return { ...f, taskId: task!.id, actionId: action!.id }; + } + + async function start(queueOnly = false) { + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] }); + const coordinator = createDeliveryWorkCoordinator({ owner: state.db, canRun: () => true, canReconcile: () => true, + onError: error => state.errors.push(error) }); + state.coordinator = coordinator; + const wake = vi.fn(async (agentId: string, opts: Parameters[1]) => { + if (!queueOnly) return state.heartbeat.wakeup(agentId, opts); + const [agent] = await state.db.select().from(agents).where(eq(agents.id, agentId)); + const [run] = await state.db.insert(heartbeatRuns).values({ companyId: agent!.companyId, agentId, invocationSource: "automation", + triggerDetail: "system", status: "queued", contextSnapshot: opts?.contextSnapshot ?? {} }).returning(); + return run!; + }); + const work = registerExecutionControlWork(coordinator, [{ queue: DELIVERY_QUEUES.executionContinuation, run: () => { + const pass = deliverReconciledExecutions(state.db, wake as never); passes.add(pass); + void pass.then(() => passes.delete(pass), () => passes.delete(pass)); + return pass; + }, next: executionControlSchedule(state.db).continuation }]); + await work.ready; + return wake; + } + + async function advance(ms: number) { + const until = Date.now() + ms; + while (Date.now() < until) { + await vi.advanceTimersByTimeAsync(Math.min(1_000, until - Date.now())); + await Promise.allSettled([...passes]); await immediate(); + } + } + + async function countStatements(work: () => Promise) { + const statements: string[] = [], prior = state.db.$client.options.debug; + state.db.$client.options.debug = (_connection, query) => { statements.push(query); }; + try { await work(); } finally { state.db.$client.options.debug = prior; } + return statements; + } + + const evidence = async (id: string) => + (await state.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, id)))[0]!.evidence; + + for (const status of ["paused", "terminated", "pending_approval"] as const) { + it(`waits instead of retrying every pass while the assignee is ${status}`, async () => { + const f = await seed(status); + const wake = await start(); + await advance(MINUTE); + const idle = await countStatements(() => advance(2 * HOUR)); + expect(wake).not.toHaveBeenCalled(); + expect(idle.length).toBeLessThan(300); + expect(await evidence(f.actionId)).toMatchObject({ continuationDelivery: "pending", + continuationDeliveryAgentWaits: expect.any(Number) }); + expect(await state.db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.agentId, f.agentId))).toHaveLength(0); + const [task] = await state.db.select().from(issues).where(eq(issues.id, f.taskId)); + expect(task).toMatchObject({ status: "in_progress", assigneeAgentId: f.agentId }); + }, 600_000); + } + + it("delivers the continuation within seconds once the paused assignee is resumed", async () => { + const f = await seed("paused"); + const wake = await start(true); + await advance(3 * HOUR); + expect(wake).not.toHaveBeenCalled(); + await createAgentLifecycle(state.db).resumeAgent(f.agentId); + await lifecycle().process(f.agentId); + expect((await state.db.select().from(agents).where(eq(agents.id, f.agentId)))[0]!.status).not.toBe("paused"); + await advance(20_000); + expect(wake).toHaveBeenCalledTimes(1); + expect(await evidence(f.actionId)).toMatchObject({ continuationDelivery: "delivered", continuationRunId: expect.any(String) }); + }, 600_000); + + it("does not store a wait when the assignee is resumed while delivery is checking it", async () => { + const f = await seed("terminated"); + const [action] = await state.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId)); + let release!: () => void; + const held = new Promise(resolve => { release = resolve; }); + let locked!: () => void; + const lockTaken = new Promise(resolve => { locked = resolve; }); + const resume = state.db.transaction(async tx => { + await tx.update(agents).set({ status: "idle" }).where(eq(agents.id, f.agentId)); + await readmitAgentContinuations(tx as never, { companyId: f.companyId, agentId: f.agentId }); + locked(); + await held; + }); + await lockTaken; + const deferred = deferForUnwakeableAssignee(state.db, action!); + await new Promise(resolve => setTimeout(resolve, 200)); + release(); + await resume; + expect(await deferred).toBe(false); + expect(await evidence(f.actionId)).not.toHaveProperty("continuationDeliveryRetryAt"); + }, 60_000); +}); diff --git a/server/src/services/agent-lifecycle.ts b/server/src/services/agent-lifecycle.ts index 619491c22d1..e38b30280d4 100644 --- a/server/src/services/agent-lifecycle.ts +++ b/server/src/services/agent-lifecycle.ts @@ -1,5 +1,6 @@ import { notifyMuseContactVerification } from "../modules/external-agents/index.js"; import { notifyBudgetResumeAgentReady } from "./budget-resume-continuation.js"; +import { readmitAgentContinuations } from "./recovery/continuation-delivery-attempt.js"; import { currentControllerProcessStamp } from "./controller-process-identity.js"; import { notifyStrandedAgentStatus } from "./live-run-recovery-notifications.js"; import { notifyDeferredExecutionEligibility } from "./deferred-execution-notifications.js"; @@ -70,6 +71,7 @@ export function createAgentLifecycleEffects(_hooks: BudgetServiceHooks = {}): li await notifyNativeReviewResponse(tx, { companyId, agentId }); await notifyChatInboundWakeNotices(tx, { companyId }); await notifyBudgetResumeAgentReady(tx, agentId); + await readmitAgentContinuations(tx, { companyId, agentId }); } }, reportFailure(context, error) { diff --git a/server/src/services/execution-control-work.ts b/server/src/services/execution-control-work.ts index 8bc16172fb9..eb3e1983964 100644 --- a/server/src/services/execution-control-work.ts +++ b/server/src/services/execution-control-work.ts @@ -10,6 +10,7 @@ import { hasPendingNativeReplacements, nextNativeOwnershipProofDeadlineAt } from import { nextAbandonedNativeOwnerReleaseAt } from "./native-runtime/native-failed-run-owner-release.js"; import { hasPendingPauseResume } from "./issue-tree-resume.js"; import { hasPendingBudgetResume } from "./budget-resume-continuation.js"; +import { continuationDeliveryDue } from "./recovery/continuation-delivery-attempt.js"; export function registerExecutionControlWork( coordinator: ReturnType, @@ -94,11 +95,16 @@ export function executionControlSchedule(db: Db) { return at === null ? null : deadline([new Date(at)]); }, async continuation() { - const rows = await db.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(and( + const pending = and( eq(issueRecoveryActions.status, "resolved"), sql`${issueRecoveryActions.evidence}->>'continuationDelivery' = 'pending'`, - )).limit(1); - return rows.length || await hasPendingPauseResume(db) || await hasPendingBudgetResume(db) ? retry() : null; + ); + const rows = await db.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions) + .where(and(pending, continuationDeliveryDue(new Date()))).limit(1); + if (rows.length || await hasPendingPauseResume(db) || await hasPendingBudgetResume(db)) return retry(); + const [waiting] = await db.select({ at: sql`min((${issueRecoveryActions.evidence}->>'continuationDeliveryRetryAt')::timestamptz)` }) + .from(issueRecoveryActions).where(pending); + return deadline([waiting?.at ? new Date(waiting.at) : null]); }, async status() { const rows = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns) diff --git a/server/src/services/execution-recovery-resolution.ts b/server/src/services/execution-recovery-resolution.ts index 250c985095d..d3dc9e68a94 100644 --- a/server/src/services/execution-recovery-resolution.ts +++ b/server/src/services/execution-recovery-resolution.ts @@ -40,7 +40,7 @@ import { readSystemRecovery } from "./system-recovery-state.js"; import { isReattachPhaseActive } from "./heartbeat/run-admission.js"; import { provenGoneLegacyControllerBootId } from "./legacy-controller-lease.js"; import { reserveAutomaticRunCreation } from "./recovery/automatic-run-brake.js"; -import { recordContinuationDeliveryAttempt } from "./recovery/continuation-delivery-attempt.js"; +import { continuationDeliveryDue, deferForUnwakeableAssignee, recordContinuationDeliveryAttempt } from "./recovery/continuation-delivery-attempt.js"; import { continueFreshInTransaction, continueHeldExecutions, freshContinuationCheckDue, recordFreshContinuationCheck, settleCoalescedFreshContinuationHolds } from "./fresh-continuation-fallback.js"; import { alias } from "drizzle-orm/pg-core"; @@ -423,6 +423,7 @@ export async function deliverReconciledExecutions( and( eq(issueRecoveryActions.status, "resolved"), sql`${issueRecoveryActions.evidence}->>'continuationDelivery' = 'pending'`, + continuationDeliveryDue(new Date()), seenActionIds.size ? not(inArray(issueRecoveryActions.id, [...seenActionIds])) : undefined, seenAgents.size ? sql`coalesce(${issueRecoveryActions.returnOwnerAgentId}::text, '') <> all(${sql.param([...seenAgents])}::text[])` : undefined, companies.length ? not(inArray(issueRecoveryActions.companyId, companies)) : undefined, @@ -545,6 +546,7 @@ export async function deliverReconciledExecutions( continue; } } + if (await deferForUnwakeableAssignee(db, action)) { seenAgents.add(action.returnOwnerAgentId); continue; } if (!await agentHasRoom(action, action.returnOwnerAgentId, kind)) { seenAgents.add(action.returnOwnerAgentId); continue; diff --git a/server/src/services/recovery/continuation-delivery-attempt.ts b/server/src/services/recovery/continuation-delivery-attempt.ts index ce3b7c2d221..c1c2d78668a 100644 --- a/server/src/services/recovery/continuation-delivery-attempt.ts +++ b/server/src/services/recovery/continuation-delivery-attempt.ts @@ -1,15 +1,69 @@ import { and, eq, sql } from "drizzle-orm"; -import { issueRecoveryActions, type Db } from "@paperclipai/db"; +import { agents, issueRecoveryActions, type Db } from "@paperclipai/db"; +import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js"; +import { evaluateAgentInvokabilityFromDb } from "../agent-invokability.js"; +import { isAgentAwaitingSetup } from "../../modules/agent-lifecycle/index.js"; + +export const CONTINUATION_AGENT_WAIT_BASE_MS = 60_000; +export const CONTINUATION_AGENT_WAIT_MAX_MS = 60 * 60_000; + +type PendingAction = Pick; + +const pendingDelivery = (action: PendingAction) => and( + eq(issueRecoveryActions.companyId, action.companyId), + eq(issueRecoveryActions.id, action.id), + eq(issueRecoveryActions.status, "resolved"), + sql`${issueRecoveryActions.evidence}->>'continuationDelivery' = 'pending'`, +); /** Records a delivery attempt for a pending continuation, so the next pass tries the records not yet tried first. */ -export function recordContinuationDeliveryAttempt(db: Db, action: Pick) { +export function recordContinuationDeliveryAttempt(db: Db, action: PendingAction) { return db.update(issueRecoveryActions) .set({ evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify( { continuationDeliveryAttemptedAt: new Date().toISOString() })}::jsonb` }) - .where(and( - eq(issueRecoveryActions.companyId, action.companyId), - eq(issueRecoveryActions.id, action.id), - eq(issueRecoveryActions.status, "resolved"), - sql`${issueRecoveryActions.evidence}->>'continuationDelivery' = 'pending'`, - )); + .where(pendingDelivery(action)); +} + +export function continuationAgentWait(evidence: Record, now = new Date()) { + const recorded = Number(evidence.continuationDeliveryAgentWaits ?? 0); + const prior = Number.isSafeInteger(recorded) && recorded > 0 ? recorded : 0; + const delay = Math.min(CONTINUATION_AGENT_WAIT_MAX_MS, CONTINUATION_AGENT_WAIT_BASE_MS * 2 ** Math.min(prior, 20)); + return { waits: prior + 1, retryAt: new Date(now.getTime() + delay) }; +} + +export async function deferForUnwakeableAssignee(db: Db, + action: PendingAction & Pick, now = new Date()) { + if (!action.returnOwnerAgentId) return false; + return db.transaction(async tx => { + const [owner] = await tx.select().from(agents).where(and(eq(agents.companyId, action.companyId), + eq(agents.id, action.returnOwnerAgentId!))).for("share"); + if (!owner || (await evaluateAgentInvokabilityFromDb(tx as unknown as Db, + isAgentAwaitingSetup(owner) ? { ...owner, status: "idle" } : owner)).invokable) return false; + const { waits, retryAt } = continuationAgentWait(action.evidence, now); + await tx.update(issueRecoveryActions) + .set({ evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify({ continuationDeliveryAttemptedAt: now.toISOString(), + continuationDeliveryRetryAt: retryAt.toISOString(), continuationDeliveryAgentWaits: waits })}::jsonb` }) + .where(pendingDelivery(action)); + return true; + }); +} + +export function continuationDeliveryDue(now: Date) { + return sql`coalesce((${issueRecoveryActions.evidence}->>'continuationDeliveryRetryAt')::timestamptz <= ${now.toISOString()}::timestamptz, true)`; +} + +export async function readmitAgentContinuations(tx: Db, input: { companyId: string; agentId: string }) { + const waiting = and( + eq(issueRecoveryActions.companyId, input.companyId), + eq(issueRecoveryActions.returnOwnerAgentId, input.agentId), + eq(issueRecoveryActions.status, "resolved"), + sql`${issueRecoveryActions.evidence}->>'continuationDelivery' = 'pending'`, + sql`${issueRecoveryActions.evidence} ? 'continuationDeliveryRetryAt'`, + ); + const [any] = await tx.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(waiting).limit(1); + if (!any) return; + await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionContinuation); + await tx.update(issueRecoveryActions) + .set({ evidence: sql`${issueRecoveryActions.evidence} - 'continuationDeliveryRetryAt' - 'continuationDeliveryAgentWaits'` }) + .where(waiting); }