mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
perf(recovery): wait for an assignee that cannot run instead of retrying its continuation every pass (#21687)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run stops and recovery decides to continue the task in a
fresh run, it records a pending continuation. The execution-continuation
worker delivers it with `heartbeat.wakeup`.
> - When the task's assignee cannot take work (paused, terminated,
waiting for hire approval, or an invalid reporting chain), the wake
refuses. The worker recorded only an "attempted at" time and tried again
on the next 15-second pass, forever.
> - Each refused wake also saved a new skipped wake row. An idle server
with one such task did about 13,000 statements and 390 wake attempts
every 2 hours.
> - This pull request makes delivery check the assignee with the same
rule the wake uses. When the assignee cannot run, the continuation waits
on a stored retry time and is re-admitted at once when the assignee can
run again.
> - The benefit is that an idle server does almost no work for these
tasks, and a resumed agent still gets its continuation within seconds.
## Linked Issues or Issue Description
No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/bug_report.yml`.
**What happened?**
`deliverReconciledExecutions`
(`server/src/services/execution-recovery-resolution.ts`) offers each
pending continuation (`continuationDelivery: "pending"`) to
`heartbeat.wakeup`. When the assignee is not invokable, the wake throws
"Agent is not invokable in its current state" and saves a skipped
`agent_wakeup_requests` row. The worker catches the refusal, records
`continuationDeliveryAttemptedAt`, and the continuation schedule
(`executionControlSchedule.continuation`) asks for another pass in 15
seconds because a pending row exists. Nothing ever changes, so the loop
never ends.
Measured on real PostgreSQL with the real coordinator and the real wake,
on a fake clock, for one pending continuation over 2 idle hours:
| Assignee status | Wake attempts (master) | Statements (master) |
| --- | --- | --- |
| paused | 380 | about 12,980 |
| terminated | 376 | about 12,820 |
| pending_approval | 395 | about 13,000 |
An agent in `error` is not affected: it is invokable, and the wake
delivers the continuation on the first pass. Only the statuses and
conditions that make an agent not invokable cause the loop.
**Expected behavior**
A continuation that cannot be delivered because its assignee cannot run
does not try again every pass. It waits for a real event (the agent can
run again) or a spaced backstop time.
**Steps to reproduce**
1. Create a task in `in_progress` assigned to an agent, with a failed
run.
2. Record a pending continuation for that run, as the fresh-continuation
fallback does.
3. Pause, terminate or un-approve the agent.
4. Run the execution-continuation worker. It calls the wake about every
18 seconds and saves a skipped wake row each time.
**Paperclip version or commit**
master at dcba9628c7.
## What Changed
- `server/src/services/execution-recovery-resolution.ts`: before the
wake, delivery calls `deferForUnwakeableAssignee`. When the assignee
cannot run, delivery skips the wake. The pending-page query skips
continuations whose stored retry time is in the future. The file stays
under 1,000 lines (997).
- `server/src/services/recovery/continuation-delivery-attempt.ts`:
`deferForUnwakeableAssignee` reads the assignee `FOR SHARE` and
evaluates it with `evaluateAgentInvokabilityFromDb`, the rule
`heartbeat.wakeup` uses (including the setup exception). In the same
transaction it stores `continuationDeliveryRetryAt` and a wait count on
the recovery action. The wait is 1 minute and doubles per refusal up to
1 hour. The row lock orders the wait against a concurrent status change,
so a resume cannot be missed. `continuationDeliveryDue` is the shared
"due now" condition. `readmitAgentContinuations` notifies the
continuation worker, then clears the stored time for one agent's waiting
continuations.
- `server/src/services/execution-control-work.ts`: the continuation
schedule asks for a pass in 15 seconds only when a continuation is due.
Otherwise it returns the earliest stored retry time.
- `server/src/services/agent-lifecycle.ts`: when an agent's status
changes from not invokable to invokable (resume, hire approval), the
existing hook also calls `readmitAgentContinuations`.
-
`server/src/__tests__/continuation-delivery-unwakeable-agent-idle.test.ts`:
five new tests on real PostgreSQL.
What the user sees does not change. The continuation stays pending and
the task stays assigned. The stranded-assignment check and the
assigned-work invariant still own the visible outcome for a task whose
assignee cannot run (the existing "can't run right now" escalation, or
the "agent paused" wait). When the agent can run again, the continuation
is delivered as before.
## Verification
- `pnpm exec vitest run
src/__tests__/continuation-delivery-unwakeable-agent-idle.test.ts` (in
`server/`):
- `waits instead of retrying every pass while the assignee is paused /
terminated / pending_approval`: 2 idle hours after the first minute.
- master: 380, 376 and 395 wake attempts (red).
- this branch: 0 wake attempts, 68 statements, no skipped wake rows, the
task stays `in_progress` and assigned (green).
- `delivers the continuation within seconds once the paused assignee is
resumed`: after 3 idle hours, the agent is resumed through the real
agent lifecycle. Within 20 seconds the worker calls the wake once and
the record says `continuationDelivery: "delivered"` (green). On master
this test fails because the wake was already called during the idle
hours.
- `does not store a wait when the assignee is resumed while delivery is
checking it`: a resume transaction holds the agent row while delivery
checks. With the row lock, delivery sees the resumed agent and stores no
wait (green). Without the lock, it stores a wait (red).
- Neighbouring suites pass locally:
`automatic-run-brake-recovery-passes`, `recovery-wake-herd`,
`execution-control-work`, `fresh-continuation-move-forward-offer`,
`fresh-continuation-retry-delivery`, `fresh-continuation-fallback`,
`issue-recovery-actions`, `resource-lifecycle-events` (345 tests).
- `npx tsc --noEmit -p server/tsconfig.json` passes.
## Risks
- Low risk. Only a continuation whose assignee is not invokable by the
wake's own rule waits. Every other refusal keeps the current behavior.
- A repair that makes the agent invokable without a status change (for
example a fixed reporting chain) does not fire the re-admit hook. The
backstop picks it up within at most 1 hour.
- A task reassigned while its continuation waits is invalidated on the
next backstop attempt instead of the next 15-second pass. The new
assignee gets its own assignment wake, so nothing waits on that.
- No schema or API change. The new fields live in the existing
`evidence` JSON.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local PostgreSQL test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Related: #21621, #21674, #21675, #21676, #21617.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
c66530da63
commit
f4e025a1cf
5 files changed
+215
-12
No files matched your search
@@ -0,0 +1,139 @@
|
||||
import { setImmediate as immediate } from "node:timers/promises";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { afterEach, expect, it, vi } from "vitest";
|
||||
import { agentWakeupRequests, agents, heartbeatRuns, issueRecoveryActions, issues } from "@paperclipai/db";
|
||||
import { configureAgentLifecycle, createAgentLifecycle } from "../services/agent-lifecycle.js";
|
||||
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
|
||||
import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
|
||||
import { executionControlSchedule, registerExecutionControlWork } from "../services/execution-control-work.js";
|
||||
import { deliverReconciledExecutions } from "../services/execution-recovery-resolution.js";
|
||||
import { deferForUnwakeableAssignee, readmitAgentContinuations } from "../services/recovery/continuation-delivery-attempt.js";
|
||||
import { describeIssueRecovery, issueRecoveryFixture } from "./issue-recovery.test-fixture.js";
|
||||
|
||||
const MINUTE = 60_000;
|
||||
const HOUR = 60 * MINUTE;
|
||||
|
||||
describeIssueRecovery("continuation delivery to an agent that cannot be woken", () => {
|
||||
const state = issueRecoveryFixture();
|
||||
const passes = new Set<Promise<unknown>>();
|
||||
let lifecycleWorker: ReturnType<typeof configureAgentLifecycle> | undefined;
|
||||
const lifecycle = () => lifecycleWorker ??= configureAgentLifecycle(state.db,
|
||||
{ requiredPluginIds: async () => [], runHost: async () => "complete", runPlugin: async () => "complete" });
|
||||
afterEach(async () => { await lifecycleWorker?.stop(); lifecycleWorker = undefined; });
|
||||
|
||||
async function seed(status: "paused" | "terminated" | "pending_approval") {
|
||||
const f = await state.fixture();
|
||||
const db = state.db;
|
||||
if (status === "paused") { await createAgentLifecycle(db).pauseAgent(f.agentId); await lifecycle().process(f.agentId); }
|
||||
else await db.update(agents).set({ status }).where(eq(agents.id, f.agentId));
|
||||
expect((await db.select().from(agents).where(eq(agents.id, f.agentId)))[0]!.status).toBe(status);
|
||||
const [task] = await db.insert(issues).values({ companyId: f.companyId, title: "Continue the work", status: "in_progress",
|
||||
assigneeAgentId: f.agentId }).returning();
|
||||
const [source] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "failed",
|
||||
errorCode: "adapter_failed", startedAt: new Date(Date.now() - 3 * MINUTE), finishedAt: new Date(Date.now() - 2 * MINUTE),
|
||||
contextSnapshot: { issueId: task!.id } }).returning();
|
||||
const recordedAt = new Date().toISOString();
|
||||
const [action] = await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: task!.id,
|
||||
kind: "active_run_watchdog", status: "resolved", outcome: "restored", resolvedAt: new Date(), ownerType: "system",
|
||||
returnOwnerAgentId: f.agentId, cause: "legacy_execution_requires_reconciliation", fingerprint: `legacy-execution:${source!.id}`,
|
||||
nextAction: "Continue the task.", attemptCount: 1,
|
||||
evidence: { runId: source!.id, originalFailureCode: "adapter_failed", continuationDelivery: "pending",
|
||||
freshContinuation: { schema: "paperclip.fresh-continuation.v1", sourceRunId: source!.id, trigger: "recovery_refused",
|
||||
delivery: "recorded", recordedAt, isolation: "writer_ended", stopProof: "writer_ended" },
|
||||
freshContinuationSettlement: { recordedAt },
|
||||
executionReconciliation: { runId: source!.id, providerStopped: true, providerStopProof: "writer_ended",
|
||||
actionOutcome: "mixed", outcomeEvidence: "The run stopped; the continuation starts from the last saved work.",
|
||||
actorId: "fresh-continuation", recordedAt } } }).returning();
|
||||
return { ...f, taskId: task!.id, actionId: action!.id };
|
||||
}
|
||||
|
||||
async function start(queueOnly = false) {
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] });
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: state.db, canRun: () => true, canReconcile: () => true,
|
||||
onError: error => state.errors.push(error) });
|
||||
state.coordinator = coordinator;
|
||||
const wake = vi.fn(async (agentId: string, opts: Parameters<typeof state.heartbeat.wakeup>[1]) => {
|
||||
if (!queueOnly) return state.heartbeat.wakeup(agentId, opts);
|
||||
const [agent] = await state.db.select().from(agents).where(eq(agents.id, agentId));
|
||||
const [run] = await state.db.insert(heartbeatRuns).values({ companyId: agent!.companyId, agentId, invocationSource: "automation",
|
||||
triggerDetail: "system", status: "queued", contextSnapshot: opts?.contextSnapshot ?? {} }).returning();
|
||||
return run!;
|
||||
});
|
||||
const work = registerExecutionControlWork(coordinator, [{ queue: DELIVERY_QUEUES.executionContinuation, run: () => {
|
||||
const pass = deliverReconciledExecutions(state.db, wake as never); passes.add(pass);
|
||||
void pass.then(() => passes.delete(pass), () => passes.delete(pass));
|
||||
return pass;
|
||||
}, next: executionControlSchedule(state.db).continuation }]);
|
||||
await work.ready;
|
||||
return wake;
|
||||
}
|
||||
|
||||
async function advance(ms: number) {
|
||||
const until = Date.now() + ms;
|
||||
while (Date.now() < until) {
|
||||
await vi.advanceTimersByTimeAsync(Math.min(1_000, until - Date.now()));
|
||||
await Promise.allSettled([...passes]); await immediate();
|
||||
}
|
||||
}
|
||||
|
||||
async function countStatements(work: () => Promise<unknown>) {
|
||||
const statements: string[] = [], prior = state.db.$client.options.debug;
|
||||
state.db.$client.options.debug = (_connection, query) => { statements.push(query); };
|
||||
try { await work(); } finally { state.db.$client.options.debug = prior; }
|
||||
return statements;
|
||||
}
|
||||
|
||||
const evidence = async (id: string) =>
|
||||
(await state.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, id)))[0]!.evidence;
|
||||
|
||||
for (const status of ["paused", "terminated", "pending_approval"] as const) {
|
||||
it(`waits instead of retrying every pass while the assignee is ${status}`, async () => {
|
||||
const f = await seed(status);
|
||||
const wake = await start();
|
||||
await advance(MINUTE);
|
||||
const idle = await countStatements(() => advance(2 * HOUR));
|
||||
expect(wake).not.toHaveBeenCalled();
|
||||
expect(idle.length).toBeLessThan(300);
|
||||
expect(await evidence(f.actionId)).toMatchObject({ continuationDelivery: "pending",
|
||||
continuationDeliveryAgentWaits: expect.any(Number) });
|
||||
expect(await state.db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.agentId, f.agentId))).toHaveLength(0);
|
||||
const [task] = await state.db.select().from(issues).where(eq(issues.id, f.taskId));
|
||||
expect(task).toMatchObject({ status: "in_progress", assigneeAgentId: f.agentId });
|
||||
}, 600_000);
|
||||
}
|
||||
|
||||
it("delivers the continuation within seconds once the paused assignee is resumed", async () => {
|
||||
const f = await seed("paused");
|
||||
const wake = await start(true);
|
||||
await advance(3 * HOUR);
|
||||
expect(wake).not.toHaveBeenCalled();
|
||||
await createAgentLifecycle(state.db).resumeAgent(f.agentId);
|
||||
await lifecycle().process(f.agentId);
|
||||
expect((await state.db.select().from(agents).where(eq(agents.id, f.agentId)))[0]!.status).not.toBe("paused");
|
||||
await advance(20_000);
|
||||
expect(wake).toHaveBeenCalledTimes(1);
|
||||
expect(await evidence(f.actionId)).toMatchObject({ continuationDelivery: "delivered", continuationRunId: expect.any(String) });
|
||||
}, 600_000);
|
||||
|
||||
it("does not store a wait when the assignee is resumed while delivery is checking it", async () => {
|
||||
const f = await seed("terminated");
|
||||
const [action] = await state.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.id, f.actionId));
|
||||
let release!: () => void;
|
||||
const held = new Promise<void>(resolve => { release = resolve; });
|
||||
let locked!: () => void;
|
||||
const lockTaken = new Promise<void>(resolve => { locked = resolve; });
|
||||
const resume = state.db.transaction(async tx => {
|
||||
await tx.update(agents).set({ status: "idle" }).where(eq(agents.id, f.agentId));
|
||||
await readmitAgentContinuations(tx as never, { companyId: f.companyId, agentId: f.agentId });
|
||||
locked();
|
||||
await held;
|
||||
});
|
||||
await lockTaken;
|
||||
const deferred = deferForUnwakeableAssignee(state.db, action!);
|
||||
await new Promise(resolve => setTimeout(resolve, 200));
|
||||
release();
|
||||
await resume;
|
||||
expect(await deferred).toBe(false);
|
||||
expect(await evidence(f.actionId)).not.toHaveProperty("continuationDeliveryRetryAt");
|
||||
}, 60_000);
|
||||
});
|
||||
@@ -1,5 +1,6 @@
|
||||
import { notifyMuseContactVerification } from "../modules/external-agents/index.js";
|
||||
import { notifyBudgetResumeAgentReady } from "./budget-resume-continuation.js";
|
||||
import { readmitAgentContinuations } from "./recovery/continuation-delivery-attempt.js";
|
||||
import { currentControllerProcessStamp } from "./controller-process-identity.js";
|
||||
import { notifyStrandedAgentStatus } from "./live-run-recovery-notifications.js";
|
||||
import { notifyDeferredExecutionEligibility } from "./deferred-execution-notifications.js";
|
||||
@@ -70,6 +71,7 @@ export function createAgentLifecycleEffects(_hooks: BudgetServiceHooks = {}): li
|
||||
await notifyNativeReviewResponse(tx, { companyId, agentId });
|
||||
await notifyChatInboundWakeNotices(tx, { companyId });
|
||||
await notifyBudgetResumeAgentReady(tx, agentId);
|
||||
await readmitAgentContinuations(tx, { companyId, agentId });
|
||||
}
|
||||
},
|
||||
reportFailure(context, error) {
|
||||
|
||||
@@ -10,6 +10,7 @@ import { hasPendingNativeReplacements, nextNativeOwnershipProofDeadlineAt } from
|
||||
import { nextAbandonedNativeOwnerReleaseAt } from "./native-runtime/native-failed-run-owner-release.js";
|
||||
import { hasPendingPauseResume } from "./issue-tree-resume.js";
|
||||
import { hasPendingBudgetResume } from "./budget-resume-continuation.js";
|
||||
import { continuationDeliveryDue } from "./recovery/continuation-delivery-attempt.js";
|
||||
|
||||
export function registerExecutionControlWork(
|
||||
coordinator: ReturnType<typeof createDeliveryWorkCoordinator>,
|
||||
@@ -94,11 +95,16 @@ export function executionControlSchedule(db: Db) {
|
||||
return at === null ? null : deadline([new Date(at)]);
|
||||
},
|
||||
async continuation() {
|
||||
const rows = await db.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(and(
|
||||
const pending = and(
|
||||
eq(issueRecoveryActions.status, "resolved"),
|
||||
sql`${issueRecoveryActions.evidence}->>'continuationDelivery' = 'pending'`,
|
||||
)).limit(1);
|
||||
return rows.length || await hasPendingPauseResume(db) || await hasPendingBudgetResume(db) ? retry() : null;
|
||||
);
|
||||
const rows = await db.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions)
|
||||
.where(and(pending, continuationDeliveryDue(new Date()))).limit(1);
|
||||
if (rows.length || await hasPendingPauseResume(db) || await hasPendingBudgetResume(db)) return retry();
|
||||
const [waiting] = await db.select({ at: sql<string | null>`min((${issueRecoveryActions.evidence}->>'continuationDeliveryRetryAt')::timestamptz)` })
|
||||
.from(issueRecoveryActions).where(pending);
|
||||
return deadline([waiting?.at ? new Date(waiting.at) : null]);
|
||||
},
|
||||
async status() {
|
||||
const rows = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
|
||||
|
||||
@@ -40,7 +40,7 @@ import { readSystemRecovery } from "./system-recovery-state.js";
|
||||
import { isReattachPhaseActive } from "./heartbeat/run-admission.js";
|
||||
import { provenGoneLegacyControllerBootId } from "./legacy-controller-lease.js";
|
||||
import { reserveAutomaticRunCreation } from "./recovery/automatic-run-brake.js";
|
||||
import { recordContinuationDeliveryAttempt } from "./recovery/continuation-delivery-attempt.js";
|
||||
import { continuationDeliveryDue, deferForUnwakeableAssignee, recordContinuationDeliveryAttempt } from "./recovery/continuation-delivery-attempt.js";
|
||||
import { continueFreshInTransaction, continueHeldExecutions, freshContinuationCheckDue, recordFreshContinuationCheck,
|
||||
settleCoalescedFreshContinuationHolds } from "./fresh-continuation-fallback.js";
|
||||
import { alias } from "drizzle-orm/pg-core";
|
||||
@@ -423,6 +423,7 @@ export async function deliverReconciledExecutions(
|
||||
and(
|
||||
eq(issueRecoveryActions.status, "resolved"),
|
||||
sql`${issueRecoveryActions.evidence}->>'continuationDelivery' = 'pending'`,
|
||||
continuationDeliveryDue(new Date()),
|
||||
seenActionIds.size ? not(inArray(issueRecoveryActions.id, [...seenActionIds])) : undefined,
|
||||
seenAgents.size ? sql`coalesce(${issueRecoveryActions.returnOwnerAgentId}::text, '') <> all(${sql.param([...seenAgents])}::text[])` : undefined,
|
||||
companies.length ? not(inArray(issueRecoveryActions.companyId, companies)) : undefined,
|
||||
@@ -545,6 +546,7 @@ export async function deliverReconciledExecutions(
|
||||
continue;
|
||||
}
|
||||
}
|
||||
if (await deferForUnwakeableAssignee(db, action)) { seenAgents.add(action.returnOwnerAgentId); continue; }
|
||||
if (!await agentHasRoom(action, action.returnOwnerAgentId, kind)) {
|
||||
seenAgents.add(action.returnOwnerAgentId);
|
||||
continue;
|
||||
|
||||
@@ -1,15 +1,69 @@
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { issueRecoveryActions, type Db } from "@paperclipai/db";
|
||||
import { agents, issueRecoveryActions, type Db } from "@paperclipai/db";
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js";
|
||||
import { evaluateAgentInvokabilityFromDb } from "../agent-invokability.js";
|
||||
import { isAgentAwaitingSetup } from "../../modules/agent-lifecycle/index.js";
|
||||
|
||||
export const CONTINUATION_AGENT_WAIT_BASE_MS = 60_000;
|
||||
export const CONTINUATION_AGENT_WAIT_MAX_MS = 60 * 60_000;
|
||||
|
||||
type PendingAction = Pick<typeof issueRecoveryActions.$inferSelect, "companyId" | "id">;
|
||||
|
||||
const pendingDelivery = (action: PendingAction) => and(
|
||||
eq(issueRecoveryActions.companyId, action.companyId),
|
||||
eq(issueRecoveryActions.id, action.id),
|
||||
eq(issueRecoveryActions.status, "resolved"),
|
||||
sql`${issueRecoveryActions.evidence}->>'continuationDelivery' = 'pending'`,
|
||||
);
|
||||
|
||||
/** Records a delivery attempt for a pending continuation, so the next pass tries the records not yet tried first. */
|
||||
export function recordContinuationDeliveryAttempt(db: Db, action: Pick<typeof issueRecoveryActions.$inferSelect, "companyId" | "id">) {
|
||||
export function recordContinuationDeliveryAttempt(db: Db, action: PendingAction) {
|
||||
return db.update(issueRecoveryActions)
|
||||
.set({ evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify(
|
||||
{ continuationDeliveryAttemptedAt: new Date().toISOString() })}::jsonb` })
|
||||
.where(and(
|
||||
eq(issueRecoveryActions.companyId, action.companyId),
|
||||
eq(issueRecoveryActions.id, action.id),
|
||||
eq(issueRecoveryActions.status, "resolved"),
|
||||
sql`${issueRecoveryActions.evidence}->>'continuationDelivery' = 'pending'`,
|
||||
));
|
||||
.where(pendingDelivery(action));
|
||||
}
|
||||
|
||||
export function continuationAgentWait(evidence: Record<string, unknown>, now = new Date()) {
|
||||
const recorded = Number(evidence.continuationDeliveryAgentWaits ?? 0);
|
||||
const prior = Number.isSafeInteger(recorded) && recorded > 0 ? recorded : 0;
|
||||
const delay = Math.min(CONTINUATION_AGENT_WAIT_MAX_MS, CONTINUATION_AGENT_WAIT_BASE_MS * 2 ** Math.min(prior, 20));
|
||||
return { waits: prior + 1, retryAt: new Date(now.getTime() + delay) };
|
||||
}
|
||||
|
||||
export async function deferForUnwakeableAssignee(db: Db,
|
||||
action: PendingAction & Pick<typeof issueRecoveryActions.$inferSelect, "evidence" | "returnOwnerAgentId">, now = new Date()) {
|
||||
if (!action.returnOwnerAgentId) return false;
|
||||
return db.transaction(async tx => {
|
||||
const [owner] = await tx.select().from(agents).where(and(eq(agents.companyId, action.companyId),
|
||||
eq(agents.id, action.returnOwnerAgentId!))).for("share");
|
||||
if (!owner || (await evaluateAgentInvokabilityFromDb(tx as unknown as Db,
|
||||
isAgentAwaitingSetup(owner) ? { ...owner, status: "idle" } : owner)).invokable) return false;
|
||||
const { waits, retryAt } = continuationAgentWait(action.evidence, now);
|
||||
await tx.update(issueRecoveryActions)
|
||||
.set({ evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify({ continuationDeliveryAttemptedAt: now.toISOString(),
|
||||
continuationDeliveryRetryAt: retryAt.toISOString(), continuationDeliveryAgentWaits: waits })}::jsonb` })
|
||||
.where(pendingDelivery(action));
|
||||
return true;
|
||||
});
|
||||
}
|
||||
|
||||
export function continuationDeliveryDue(now: Date) {
|
||||
return sql`coalesce((${issueRecoveryActions.evidence}->>'continuationDeliveryRetryAt')::timestamptz <= ${now.toISOString()}::timestamptz, true)`;
|
||||
}
|
||||
|
||||
export async function readmitAgentContinuations(tx: Db, input: { companyId: string; agentId: string }) {
|
||||
const waiting = and(
|
||||
eq(issueRecoveryActions.companyId, input.companyId),
|
||||
eq(issueRecoveryActions.returnOwnerAgentId, input.agentId),
|
||||
eq(issueRecoveryActions.status, "resolved"),
|
||||
sql`${issueRecoveryActions.evidence}->>'continuationDelivery' = 'pending'`,
|
||||
sql`${issueRecoveryActions.evidence} ? 'continuationDeliveryRetryAt'`,
|
||||
);
|
||||
const [any] = await tx.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(waiting).limit(1);
|
||||
if (!any) return;
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.executionContinuation);
|
||||
await tx.update(issueRecoveryActions)
|
||||
.set({ evidence: sql`${issueRecoveryActions.evidence} - 'continuationDeliveryRetryAt' - 'continuationDeliveryAgentWaits'` })
|
||||
.where(waiting);
|
||||
}
|
||||
Reference in new issue
Block a user