mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
fix(recovery): deliver every pending continuation within a bounded time (#21324)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Execution recovery continues a task in a fresh run after a restart or a lost run left it unfinished > - #21234 paced these continuations: at most 1 per agent and 5 per company per pass, and a continuation waits while its agent already has a queued recovery run or a full queue > - A continuation that waited 5 minutes must enter the queue anyway, but that bound applied only to the full-queue check. The "agent already has a queued recovery run" check was in the page query, and it had no bound. > - On staging, an agent had queued recovery runs that did not start for hours. Its pending continuation was never read again, and its task showed "working" with nothing running or queued > - This pull request applies the 5-minute bound to both checks, stops one undeliverable continuation from using its agent's or company's slot, and stops the task view from showing "working" for a continuation that is overdue > - The benefit is that every pending continuation enters the queue within a bounded time, also after a restart, and the pacing still protects people from recovery herds ## Linked Issues or Issue Description Refs #21234, #21135 **What happened?** On staging, a task had a fresh-continuation hold with `continuationDelivery: "pending"` since about 03:30Z. More than 3.5 hours later, no run had started for the task. The task view showed the interruption state `working` with `canRetry: false`, so the user saw nothing wrong. The task's agent had 20 concurrency slots and 11 running runs. It also had queued `issue_continuation_needed` runs from 22:54Z that had not started. The delivery pass reads pending continuations with a query that leaves out every agent that has a queued recovery run (`issue_recovery_action_restored` or `issue_continuation_needed`). That condition had no time bound, so the pass never read this continuation again. The 5-minute admission bound was never reached. Across the company, 203 open tasks showed `working` while the API showed no queued or running run for them. 162 queued runs were more than 1 hour old. **Expected behavior** Every pending continuation enters its agent's queue within a bounded time, also when the agent's queue holds recovery runs that do not start, and also after a server restart. Pacing still limits how many continuations enter a queue at the same time. A task does not show "working" when its continuation is overdue and nothing runs or is queued. **Steps to reproduce** 1. Give an agent a concurrency limit of 1. 2. Put one queued `issue_recovery_action_restored` run on that agent that does not start. 3. Record a pending fresh continuation for another task of the same agent. 4. Wait more than 5 minutes, and let the 15-second continuation pass run. 5. See that the continuation stays `pending`, the task shows `working`, and no run starts. **Paperclip version or commit** master atbbafa78fe2(includes #21234,ade7e0643f) **Deployment mode** Hosted cloud (staging) **Agent adapter(s) involved** paperclip_runner (native runtime). The problem is in the shared continuation delivery pass, so it applies to all adapters. ## What Changed - `deliverReconciledExecutions` in `server/src/services/execution-recovery-resolution.ts`: a continuation held for 5 minutes or more is read even when its agent has a queued recovery run. Before, only the full-queue check let it through. - Each page of the pass now reads at most one candidate per agent (`row_number() over (partition by agent)`), so one agent's backlog cannot fill the page budget and hide other agents' continuations. - A continuation counts against the company's limit only when its wake puts a run in the queue. A continuation that was invalidated or replaced by a newer run does not use its agent's turn. - When an agent refuses a wake (no run, or an error), the pass skips that agent for the rest of the pass and stamps the row with `continuationDeliveryAttemptedAt`. Overdue rows always rank before rows that are not overdue. Among overdue rows, the turn is the later of the hold time and the last attempt, so a refused row goes behind rows held before that attempt and ahead of rows held after it. One refused row cannot pin its agent, and new arrivals cannot keep an older refused row waiting. - New module `server/src/services/recovery-continuation-pacing.ts` holds the pacing constants and the "held since" rule. It adds `RECOVERY_CONTINUATION_DELIVERY_BOUND_MS` (15 minutes). The old constants are still exported from `execution-recovery-resolution.ts`. - `taskInterruptionPresentations` in `server/src/services/task-interruption-presentation.ts` shows a pending continuation as "continuing" only within the delivery bound. After that, the task shows its real state instead of `working`. - No change to the schedule: the continuation duty runs only while pending continuations exist, and it checks the database on start, so a restart does not lose pending rows. ## Verification - `pnpm exec vitest run src/__tests__/recovery-wake-herd.test.ts` (in `server/`): 15 tests pass. New tests: - A continuation held past the bound is delivered while the agent's queue holds recovery runs that never start. - After a restart with a continuation pending past the bound and a full queue, a new schedule is due within one pass, the pass delivers the continuation, and the schedule then goes idle. - Continuations whose wake returns no run do not use the per-company count. Another continuation of the same agent is delivered on the next pass. - An overdue continuation is refused once while the agent's queue is full. On the next pass, it is delivered even though new continuations keep arriving for the same agent. - One agent refuses a backlog of 210 overdue continuations. Another agent's continuation is delivered in the same pass, and the refusing agent gets one wake attempt in that pass. - A task whose continuation is pending past the delivery bound with nothing running does not show `working`. A fresh pending continuation still shows `working` with `canRetry: false`. - Red/green: with the two source files reverted to master (and only the new constant exported), all 4 first new tests fail. The backlog test fails on the first PR head (200 refused wake attempts in one pass). The temporary-refusal test fails on the second PR head (the overdue row stays pending). With this change, all 15 pass. - Related suites pass: fresh-continuation fallback, workspace recovery fallback, native safe replacement, native workspace reconstruction, task interruption presentation, queued-run dispatch order, execution recovery resolution, assigned-work invariant, execution control work, task interruption automatic path and query. - `node scripts/typecheck-layers.mjs` (in `server/`) passes. ## Risks - A continuation that waited 5 minutes now enters a queue that already has recovery runs. The per-agent (1) and per-company (5) limits per pass still apply, and the queue still starts person-initiated work first. - An agent that refuses wakes gets one wake attempt per pass. Its refused rows rotate through `continuationDeliveryAttemptedAt`, and they do not block other agents. - After 15 minutes without delivery, the task no longer shows `working`. With this fix, delivery happens well within that bound, so this is a safety net. - No schema or migration changes. No new polling. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs, and read-only staging API requests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
2483cb46c2
commit
fde0328ff5
4 files changed
+236
-49
No files matched your search
@@ -7,7 +7,8 @@ import {
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { heartbeatService } from "../services/heartbeat.js";
|
||||
import {
|
||||
RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS, RECOVERY_CONTINUATION_MAX_HOLD_MS, deliverReconciledExecutions,
|
||||
RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS, RECOVERY_CONTINUATION_DELIVERY_BOUND_MS, RECOVERY_CONTINUATION_MAX_HOLD_MS,
|
||||
deliverReconciledExecutions,
|
||||
} from "../services/execution-recovery-resolution.js";
|
||||
import { assignedWorkSweep } from "../services/recovery/assigned-work-sweep.js";
|
||||
import { evaluateAssignedWork } from "../services/recovery/assigned-work-invariant.js";
|
||||
@@ -318,6 +319,135 @@ suite("recovery wake herd", () => {
|
||||
expect(await Promise.all(old.map(record => deliveryOf(record.action.id)))).toEqual(old.map(() => "delivered"));
|
||||
}, 60_000);
|
||||
|
||||
async function holdPastBound(record: Awaited<ReturnType<typeof pendingContinuation>>, extraMs = 1_000) {
|
||||
const heldSince = new Date(Date.now() - RECOVERY_CONTINUATION_MAX_HOLD_MS - extraMs).toISOString();
|
||||
await db.update(issueRecoveryActions).set({ evidence: { ...record.action.evidence,
|
||||
freshContinuation: { ...(record.action.evidence.freshContinuation as object), recordedAt: heldSince } } })
|
||||
.where(eq(issueRecoveryActions.id, record.action.id));
|
||||
}
|
||||
|
||||
it("delivers a continuation held past the bound while the agent's queue is full of recovery runs that never start", async () => {
|
||||
const agentId = await seedCompany(1);
|
||||
await queuedRecoveryRun(agentId, 240);
|
||||
await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: (await task(agentId)).id, invocationSource: "automation",
|
||||
triggerDetail: "system", status: "queued", createdAt: new Date(Date.now() - 230 * 60_000),
|
||||
contextSnapshot: { wakeReason: "issue_continuation_needed" } });
|
||||
const record = await pendingContinuation(agentId, 200);
|
||||
const wake = fakeWake();
|
||||
await deliverReconciledExecutions(db, wake as never);
|
||||
expect(wake).not.toHaveBeenCalled();
|
||||
expect(await deliveryOf(record.action.id)).toBe("pending");
|
||||
|
||||
await holdPastBound(record);
|
||||
await deliverReconciledExecutions(db, wake as never);
|
||||
expect(wake).toHaveBeenCalledTimes(1);
|
||||
expect(wake.mock.calls[0]![0]).toBe(agentId);
|
||||
expect(await deliveryOf(record.action.id)).toBe("delivered");
|
||||
}, 60_000);
|
||||
|
||||
it("delivers the continuation after a restart while it was pending past the bound with a full queue", async () => {
|
||||
const agentId = await seedCompany(1);
|
||||
await queuedRecoveryRun(agentId, 240);
|
||||
const record = await pendingContinuation(agentId, 200);
|
||||
await deliverReconciledExecutions(db, fakeWake() as never);
|
||||
expect(await deliveryOf(record.action.id)).toBe("pending");
|
||||
await holdPastBound(record, 3 * 60 * 60_000);
|
||||
|
||||
const restartedSchedule = executionControlSchedule(db);
|
||||
const due = await restartedSchedule.continuation();
|
||||
expect(due).not.toBeNull();
|
||||
expect(due!).toBeLessThanOrEqual(Date.now() + 60_000);
|
||||
const restartedWake = fakeWake();
|
||||
await deliverReconciledExecutions(db, restartedWake as never);
|
||||
expect(restartedWake).toHaveBeenCalledTimes(1);
|
||||
expect(await deliveryOf(record.action.id)).toBe("delivered");
|
||||
expect(await restartedSchedule.continuation()).toBeNull();
|
||||
}, 60_000);
|
||||
|
||||
it("does not let a continuation whose wake cannot enter the queue hold back other continuations", async () => {
|
||||
const agentId = await seedCompany(1);
|
||||
const stuck: Array<Awaited<ReturnType<typeof pendingContinuation>>> = [];
|
||||
for (let index = 0; index < RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS; index += 1) {
|
||||
const record = await pendingContinuation(await addAgent(1), 300);
|
||||
await holdPastBound(record, 60 * 60_000);
|
||||
stuck.push(record);
|
||||
}
|
||||
const sameAgentStuck = await pendingContinuation(agentId, 300);
|
||||
await holdPastBound(sameAgentStuck, 60 * 60_000);
|
||||
const waiting = await pendingContinuation(agentId, 10);
|
||||
await holdPastBound(waiting);
|
||||
const stuckIds = new Set([...stuck, sameAgentStuck].map(record => record.action.id));
|
||||
const created = fakeWake();
|
||||
const wake = vi.fn(async (wakeAgentId: string, opts: { contextSnapshot: Record<string, unknown> }) =>
|
||||
stuckIds.has(String(opts.contextSnapshot.recoveryActionId)) ? null : created(wakeAgentId, opts));
|
||||
await deliverReconciledExecutions(db, wake as never);
|
||||
expect(created).not.toHaveBeenCalled();
|
||||
expect(await deliveryOf(waiting.action.id)).toBe("pending");
|
||||
await deliverReconciledExecutions(db, wake as never);
|
||||
expect(created).toHaveBeenCalledTimes(1);
|
||||
expect(await deliveryOf(waiting.action.id)).toBe("delivered");
|
||||
}, 60_000);
|
||||
|
||||
it("delivers an overdue continuation after a temporary refusal while new continuations keep arriving for a full queue", async () => {
|
||||
const agentId = await seedCompany(1);
|
||||
const busy = await task(agentId);
|
||||
await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: busy.id, invocationSource: "on_demand", status: "queued",
|
||||
contextSnapshot: { issueId: busy.id, wakeReason: "task_retry" } });
|
||||
const overdue = await pendingContinuation(agentId, 240);
|
||||
await holdPastBound(overdue);
|
||||
const created = fakeWake();
|
||||
let refusing = true;
|
||||
const wake = vi.fn(async (wakeAgentId: string, opts: { contextSnapshot: Record<string, unknown> }) =>
|
||||
refusing ? null : created(wakeAgentId, opts));
|
||||
await deliverReconciledExecutions(db, wake as never);
|
||||
expect(await deliveryOf(overdue.action.id)).toBe("pending");
|
||||
refusing = false;
|
||||
for (let pass = 0; pass < 3 && await deliveryOf(overdue.action.id) === "pending"; pass += 1) {
|
||||
await pendingContinuation(agentId, 1);
|
||||
await deliverReconciledExecutions(db, wake as never);
|
||||
}
|
||||
expect(await deliveryOf(overdue.action.id)).toBe("delivered");
|
||||
expect(created.mock.calls[0]![1].contextSnapshot.recoveryActionId).toBe(overdue.action.id);
|
||||
}, 60_000);
|
||||
|
||||
it("delivers another agent's continuation in one pass while one agent refuses a backlog of more than 200", async () => {
|
||||
const refusingAgent = await seedCompany(1);
|
||||
const heldSince = new Date(Date.now() - 3 * 60 * 60_000).toISOString();
|
||||
const backlog = await db.insert(issues).values(Array.from({ length: 210 }, () => ({ companyId, title: "Continue the work",
|
||||
status: "todo" as const, assigneeAgentId: refusingAgent }))).returning();
|
||||
await db.insert(issueRecoveryActions).values(backlog.map(issue => {
|
||||
const runId = randomUUID();
|
||||
return { companyId, sourceIssueId: issue.id, kind: "active_run_watchdog" as const, status: "resolved" as const,
|
||||
outcome: "handed_back" as const, resolvedAt: new Date(heldSince), ownerType: "board" as const, returnOwnerAgentId: refusingAgent,
|
||||
cause: "legacy_execution_requires_reconciliation", fingerprint: `legacy-execution:${runId}`, nextAction: "Continue the task.",
|
||||
attemptCount: 1, evidence: { runId, continuationDelivery: "pending",
|
||||
freshContinuation: { schema: "paperclip.fresh-continuation.v1", reason: "exact_resumption_unavailable", recordedAt: heldSince },
|
||||
executionReconciliation: { runId, observedOutcome: "unknown", disposition: "continue_fresh",
|
||||
actorId: "execution-recovery", recordedAt: heldSince } } };
|
||||
}));
|
||||
const healthy = await pendingContinuation(await addAgent(1), 30);
|
||||
const created = fakeWake();
|
||||
const wake = vi.fn(async (wakeAgentId: string, opts: { contextSnapshot: Record<string, unknown> }) =>
|
||||
wakeAgentId === refusingAgent ? null : created(wakeAgentId, opts));
|
||||
await deliverReconciledExecutions(db, wake as never);
|
||||
expect(wake.mock.calls.filter(call => call[0] === refusingAgent)).toHaveLength(1);
|
||||
expect(created).toHaveBeenCalledTimes(1);
|
||||
expect(await deliveryOf(healthy.action.id)).toBe("delivered");
|
||||
}, 120_000);
|
||||
|
||||
it("does not show a task as working when its continuation is pending past the delivery bound and nothing runs", async () => {
|
||||
const agentId = await seedCompany(1);
|
||||
const record = await pendingContinuation(agentId, 240);
|
||||
await holdPastBound(record, RECOVERY_CONTINUATION_DELIVERY_BOUND_MS);
|
||||
const [issueRow] = await db.select().from(issues).where(eq(issues.id, record.issue.id));
|
||||
const presentation = await taskInterruptionPresentation(db, issueRow!);
|
||||
expect(presentation?.state).not.toBe("working");
|
||||
|
||||
const fresh = await pendingContinuation(agentId, 30);
|
||||
const [freshRow] = await db.select().from(issues).where(eq(issues.id, fresh.issue.id));
|
||||
expect(await taskInterruptionPresentation(db, freshRow!)).toMatchObject({ state: "working", canRetry: false });
|
||||
}, 60_000);
|
||||
|
||||
it("reads a task whose continuation is held back as waiting for its agent, not idle and not needing Retry", async () => {
|
||||
const agentId = await seedCompany(1);
|
||||
const queued = await queuedRecoveryRun(agentId, 1);
|
||||
|
||||
@@ -43,8 +43,11 @@ import { alias } from "drizzle-orm/pg-core";
|
||||
import { RECOVERY_WAKE_REASONS } from "./heartbeat/queued-run-dispatch-order.js";
|
||||
import { parseHeartbeatPolicy } from "./heartbeat/timer-policy.js";
|
||||
|
||||
export const RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS = 5;
|
||||
export const RECOVERY_CONTINUATION_MAX_HOLD_MS = 5 * 60_000;
|
||||
import { RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS, RECOVERY_CONTINUATION_MAX_HOLD_MS,
|
||||
recoveryContinuationHeldSince } from "./recovery-continuation-pacing.js";
|
||||
|
||||
export { RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS, RECOVERY_CONTINUATION_DELIVERY_BOUND_MS,
|
||||
RECOVERY_CONTINUATION_MAX_HOLD_MS } from "./recovery-continuation-pacing.js";
|
||||
const RECOVERY_CONTINUATION_PAGES_PER_PASS = 8;
|
||||
|
||||
/** An operator records observed outcomes; this is not permission to blindly retry. */
|
||||
@@ -375,8 +378,27 @@ export async function deliverReconciledExecutions(
|
||||
const readPendingPage = async () => {
|
||||
const companies = fullCompanies();
|
||||
const heldBound = new Date(Date.now() - RECOVERY_CONTINUATION_MAX_HOLD_MS).toISOString();
|
||||
const rows = await db
|
||||
.select({ action: issueRecoveryActions })
|
||||
const pastBound = sql`${heldSince} <= ${heldBound}::timestamptz`;
|
||||
const overdueTurn = sql`case when ${pastBound} then greatest(${heldSince},
|
||||
(${issueRecoveryActions.evidence}->>'continuationDeliveryAttemptedAt')::timestamptz) end`;
|
||||
const priority = [
|
||||
sql`${pastBound} desc`,
|
||||
sql`${overdueTurn} asc nulls last`,
|
||||
sql`${sourceRun.finishedAt} desc nulls last`,
|
||||
sql`${issueRecoveryActions.updatedAt} asc`,
|
||||
];
|
||||
const candidates = db
|
||||
.select({
|
||||
id: issueRecoveryActions.id,
|
||||
pastBound: sql<boolean>`${pastBound}`.as("past_bound"),
|
||||
heldAt: sql<Date | null>`${overdueTurn}`.as("held_at"),
|
||||
sourceFinishedAt: sql<Date | null>`${sourceRun.finishedAt}`.as("source_finished_at"),
|
||||
actionUpdatedAt: sql<Date>`${issueRecoveryActions.updatedAt}`.as("action_updated_at"),
|
||||
agentRank: sql<number>`row_number() over (
|
||||
partition by coalesce(${issueRecoveryActions.returnOwnerAgentId}::text, ${issueRecoveryActions.id}::text)
|
||||
order by ${sql.join(priority, sql`, `)})`
|
||||
.as("agent_rank"),
|
||||
})
|
||||
.from(issueRecoveryActions)
|
||||
.leftJoin(sourceRun, and(
|
||||
eq(sourceRun.companyId, issueRecoveryActions.companyId),
|
||||
@@ -389,22 +411,37 @@ export async function deliverReconciledExecutions(
|
||||
seenActionIds.size ? not(inArray(issueRecoveryActions.id, [...seenActionIds])) : undefined,
|
||||
seenAgentIds.size ? sql`coalesce(${issueRecoveryActions.returnOwnerAgentId}::text, '') <> all(${sql.param([...seenAgentIds])}::text[])` : undefined,
|
||||
companies.length ? not(inArray(issueRecoveryActions.companyId, companies)) : undefined,
|
||||
sql`not exists (select 1 from ${heartbeatRuns} queued_recovery
|
||||
sql`(${pastBound} or not exists (select 1 from ${heartbeatRuns} queued_recovery
|
||||
where queued_recovery.company_id = ${issueRecoveryActions.companyId}
|
||||
and queued_recovery.agent_id = ${issueRecoveryActions.returnOwnerAgentId}
|
||||
and queued_recovery.status = 'queued'
|
||||
and queued_recovery.context_snapshot->>'wakeReason' in (${sql.join(recoveryReasons.map(reason => sql`${reason}`), sql`, `)}))`,
|
||||
and queued_recovery.context_snapshot->>'wakeReason' in (${sql.join(recoveryReasons.map(reason => sql`${reason}`), sql`, `)})))`,
|
||||
),
|
||||
)
|
||||
.as("continuation_candidates");
|
||||
const rows = await db
|
||||
.select({ action: issueRecoveryActions })
|
||||
.from(candidates)
|
||||
.innerJoin(issueRecoveryActions, eq(issueRecoveryActions.id, candidates.id))
|
||||
.where(eq(candidates.agentRank, 1))
|
||||
.orderBy(
|
||||
sql`${heldSince} <= ${heldBound}::timestamptz desc`,
|
||||
sql`case when ${heldSince} <= ${heldBound}::timestamptz then ${heldSince} end asc nulls last`,
|
||||
sql`${sourceRun.finishedAt} desc nulls last`,
|
||||
asc(issueRecoveryActions.updatedAt),
|
||||
sql`${candidates.pastBound} desc`,
|
||||
sql`${candidates.heldAt} asc nulls last`,
|
||||
sql`${candidates.sourceFinishedAt} desc nulls last`,
|
||||
sql`${candidates.actionUpdatedAt} asc`,
|
||||
)
|
||||
.limit(25);
|
||||
return rows.map(row => row.action);
|
||||
};
|
||||
const deferRefusedContinuation = (action: typeof issueRecoveryActions.$inferSelect) => db.update(issueRecoveryActions)
|
||||
.set({ evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify(
|
||||
{ continuationDeliveryAttemptedAt: new Date().toISOString() })}::jsonb` })
|
||||
.where(and(
|
||||
eq(issueRecoveryActions.companyId, action.companyId),
|
||||
eq(issueRecoveryActions.id, action.id),
|
||||
eq(issueRecoveryActions.status, "resolved"),
|
||||
sql`${issueRecoveryActions.evidence}->>'continuationDelivery' = 'pending'`,
|
||||
));
|
||||
const agentHasRoom = async (action: typeof issueRecoveryActions.$inferSelect, agentId: string) => {
|
||||
if ((admittedByCompany.get(action.companyId) ?? 0) >= RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS) return false;
|
||||
const [agent] = await db.select().from(agents).where(and(eq(agents.companyId, action.companyId), eq(agents.id, agentId)));
|
||||
@@ -412,19 +449,14 @@ export async function deliverReconciledExecutions(
|
||||
const [queued] = await db.select({ count: sql<number>`count(*)::int` }).from(heartbeatRuns).where(and(
|
||||
eq(heartbeatRuns.companyId, action.companyId), eq(heartbeatRuns.agentId, agentId), eq(heartbeatRuns.status, "queued")));
|
||||
if (Number(queued?.count ?? 0) < parseHeartbeatPolicy(agent).maxConcurrentRuns) return true;
|
||||
const recordedAt = Date.parse(String((action.evidence.freshContinuation as { recordedAt?: unknown } | undefined)?.recordedAt ?? ""));
|
||||
const heldSince = Number.isFinite(recordedAt) ? recordedAt : (action.resolvedAt ?? action.updatedAt).getTime();
|
||||
return Date.now() - heldSince >= RECOVERY_CONTINUATION_MAX_HOLD_MS;
|
||||
return Date.now() - recoveryContinuationHeldSince(action) >= RECOVERY_CONTINUATION_MAX_HOLD_MS;
|
||||
};
|
||||
for (let page = 0; page < RECOVERY_CONTINUATION_PAGES_PER_PASS; page += 1) {
|
||||
const pending = await readPendingPage();
|
||||
if (!pending.length) break;
|
||||
for (const action of pending) {
|
||||
seenActionIds.add(action.id);
|
||||
if (action.returnOwnerAgentId) {
|
||||
if (seenAgentIds.has(action.returnOwnerAgentId)) continue;
|
||||
seenAgentIds.add(action.returnOwnerAgentId);
|
||||
}
|
||||
if (action.returnOwnerAgentId && seenAgentIds.has(action.returnOwnerAgentId)) continue;
|
||||
try {
|
||||
const decision = action.evidence.executionReconciliation as
|
||||
ExecutionReconciliation | undefined;
|
||||
@@ -478,8 +510,10 @@ export async function deliverReconciledExecutions(
|
||||
continue;
|
||||
}
|
||||
}
|
||||
if (!await agentHasRoom(action, action.returnOwnerAgentId)) continue;
|
||||
admittedByCompany.set(action.companyId, (admittedByCompany.get(action.companyId) ?? 0) + 1);
|
||||
if (!await agentHasRoom(action, action.returnOwnerAgentId)) {
|
||||
seenAgentIds.add(action.returnOwnerAgentId);
|
||||
continue;
|
||||
}
|
||||
const run = await wake(action.returnOwnerAgentId, {
|
||||
source: "automation",
|
||||
triggerDetail: "system",
|
||||
@@ -501,35 +535,42 @@ export async function deliverReconciledExecutions(
|
||||
? { freshContinuationIsolation: action.id } : {}),
|
||||
},
|
||||
});
|
||||
if (run)
|
||||
await db.transaction(async (tx) => {
|
||||
await tx
|
||||
.update(heartbeatRuns)
|
||||
.set({ retryOfRunId: decision.runId })
|
||||
.where(
|
||||
and(
|
||||
eq(heartbeatRuns.companyId, action.companyId),
|
||||
eq(heartbeatRuns.id, run.id),
|
||||
eq(heartbeatRuns.agentId, action.returnOwnerAgentId!),
|
||||
sql`${heartbeatRuns.contextSnapshot}->>'recoveryActionId' = ${action.id}`,
|
||||
sql`${heartbeatRuns.contextSnapshot}->>'previousRunId' = ${decision.runId}`,
|
||||
),
|
||||
);
|
||||
await tx
|
||||
.update(issueRecoveryActions)
|
||||
.set({
|
||||
evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify(
|
||||
{
|
||||
continuationDelivery: "delivered",
|
||||
continuationRunId: run.id,
|
||||
},
|
||||
)}::jsonb`,
|
||||
})
|
||||
.where(pendingDecision);
|
||||
await settleCoalescedFreshContinuationHolds(tx as unknown as Db, { companyId: action.companyId, primaryId: action.id,
|
||||
delivery: "delivered", continuationRunId: run.id });
|
||||
});
|
||||
seenAgentIds.add(action.returnOwnerAgentId);
|
||||
if (!run) {
|
||||
await deferRefusedContinuation(action);
|
||||
continue;
|
||||
}
|
||||
admittedByCompany.set(action.companyId, (admittedByCompany.get(action.companyId) ?? 0) + 1);
|
||||
await db.transaction(async (tx) => {
|
||||
await tx
|
||||
.update(heartbeatRuns)
|
||||
.set({ retryOfRunId: decision.runId })
|
||||
.where(
|
||||
and(
|
||||
eq(heartbeatRuns.companyId, action.companyId),
|
||||
eq(heartbeatRuns.id, run.id),
|
||||
eq(heartbeatRuns.agentId, action.returnOwnerAgentId!),
|
||||
sql`${heartbeatRuns.contextSnapshot}->>'recoveryActionId' = ${action.id}`,
|
||||
sql`${heartbeatRuns.contextSnapshot}->>'previousRunId' = ${decision.runId}`,
|
||||
),
|
||||
);
|
||||
await tx
|
||||
.update(issueRecoveryActions)
|
||||
.set({
|
||||
evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify(
|
||||
{
|
||||
continuationDelivery: "delivered",
|
||||
continuationRunId: run.id,
|
||||
},
|
||||
)}::jsonb`,
|
||||
})
|
||||
.where(pendingDecision);
|
||||
await settleCoalescedFreshContinuationHolds(tx as unknown as Db, { companyId: action.companyId, primaryId: action.id,
|
||||
delivery: "delivered", continuationRunId: run.id });
|
||||
});
|
||||
} catch {
|
||||
if (action.returnOwnerAgentId) seenAgentIds.add(action.returnOwnerAgentId);
|
||||
await deferRefusedContinuation(action).catch(() => undefined);
|
||||
logger.warn(
|
||||
{ recoveryActionId: action.id },
|
||||
"Reconciled execution continuation remains pending for retry",
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
export const RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS = 5;
|
||||
export const RECOVERY_CONTINUATION_MAX_HOLD_MS = 5 * 60_000;
|
||||
export const RECOVERY_CONTINUATION_DELIVERY_BOUND_MS = 15 * 60_000;
|
||||
|
||||
export function recoveryContinuationHeldSince(input: {
|
||||
evidence: Record<string, unknown>;
|
||||
resolvedAt: Date | null;
|
||||
updatedAt: Date;
|
||||
}) {
|
||||
const recordedAt = Date.parse(String((input.evidence.freshContinuation as { recordedAt?: unknown } | undefined)?.recordedAt ?? ""));
|
||||
return Number.isFinite(recordedAt) ? recordedAt : (input.resolvedAt ?? input.updatedAt).getTime();
|
||||
}
|
||||
@@ -11,6 +11,7 @@ import { EXECUTION_RECONCILIATION_CAUSES, HEARTBEAT_RUN_STATUSES, readProviderFa
|
||||
import { captureTaskInterruptionFailure } from "../sentry.js";
|
||||
import { isAiAuthenticationBlocked } from "./ai-auth-failure.js";
|
||||
import { readSystemRecovery } from "./system-recovery-state.js";
|
||||
import { RECOVERY_CONTINUATION_DELIVERY_BOUND_MS, recoveryContinuationHeldSince } from "./recovery-continuation-pacing.js";
|
||||
import { PROJECT_ACCESS_DENIED } from "./project-access-denial.js";
|
||||
import { authorizationService } from "./authorization.js";
|
||||
import { taskWorkspaceProjectRequirements } from "./task-workspace-access.js";
|
||||
@@ -582,6 +583,8 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i
|
||||
runId: sql<string>`${issueRecoveryActions.evidence}->>'runId'`,
|
||||
delivery: sql<string | null>`${issueRecoveryActions.evidence}->>'continuationDelivery'`,
|
||||
recorded: sql<boolean>`${issueRecoveryActions.evidence} ? 'freshContinuation'`,
|
||||
evidence: sql<Record<string, unknown>>`jsonb_build_object('freshContinuation', ${issueRecoveryActions.evidence}->'freshContinuation')`,
|
||||
resolvedAt: issueRecoveryActions.resolvedAt, updatedAt: issueRecoveryActions.updatedAt,
|
||||
checkKind: sql<string | null>`${issueRecoveryActions.evidence}#>>'{freshContinuationCheck,kind}'`,
|
||||
nextCheckAt: sql<string | null>`${issueRecoveryActions.evidence}#>>'{freshContinuationCheck,nextCheckAt}'` })
|
||||
.from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, companyId), inArray(issueRecoveryActions.sourceIssueId, ids),
|
||||
@@ -591,7 +594,8 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i
|
||||
const rows = freshRows.filter(row => row.issueId === issueId && row.runId === runId);
|
||||
// Only the primary record (it carries `freshContinuation`) says whether a
|
||||
// continuation is still on its way; holds coalesced into it never do.
|
||||
if (rows.some(row => row.recorded && ["pending", "not_requested"].includes(row.delivery ?? ""))) return { state: "continuing" };
|
||||
if (rows.some(row => row.recorded && (row.delivery === "not_requested" || (row.delivery === "pending"
|
||||
&& Date.now() - recoveryContinuationHeldSince(row) < RECOVERY_CONTINUATION_DELIVERY_BOUND_MS)))) return { state: "continuing" };
|
||||
const scheduled = rows.filter(row => ["exact_recovery_pending", "backoff"].includes(row.checkKind ?? "") && row.nextCheckAt)
|
||||
.map(row => Number(row.nextCheckAt)).sort((a, b) => a - b)[0];
|
||||
return scheduled ? { state: "scheduled", nextCheckAt: scheduled } : undefined;
|
||||
|
||||
Reference in new issue
Block a user