fix(recovery): deliver every pending continuation within a bounded time (#21324)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Execution recovery continues a task in a fresh run after a restart
or a lost run left it unfinished
> - #21234 paced these continuations: at most 1 per agent and 5 per
company per pass, and a continuation waits while its agent already has a
queued recovery run or a full queue
> - A continuation that waited 5 minutes must enter the queue anyway,
but that bound applied only to the full-queue check. The "agent already
has a queued recovery run" check was in the page query, and it had no
bound.
> - On staging, an agent had queued recovery runs that did not start for
hours. Its pending continuation was never read again, and its task
showed "working" with nothing running or queued
> - This pull request applies the 5-minute bound to both checks, stops
one undeliverable continuation from using its agent's or company's slot,
and stops the task view from showing "working" for a continuation that
is overdue
> - The benefit is that every pending continuation enters the queue
within a bounded time, also after a restart, and the pacing still
protects people from recovery herds

## Linked Issues or Issue Description

Refs #21234, #21135

**What happened?**

On staging, a task had a fresh-continuation hold with
`continuationDelivery: "pending"` since about 03:30Z. More than 3.5
hours later, no run had started for the task. The task view showed the
interruption state `working` with `canRetry: false`, so the user saw
nothing wrong. The task's agent had 20 concurrency slots and 11 running
runs. It also had queued `issue_continuation_needed` runs from 22:54Z
that had not started. The delivery pass reads pending continuations with
a query that leaves out every agent that has a queued recovery run
(`issue_recovery_action_restored` or `issue_continuation_needed`). That
condition had no time bound, so the pass never read this continuation
again. The 5-minute admission bound was never reached. Across the
company, 203 open tasks showed `working` while the API showed no queued
or running run for them. 162 queued runs were more than 1 hour old.

**Expected behavior**

Every pending continuation enters its agent's queue within a bounded
time, also when the agent's queue holds recovery runs that do not start,
and also after a server restart. Pacing still limits how many
continuations enter a queue at the same time. A task does not show
"working" when its continuation is overdue and nothing runs or is
queued.

**Steps to reproduce**

1. Give an agent a concurrency limit of 1.
2. Put one queued `issue_recovery_action_restored` run on that agent
that does not start.
3. Record a pending fresh continuation for another task of the same
agent.
4. Wait more than 5 minutes, and let the 15-second continuation pass
run.
5. See that the continuation stays `pending`, the task shows `working`,
and no run starts.

**Paperclip version or commit**

master at bbafa78fe2 (includes #21234, ade7e0643f)

**Deployment mode**

Hosted cloud (staging)

**Agent adapter(s) involved**

paperclip_runner (native runtime). The problem is in the shared
continuation delivery pass, so it applies to all adapters.

## What Changed

- `deliverReconciledExecutions` in
`server/src/services/execution-recovery-resolution.ts`: a continuation
held for 5 minutes or more is read even when its agent has a queued
recovery run. Before, only the full-queue check let it through.
- Each page of the pass now reads at most one candidate per agent
(`row_number() over (partition by agent)`), so one agent's backlog
cannot fill the page budget and hide other agents' continuations.
- A continuation counts against the company's limit only when its wake
puts a run in the queue. A continuation that was invalidated or replaced
by a newer run does not use its agent's turn.
- When an agent refuses a wake (no run, or an error), the pass skips
that agent for the rest of the pass and stamps the row with
`continuationDeliveryAttemptedAt`. Overdue rows always rank before rows
that are not overdue. Among overdue rows, the turn is the later of the
hold time and the last attempt, so a refused row goes behind rows held
before that attempt and ahead of rows held after it. One refused row
cannot pin its agent, and new arrivals cannot keep an older refused row
waiting.
- New module `server/src/services/recovery-continuation-pacing.ts` holds
the pacing constants and the "held since" rule. It adds
`RECOVERY_CONTINUATION_DELIVERY_BOUND_MS` (15 minutes). The old
constants are still exported from `execution-recovery-resolution.ts`.
- `taskInterruptionPresentations` in
`server/src/services/task-interruption-presentation.ts` shows a pending
continuation as "continuing" only within the delivery bound. After that,
the task shows its real state instead of `working`.
- No change to the schedule: the continuation duty runs only while
pending continuations exist, and it checks the database on start, so a
restart does not lose pending rows.

## Verification

- `pnpm exec vitest run src/__tests__/recovery-wake-herd.test.ts` (in
`server/`): 15 tests pass. New tests:
- A continuation held past the bound is delivered while the agent's
queue holds recovery runs that never start.
- After a restart with a continuation pending past the bound and a full
queue, a new schedule is due within one pass, the pass delivers the
continuation, and the schedule then goes idle.
- Continuations whose wake returns no run do not use the per-company
count. Another continuation of the same agent is delivered on the next
pass.
- An overdue continuation is refused once while the agent's queue is
full. On the next pass, it is delivered even though new continuations
keep arriving for the same agent.
- One agent refuses a backlog of 210 overdue continuations. Another
agent's continuation is delivered in the same pass, and the refusing
agent gets one wake attempt in that pass.
- A task whose continuation is pending past the delivery bound with
nothing running does not show `working`. A fresh pending continuation
still shows `working` with `canRetry: false`.
- Red/green: with the two source files reverted to master (and only the
new constant exported), all 4 first new tests fail. The backlog test
fails on the first PR head (200 refused wake attempts in one pass). The
temporary-refusal test fails on the second PR head (the overdue row
stays pending). With this change, all 15 pass.
- Related suites pass: fresh-continuation fallback, workspace recovery
fallback, native safe replacement, native workspace reconstruction, task
interruption presentation, queued-run dispatch order, execution recovery
resolution, assigned-work invariant, execution control work, task
interruption automatic path and query.
- `node scripts/typecheck-layers.mjs` (in `server/`) passes.

## Risks

- A continuation that waited 5 minutes now enters a queue that already
has recovery runs. The per-agent (1) and per-company (5) limits per pass
still apply, and the queue still starts person-initiated work first.
- An agent that refuses wakes gets one wake attempt per pass. Its
refused rows rotate through `continuationDeliveryAttemptedAt`, and they
do not block other agents.
- After 15 minutes without delivery, the task no longer shows `working`.
With this fix, delivery happens well within that bound, so this is a
safety net.
- No schema or migration changes. No new polling.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs, and read-only staging
API requests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-13 02:12:13 -06:00
1 parent 2483cb46c2
commit fde0328ff5
4 files changed
+236 -49

No files matched your search

+131 -1
View File
@@ -7,7 +7,8 @@ import {
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { heartbeatService } from "../services/heartbeat.js";
import {
RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS, RECOVERY_CONTINUATION_MAX_HOLD_MS, deliverReconciledExecutions,
RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS, RECOVERY_CONTINUATION_DELIVERY_BOUND_MS, RECOVERY_CONTINUATION_MAX_HOLD_MS,
deliverReconciledExecutions,
} from "../services/execution-recovery-resolution.js";
import { assignedWorkSweep } from "../services/recovery/assigned-work-sweep.js";
import { evaluateAssignedWork } from "../services/recovery/assigned-work-invariant.js";
@@ -318,6 +319,135 @@ suite("recovery wake herd", () => {
expect(await Promise.all(old.map(record => deliveryOf(record.action.id)))).toEqual(old.map(() => "delivered"));
}, 60_000);
async function holdPastBound(record: Awaited<ReturnType<typeof pendingContinuation>>, extraMs = 1_000) {
const heldSince = new Date(Date.now() - RECOVERY_CONTINUATION_MAX_HOLD_MS - extraMs).toISOString();
await db.update(issueRecoveryActions).set({ evidence: { ...record.action.evidence,
freshContinuation: { ...(record.action.evidence.freshContinuation as object), recordedAt: heldSince } } })
.where(eq(issueRecoveryActions.id, record.action.id));
}
it("delivers a continuation held past the bound while the agent's queue is full of recovery runs that never start", async () => {
const agentId = await seedCompany(1);
await queuedRecoveryRun(agentId, 240);
await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: (await task(agentId)).id, invocationSource: "automation",
triggerDetail: "system", status: "queued", createdAt: new Date(Date.now() - 230 * 60_000),
contextSnapshot: { wakeReason: "issue_continuation_needed" } });
const record = await pendingContinuation(agentId, 200);
const wake = fakeWake();
await deliverReconciledExecutions(db, wake as never);
expect(wake).not.toHaveBeenCalled();
expect(await deliveryOf(record.action.id)).toBe("pending");
await holdPastBound(record);
await deliverReconciledExecutions(db, wake as never);
expect(wake).toHaveBeenCalledTimes(1);
expect(wake.mock.calls[0]![0]).toBe(agentId);
expect(await deliveryOf(record.action.id)).toBe("delivered");
}, 60_000);
it("delivers the continuation after a restart while it was pending past the bound with a full queue", async () => {
const agentId = await seedCompany(1);
await queuedRecoveryRun(agentId, 240);
const record = await pendingContinuation(agentId, 200);
await deliverReconciledExecutions(db, fakeWake() as never);
expect(await deliveryOf(record.action.id)).toBe("pending");
await holdPastBound(record, 3 * 60 * 60_000);
const restartedSchedule = executionControlSchedule(db);
const due = await restartedSchedule.continuation();
expect(due).not.toBeNull();
expect(due!).toBeLessThanOrEqual(Date.now() + 60_000);
const restartedWake = fakeWake();
await deliverReconciledExecutions(db, restartedWake as never);
expect(restartedWake).toHaveBeenCalledTimes(1);
expect(await deliveryOf(record.action.id)).toBe("delivered");
expect(await restartedSchedule.continuation()).toBeNull();
}, 60_000);
it("does not let a continuation whose wake cannot enter the queue hold back other continuations", async () => {
const agentId = await seedCompany(1);
const stuck: Array<Awaited<ReturnType<typeof pendingContinuation>>> = [];
for (let index = 0; index < RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS; index += 1) {
const record = await pendingContinuation(await addAgent(1), 300);
await holdPastBound(record, 60 * 60_000);
stuck.push(record);
}
const sameAgentStuck = await pendingContinuation(agentId, 300);
await holdPastBound(sameAgentStuck, 60 * 60_000);
const waiting = await pendingContinuation(agentId, 10);
await holdPastBound(waiting);
const stuckIds = new Set([...stuck, sameAgentStuck].map(record => record.action.id));
const created = fakeWake();
const wake = vi.fn(async (wakeAgentId: string, opts: { contextSnapshot: Record<string, unknown> }) =>
stuckIds.has(String(opts.contextSnapshot.recoveryActionId)) ? null : created(wakeAgentId, opts));
await deliverReconciledExecutions(db, wake as never);
expect(created).not.toHaveBeenCalled();
expect(await deliveryOf(waiting.action.id)).toBe("pending");
await deliverReconciledExecutions(db, wake as never);
expect(created).toHaveBeenCalledTimes(1);
expect(await deliveryOf(waiting.action.id)).toBe("delivered");
}, 60_000);
it("delivers an overdue continuation after a temporary refusal while new continuations keep arriving for a full queue", async () => {
const agentId = await seedCompany(1);
const busy = await task(agentId);
await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: busy.id, invocationSource: "on_demand", status: "queued",
contextSnapshot: { issueId: busy.id, wakeReason: "task_retry" } });
const overdue = await pendingContinuation(agentId, 240);
await holdPastBound(overdue);
const created = fakeWake();
let refusing = true;
const wake = vi.fn(async (wakeAgentId: string, opts: { contextSnapshot: Record<string, unknown> }) =>
refusing ? null : created(wakeAgentId, opts));
await deliverReconciledExecutions(db, wake as never);
expect(await deliveryOf(overdue.action.id)).toBe("pending");
refusing = false;
for (let pass = 0; pass < 3 && await deliveryOf(overdue.action.id) === "pending"; pass += 1) {
await pendingContinuation(agentId, 1);
await deliverReconciledExecutions(db, wake as never);
}
expect(await deliveryOf(overdue.action.id)).toBe("delivered");
expect(created.mock.calls[0]![1].contextSnapshot.recoveryActionId).toBe(overdue.action.id);
}, 60_000);
it("delivers another agent's continuation in one pass while one agent refuses a backlog of more than 200", async () => {
const refusingAgent = await seedCompany(1);
const heldSince = new Date(Date.now() - 3 * 60 * 60_000).toISOString();
const backlog = await db.insert(issues).values(Array.from({ length: 210 }, () => ({ companyId, title: "Continue the work",
status: "todo" as const, assigneeAgentId: refusingAgent }))).returning();
await db.insert(issueRecoveryActions).values(backlog.map(issue => {
const runId = randomUUID();
return { companyId, sourceIssueId: issue.id, kind: "active_run_watchdog" as const, status: "resolved" as const,
outcome: "handed_back" as const, resolvedAt: new Date(heldSince), ownerType: "board" as const, returnOwnerAgentId: refusingAgent,
cause: "legacy_execution_requires_reconciliation", fingerprint: `legacy-execution:${runId}`, nextAction: "Continue the task.",
attemptCount: 1, evidence: { runId, continuationDelivery: "pending",
freshContinuation: { schema: "paperclip.fresh-continuation.v1", reason: "exact_resumption_unavailable", recordedAt: heldSince },
executionReconciliation: { runId, observedOutcome: "unknown", disposition: "continue_fresh",
actorId: "execution-recovery", recordedAt: heldSince } } };
}));
const healthy = await pendingContinuation(await addAgent(1), 30);
const created = fakeWake();
const wake = vi.fn(async (wakeAgentId: string, opts: { contextSnapshot: Record<string, unknown> }) =>
wakeAgentId === refusingAgent ? null : created(wakeAgentId, opts));
await deliverReconciledExecutions(db, wake as never);
expect(wake.mock.calls.filter(call => call[0] === refusingAgent)).toHaveLength(1);
expect(created).toHaveBeenCalledTimes(1);
expect(await deliveryOf(healthy.action.id)).toBe("delivered");
}, 120_000);
it("does not show a task as working when its continuation is pending past the delivery bound and nothing runs", async () => {
const agentId = await seedCompany(1);
const record = await pendingContinuation(agentId, 240);
await holdPastBound(record, RECOVERY_CONTINUATION_DELIVERY_BOUND_MS);
const [issueRow] = await db.select().from(issues).where(eq(issues.id, record.issue.id));
const presentation = await taskInterruptionPresentation(db, issueRow!);
expect(presentation?.state).not.toBe("working");
const fresh = await pendingContinuation(agentId, 30);
const [freshRow] = await db.select().from(issues).where(eq(issues.id, fresh.issue.id));
expect(await taskInterruptionPresentation(db, freshRow!)).toMatchObject({ state: "working", canRetry: false });
}, 60_000);
it("reads a task whose continuation is held back as waiting for its agent, not idle and not needing Retry", async () => {
const agentId = await seedCompany(1);
const queued = await queuedRecoveryRun(agentId, 1);
@@ -43,8 +43,11 @@ import { alias } from "drizzle-orm/pg-core";
import { RECOVERY_WAKE_REASONS } from "./heartbeat/queued-run-dispatch-order.js";
import { parseHeartbeatPolicy } from "./heartbeat/timer-policy.js";
export const RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS = 5;
export const RECOVERY_CONTINUATION_MAX_HOLD_MS = 5 * 60_000;
import { RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS, RECOVERY_CONTINUATION_MAX_HOLD_MS,
recoveryContinuationHeldSince } from "./recovery-continuation-pacing.js";
export { RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS, RECOVERY_CONTINUATION_DELIVERY_BOUND_MS,
RECOVERY_CONTINUATION_MAX_HOLD_MS } from "./recovery-continuation-pacing.js";
const RECOVERY_CONTINUATION_PAGES_PER_PASS = 8;
/** An operator records observed outcomes; this is not permission to blindly retry. */
@@ -375,8 +378,27 @@ export async function deliverReconciledExecutions(
const readPendingPage = async () => {
const companies = fullCompanies();
const heldBound = new Date(Date.now() - RECOVERY_CONTINUATION_MAX_HOLD_MS).toISOString();
const rows = await db
.select({ action: issueRecoveryActions })
const pastBound = sql`${heldSince} <= ${heldBound}::timestamptz`;
const overdueTurn = sql`case when ${pastBound} then greatest(${heldSince},
(${issueRecoveryActions.evidence}->>'continuationDeliveryAttemptedAt')::timestamptz) end`;
const priority = [
sql`${pastBound} desc`,
sql`${overdueTurn} asc nulls last`,
sql`${sourceRun.finishedAt} desc nulls last`,
sql`${issueRecoveryActions.updatedAt} asc`,
];
const candidates = db
.select({
id: issueRecoveryActions.id,
pastBound: sql<boolean>`${pastBound}`.as("past_bound"),
heldAt: sql<Date | null>`${overdueTurn}`.as("held_at"),
sourceFinishedAt: sql<Date | null>`${sourceRun.finishedAt}`.as("source_finished_at"),
actionUpdatedAt: sql<Date>`${issueRecoveryActions.updatedAt}`.as("action_updated_at"),
agentRank: sql<number>`row_number() over (
partition by coalesce(${issueRecoveryActions.returnOwnerAgentId}::text, ${issueRecoveryActions.id}::text)
order by ${sql.join(priority, sql`, `)})`
.as("agent_rank"),
})
.from(issueRecoveryActions)
.leftJoin(sourceRun, and(
eq(sourceRun.companyId, issueRecoveryActions.companyId),
@@ -389,22 +411,37 @@ export async function deliverReconciledExecutions(
seenActionIds.size ? not(inArray(issueRecoveryActions.id, [...seenActionIds])) : undefined,
seenAgentIds.size ? sql`coalesce(${issueRecoveryActions.returnOwnerAgentId}::text, '') <> all(${sql.param([...seenAgentIds])}::text[])` : undefined,
companies.length ? not(inArray(issueRecoveryActions.companyId, companies)) : undefined,
sql`not exists (select 1 from ${heartbeatRuns} queued_recovery
sql`(${pastBound} or not exists (select 1 from ${heartbeatRuns} queued_recovery
where queued_recovery.company_id = ${issueRecoveryActions.companyId}
and queued_recovery.agent_id = ${issueRecoveryActions.returnOwnerAgentId}
and queued_recovery.status = 'queued'
and queued_recovery.context_snapshot->>'wakeReason' in (${sql.join(recoveryReasons.map(reason => sql`${reason}`), sql`, `)}))`,
and queued_recovery.context_snapshot->>'wakeReason' in (${sql.join(recoveryReasons.map(reason => sql`${reason}`), sql`, `)})))`,
),
)
.as("continuation_candidates");
const rows = await db
.select({ action: issueRecoveryActions })
.from(candidates)
.innerJoin(issueRecoveryActions, eq(issueRecoveryActions.id, candidates.id))
.where(eq(candidates.agentRank, 1))
.orderBy(
sql`${heldSince} <= ${heldBound}::timestamptz desc`,
sql`case when ${heldSince} <= ${heldBound}::timestamptz then ${heldSince} end asc nulls last`,
sql`${sourceRun.finishedAt} desc nulls last`,
asc(issueRecoveryActions.updatedAt),
sql`${candidates.pastBound} desc`,
sql`${candidates.heldAt} asc nulls last`,
sql`${candidates.sourceFinishedAt} desc nulls last`,
sql`${candidates.actionUpdatedAt} asc`,
)
.limit(25);
return rows.map(row => row.action);
};
const deferRefusedContinuation = (action: typeof issueRecoveryActions.$inferSelect) => db.update(issueRecoveryActions)
.set({ evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify(
{ continuationDeliveryAttemptedAt: new Date().toISOString() })}::jsonb` })
.where(and(
eq(issueRecoveryActions.companyId, action.companyId),
eq(issueRecoveryActions.id, action.id),
eq(issueRecoveryActions.status, "resolved"),
sql`${issueRecoveryActions.evidence}->>'continuationDelivery' = 'pending'`,
));
const agentHasRoom = async (action: typeof issueRecoveryActions.$inferSelect, agentId: string) => {
if ((admittedByCompany.get(action.companyId) ?? 0) >= RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS) return false;
const [agent] = await db.select().from(agents).where(and(eq(agents.companyId, action.companyId), eq(agents.id, agentId)));
@@ -412,19 +449,14 @@ export async function deliverReconciledExecutions(
const [queued] = await db.select({ count: sql<number>`count(*)::int` }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, action.companyId), eq(heartbeatRuns.agentId, agentId), eq(heartbeatRuns.status, "queued")));
if (Number(queued?.count ?? 0) < parseHeartbeatPolicy(agent).maxConcurrentRuns) return true;
const recordedAt = Date.parse(String((action.evidence.freshContinuation as { recordedAt?: unknown } | undefined)?.recordedAt ?? ""));
const heldSince = Number.isFinite(recordedAt) ? recordedAt : (action.resolvedAt ?? action.updatedAt).getTime();
return Date.now() - heldSince >= RECOVERY_CONTINUATION_MAX_HOLD_MS;
return Date.now() - recoveryContinuationHeldSince(action) >= RECOVERY_CONTINUATION_MAX_HOLD_MS;
};
for (let page = 0; page < RECOVERY_CONTINUATION_PAGES_PER_PASS; page += 1) {
const pending = await readPendingPage();
if (!pending.length) break;
for (const action of pending) {
seenActionIds.add(action.id);
if (action.returnOwnerAgentId) {
if (seenAgentIds.has(action.returnOwnerAgentId)) continue;
seenAgentIds.add(action.returnOwnerAgentId);
}
if (action.returnOwnerAgentId && seenAgentIds.has(action.returnOwnerAgentId)) continue;
try {
const decision = action.evidence.executionReconciliation as
ExecutionReconciliation | undefined;
@@ -478,8 +510,10 @@ export async function deliverReconciledExecutions(
continue;
}
}
if (!await agentHasRoom(action, action.returnOwnerAgentId)) continue;
admittedByCompany.set(action.companyId, (admittedByCompany.get(action.companyId) ?? 0) + 1);
if (!await agentHasRoom(action, action.returnOwnerAgentId)) {
seenAgentIds.add(action.returnOwnerAgentId);
continue;
}
const run = await wake(action.returnOwnerAgentId, {
source: "automation",
triggerDetail: "system",
@@ -501,35 +535,42 @@ export async function deliverReconciledExecutions(
? { freshContinuationIsolation: action.id } : {}),
},
});
if (run)
await db.transaction(async (tx) => {
await tx
.update(heartbeatRuns)
.set({ retryOfRunId: decision.runId })
.where(
and(
eq(heartbeatRuns.companyId, action.companyId),
eq(heartbeatRuns.id, run.id),
eq(heartbeatRuns.agentId, action.returnOwnerAgentId!),
sql`${heartbeatRuns.contextSnapshot}->>'recoveryActionId' = ${action.id}`,
sql`${heartbeatRuns.contextSnapshot}->>'previousRunId' = ${decision.runId}`,
),
);
await tx
.update(issueRecoveryActions)
.set({
evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify(
{
continuationDelivery: "delivered",
continuationRunId: run.id,
},
)}::jsonb`,
})
.where(pendingDecision);
await settleCoalescedFreshContinuationHolds(tx as unknown as Db, { companyId: action.companyId, primaryId: action.id,
delivery: "delivered", continuationRunId: run.id });
});
seenAgentIds.add(action.returnOwnerAgentId);
if (!run) {
await deferRefusedContinuation(action);
continue;
}
admittedByCompany.set(action.companyId, (admittedByCompany.get(action.companyId) ?? 0) + 1);
await db.transaction(async (tx) => {
await tx
.update(heartbeatRuns)
.set({ retryOfRunId: decision.runId })
.where(
and(
eq(heartbeatRuns.companyId, action.companyId),
eq(heartbeatRuns.id, run.id),
eq(heartbeatRuns.agentId, action.returnOwnerAgentId!),
sql`${heartbeatRuns.contextSnapshot}->>'recoveryActionId' = ${action.id}`,
sql`${heartbeatRuns.contextSnapshot}->>'previousRunId' = ${decision.runId}`,
),
);
await tx
.update(issueRecoveryActions)
.set({
evidence: sql`${issueRecoveryActions.evidence} || ${JSON.stringify(
{
continuationDelivery: "delivered",
continuationRunId: run.id,
},
)}::jsonb`,
})
.where(pendingDecision);
await settleCoalescedFreshContinuationHolds(tx as unknown as Db, { companyId: action.companyId, primaryId: action.id,
delivery: "delivered", continuationRunId: run.id });
});
} catch {
if (action.returnOwnerAgentId) seenAgentIds.add(action.returnOwnerAgentId);
await deferRefusedContinuation(action).catch(() => undefined);
logger.warn(
{ recoveryActionId: action.id },
"Reconciled execution continuation remains pending for retry",
@@ -0,0 +1,12 @@
export const RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS = 5;
export const RECOVERY_CONTINUATION_MAX_HOLD_MS = 5 * 60_000;
export const RECOVERY_CONTINUATION_DELIVERY_BOUND_MS = 15 * 60_000;
export function recoveryContinuationHeldSince(input: {
evidence: Record<string, unknown>;
resolvedAt: Date | null;
updatedAt: Date;
}) {
const recordedAt = Date.parse(String((input.evidence.freshContinuation as { recordedAt?: unknown } | undefined)?.recordedAt ?? ""));
return Number.isFinite(recordedAt) ? recordedAt : (input.resolvedAt ?? input.updatedAt).getTime();
}
@@ -11,6 +11,7 @@ import { EXECUTION_RECONCILIATION_CAUSES, HEARTBEAT_RUN_STATUSES, readProviderFa
import { captureTaskInterruptionFailure } from "../sentry.js";
import { isAiAuthenticationBlocked } from "./ai-auth-failure.js";
import { readSystemRecovery } from "./system-recovery-state.js";
import { RECOVERY_CONTINUATION_DELIVERY_BOUND_MS, recoveryContinuationHeldSince } from "./recovery-continuation-pacing.js";
import { PROJECT_ACCESS_DENIED } from "./project-access-denial.js";
import { authorizationService } from "./authorization.js";
import { taskWorkspaceProjectRequirements } from "./task-workspace-access.js";
@@ -582,6 +583,8 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i
runId: sql<string>`${issueRecoveryActions.evidence}->>'runId'`,
delivery: sql<string | null>`${issueRecoveryActions.evidence}->>'continuationDelivery'`,
recorded: sql<boolean>`${issueRecoveryActions.evidence} ? 'freshContinuation'`,
evidence: sql<Record<string, unknown>>`jsonb_build_object('freshContinuation', ${issueRecoveryActions.evidence}->'freshContinuation')`,
resolvedAt: issueRecoveryActions.resolvedAt, updatedAt: issueRecoveryActions.updatedAt,
checkKind: sql<string | null>`${issueRecoveryActions.evidence}#>>'{freshContinuationCheck,kind}'`,
nextCheckAt: sql<string | null>`${issueRecoveryActions.evidence}#>>'{freshContinuationCheck,nextCheckAt}'` })
.from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, companyId), inArray(issueRecoveryActions.sourceIssueId, ids),
@@ -591,7 +594,8 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i
const rows = freshRows.filter(row => row.issueId === issueId && row.runId === runId);
// Only the primary record (it carries `freshContinuation`) says whether a
// continuation is still on its way; holds coalesced into it never do.
if (rows.some(row => row.recorded && ["pending", "not_requested"].includes(row.delivery ?? ""))) return { state: "continuing" };
if (rows.some(row => row.recorded && (row.delivery === "not_requested" || (row.delivery === "pending"
&& Date.now() - recoveryContinuationHeldSince(row) < RECOVERY_CONTINUATION_DELIVERY_BOUND_MS)))) return { state: "continuing" };
const scheduled = rows.filter(row => ["exact_recovery_pending", "backoff"].includes(row.checkKind ?? "") && row.nextCheckAt)
.map(row => Number(row.nextCheckAt)).sort((a, b) => a - b)[0];
return scheduled ? { state: "scheduled", nextCheckAt: scheduled } : undefined;