fix(recovery): enqueue a fresh interruption's continuation at once and record why a folded wake did not start (#21354)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run stops before it finishes, for example when a server swap
reaches a sandbox late, the fresh-continuation fallback records one
continuation of the task from its saved work
> - The execution-continuation worker turns that record into a queued
run. It took one record per agent per pass, and it took the oldest held
record first
> - After a large swap, the backlog of old records is hours deep. A
record for a run that ended a few minutes ago waited behind the whole
backlog. The record also stayed outside the queue while the agent's
queue was deeper than its concurrency limit
> - Meanwhile the task showed "todo" with nothing running, and the
assigned-work sweep's wakes for it were folded into the pending record
without any trace
> - This pull request queues a fresh interruption's continuation on the
next pass, keeps a share of every pass for the old backlog, and records
a skipped wake row when a system wake is folded into a pending
continuation
> - The benefit is that an interrupted task gets its next run promptly,
it waits in the agent's queue where the dispatcher ranks it, and nothing
is lost or invisible

## Linked Issues or Issue Description

Refs #21324, #21257, #21234 (earlier pacing and dispatch ranking that
this builds on).

**What happened?**
On a hosted staging instance, five tasks had runs that ended after a
server swap ("The runner restored the conversation after process loss,
but the previous turn is no longer active."). The fallback recorded a
continuation for each task 3 to 6 minutes after the run ended. Each
record stayed `continuationDelivery: pending` for 30 to 110 minutes with
no delivery attempt. During that time the worker delivered continuations
whose source runs ended 4 to 7 hours earlier, one per agent per pass.
The tasks read as failed and needing Retry. The sweep tried 3 to 4 times
per task, and each wake returned `null` without a wake row.

**Expected behavior**
A recorded continuation for a recent interruption becomes a queued run
within about one delivery pass. Old backlog continues to drain and is
not starved. A wake that is not started records why.

**Steps to reproduce**
1. Give an agent a queue that is deeper than its concurrency limit.
2. Record pending continuations for older interruptions that are held
past the hold bound.
3. Record a continuation for a run that ended a few minutes ago.
4. Run `deliverReconciledExecutions`. Before this change the fresh
record is not delivered. The oldest held record is delivered first, one
per agent per pass.

**Agent adapter(s) involved**
Not adapter-specific (core bug).

## What Changed

- `deliverReconciledExecutions` now sorts pending continuations into two
classes by when the source run ended. "Fresh" means that the source run
ended within `RECOVERY_BACKLOG_AGE_MS` (30 minutes), which is the same
value the dispatcher uses to tell fresh recovery from backlog.
- A fresh continuation is enqueued without the queue-depth room check.
It is limited only by the per-agent limit (one fresh continuation per
agent per pass) and the per-company cap per pass. Among fresh records,
the newest interruption goes first. A record whose wake was refused
moves behind records that were not attempted yet.
- Backlog continuations keep their earlier rules. These are the
queue-depth room check, the bypass after the 5-minute hold bound, oldest
held first, and one per agent per pass.
- Each pass runs three phases: fresh up to the company cap minus
`RECOVERY_CONTINUATION_BACKLOG_SHARE_PER_PASS` (1), then backlog up to
the cap, then fresh again up to the cap. When both classes wait, the
backlog gets at least 1 of the 5 slots per company per pass. The
per-company cap (5) does not change, so a swap that interrupts many runs
does not enqueue them all at once. All admissions still go through the
one `wake(...)` call.
- In `enqueueWakeup`, a system wake that is folded into the task's
pending fresh continuation now writes a `skipped` wake row with reason
`issue_continuation_pending`. Before this change it returned `null` with
no trace.
- Tests in `recovery-wake-herd.test.ts`:
- New: a fresh continuation is queued on the first pass while the
agent's queue is deeper than its limit.
- New: a fresh continuation is delivered ahead of an hours-old backlog
for the same agent, and the backlog keeps draining.
- New: many fresh continuations for one agent are enqueued one per pass,
newest first, and none is left behind.
- New: the backlog keeps a share of each pass while fresh continuations
fill the company cap.
- New: a refused fresh wake stays pending and is enqueued on the next
pass.
  - New: a folded system wake leaves a skipped row.
- Updated: existing tests that used a recent source run to mean "held
because the queue is full" now use backlog-age sources, and assertions
that depended on exactly one admission per agent per pass now check the
record under test.

## Verification

- Red before the fix: the 6 new tests failed. The 18 earlier tests still
passed.
- Green after the fix: `cd server && npx vitest run
src/__tests__/recovery-wake-herd.test.ts` gives 24 passed.
- `npx vitest run src/__tests__/execution-control-work.test.ts` gives 27
passed. `src/__tests__/issue-recovery-actions.test.ts` gives 112 passed.
`src/services/fresh-continuation-fallback.test.ts` gives 62 passed.
`src/services/workspace-recovery-fallback.test.ts` gives 19 passed.
`src/services/native-runtime/native-safe-replacement.test.ts` gives 62
passed.
- `cd server && npx tsc --noEmit -p tsconfig.json` passes.

## Risks

- Behavior shift: an agent with a full queue now gets fresh recovery
continuations queued at once. They wait in its queue, where the
dispatcher ranking (person-initiated first, recovery share) decides when
they start. They are no longer held outside the queue. The per-company
cap per pass still limits the burst.
- Old backlog drains at a similar pace as before. It always gets at
least one slot per company per pass when it waits.
- The skipped wake row adds one small row each time the sweep retries a
task that has a pending continuation. The sweep spaces its retries, so
the volume is low.
- No migration. No API or UI contract change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, read-only staging API calls).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-13 05:46:12 -06:00
1 parent e23549cd9c
commit 0ec155ac31
4 files changed
+176 -31

No files matched your search

+118 -8
View File
@@ -309,12 +309,12 @@ suite("recovery wake herd", () => {
expect(await executionControlSchedule(db).continuation()).toBeNull();
}, 60_000);
it("leaves a continuation pending while the agent's queue is at its concurrency limit", async () => {
it("leaves a backlog continuation pending while the agent's queue is at its concurrency limit", async () => {
const agentId = await seedCompany(1);
const busy = await task(agentId);
await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: busy.id, invocationSource: "assignment", status: "queued",
contextSnapshot: { issueId: busy.id, wakeReason: "issue_assigned" } });
const record = await pendingContinuation(agentId, 5);
const record = await pendingContinuation(agentId, 120);
const wake = fakeWake();
await deliverReconciledExecutions(db, wake as never);
expect(wake).not.toHaveBeenCalled();
@@ -355,9 +355,9 @@ suite("recovery wake herd", () => {
const fresh = await pendingContinuation(agentId, 1);
const wake = fakeWake();
await deliverReconciledExecutions(db, wake as never);
expect(wake).toHaveBeenCalledTimes(1);
expect(wake).toHaveBeenCalledTimes(2);
expect(await deliveryOf(held.action.id)).toBe("delivered");
expect(await deliveryOf(fresh.action.id)).toBe("pending");
expect(await deliveryOf(fresh.action.id)).toBe("delivered");
}, 60_000);
it("admits an older continuation within a bounded number of passes while fresh interruptions keep arriving", async () => {
@@ -381,7 +381,7 @@ suite("recovery wake herd", () => {
}
expect(admittedAt).toBeGreaterThan(0);
expect(admittedAt).toBeLessThanOrEqual(bound);
expect(wake).toHaveBeenCalledTimes(1);
expect(wake.mock.calls.filter(call => call[1].contextSnapshot.recoveryActionId === old.action.id)).toHaveLength(1);
} finally {
vi.useRealTimers();
}
@@ -394,7 +394,7 @@ suite("recovery wake herd", () => {
const busy = await task(agentId);
await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: busy.id, invocationSource: "assignment", status: "queued",
contextSnapshot: { issueId: busy.id, wakeReason: "issue_assigned" } });
for (let index = 0; index < 8; index += 1) await pendingContinuation(agentId, 2 + index);
for (let index = 0; index < 8; index += 1) await pendingContinuation(agentId, 60 + index);
}
const idleAgent = await addAgent(1);
const idle = await pendingContinuation(idleAgent, 600);
@@ -474,7 +474,7 @@ suite("recovery wake herd", () => {
}
const sameAgentStuck = await pendingContinuation(agentId, 300);
await holdPastBound(sameAgentStuck, 60 * 60_000);
const waiting = await pendingContinuation(agentId, 10);
const waiting = await pendingContinuation(agentId, 60);
await holdPastBound(waiting);
const stuckIds = new Set([...stuck, sameAgentStuck].map(record => record.action.id));
const created = fakeWake();
@@ -507,7 +507,7 @@ suite("recovery wake herd", () => {
await deliverReconciledExecutions(db, wake as never);
}
expect(await deliveryOf(overdue.action.id)).toBe("delivered");
expect(created.mock.calls[0]![1].contextSnapshot.recoveryActionId).toBe(overdue.action.id);
expect(created.mock.calls.map(call => call[1].contextSnapshot.recoveryActionId)).toContain(overdue.action.id);
}, 60_000);
it("delivers another agent's continuation in one pass while one agent refuses a backlog of more than 200", async () => {
@@ -592,4 +592,114 @@ suite("recovery wake herd", () => {
expect(wake).toHaveBeenCalledTimes(1);
expect(await deliveryOf(held.action.id)).toBe("delivered");
}, 60_000);
async function queueOrdinaryWork(agentId: string, count: number) {
for (let index = 0; index < count; index += 1) {
const busy = await task(agentId);
await db.insert(heartbeatRuns).values({ companyId, agentId, issueId: busy.id, invocationSource: "assignment", status: "queued",
contextSnapshot: { issueId: busy.id, wakeReason: "issue_assigned" } });
}
}
const deliveredIds = (wake: ReturnType<typeof fakeWake>) =>
wake.mock.calls.map(call => String(call[1].contextSnapshot.recoveryActionId));
it("enqueues a fresh interruption's continuation at once while the agent's queue is deeper than its concurrency limit", async () => {
const agentId = await seedCompany(1);
await queueOrdinaryWork(agentId, 3);
const record = await pendingContinuation(agentId, 3);
const wake = fakeWake();
await deliverReconciledExecutions(db, wake as never);
expect(deliveredIds(wake)).toEqual([record.action.id]);
expect(await deliveryOf(record.action.id)).toBe("delivered");
}, 60_000);
it("delivers a fresh interruption ahead of an hours-old backlog for the same agent and keeps draining the backlog", async () => {
const agentId = await seedCompany(2);
await queueOrdinaryWork(agentId, 4);
const backlog: Array<Awaited<ReturnType<typeof pendingContinuation>>> = [];
for (let index = 0; index < 4; index += 1) {
const record = await pendingContinuation(agentId, 300 + index);
await holdPastBound(record, (4 * 60 - index) * 60_000);
backlog.push(record);
}
const fresh = await pendingContinuation(agentId, 3);
const wake = fakeWake();
await deliverReconciledExecutions(db, wake as never);
expect(deliveredIds(wake)).toEqual([fresh.action.id, backlog[0]!.action.id]);
for (let pass = 0; pass < backlog.length; pass += 1) await deliverReconciledExecutions(db, wake as never);
expect(await Promise.all(backlog.map(record => deliveryOf(record.action.id)))).toEqual(backlog.map(() => "delivered"));
expect(wake).toHaveBeenCalledTimes(backlog.length + 1);
}, 60_000);
it("enqueues many fresh continuations for one agent one per pass, newest first, and leaves none behind", async () => {
const agentId = await seedCompany(1);
await queueOrdinaryWork(agentId, 2);
const records: Array<Awaited<ReturnType<typeof pendingContinuation>>> = [];
for (let index = 0; index < 5; index += 1) records.push(await pendingContinuation(agentId, 1 + index));
const wake = fakeWake();
for (let pass = 0; pass < records.length; pass += 1) {
await deliverReconciledExecutions(db, wake as never);
expect(wake).toHaveBeenCalledTimes(pass + 1);
}
expect(deliveredIds(wake)).toEqual(records.map(record => record.action.id));
await deliverReconciledExecutions(db, wake as never);
expect(wake).toHaveBeenCalledTimes(records.length);
}, 60_000);
it("keeps a share of each pass for backlog while fresh interruptions fill the company cap", async () => {
const freshAgents = [await seedCompany(1)];
for (let index = 1; index < RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS + 2; index += 1) freshAgents.push(await addAgent(1));
for (const agentId of freshAgents) await pendingContinuation(agentId, 2);
const backlogAgent = await addAgent(1);
const backlog = await pendingContinuation(backlogAgent, 240);
const wake = fakeWake();
await deliverReconciledExecutions(db, wake as never);
expect(wake).toHaveBeenCalledTimes(RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS);
expect(await deliveryOf(backlog.action.id)).toBe("delivered");
expect(wake.mock.calls.filter(call => call[0] !== backlogAgent)).toHaveLength(RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS - 1);
}, 60_000);
it("delivers a fresh continuation for every agent up to the company cap in one pass when no backlog waits", async () => {
const agentIds = [await seedCompany(1)];
for (let index = 1; index < RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS; index += 1) agentIds.push(await addAgent(1));
const records = [];
for (const agentId of agentIds) records.push(await pendingContinuation(agentId, 2));
const wake = fakeWake();
await deliverReconciledExecutions(db, wake as never);
expect(wake).toHaveBeenCalledTimes(RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS);
expect(await Promise.all(records.map(record => deliveryOf(record.action.id)))).toEqual(records.map(() => "delivered"));
}, 60_000);
it("keeps a fresh continuation whose wake was refused pending and enqueues it on the next pass", async () => {
const agentId = await seedCompany(1);
await queueOrdinaryWork(agentId, 3);
const record = await pendingContinuation(agentId, 2);
const created = fakeWake();
let refusing = true;
const wake = vi.fn(async (wakeAgentId: string, opts: { contextSnapshot: Record<string, unknown> }) =>
refusing ? null : created(wakeAgentId, opts));
await deliverReconciledExecutions(db, wake as never);
expect(wake).toHaveBeenCalledTimes(1);
expect(await deliveryOf(record.action.id)).toBe("pending");
refusing = false;
await deliverReconciledExecutions(db, wake as never);
expect(created).toHaveBeenCalledTimes(1);
expect(await deliveryOf(record.action.id)).toBe("delivered");
}, 60_000);
it("records why a system wake was folded into the task's pending continuation instead of leaving no trace", async () => {
const agentId = await seedCompany(1);
const record = await pendingContinuation(agentId, 30);
const heartbeat = heartbeatService(db);
try {
const run = await heartbeat.wakeup(agentId, { source: "automation", triggerDetail: "system", reason: "issue_continuation_needed",
payload: { issueId: record.issue.id, mutation: "assigned_work_invariant" }, requestedByActorType: "system",
contextSnapshot: { issueId: record.issue.id, taskId: record.issue.id, wakeReason: "issue_continuation_needed" } });
expect(run).toBeNull();
const rows = await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.agentId, agentId),
sql`${agentWakeupRequests.payload}->>'issueId' = ${record.issue.id}`));
expect(rows).toMatchObject([{ status: "skipped", reason: "issue_continuation_pending" }]);
expect(await deliveryOf(record.action.id)).toBe("pending");
} finally {
await heartbeat.drainActiveRunExecutions();
}
}, 60_000);
});
@@ -40,10 +40,10 @@ import { provenGoneLegacyControllerBootId } from "./legacy-controller-lease.js";
import { continueFreshInTransaction, continueHeldExecutions, freshContinuationCheckDue, recordFreshContinuationCheck,
settleCoalescedFreshContinuationHolds } from "./fresh-continuation-fallback.js";
import { alias } from "drizzle-orm/pg-core";
import { RECOVERY_WAKE_REASONS } from "./heartbeat/queued-run-dispatch-order.js";
import { RECOVERY_BACKLOG_AGE_MS, RECOVERY_WAKE_REASONS } from "./heartbeat/queued-run-dispatch-order.js";
import { parseHeartbeatPolicy } from "./heartbeat/timer-policy.js";
import { RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS, RECOVERY_CONTINUATION_MAX_HOLD_MS,
import { RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS, RECOVERY_CONTINUATION_BACKLOG_SHARE_PER_PASS, RECOVERY_CONTINUATION_MAX_HOLD_MS,
recoveryContinuationHeldSince } from "./recovery-continuation-pacing.js";
export { RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS, RECOVERY_CONTINUATION_DELIVERY_BOUND_MS,
@@ -370,18 +370,25 @@ export async function deliverReconciledExecutions(
const recoveryReasons = [...RECOVERY_WAKE_REASONS];
const sourceRun = alias(heartbeatRuns, "continuation_source_run");
const seenActionIds = new Set<string>();
const seenAgentIds = new Set<string>();
const seenAgentIds: Record<ContinuationClass, Set<string>> = { fresh: new Set(), backlog: new Set() };
const admittedByCompany = new Map<string, number>();
let companyLimit = RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS;
const fullCompanies = () => [...admittedByCompany].flatMap(([companyId, count]) =>
count >= RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS ? [companyId] : []);
count >= companyLimit ? [companyId] : []);
const heldSince = sql`coalesce(${issueRecoveryActions.evidence}#>>'{freshContinuation,recordedAt}', ${issueRecoveryActions.resolvedAt}::text, ${issueRecoveryActions.updatedAt}::text)::timestamptz`;
const readPendingPage = async () => {
const attemptedAt = sql`(${issueRecoveryActions.evidence}->>'continuationDeliveryAttemptedAt')::timestamptz`;
const freshSource = () => sql`coalesce(${sourceRun.finishedAt} > ${new Date(Date.now() - RECOVERY_BACKLOG_AGE_MS).toISOString()}::timestamptz, false)`;
const readPendingPage = async (kind: ContinuationClass) => {
const companies = fullCompanies();
const seenAgents = seenAgentIds[kind];
const heldBound = new Date(Date.now() - RECOVERY_CONTINUATION_MAX_HOLD_MS).toISOString();
const pastBound = sql`${heldSince} <= ${heldBound}::timestamptz`;
const overdueTurn = sql`case when ${pastBound} then greatest(${heldSince},
(${issueRecoveryActions.evidence}->>'continuationDeliveryAttemptedAt')::timestamptz) end`;
const priority = [
const overdueTurn = sql`case when ${pastBound} then greatest(${heldSince}, ${attemptedAt}) end`;
const priority = kind === "fresh" ? [
sql`${attemptedAt} asc nulls first`,
sql`${sourceRun.finishedAt} desc`,
sql`${issueRecoveryActions.updatedAt} asc`,
] : [
sql`${pastBound} desc`,
sql`${overdueTurn} asc nulls last`,
sql`${sourceRun.finishedAt} desc nulls last`,
@@ -393,6 +400,7 @@ export async function deliverReconciledExecutions(
pastBound: sql<boolean>`${pastBound}`.as("past_bound"),
heldAt: sql<Date | null>`${overdueTurn}`.as("held_at"),
sourceFinishedAt: sql<Date | null>`${sourceRun.finishedAt}`.as("source_finished_at"),
attemptedAt: sql<Date | null>`${attemptedAt}`.as("attempted_at"),
actionUpdatedAt: sql<Date>`${issueRecoveryActions.updatedAt}`.as("action_updated_at"),
agentRank: sql<number>`row_number() over (
partition by coalesce(${issueRecoveryActions.returnOwnerAgentId}::text, ${issueRecoveryActions.id}::text)
@@ -409,9 +417,10 @@ export async function deliverReconciledExecutions(
eq(issueRecoveryActions.status, "resolved"),
sql`${issueRecoveryActions.evidence}->>'continuationDelivery' = 'pending'`,
seenActionIds.size ? not(inArray(issueRecoveryActions.id, [...seenActionIds])) : undefined,
seenAgentIds.size ? sql`coalesce(${issueRecoveryActions.returnOwnerAgentId}::text, '') <> all(${sql.param([...seenAgentIds])}::text[])` : undefined,
seenAgents.size ? sql`coalesce(${issueRecoveryActions.returnOwnerAgentId}::text, '') <> all(${sql.param([...seenAgents])}::text[])` : undefined,
companies.length ? not(inArray(issueRecoveryActions.companyId, companies)) : undefined,
sql`(${pastBound} or not exists (select 1 from ${heartbeatRuns} queued_recovery
kind === "fresh" ? freshSource() : sql`not ${freshSource()}`,
kind === "fresh" ? undefined : sql`(${pastBound} or not exists (select 1 from ${heartbeatRuns} queued_recovery
where queued_recovery.company_id = ${issueRecoveryActions.companyId}
and queued_recovery.agent_id = ${issueRecoveryActions.returnOwnerAgentId}
and queued_recovery.status = 'queued'
@@ -424,12 +433,16 @@ export async function deliverReconciledExecutions(
.from(candidates)
.innerJoin(issueRecoveryActions, eq(issueRecoveryActions.id, candidates.id))
.where(eq(candidates.agentRank, 1))
.orderBy(
.orderBy(...(kind === "fresh" ? [
sql`${candidates.attemptedAt} asc nulls first`,
sql`${candidates.sourceFinishedAt} desc`,
sql`${candidates.actionUpdatedAt} asc`,
] : [
sql`${candidates.pastBound} desc`,
sql`${candidates.heldAt} asc nulls last`,
sql`${candidates.sourceFinishedAt} desc nulls last`,
sql`${candidates.actionUpdatedAt} asc`,
)
]))
.limit(25);
return rows.map(row => row.action);
};
@@ -442,8 +455,9 @@ export async function deliverReconciledExecutions(
eq(issueRecoveryActions.status, "resolved"),
sql`${issueRecoveryActions.evidence}->>'continuationDelivery' = 'pending'`,
));
const agentHasRoom = async (action: typeof issueRecoveryActions.$inferSelect, agentId: string) => {
if ((admittedByCompany.get(action.companyId) ?? 0) >= RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS) return false;
const agentHasRoom = async (action: typeof issueRecoveryActions.$inferSelect, agentId: string, kind: ContinuationClass) => {
if ((admittedByCompany.get(action.companyId) ?? 0) >= companyLimit) return false;
if (kind === "fresh") return true;
const [agent] = await db.select().from(agents).where(and(eq(agents.companyId, action.companyId), eq(agents.id, agentId)));
if (!agent) return true;
const [queued] = await db.select({ count: sql<number>`count(*)::int` }).from(heartbeatRuns).where(and(
@@ -451,12 +465,23 @@ export async function deliverReconciledExecutions(
if (Number(queued?.count ?? 0) < parseHeartbeatPolicy(agent).maxConcurrentRuns) return true;
return Date.now() - recoveryContinuationHeldSince(action) >= RECOVERY_CONTINUATION_MAX_HOLD_MS;
};
for (let page = 0; page < RECOVERY_CONTINUATION_PAGES_PER_PASS; page += 1) {
const pending = await readPendingPage();
if (!pending.length) break;
const cap = RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS;
const phases: Array<[ContinuationClass, number]> = [
["fresh", cap - RECOVERY_CONTINUATION_BACKLOG_SHARE_PER_PASS], ["backlog", cap], ["fresh", cap]];
for (let page = 0; page < RECOVERY_CONTINUATION_PAGES_PER_PASS * phases.length; page += 1) {
const phase = Math.floor(page / RECOVERY_CONTINUATION_PAGES_PER_PASS);
const [kind, limit] = phases[phase]!;
companyLimit = limit;
const seenAgents = seenAgentIds[kind];
const pending = await readPendingPage(kind);
if (!pending.length) {
page = (phase + 1) * RECOVERY_CONTINUATION_PAGES_PER_PASS - 1;
continue;
}
for (const action of pending) {
if ((admittedByCompany.get(action.companyId) ?? 0) >= companyLimit) continue;
seenActionIds.add(action.id);
if (action.returnOwnerAgentId && seenAgentIds.has(action.returnOwnerAgentId)) continue;
if (action.returnOwnerAgentId && seenAgents.has(action.returnOwnerAgentId)) continue;
try {
const decision = action.evidence.executionReconciliation as
ExecutionReconciliation | undefined;
@@ -510,8 +535,8 @@ export async function deliverReconciledExecutions(
continue;
}
}
if (!await agentHasRoom(action, action.returnOwnerAgentId)) {
seenAgentIds.add(action.returnOwnerAgentId);
if (!await agentHasRoom(action, action.returnOwnerAgentId, kind)) {
seenAgents.add(action.returnOwnerAgentId);
continue;
}
const run = await wake(action.returnOwnerAgentId, {
@@ -535,7 +560,7 @@ export async function deliverReconciledExecutions(
? { freshContinuationIsolation: action.id } : {}),
},
});
seenAgentIds.add(action.returnOwnerAgentId);
seenAgents.add(action.returnOwnerAgentId);
if (!run) {
await deferRefusedContinuation(action);
continue;
@@ -569,7 +594,7 @@ export async function deliverReconciledExecutions(
delivery: "delivered", continuationRunId: run.id });
});
} catch {
if (action.returnOwnerAgentId) seenAgentIds.add(action.returnOwnerAgentId);
if (action.returnOwnerAgentId) seenAgents.add(action.returnOwnerAgentId);
await deferRefusedContinuation(action).catch(() => undefined);
logger.warn(
{ recoveryActionId: action.id },
@@ -580,6 +605,8 @@ export async function deliverReconciledExecutions(
}
}
type ContinuationClass = "fresh" | "backlog";
const supersedingWorkspaceHold = alias(issueRecoveryActions, "superseding_workspace_hold");
function supersedingNativeWorkspaceHold(db: Db, input: {
+8 -1
View File
@@ -4018,8 +4018,15 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
&& !executionReconciliationWake;
// A recorded fresh continuation that has no run yet is this task's
// next run: another system wake is coalesced into it, never a second run.
if (systemWake && !executionBlocker && await hasPendingFreshContinuation(tx as unknown as Db, issue.companyId, issue.id))
if (systemWake && !executionBlocker && await hasPendingFreshContinuation(tx as unknown as Db, issue.companyId, issue.id)) {
await tx.insert(agentWakeupRequests).values({
...durableReceiptFields, companyId: agent.companyId, agentId, source, triggerDetail,
reason: "issue_continuation_pending", payload, status: "skipped",
requestedByActorType: opts.requestedByActorType ?? null, requestedByActorId: opts.requestedByActorId ?? null,
idempotencyKey: opts.idempotencyKey ?? null, finishedAt: new Date(),
});
return { kind: "skipped" as const };
}
if ((executionBlocker || opts.queuedCommentInterruptId) && !ownsReconstructionContinuation && !(await admitExplicitNativeContinuation({
db: tx as unknown as Db, companyId: issue.companyId, issueId: issue.id,
agentId, actorType: taskRetryActorId ? "user" : opts.requestedByActorType, actorId: taskRetryActorId ?? opts.requestedByActorId,
@@ -1,4 +1,5 @@
export const RECOVERY_CONTINUATIONS_PER_COMPANY_PER_PASS = 5;
export const RECOVERY_CONTINUATION_BACKLOG_SHARE_PER_PASS = 1;
export const RECOVERY_CONTINUATION_MAX_HOLD_MS = 5 * 60_000;
export const RECOVERY_CONTINUATION_DELIVERY_BOUND_MS = 15 * 60_000;
export const PENDING_CONTINUATION_BOUND_MS = RECOVERY_CONTINUATION_MAX_HOLD_MS + 2 * 60_000;