mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-20 15:09:57 +02:00
fix: prevent retry-exhaustion events from exhausting attention-feed memory (#13451)
## Thinking Path > - Paperclip manages AI agents and their work. > - Its attention feed shows failed runs whose retry budget is exhausted. > - Startup retention builds this feed before startup completes. > - The feed joins every exhaustion event to the full run context, then removes duplicate runs in JavaScript. > - Recovery can revisit an exhausted run and append the same event again. This multiplies the data loaded into memory. > - This pull request selects one small row per run in PostgreSQL and makes exhaustion writes idempotent. > - Existing duplicate events can stay in the database without multiplying run contexts in server memory. ## Linked Issues or Issue Description Refs #13367. This fixes the repeated-event allocation path in the retention feed. Other full-feed sources and sweep cadence remain separate concerns. **What happened?** The attention query loaded one full run context for every matching exhaustion event. Deduplication ran only after the driver had loaded those rows. A run with thousands of exhaustion events therefore produced thousands of context copies. The startup retention sweep can exhaust the server heap while reading this result. **Expected behavior** The query should return one row per exhausted run and only the context fields that the feed needs. Repeated checks of the same exhausted retry budget should reuse the original event. **Steps to reproduce** 1. Create a failed run with a 32 KB context and 2,500 matching exhaustion events. 2. Build the attention feed, including dismissed items, as startup retention does. 3. Inspect the database result before JavaScript feed processing. The old join returns 2,500 copies of the run context. 4. Call bounded retry scheduling repeatedly for a run at its retry limit. The old writer appends another exhaustion event on every call. **Paperclip version or commit** The attention query was introduced by #9380 and is present in stable `v2026.831.1`. The retention caller is also present in that stable release. The later recovery callback added by #13075 provides a repeated path into the exhausted-budget writer. This change is based on `c0fda8fac` after rebasing onto current master. Related PR: [#12162](https://github.com/paperclipai/paperclip/pull/12162) changes retention cadence and newer-run suppression queries. This change addresses the exhaustion-event join and duplicate event writes. ## What Changed - Select the newest company-scoped exhaustion event per run with a PostgreSQL `DISTINCT ON` subquery before joining run data. - Project only `issueId` and `taskId` from run context. Preserve JSON types, fallback behavior, run ordering, company filters, and run/agent status filters. - Reuse an exhaustion event for the same run, reason, attempt, and retry limit under the existing run-row lock. Recognize historical events without a migration. - Skip sequence allocation and live publication when a receipt already exists. - Document the run-log behavior and add PostgreSQL regression tests. ## Verification - Focused attention, retry-scheduling, and event-sequencing suites: 72 tests pass on the rebased head `39ce77158`. - The large-history fixture verifies two database result rows under 4 KB, newest-message selection, task-ID fallback, and company/status filtering. This assertion runs before feed deduplication. - Concurrent event tests verify one receipt across two database clients, reuse of historical receipts, distinct reason/attempt/budget keys, and stable event sequences. - Repeated scheduling through a new service instance produces no extra run-log or live events. - `pnpm --filter @paperclipai/server exec tsc --noEmit`: passes. - `pnpm -r typecheck` and `pnpm build`: pass on the rebased head, including native runner checks. - Greptile: 5/5 on `39ce77158`, with no review threads. - [GitHub CI](https://github.com/paperclipai/paperclip/actions/runs/34929002304): all 25 workflow jobs pass, including all server/workspace test shards, serialized server suites, browser tests, typecheck, build, native-runner verification, and the release dry run. Security checks also pass. The branch has no merge conflicts. - `pnpm test:run`: stopped with unrelated failures. Two chat integration cases passed when rerun separately (2 passed, 993 unselected). Three skill-cache cases failed on both this branch and the unmodified parent commit, with `EACCES` during directory rename. The full suite is not reported as green. ## Risks - No schema migration or data cleanup is required. The query still scans matching event history in PostgreSQL; its result size now scales with exhausted runs. - This does not bound every source in the attention feed or change retention scheduling. - An exhaustion receipt is emitted once per retry decision. Consumers that observed repeated copies will now receive one event. - Native source-event replay handling stays on its existing path. - A deployed application boot has not been verified. ## Model Used OpenAI GPT-6, used through Codex with reasoning, repository inspection, code editing, and test execution. The runtime does not expose a more specific model snapshot or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run the focused tests locally and they pass (full-suite limitations are documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
c0fda8fac5
commit
667c79ded2
8 files changed
+242
-41
No files matched your search
@@ -159,6 +159,14 @@ Provider identity diagnostics remain in the local run log. They record the notif
|
||||
|
||||
Recovery lifecycle events retain the original structured failure code, retry attempt, next retry time, and predecessor/successor identifiers. Durable status delivery uses an idempotency marker; delivery grants no provider authority. Failed publication is retried without repeating provider work. These records are not first-party Telemetry.
|
||||
|
||||
Bounded retry exhaustion writes one lifecycle receipt per run, retry reason,
|
||||
scheduled attempt, and retry limit. Repeated or concurrent recovery checks reuse
|
||||
that receipt, including receipts from earlier builds, without advancing the event
|
||||
sequence or publishing another live event. Attention reads select the latest
|
||||
matching receipt in PostgreSQL and project only the run's issue/task identifiers
|
||||
from its context, so historical duplicate receipts cannot multiply run contexts
|
||||
in server memory. Existing duplicate events do not require deletion or migration.
|
||||
|
||||
## Codex resume usage snapshot
|
||||
|
||||
The native runner retains a bounded local `harness.diagnostic` event with code
|
||||
|
||||
@@ -42,6 +42,7 @@ import {
|
||||
import { errorHandler } from "../middleware/index.js";
|
||||
import { attentionRoutes } from "../routes/attention.js";
|
||||
import { attentionService } from "../services/attention.js";
|
||||
import { listAttentionExhaustedRuns } from "../services/attention-exhausted-runs.js";
|
||||
import { agentService } from "../services/agents.js";
|
||||
import { ROUTABLE_BLOCKED_ROLLOUT_AT } from "../services/routable-blocked.js";
|
||||
|
||||
@@ -918,6 +919,63 @@ describeEmbeddedPostgres("attention service", () => {
|
||||
.toBeNull();
|
||||
});
|
||||
|
||||
it("reads one compact row per exhausted run despite thousands of historical receipts", async () => {
|
||||
const { companyId, workerId, reviewerId } = await seedCompany("MEM");
|
||||
const other = await seedCompany("OTH");
|
||||
await db.update(agents).set({ status: "terminated" }).where(eq(agents.id, reviewerId));
|
||||
const issueId = await insertIssue({
|
||||
companyId, identifier: "MEM-1", title: "Failed task", status: "in_progress",
|
||||
});
|
||||
const taskId = await insertIssue({
|
||||
companyId, identifier: "MEM-2", title: "Timed out task", status: "in_progress",
|
||||
});
|
||||
const [failedId, timedOutId, succeededId, terminatedId, foreignId, noReceiptId] =
|
||||
Array.from({ length: 6 }, () => randomUUID());
|
||||
const createdAt = new Date("2026-07-09T12:00:00.000Z");
|
||||
await db.insert(heartbeatRuns).values([
|
||||
{ id: failedId, companyId, agentId: workerId, status: "failed", contextSnapshot: { issueId, prompt: "x".repeat(32_000) } },
|
||||
{ id: timedOutId, companyId, agentId: workerId, status: "timed_out", contextSnapshot: { taskId, prompt: "y".repeat(32_000) } },
|
||||
{ id: succeededId, companyId, agentId: workerId, status: "succeeded" },
|
||||
{ id: terminatedId, companyId, agentId: reviewerId, status: "failed" },
|
||||
{ id: foreignId, companyId: other.companyId, agentId: other.workerId, status: "failed" },
|
||||
{ id: noReceiptId, companyId, agentId: workerId, status: "failed" },
|
||||
].map((run) => ({ ...run, createdAt, updatedAt: createdAt, finishedAt: createdAt })));
|
||||
for (let batch = 0; batch < 5; batch += 1) {
|
||||
await db.insert(heartbeatRunEvents).values(Array.from({ length: 500 }, (_, index) => ({
|
||||
companyId, agentId: workerId, runId: failedId, seq: batch * 500 + index + 1,
|
||||
eventType: "lifecycle", message: `Bounded retry exhausted receipt ${batch * 500 + index + 1}`,
|
||||
})));
|
||||
}
|
||||
await db.insert(heartbeatRunEvents).values([
|
||||
{ companyId, agentId: workerId, runId: timedOutId, seq: 1, eventType: "lifecycle", message: "Bounded retry exhausted timeout" },
|
||||
{ companyId, agentId: workerId, runId: succeededId, seq: 1, eventType: "lifecycle", message: "Bounded retry exhausted success" },
|
||||
{ companyId, agentId: reviewerId, runId: terminatedId, seq: 1, eventType: "lifecycle", message: "Bounded retry exhausted terminated" },
|
||||
{ companyId: other.companyId, agentId: other.workerId, runId: foreignId, seq: 1, eventType: "lifecycle", message: "Bounded retry exhausted other company" },
|
||||
// A newer event must not replace the latest matching, company-scoped receipt.
|
||||
{ companyId: other.companyId, agentId: other.workerId, runId: failedId, seq: 2501, eventType: "lifecycle", message: "Bounded retry exhausted foreign receipt" },
|
||||
{ companyId, agentId: workerId, runId: failedId, seq: 2502, eventType: "stdout", message: "Bounded retry exhausted quoted output" },
|
||||
{ companyId, agentId: workerId, runId: failedId, seq: 2503, eventType: "lifecycle", message: "Unrelated lifecycle event" },
|
||||
]);
|
||||
|
||||
// Assert the database result itself: JavaScript feed deduplication used to
|
||||
// hide the thousands of full run contexts already loaded into memory.
|
||||
const rows = await listAttentionExhaustedRuns(db, companyId);
|
||||
expect(rows).toHaveLength(2);
|
||||
expect(rows.find((row) => row.id === failedId)).toMatchObject({
|
||||
exhaustionMessage: "Bounded retry exhausted receipt 2500",
|
||||
contextSnapshot: { issueId, taskId: null },
|
||||
});
|
||||
expect(rows.find((row) => row.id === timedOutId)?.contextSnapshot).toEqual({ issueId: null, taskId });
|
||||
expect(Buffer.byteLength(JSON.stringify(rows))).toBeLessThan(4096);
|
||||
|
||||
const feed = await attentionService(db).list(companyId, {
|
||||
includeDismissed: true, all: true, allowUnscopedAll: true,
|
||||
});
|
||||
const failures = feed.items.filter((item) => item.sourceKind === "failed_run");
|
||||
expect(failures.map((item) => item.subject.id).sort()).toEqual([failedId, timedOutId].sort());
|
||||
expect(failures.find((item) => item.subject.id === timedOutId)?.relatedIssue?.id).toBe(taskId);
|
||||
});
|
||||
|
||||
it("suppresses failed-run attention after a newer run for the same issue", async () => {
|
||||
const { companyId, workerId } = await seedCompany("ATN");
|
||||
const issueId = await insertIssue({
|
||||
|
||||
@@ -55,6 +55,7 @@ vi.mock("../services/heartbeat-run-events.js", async (importOriginal) => {
|
||||
});
|
||||
|
||||
import { appendHeartbeatRunEvent } from "../services/heartbeat-run-events.js";
|
||||
import { subscribeCompanyLiveEvents } from "../services/live-events.js";
|
||||
import {
|
||||
BOUNDED_TRANSIENT_HEARTBEAT_RETRY_DELAYS_MS,
|
||||
INTERACTION_CONTINUATION_INFRA_RETRY_REASON,
|
||||
@@ -1835,6 +1836,23 @@ describeEmbeddedPostgres("heartbeat bounded retry scheduling", () => {
|
||||
scheduledRetryAttempt: BOUNDED_TRANSIENT_HEARTBEAT_RETRY_DELAYS_MS.length,
|
||||
maxAttempts: BOUNDED_TRANSIENT_HEARTBEAT_RETRY_DELAYS_MS.length,
|
||||
});
|
||||
|
||||
const onLiveEvent = vi.fn();
|
||||
const unsubscribe = subscribeCompanyLiveEvents(companyId, onLiveEvent);
|
||||
try {
|
||||
const restartedHeartbeat = heartbeatService(createDb(tempDb!.connectionString));
|
||||
const repeated = await Promise.all(Array.from({ length: 8 }, (_, index) =>
|
||||
(index % 2 ? heartbeat : restartedHeartbeat).scheduleBoundedRetry(cappedRunId, {
|
||||
now, random: () => 0.5,
|
||||
})));
|
||||
expect(repeated).toEqual(Array.from({ length: 8 }, () => exhausted));
|
||||
expect(await db.select().from(heartbeatRunEvents)
|
||||
.where(eq(heartbeatRunEvents.runId, cappedRunId))).toHaveLength(1);
|
||||
expect((await heartbeat.getRun(cappedRunId))?.nextEventSeq).toBe(2);
|
||||
expect(onLiveEvent).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
unsubscribe();
|
||||
}
|
||||
});
|
||||
|
||||
it("advances codex transient fallback stages across bounded retry attempts", async () => {
|
||||
|
||||
@@ -15,6 +15,67 @@ import {
|
||||
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
|
||||
describe("P6-11..13 / P6-17 canonical event allocator", () => {
|
||||
it.each([false, true])("deduplicates concurrent retry exhaustion (historical receipt: %s)", async (historical) => {
|
||||
const temporary = await startEmbeddedPostgresTestDatabase("paperclip-exhaustion-events-");
|
||||
const db = createDb(temporary.connectionString);
|
||||
const otherDb = createDb(temporary.connectionString);
|
||||
const companyId = "20000000-0000-4000-8000-000000000001";
|
||||
const agentId = "20000000-0000-4000-8000-000000000002";
|
||||
const runId = "20000000-0000-4000-8000-000000000003";
|
||||
const otherRunId = "20000000-0000-4000-8000-000000000004";
|
||||
const exhaustion = { retryReason: "transient_failure", scheduledRetryAttempt: 2, maxAttempts: 2 };
|
||||
const event = {
|
||||
companyId, agentId, runId, eventType: "lifecycle",
|
||||
message: "Bounded retry exhausted after 2 scheduled attempts; no further automatic retry will be queued",
|
||||
payload: exhaustion,
|
||||
};
|
||||
try {
|
||||
await db.insert(companies).values({ id: companyId, name: "Exhaustion fixture", issuePrefix: "EXH" });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Exhaustion agent" });
|
||||
await db.insert(heartbeatRuns).values([runId, otherRunId].map((id) => ({
|
||||
id, companyId, agentId, status: "failed",
|
||||
})));
|
||||
if (historical) {
|
||||
// Old builds wrote ordinary lifecycle rows with no new idempotency field.
|
||||
await appendHeartbeatRunEvent(db, event);
|
||||
}
|
||||
const receipts = await Promise.all(Array.from({ length: 16 }, (_, index) =>
|
||||
appendHeartbeatRunEvent(index % 2 ? db : otherDb, {
|
||||
...event, retryExhaustion: exhaustion,
|
||||
})));
|
||||
expect(receipts.filter((receipt) => receipt.disposition === "committed"))
|
||||
.toHaveLength(historical ? 0 : 1);
|
||||
expect(new Set(receipts.map((receipt) => receipt.row.id)).size).toBe(1);
|
||||
expect(await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, runId)))
|
||||
.toHaveLength(1);
|
||||
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0]?.nextEventSeq)
|
||||
.toBe(2);
|
||||
|
||||
// The key is per run, reason, attempt, and budget. Distinct exhaustion
|
||||
// decisions still get their own receipt and an ordinary sequence number.
|
||||
for (const changed of [
|
||||
{ ...exhaustion, retryReason: "bootstrap_failure" },
|
||||
{ ...exhaustion, scheduledRetryAttempt: 3 },
|
||||
{ ...exhaustion, maxAttempts: 3 },
|
||||
]) {
|
||||
expect((await appendHeartbeatRunEvent(db, {
|
||||
...event, payload: changed, retryExhaustion: changed,
|
||||
})).disposition).toBe("committed");
|
||||
}
|
||||
expect((await appendHeartbeatRunEvent(db, {
|
||||
...event, runId: otherRunId, retryExhaustion: exhaustion,
|
||||
})).disposition).toBe("committed");
|
||||
const rows = await db.select().from(heartbeatRunEvents)
|
||||
.where(eq(heartbeatRunEvents.runId, runId)).orderBy(heartbeatRunEvents.seq);
|
||||
expect(rows.map((row) => row.seq)).toEqual([1, 2, 3, 4]);
|
||||
await expect(appendHeartbeatRunEvent(db, {
|
||||
...event, companyId: "20000000-0000-4000-8000-000000000099", retryExhaustion: exhaustion,
|
||||
})).rejects.toThrow("heartbeat_run_event_binding_mismatch");
|
||||
} finally {
|
||||
await temporary.cleanup();
|
||||
}
|
||||
}, 60_000);
|
||||
|
||||
it("serializes concurrent writers and rejects conflicting replay without cursor drift", async () => {
|
||||
const temporary = await startEmbeddedPostgresTestDatabase("paperclip-native-events-");
|
||||
const db = createDb(temporary.connectionString);
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
import { and, asc, desc, eq, inArray, notInArray, sql } from "drizzle-orm";
|
||||
import { agents, heartbeatRunEvents, heartbeatRuns, type Db } from "@paperclipai/db";
|
||||
|
||||
export function listAttentionExhaustedRuns(db: Db, companyId: string) {
|
||||
// Recovery can revisit an exhausted run. Deduplicate its historical events
|
||||
// before joining run data so duplicate events never multiply the wire payload.
|
||||
const latestExhaustion = db
|
||||
.selectDistinctOn([heartbeatRunEvents.runId], {
|
||||
runId: heartbeatRunEvents.runId,
|
||||
eventId: heartbeatRunEvents.id,
|
||||
message: heartbeatRunEvents.message,
|
||||
})
|
||||
.from(heartbeatRunEvents)
|
||||
.where(and(
|
||||
eq(heartbeatRunEvents.companyId, companyId),
|
||||
eq(heartbeatRunEvents.eventType, "lifecycle"),
|
||||
sql`${heartbeatRunEvents.message} like 'Bounded retry exhausted%'`,
|
||||
))
|
||||
.orderBy(asc(heartbeatRunEvents.runId), desc(heartbeatRunEvents.id))
|
||||
.as("latest_exhaustion");
|
||||
|
||||
return db
|
||||
.select({
|
||||
id: heartbeatRuns.id,
|
||||
companyId: heartbeatRuns.companyId,
|
||||
agentId: heartbeatRuns.agentId,
|
||||
agentName: agents.name,
|
||||
status: heartbeatRuns.status,
|
||||
error: heartbeatRuns.error,
|
||||
errorCode: heartbeatRuns.errorCode,
|
||||
// Preserve JSON types and issueId/taskId fallback semantics without
|
||||
// transferring prompts, transcripts, or the rest of the run context.
|
||||
contextSnapshot: sql<Record<string, unknown>>`jsonb_build_object(
|
||||
'issueId', ${heartbeatRuns.contextSnapshot} -> 'issueId',
|
||||
'taskId', ${heartbeatRuns.contextSnapshot} -> 'taskId'
|
||||
)`,
|
||||
createdAt: heartbeatRuns.createdAt,
|
||||
updatedAt: heartbeatRuns.updatedAt,
|
||||
finishedAt: heartbeatRuns.finishedAt,
|
||||
exhaustionMessage: latestExhaustion.message,
|
||||
})
|
||||
.from(heartbeatRuns)
|
||||
.innerJoin(agents, eq(heartbeatRuns.agentId, agents.id))
|
||||
.innerJoin(latestExhaustion, eq(latestExhaustion.runId, heartbeatRuns.id))
|
||||
.where(and(
|
||||
eq(heartbeatRuns.companyId, companyId),
|
||||
eq(agents.companyId, companyId),
|
||||
notInArray(agents.status, ["terminated"]),
|
||||
inArray(heartbeatRuns.status, ["failed", "timed_out"]),
|
||||
))
|
||||
.orderBy(desc(heartbeatRuns.createdAt), desc(latestExhaustion.eventId));
|
||||
}
|
||||
@@ -11,7 +11,6 @@ import {
|
||||
decisionTrainingExamples,
|
||||
decisionTriage,
|
||||
decisions,
|
||||
heartbeatRunEvents,
|
||||
heartbeatRuns,
|
||||
inboxDismissals,
|
||||
invites,
|
||||
@@ -49,6 +48,7 @@ import type {
|
||||
IssueReviewPolicy,
|
||||
} from "@paperclipai/shared";
|
||||
import { badRequest } from "../errors.js";
|
||||
import { listAttentionExhaustedRuns } from "./attention-exhausted-runs.js";
|
||||
import { budgetService } from "./budgets.js";
|
||||
import {
|
||||
BLOCKER_ATTENTION_MAX_DEPTH,
|
||||
@@ -104,7 +104,6 @@ const SOURCE_RANK: Record<AttentionSourceKind, number> = {
|
||||
const PENDING_INTERACTION_STATUSES = ["pending"] as const;
|
||||
const OPEN_RECOVERY_STATUSES = ["active", "escalated"] as const;
|
||||
const HUMAN_RECOVERY_OWNER_TYPES = ["user", "board"] as const;
|
||||
const FAILED_RUN_STATUSES = ["failed", "timed_out"] as const;
|
||||
const DETAIL_EXCERPT_LENGTH = 160;
|
||||
const DETAIL_IMAGE_LIMIT = 3;
|
||||
const OPEN_DECISION_DEFAULT_LIMIT = 500;
|
||||
@@ -1662,40 +1661,7 @@ export function attentionService(db: Db, serviceOptions: AttentionServiceOptions
|
||||
}));
|
||||
}
|
||||
|
||||
const exhaustedRunRows = await db
|
||||
.select({
|
||||
id: heartbeatRuns.id,
|
||||
companyId: heartbeatRuns.companyId,
|
||||
agentId: heartbeatRuns.agentId,
|
||||
agentName: agents.name,
|
||||
status: heartbeatRuns.status,
|
||||
error: heartbeatRuns.error,
|
||||
errorCode: heartbeatRuns.errorCode,
|
||||
contextSnapshot: heartbeatRuns.contextSnapshot,
|
||||
createdAt: heartbeatRuns.createdAt,
|
||||
updatedAt: heartbeatRuns.updatedAt,
|
||||
finishedAt: heartbeatRuns.finishedAt,
|
||||
exhaustionMessage: heartbeatRunEvents.message,
|
||||
})
|
||||
.from(heartbeatRuns)
|
||||
.innerJoin(agents, eq(heartbeatRuns.agentId, agents.id))
|
||||
.innerJoin(heartbeatRunEvents, eq(heartbeatRunEvents.runId, heartbeatRuns.id))
|
||||
.where(and(
|
||||
eq(heartbeatRuns.companyId, companyId),
|
||||
eq(agents.companyId, companyId),
|
||||
notInArray(agents.status, ["terminated"]),
|
||||
inArray(heartbeatRuns.status, [...FAILED_RUN_STATUSES]),
|
||||
eq(heartbeatRunEvents.companyId, companyId),
|
||||
eq(heartbeatRunEvents.eventType, "lifecycle"),
|
||||
sql`${heartbeatRunEvents.message} like 'Bounded retry exhausted%'`,
|
||||
))
|
||||
.orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRunEvents.id));
|
||||
|
||||
const latestExhaustedByRunId = new Map<string, (typeof exhaustedRunRows)[number]>();
|
||||
for (const row of exhaustedRunRows) {
|
||||
if (!latestExhaustedByRunId.has(row.id)) latestExhaustedByRunId.set(row.id, row);
|
||||
}
|
||||
const failedRows = [...latestExhaustedByRunId.values()];
|
||||
const failedRows = await listAttentionExhaustedRuns(db, companyId);
|
||||
const failedIssueIds = failedRows.map((row) => readRunIssueId(row.contextSnapshot));
|
||||
const failedAgentIds = [...new Set(failedRows.map((row) => row.agentId))];
|
||||
const oldestFailedRunCreatedAt = failedRows.reduce<Date | null>((oldest, row) => {
|
||||
|
||||
@@ -13,6 +13,12 @@ export interface AppendHeartbeatRunEventInput {
|
||||
color?: string | null;
|
||||
message?: string | null;
|
||||
payload?: Record<string, unknown> | null;
|
||||
/** Reuse an existing exhaustion receipt, including receipts from older builds. */
|
||||
retryExhaustion?: {
|
||||
retryReason: string;
|
||||
scheduledRetryAttempt: number;
|
||||
maxAttempts: number;
|
||||
};
|
||||
nativeSource?: {
|
||||
sourceInstanceId: string;
|
||||
sourceEventId: string;
|
||||
@@ -74,6 +80,32 @@ export async function appendHeartbeatRunEvent(
|
||||
throw new Error("heartbeat_run_event_binding_mismatch");
|
||||
}
|
||||
|
||||
if (input.retryExhaustion && !input.nativeSource) {
|
||||
// The run lock also serializes concurrent recovery checks across server
|
||||
// instances. Reusing the receipt must not allocate a sequence or publish
|
||||
// another live event on each scheduler tick.
|
||||
const existing = await tx
|
||||
.select()
|
||||
.from(heartbeatRunEvents)
|
||||
.where(and(
|
||||
eq(heartbeatRunEvents.companyId, input.companyId),
|
||||
eq(heartbeatRunEvents.runId, input.runId),
|
||||
eq(heartbeatRunEvents.agentId, input.agentId),
|
||||
eq(heartbeatRunEvents.eventType, "lifecycle"),
|
||||
sql`${heartbeatRunEvents.message} like 'Bounded retry exhausted%'`,
|
||||
sql`${heartbeatRunEvents.payload} @> ${JSON.stringify(input.retryExhaustion)}::jsonb`,
|
||||
))
|
||||
.limit(1)
|
||||
.then((rows) => rows[0] ?? null);
|
||||
if (existing) {
|
||||
return {
|
||||
row: existing,
|
||||
disposition: "duplicate" as const,
|
||||
highestContiguousSourceSeq: 0,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
const sourceHash = input.nativeSource
|
||||
? nativeSha256(input.nativeSource.canonicalPayload)
|
||||
: null;
|
||||
|
||||
@@ -176,6 +176,7 @@ import { publishLiveEvent } from "./live-events.js";
|
||||
import {
|
||||
allocateHeartbeatRunEventSeq,
|
||||
appendHeartbeatRunEvent,
|
||||
type AppendHeartbeatRunEventInput,
|
||||
} from "./heartbeat-run-events.js";
|
||||
import {
|
||||
queuedCommentIdsFromWakePayload,
|
||||
@@ -13594,6 +13595,7 @@ export function heartbeatService(
|
||||
color?: string;
|
||||
message?: string;
|
||||
payload?: Record<string, unknown>;
|
||||
retryExhaustion?: AppendHeartbeatRunEventInput["retryExhaustion"];
|
||||
},
|
||||
) {
|
||||
const eventAt = new Date();
|
||||
@@ -13632,7 +13634,9 @@ export function heartbeatService(
|
||||
color: event.color,
|
||||
message: sanitizedMessage,
|
||||
payload: sanitizedPayload,
|
||||
retryExhaustion: event.retryExhaustion,
|
||||
});
|
||||
if (persistedEvent.disposition === "duplicate") return;
|
||||
const seq = persistedEvent.row.seq;
|
||||
|
||||
publishLiveEvent({
|
||||
@@ -15150,16 +15154,18 @@ export function heartbeatService(
|
||||
const issueId = readNonEmptyString(contextSnapshot.issueId);
|
||||
|
||||
if (!baseSchedule) {
|
||||
const exhaustion = {
|
||||
retryReason,
|
||||
scheduledRetryAttempt: run.scheduledRetryAttempt ?? 0,
|
||||
maxAttempts,
|
||||
};
|
||||
await appendRunEvent(run, {
|
||||
eventType: "lifecycle",
|
||||
stream: "system",
|
||||
level: "warn",
|
||||
message: `Bounded retry exhausted after ${run.scheduledRetryAttempt ?? 0} scheduled attempts; no further automatic retry will be queued`,
|
||||
payload: {
|
||||
retryReason,
|
||||
scheduledRetryAttempt: run.scheduledRetryAttempt ?? 0,
|
||||
maxAttempts,
|
||||
},
|
||||
payload: exhaustion,
|
||||
retryExhaustion: exhaustion,
|
||||
});
|
||||
if (retryReason === INTERACTION_CONTINUATION_INFRA_RETRY_REASON) {
|
||||
await escalatePlanApprovalResumeFailureNeedsAttention({
|
||||
|
||||
Reference in new issue
Block a user