fix: prevent retry-exhaustion events from exhausting attention-feed memory (#13451)

## Thinking Path

> - Paperclip manages AI agents and their work.
> - Its attention feed shows failed runs whose retry budget is
exhausted.
> - Startup retention builds this feed before startup completes.
> - The feed joins every exhaustion event to the full run context, then
removes duplicate runs in JavaScript.
> - Recovery can revisit an exhausted run and append the same event
again. This multiplies the data loaded into memory.
> - This pull request selects one small row per run in PostgreSQL and
makes exhaustion writes idempotent.
> - Existing duplicate events can stay in the database without
multiplying run contexts in server memory.

## Linked Issues or Issue Description

Refs #13367. This fixes the repeated-event allocation path in the
retention feed. Other full-feed sources and sweep cadence remain
separate concerns.

**What happened?**

The attention query loaded one full run context for every matching
exhaustion event. Deduplication ran only after the driver had loaded
those rows. A run with thousands of exhaustion events therefore produced
thousands of context copies. The startup retention sweep can exhaust the
server heap while reading this result.

**Expected behavior**

The query should return one row per exhausted run and only the context
fields that the feed needs. Repeated checks of the same exhausted retry
budget should reuse the original event.

**Steps to reproduce**

1. Create a failed run with a 32 KB context and 2,500 matching
exhaustion events.
2. Build the attention feed, including dismissed items, as startup
retention does.
3. Inspect the database result before JavaScript feed processing. The
old join returns 2,500 copies of the run context.
4. Call bounded retry scheduling repeatedly for a run at its retry
limit. The old writer appends another exhaustion event on every call.

**Paperclip version or commit**

The attention query was introduced by #9380 and is present in stable
`v2026.831.1`. The retention caller is also present in that stable
release. The later recovery callback added by #13075 provides a repeated
path into the exhausted-budget writer. This change is based on
`c0fda8fac` after rebasing onto current master.

Related PR:
[#12162](https://github.com/paperclipai/paperclip/pull/12162) changes
retention cadence and newer-run suppression queries. This change
addresses the exhaustion-event join and duplicate event writes.

## What Changed

- Select the newest company-scoped exhaustion event per run with a
PostgreSQL `DISTINCT ON` subquery before joining run data.
- Project only `issueId` and `taskId` from run context. Preserve JSON
types, fallback behavior, run ordering, company filters, and run/agent
status filters.
- Reuse an exhaustion event for the same run, reason, attempt, and retry
limit under the existing run-row lock. Recognize historical events
without a migration.
- Skip sequence allocation and live publication when a receipt already
exists.
- Document the run-log behavior and add PostgreSQL regression tests.

## Verification

- Focused attention, retry-scheduling, and event-sequencing suites: 72
tests pass on the rebased head `39ce77158`.
- The large-history fixture verifies two database result rows under 4
KB, newest-message selection, task-ID fallback, and company/status
filtering. This assertion runs before feed deduplication.
- Concurrent event tests verify one receipt across two database clients,
reuse of historical receipts, distinct reason/attempt/budget keys, and
stable event sequences.
- Repeated scheduling through a new service instance produces no extra
run-log or live events.
- `pnpm --filter @paperclipai/server exec tsc --noEmit`: passes.
- `pnpm -r typecheck` and `pnpm build`: pass on the rebased head,
including native runner checks.
- Greptile: 5/5 on `39ce77158`, with no review threads.
- [GitHub
CI](https://github.com/paperclipai/paperclip/actions/runs/34929002304):
all 25 workflow jobs pass, including all server/workspace test shards,
serialized server suites, browser tests, typecheck, build, native-runner
verification, and the release dry run. Security checks also pass. The
branch has no merge conflicts.
- `pnpm test:run`: stopped with unrelated failures. Two chat integration
cases passed when rerun separately (2 passed, 993 unselected). Three
skill-cache cases failed on both this branch and the unmodified parent
commit, with `EACCES` during directory rename. The full suite is not
reported as green.

## Risks

- No schema migration or data cleanup is required. The query still scans
matching event history in PostgreSQL; its result size now scales with
exhausted runs.
- This does not bound every source in the attention feed or change
retention scheduling.
- An exhaustion receipt is emitted once per retry decision. Consumers
that observed repeated copies will now receive one event.
- Native source-event replay handling stays on its existing path.
- A deployed application boot has not been verified.

## Model Used

OpenAI GPT-6, used through Codex with reasoning, repository inspection,
code editing, and test execution. The runtime does not expose a more
specific model snapshot or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run the focused tests locally and they pass (full-suite
limitations are documented above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Devin FoleyandPaperclip authored and GitHub committed 2026-09-14 21:58:34 -07:00
1 parent c0fda8fac5
commit 667c79ded2
8 files changed
+242 -41

No files matched your search

+8
View File
@@ -159,6 +159,14 @@ Provider identity diagnostics remain in the local run log. They record the notif
Recovery lifecycle events retain the original structured failure code, retry attempt, next retry time, and predecessor/successor identifiers. Durable status delivery uses an idempotency marker; delivery grants no provider authority. Failed publication is retried without repeating provider work. These records are not first-party Telemetry.
Bounded retry exhaustion writes one lifecycle receipt per run, retry reason,
scheduled attempt, and retry limit. Repeated or concurrent recovery checks reuse
that receipt, including receipts from earlier builds, without advancing the event
sequence or publishing another live event. Attention reads select the latest
matching receipt in PostgreSQL and project only the run's issue/task identifiers
from its context, so historical duplicate receipts cannot multiply run contexts
in server memory. Existing duplicate events do not require deletion or migration.
## Codex resume usage snapshot
The native runner retains a bounded local `harness.diagnostic` event with code
@@ -42,6 +42,7 @@ import {
import { errorHandler } from "../middleware/index.js";
import { attentionRoutes } from "../routes/attention.js";
import { attentionService } from "../services/attention.js";
import { listAttentionExhaustedRuns } from "../services/attention-exhausted-runs.js";
import { agentService } from "../services/agents.js";
import { ROUTABLE_BLOCKED_ROLLOUT_AT } from "../services/routable-blocked.js";
@@ -918,6 +919,63 @@ describeEmbeddedPostgres("attention service", () => {
.toBeNull();
});
it("reads one compact row per exhausted run despite thousands of historical receipts", async () => {
const { companyId, workerId, reviewerId } = await seedCompany("MEM");
const other = await seedCompany("OTH");
await db.update(agents).set({ status: "terminated" }).where(eq(agents.id, reviewerId));
const issueId = await insertIssue({
companyId, identifier: "MEM-1", title: "Failed task", status: "in_progress",
});
const taskId = await insertIssue({
companyId, identifier: "MEM-2", title: "Timed out task", status: "in_progress",
});
const [failedId, timedOutId, succeededId, terminatedId, foreignId, noReceiptId] =
Array.from({ length: 6 }, () => randomUUID());
const createdAt = new Date("2026-07-09T12:00:00.000Z");
await db.insert(heartbeatRuns).values([
{ id: failedId, companyId, agentId: workerId, status: "failed", contextSnapshot: { issueId, prompt: "x".repeat(32_000) } },
{ id: timedOutId, companyId, agentId: workerId, status: "timed_out", contextSnapshot: { taskId, prompt: "y".repeat(32_000) } },
{ id: succeededId, companyId, agentId: workerId, status: "succeeded" },
{ id: terminatedId, companyId, agentId: reviewerId, status: "failed" },
{ id: foreignId, companyId: other.companyId, agentId: other.workerId, status: "failed" },
{ id: noReceiptId, companyId, agentId: workerId, status: "failed" },
].map((run) => ({ ...run, createdAt, updatedAt: createdAt, finishedAt: createdAt })));
for (let batch = 0; batch < 5; batch += 1) {
await db.insert(heartbeatRunEvents).values(Array.from({ length: 500 }, (_, index) => ({
companyId, agentId: workerId, runId: failedId, seq: batch * 500 + index + 1,
eventType: "lifecycle", message: `Bounded retry exhausted receipt ${batch * 500 + index + 1}`,
})));
}
await db.insert(heartbeatRunEvents).values([
{ companyId, agentId: workerId, runId: timedOutId, seq: 1, eventType: "lifecycle", message: "Bounded retry exhausted timeout" },
{ companyId, agentId: workerId, runId: succeededId, seq: 1, eventType: "lifecycle", message: "Bounded retry exhausted success" },
{ companyId, agentId: reviewerId, runId: terminatedId, seq: 1, eventType: "lifecycle", message: "Bounded retry exhausted terminated" },
{ companyId: other.companyId, agentId: other.workerId, runId: foreignId, seq: 1, eventType: "lifecycle", message: "Bounded retry exhausted other company" },
// A newer event must not replace the latest matching, company-scoped receipt.
{ companyId: other.companyId, agentId: other.workerId, runId: failedId, seq: 2501, eventType: "lifecycle", message: "Bounded retry exhausted foreign receipt" },
{ companyId, agentId: workerId, runId: failedId, seq: 2502, eventType: "stdout", message: "Bounded retry exhausted quoted output" },
{ companyId, agentId: workerId, runId: failedId, seq: 2503, eventType: "lifecycle", message: "Unrelated lifecycle event" },
]);
// Assert the database result itself: JavaScript feed deduplication used to
// hide the thousands of full run contexts already loaded into memory.
const rows = await listAttentionExhaustedRuns(db, companyId);
expect(rows).toHaveLength(2);
expect(rows.find((row) => row.id === failedId)).toMatchObject({
exhaustionMessage: "Bounded retry exhausted receipt 2500",
contextSnapshot: { issueId, taskId: null },
});
expect(rows.find((row) => row.id === timedOutId)?.contextSnapshot).toEqual({ issueId: null, taskId });
expect(Buffer.byteLength(JSON.stringify(rows))).toBeLessThan(4096);
const feed = await attentionService(db).list(companyId, {
includeDismissed: true, all: true, allowUnscopedAll: true,
});
const failures = feed.items.filter((item) => item.sourceKind === "failed_run");
expect(failures.map((item) => item.subject.id).sort()).toEqual([failedId, timedOutId].sort());
expect(failures.find((item) => item.subject.id === timedOutId)?.relatedIssue?.id).toBe(taskId);
});
it("suppresses failed-run attention after a newer run for the same issue", async () => {
const { companyId, workerId } = await seedCompany("ATN");
const issueId = await insertIssue({
@@ -55,6 +55,7 @@ vi.mock("../services/heartbeat-run-events.js", async (importOriginal) => {
});
import { appendHeartbeatRunEvent } from "../services/heartbeat-run-events.js";
import { subscribeCompanyLiveEvents } from "../services/live-events.js";
import {
BOUNDED_TRANSIENT_HEARTBEAT_RETRY_DELAYS_MS,
INTERACTION_CONTINUATION_INFRA_RETRY_REASON,
@@ -1835,6 +1836,23 @@ describeEmbeddedPostgres("heartbeat bounded retry scheduling", () => {
scheduledRetryAttempt: BOUNDED_TRANSIENT_HEARTBEAT_RETRY_DELAYS_MS.length,
maxAttempts: BOUNDED_TRANSIENT_HEARTBEAT_RETRY_DELAYS_MS.length,
});
const onLiveEvent = vi.fn();
const unsubscribe = subscribeCompanyLiveEvents(companyId, onLiveEvent);
try {
const restartedHeartbeat = heartbeatService(createDb(tempDb!.connectionString));
const repeated = await Promise.all(Array.from({ length: 8 }, (_, index) =>
(index % 2 ? heartbeat : restartedHeartbeat).scheduleBoundedRetry(cappedRunId, {
now, random: () => 0.5,
})));
expect(repeated).toEqual(Array.from({ length: 8 }, () => exhausted));
expect(await db.select().from(heartbeatRunEvents)
.where(eq(heartbeatRunEvents.runId, cappedRunId))).toHaveLength(1);
expect((await heartbeat.getRun(cappedRunId))?.nextEventSeq).toBe(2);
expect(onLiveEvent).not.toHaveBeenCalled();
} finally {
unsubscribe();
}
});
it("advances codex transient fallback stages across bounded retry attempts", async () => {
@@ -15,6 +15,67 @@ import {
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
describe("P6-11..13 / P6-17 canonical event allocator", () => {
it.each([false, true])("deduplicates concurrent retry exhaustion (historical receipt: %s)", async (historical) => {
const temporary = await startEmbeddedPostgresTestDatabase("paperclip-exhaustion-events-");
const db = createDb(temporary.connectionString);
const otherDb = createDb(temporary.connectionString);
const companyId = "20000000-0000-4000-8000-000000000001";
const agentId = "20000000-0000-4000-8000-000000000002";
const runId = "20000000-0000-4000-8000-000000000003";
const otherRunId = "20000000-0000-4000-8000-000000000004";
const exhaustion = { retryReason: "transient_failure", scheduledRetryAttempt: 2, maxAttempts: 2 };
const event = {
companyId, agentId, runId, eventType: "lifecycle",
message: "Bounded retry exhausted after 2 scheduled attempts; no further automatic retry will be queued",
payload: exhaustion,
};
try {
await db.insert(companies).values({ id: companyId, name: "Exhaustion fixture", issuePrefix: "EXH" });
await db.insert(agents).values({ id: agentId, companyId, name: "Exhaustion agent" });
await db.insert(heartbeatRuns).values([runId, otherRunId].map((id) => ({
id, companyId, agentId, status: "failed",
})));
if (historical) {
// Old builds wrote ordinary lifecycle rows with no new idempotency field.
await appendHeartbeatRunEvent(db, event);
}
const receipts = await Promise.all(Array.from({ length: 16 }, (_, index) =>
appendHeartbeatRunEvent(index % 2 ? db : otherDb, {
...event, retryExhaustion: exhaustion,
})));
expect(receipts.filter((receipt) => receipt.disposition === "committed"))
.toHaveLength(historical ? 0 : 1);
expect(new Set(receipts.map((receipt) => receipt.row.id)).size).toBe(1);
expect(await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, runId)))
.toHaveLength(1);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0]?.nextEventSeq)
.toBe(2);
// The key is per run, reason, attempt, and budget. Distinct exhaustion
// decisions still get their own receipt and an ordinary sequence number.
for (const changed of [
{ ...exhaustion, retryReason: "bootstrap_failure" },
{ ...exhaustion, scheduledRetryAttempt: 3 },
{ ...exhaustion, maxAttempts: 3 },
]) {
expect((await appendHeartbeatRunEvent(db, {
...event, payload: changed, retryExhaustion: changed,
})).disposition).toBe("committed");
}
expect((await appendHeartbeatRunEvent(db, {
...event, runId: otherRunId, retryExhaustion: exhaustion,
})).disposition).toBe("committed");
const rows = await db.select().from(heartbeatRunEvents)
.where(eq(heartbeatRunEvents.runId, runId)).orderBy(heartbeatRunEvents.seq);
expect(rows.map((row) => row.seq)).toEqual([1, 2, 3, 4]);
await expect(appendHeartbeatRunEvent(db, {
...event, companyId: "20000000-0000-4000-8000-000000000099", retryExhaustion: exhaustion,
})).rejects.toThrow("heartbeat_run_event_binding_mismatch");
} finally {
await temporary.cleanup();
}
}, 60_000);
it("serializes concurrent writers and rejects conflicting replay without cursor drift", async () => {
const temporary = await startEmbeddedPostgresTestDatabase("paperclip-native-events-");
const db = createDb(temporary.connectionString);
@@ -0,0 +1,52 @@
import { and, asc, desc, eq, inArray, notInArray, sql } from "drizzle-orm";
import { agents, heartbeatRunEvents, heartbeatRuns, type Db } from "@paperclipai/db";
export function listAttentionExhaustedRuns(db: Db, companyId: string) {
// Recovery can revisit an exhausted run. Deduplicate its historical events
// before joining run data so duplicate events never multiply the wire payload.
const latestExhaustion = db
.selectDistinctOn([heartbeatRunEvents.runId], {
runId: heartbeatRunEvents.runId,
eventId: heartbeatRunEvents.id,
message: heartbeatRunEvents.message,
})
.from(heartbeatRunEvents)
.where(and(
eq(heartbeatRunEvents.companyId, companyId),
eq(heartbeatRunEvents.eventType, "lifecycle"),
sql`${heartbeatRunEvents.message} like 'Bounded retry exhausted%'`,
))
.orderBy(asc(heartbeatRunEvents.runId), desc(heartbeatRunEvents.id))
.as("latest_exhaustion");
return db
.select({
id: heartbeatRuns.id,
companyId: heartbeatRuns.companyId,
agentId: heartbeatRuns.agentId,
agentName: agents.name,
status: heartbeatRuns.status,
error: heartbeatRuns.error,
errorCode: heartbeatRuns.errorCode,
// Preserve JSON types and issueId/taskId fallback semantics without
// transferring prompts, transcripts, or the rest of the run context.
contextSnapshot: sql<Record<string, unknown>>`jsonb_build_object(
'issueId', ${heartbeatRuns.contextSnapshot} -> 'issueId',
'taskId', ${heartbeatRuns.contextSnapshot} -> 'taskId'
)`,
createdAt: heartbeatRuns.createdAt,
updatedAt: heartbeatRuns.updatedAt,
finishedAt: heartbeatRuns.finishedAt,
exhaustionMessage: latestExhaustion.message,
})
.from(heartbeatRuns)
.innerJoin(agents, eq(heartbeatRuns.agentId, agents.id))
.innerJoin(latestExhaustion, eq(latestExhaustion.runId, heartbeatRuns.id))
.where(and(
eq(heartbeatRuns.companyId, companyId),
eq(agents.companyId, companyId),
notInArray(agents.status, ["terminated"]),
inArray(heartbeatRuns.status, ["failed", "timed_out"]),
))
.orderBy(desc(heartbeatRuns.createdAt), desc(latestExhaustion.eventId));
}
+2 -36
View File
@@ -11,7 +11,6 @@ import {
decisionTrainingExamples,
decisionTriage,
decisions,
heartbeatRunEvents,
heartbeatRuns,
inboxDismissals,
invites,
@@ -49,6 +48,7 @@ import type {
IssueReviewPolicy,
} from "@paperclipai/shared";
import { badRequest } from "../errors.js";
import { listAttentionExhaustedRuns } from "./attention-exhausted-runs.js";
import { budgetService } from "./budgets.js";
import {
BLOCKER_ATTENTION_MAX_DEPTH,
@@ -104,7 +104,6 @@ const SOURCE_RANK: Record<AttentionSourceKind, number> = {
const PENDING_INTERACTION_STATUSES = ["pending"] as const;
const OPEN_RECOVERY_STATUSES = ["active", "escalated"] as const;
const HUMAN_RECOVERY_OWNER_TYPES = ["user", "board"] as const;
const FAILED_RUN_STATUSES = ["failed", "timed_out"] as const;
const DETAIL_EXCERPT_LENGTH = 160;
const DETAIL_IMAGE_LIMIT = 3;
const OPEN_DECISION_DEFAULT_LIMIT = 500;
@@ -1662,40 +1661,7 @@ export function attentionService(db: Db, serviceOptions: AttentionServiceOptions
}));
}
const exhaustedRunRows = await db
.select({
id: heartbeatRuns.id,
companyId: heartbeatRuns.companyId,
agentId: heartbeatRuns.agentId,
agentName: agents.name,
status: heartbeatRuns.status,
error: heartbeatRuns.error,
errorCode: heartbeatRuns.errorCode,
contextSnapshot: heartbeatRuns.contextSnapshot,
createdAt: heartbeatRuns.createdAt,
updatedAt: heartbeatRuns.updatedAt,
finishedAt: heartbeatRuns.finishedAt,
exhaustionMessage: heartbeatRunEvents.message,
})
.from(heartbeatRuns)
.innerJoin(agents, eq(heartbeatRuns.agentId, agents.id))
.innerJoin(heartbeatRunEvents, eq(heartbeatRunEvents.runId, heartbeatRuns.id))
.where(and(
eq(heartbeatRuns.companyId, companyId),
eq(agents.companyId, companyId),
notInArray(agents.status, ["terminated"]),
inArray(heartbeatRuns.status, [...FAILED_RUN_STATUSES]),
eq(heartbeatRunEvents.companyId, companyId),
eq(heartbeatRunEvents.eventType, "lifecycle"),
sql`${heartbeatRunEvents.message} like 'Bounded retry exhausted%'`,
))
.orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRunEvents.id));
const latestExhaustedByRunId = new Map<string, (typeof exhaustedRunRows)[number]>();
for (const row of exhaustedRunRows) {
if (!latestExhaustedByRunId.has(row.id)) latestExhaustedByRunId.set(row.id, row);
}
const failedRows = [...latestExhaustedByRunId.values()];
const failedRows = await listAttentionExhaustedRuns(db, companyId);
const failedIssueIds = failedRows.map((row) => readRunIssueId(row.contextSnapshot));
const failedAgentIds = [...new Set(failedRows.map((row) => row.agentId))];
const oldestFailedRunCreatedAt = failedRows.reduce<Date | null>((oldest, row) => {
@@ -13,6 +13,12 @@ export interface AppendHeartbeatRunEventInput {
color?: string | null;
message?: string | null;
payload?: Record<string, unknown> | null;
/** Reuse an existing exhaustion receipt, including receipts from older builds. */
retryExhaustion?: {
retryReason: string;
scheduledRetryAttempt: number;
maxAttempts: number;
};
nativeSource?: {
sourceInstanceId: string;
sourceEventId: string;
@@ -74,6 +80,32 @@ export async function appendHeartbeatRunEvent(
throw new Error("heartbeat_run_event_binding_mismatch");
}
if (input.retryExhaustion && !input.nativeSource) {
// The run lock also serializes concurrent recovery checks across server
// instances. Reusing the receipt must not allocate a sequence or publish
// another live event on each scheduler tick.
const existing = await tx
.select()
.from(heartbeatRunEvents)
.where(and(
eq(heartbeatRunEvents.companyId, input.companyId),
eq(heartbeatRunEvents.runId, input.runId),
eq(heartbeatRunEvents.agentId, input.agentId),
eq(heartbeatRunEvents.eventType, "lifecycle"),
sql`${heartbeatRunEvents.message} like 'Bounded retry exhausted%'`,
sql`${heartbeatRunEvents.payload} @> ${JSON.stringify(input.retryExhaustion)}::jsonb`,
))
.limit(1)
.then((rows) => rows[0] ?? null);
if (existing) {
return {
row: existing,
disposition: "duplicate" as const,
highestContiguousSourceSeq: 0,
};
}
}
const sourceHash = input.nativeSource
? nativeSha256(input.nativeSource.canonicalPayload)
: null;
+11 -5
View File
@@ -176,6 +176,7 @@ import { publishLiveEvent } from "./live-events.js";
import {
allocateHeartbeatRunEventSeq,
appendHeartbeatRunEvent,
type AppendHeartbeatRunEventInput,
} from "./heartbeat-run-events.js";
import {
queuedCommentIdsFromWakePayload,
@@ -13594,6 +13595,7 @@ export function heartbeatService(
color?: string;
message?: string;
payload?: Record<string, unknown>;
retryExhaustion?: AppendHeartbeatRunEventInput["retryExhaustion"];
},
) {
const eventAt = new Date();
@@ -13632,7 +13634,9 @@ export function heartbeatService(
color: event.color,
message: sanitizedMessage,
payload: sanitizedPayload,
retryExhaustion: event.retryExhaustion,
});
if (persistedEvent.disposition === "duplicate") return;
const seq = persistedEvent.row.seq;
publishLiveEvent({
@@ -15150,16 +15154,18 @@ export function heartbeatService(
const issueId = readNonEmptyString(contextSnapshot.issueId);
if (!baseSchedule) {
const exhaustion = {
retryReason,
scheduledRetryAttempt: run.scheduledRetryAttempt ?? 0,
maxAttempts,
};
await appendRunEvent(run, {
eventType: "lifecycle",
stream: "system",
level: "warn",
message: `Bounded retry exhausted after ${run.scheduledRetryAttempt ?? 0} scheduled attempts; no further automatic retry will be queued`,
payload: {
retryReason,
scheduledRetryAttempt: run.scheduledRetryAttempt ?? 0,
maxAttempts,
},
payload: exhaustion,
retryExhaustion: exhaustion,
});
if (retryReason === INTERACTION_CONTINUATION_INFRA_RETRY_REASON) {
await escalatePlanApprovalResumeFailureNeedsAttention({