perf(attention): scope run lifecycle retention hints and read only used columns (#21579)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The `attention-retention` queue archives old attention items (failed
runs, reviews, questions, recovery actions, blocked tasks). A company
hint makes it reload every candidate of the company, 100 per page, with
eligibility reads for each page.
> - Several run lifecycle producers sent that company hint: every agent
status change after a run, the execution-control writes and legacy run
terminalization when they name only a run, scheduled retries, and the
Runner's task monitor tool.
> - Each page also read whole rows it did not use: full `heartbeat_runs`
rows (recent ones carry tens of KB of `context_snapshot` and
`runner_profile_json`), full `issues` rows (descriptions), and the whole
agent roster with its configuration.
> - This pull request names the task graph or the source kind in those
hints, and reads only the columns each eligibility decision uses.
> - The benefit is fewer company reloads per run event and far fewer
result bytes per reconstruction, with the same decisions.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`. Stacked on #21552 (it builds
on that pull request's change to `execution-control-notifications.ts`);
review that one first.

**What existing behavior does this improve?**

The attention retention queue
(`server/src/services/attention-retention.ts`) and the producers that
wake it on run lifecycle paths.

**Subsystem affected**

Server background retention (`attention-retention.ts`,
`retention-work-notifications.ts`, `execution-control-notifications.ts`,
`heartbeat/agent-status.ts`, `heartbeat/retries.ts`,
`native-runtime/paperclip-runner-tool-authority.ts`).

**Current behavior**

- `heartbeat/agent-status.ts` sends `{ companyId }` on every agent
status change after a run (two places). `heartbeat/retries.ts` sends it
for each scheduled retry. `writeExecutionControl` and
`notifyExecutionRunWork` send it when the scope names only a run. The
Runner's `set_task_monitor` tool sends it for one task.
- On a local fixture with 1,500 open tasks and run churn, the queue
loaded 47 company candidate rows per second from these hints.
- `pageEvidence` read `select *` from `heartbeat_runs`,
`issue_recovery_actions` and (through `getTableColumns`) `issues`;
`reviewEligibility` read `select *` from `issues`;
`interactionEligibility` read the full agent roster; the per-entry
`check` repeated full-row reads.

**Proposed behavior**

- Agent status changes send `{ companyId, sourceKind:
"agent_error_alert" }`. The status moves between `running`, `idle` and
`error`. All three are invokable for blocker attention
(`BLOCKER_ATTENTION_INVOKABLE_AGENT_STATUSES`) and for questions
(`agent-eligibility.ts`). Failed-run attention only excludes terminated
agents, and this path never sets that. So only agent error alerts can
change.
- Run-scoped execution-control writes, legacy run terminalization and
scheduled retries use `notifyIssueAttentionRetentionWork` with the run
id (the run's task, its parents and the tasks it blocks; company
fallback past 100). This is the same helper as #21521. The task monitor
tool scopes to its task.
- `notifyIssueAttentionRetentionWork` matched the run's saved task id
with `issue.id::text = …`. It now validates and casts the id, so the
primary key is used, as #21552 does for recovery.
- Narrow reads, same decisions:
- runs: `id`, `agent_id`, `created_at` and the saved task id computed in
SQL with the same rules as `readRunIssueId` (`issueId` unless it is
absent or JSON null, then `taskId`; only non-empty strings).
  - recovery actions: `id`, `source_issue_id`.
- issues for evidence and `visibleIssue`: the blocker-attention input
columns plus `unblock_descriptor` and `blocked_transition_at`.
- review issues: `id`, `company_id`, `status`, `execution_state`,
`assignee_user_id`.
- agent roster for question addressees: `id`, `company_id`, `name`,
`reports_to`, `status` (the `AgentOrgRow` that
`evaluateAgentInvokability` takes).

**Reason and benefit**

A run event no longer reloads every attention candidate of the company.
A reconstruction no longer transfers whole run, task and agent rows.

## What Changed

- `server/src/services/attention-retention.ts`: column lists for runs,
recovery actions, issues and the roster; the run probe type replaces
whole rows.
- `server/src/services/heartbeat/agent-status.ts`,
`heartbeat/retries.ts`, `execution-control-notifications.ts`,
`native-runtime/paperclip-runner-tool-authority.ts`: scoped attention
hints.
- `server/src/services/retention-work-notifications.ts`: indexed lookup
of a run's saved task.
- `server/src/__tests__/attention-retention-work.test.ts`: two tests.

## Verification

- `reads only the fields its decisions use from large runs, reviews and
agents` (I saw it fail first): 50 failed runs, 50 tasks in review with
large descriptions, 10 questions to a paused agent, about 40 KB of
payload in each run's snapshot, profile and stdout and in the agents'
configuration. It loads the three kinds with the full-size response
capture (`full-database-response-capture.ts`). Before: `heartbeat_runs`
6,317,700 bytes, `agents` 84,336 bytes, `issues` 4,235,810 bytes. After:
under the test bounds of 20,000, 5,000 and 60,000 bytes. The decisions
are the same before and after: 50 failed runs, 40 reviews and 10
questions.
- `scopes run lifecycle attention hints to the run's task graph and
agent status to agent alerts` (I saw it fail first):
`writeExecutionControl` and `notifyExecutionRunWork` with a run id send
the hint for the task and its parent. `finalizeAgentStatus` sends the
agent-alert hint. Before, all three sent `{ companyId }`.
- `attention-retention-work` (57 tests),
`attention-retention-candidates`, `attention-service`,
`execution-control-work` and `heartbeat-agent-status-recovery` pass,
each file run alone (in `server/`, against a dedicated local `_test`
database).
- `pnpm check:module-boundaries` passes.

Not done in this pull request, and why:

- I did not reproduce `syncItems` re-syncing unchanged rows from the
retention `check`. On my fixture (300 stalled blocked tasks, 4,500 old
recovery-action and blocker retention rows, run churn), `syncItems` ran
0 times in a 180-second window and `decision_retention` reads were 0 per
second. `check` calls `syncItems` only for entries that are already due
(older than the 90-day archive delay), and a due entry is archived and
leaves the candidates. So I did not batch `syncItems` or skip checks of
unchanged rows.
- `readIssues` still selects the `issues` columns that blocker attention
needs. I did not narrow `issueService.listBlockerAttentionBatch`.

## Risks

- A producer that changes more than the run's task graph would now miss
the wider effect. Company-level changes (agent configuration, company
status, budgets) keep their company hints. The 90-day archive delay
makes a missed reload cheap: the item is reloaded on the next hint that
covers it.
- The saved task id now comes from SQL. The SQL mirrors `readRunIssueId`
for strings, absent keys and JSON null. A non-string `issueId` value now
yields no task, as before.
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-14 02:17:21 -06:00
1 parent a1c37b5cc4
commit e8c79235ca
8 files changed
+120 -28

No files matched your search

@@ -276,6 +276,23 @@ turn, so memory per session counts for idle agents too.
| 1g | Size the shared limiters (log uploads, checkpoint pool, recovery pass) from the active run count | no fixed slot count below the run count |
| 1h | Keep one frozen, parsed tool catalog per catalog hash and company, shared by the sessions that use it, in place of two private copies per session (`assigned-mcp-tools.ts:146`) | server memory per live or warm session from about 5–11 MB (measured at 400–800 tools) to near zero; at 1,000 agents that is 5–11 GB |
### Background recovery sweeps
Run events used to send a company recovery hint, so the stranded-assignment
and assigned-work queues re-checked every open task of the company on each
run event (#21552). Local fixture: 1,500 open tasks, 60 agents, process agents
that fail at once, 180-second window that includes one 5-minute backstop pass.
| Build | Runs created (/s) | Stranded checks per window | Assigned-work checks per window | `agent_task_sessions` reads (/s) | Statements (/s) |
| --- | --- | --- | --- | --- | --- |
| Before #21552 (54c3616ad9) | 1.09 | 4,327 | 245 | 26.5 | 1,431 |
| #21552 at dcd170e29f | 1.00 | 702 | 276 | 7.2 | 1,060 |
| master at 340157d1d6 | 0.24 | 330 | 72 | 2.5 | 409 |
The master row has much less run churn than the earlier rows, so compare the
per-window totals with care. In that window the one backstop pass loaded 260
tasks into the stranded queue; run events loaded 147.
### Phase 2: several controller processes (target: 1,000)
| # | Change |
@@ -9,7 +9,8 @@ import { attentionRetentionService, registerAttentionRetentionWork } from "../se
import { workspaceOperationService } from "../services/workspace-operations.js";
import * as workspaceOperationLogs from "../services/workspace-operation-log-store.js";
import { budgetService, flushBudgetEvaluationWork } from "../services/budgets.js";
import { writeExecutionControl } from "../services/execution-control-notifications.js";
import { notifyExecutionRunWork, writeExecutionControl } from "../services/execution-control-notifications.js";
import { createHeartbeatAgentStatus } from "../services/heartbeat/agent-status.js";
import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
import { attentionService } from "../services/attention.js";
import { companyService } from "../services/companies.js";
@@ -26,6 +27,7 @@ import { idleWorkSnapshot } from "../services/task-admission.js";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
import { createFullDatabaseResponseCapture, installDatabaseResponseCapture } from "./helpers/full-database-response-capture.js";
const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_ATTENTION_RETENTION_TEST_DATABASE_URL");
const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
@@ -249,6 +251,45 @@ const delay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000;
expect(await db.select().from(decisionArchiveNotificationOutbox)).toHaveLength(0);
});
it("reads only the fields its decisions use from large runs, reviews and agents", async () => {
const payload = "saved-execution-input".repeat(2_000);
await db.update(agents).set({ adapterConfig: { instructions: payload }, runtimeConfig: { notes: payload } }).where(eq(agents.id, agentId));
const origins = await db.insert(issues).values(Array.from({ length: 50 }, () => ({ companyId,
title: randomUUID(), status: "in_review", assigneeAgentId: agentId, description: payload, updatedAt: old() }))).returning();
const runs = await db.insert(heartbeatRuns).values(origins.map(origin => ({ companyId, agentId, status: "failed",
contextSnapshot: { issueId: origin.id, prompt: payload }, runnerProfileJson: { instructions: payload },
stdoutExcerpt: payload, createdAt: old(), finishedAt: old() }))).returning({ id: heartbeatRuns.id });
await db.insert(heartbeatRunEvents).values(runs.map(run => ({ companyId, agentId, runId: run.id,
seq: 1, eventType: "lifecycle", message: "Bounded retry exhausted: byte fixture" })));
const [paused] = await db.insert(agents).values({ companyId, name: "Paused", role: "engineer", status: "paused",
adapterConfig: { instructions: payload } }).returning({ id: agents.id });
await db.insert(issueThreadInteractions).values(origins.slice(0, 10).map(row => ({ companyId, issueId: row.id,
kind: "ask_user_questions" as const, addresseeAgentId: paused.id, payload: { version: 1 as const, questions: [] } })));
const observed = createDb(externalUrl ?? temporary!.connectionString), events: Array<Record<string, unknown>> = [], now = Date.now();
const capture = createFullDatabaseResponseCapture({ start: now - 100, end: now + 59_000 }, event => events.push(event));
installDatabaseResponseCapture(observed, { heartbeatRuns, issues, agents }, capture);
const decisions = new Map<string, string>();
try {
const service = attentionRetentionService(observed);
for (const sourceKind of ["failed_run", "review", "issue_thread_interaction"] as const) {
for (let page = 0; page < 3; page++) {
const loaded = await service.load({ companyId, sourceKind });
for (const entry of loaded.rows ?? []) decisions.set(entry.id, `${entry.sourceKind}:${entry.dueAt}:${entry.activityAt}`);
if (loaded.complete) break;
}
}
capture.flush();
} finally { capture.flush(); await observed.$client.end(); }
const bytes = (table: string) => events.filter(event => event.table === table && event.operation === "select")
.reduce((total, event) => total + Number(event.responseValueBytes), 0);
const kinds = [...decisions.values()].reduce((counts, value) => {
const kind = value.split(":")[0]; counts[kind] = (counts[kind] ?? 0) + 1; return counts; }, {} as Record<string, number>);
expect(kinds).toEqual({ failed_run: 50, review: 40, issue_thread_interaction: 10 });
expect(bytes("heartbeat_runs")).toBeLessThan(20_000);
expect(bytes("agents")).toBeLessThan(5_000);
expect(bytes("issues")).toBeLessThan(60_000);
});
it("batches 100 failed runs and rechecks late visibility and supersession before archive", async () => {
const origins = await db.insert(issues).values(Array.from({ length: 100 }, () => ({ companyId,
title: randomUUID(), status: "todo", updatedAt: old() }))).returning();
@@ -877,6 +918,24 @@ const delay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000;
expect(hints).toEqual([scoped, scoped]);
});
it("scopes run lifecycle attention hints to the run's task graph and agent status to agent alerts", async () => {
const parent = await insertIssue("blocked");
const issue = await insertIssue("todo", { parentId: parent.id });
const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "failed",
contextSnapshot: { issueId: issue.id } }).returning();
const hints: string[] = [];
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.attentionRetention, (event, values) => {
if (event === "settled") hints.push(...values ?? []);
});
try {
await writeExecutionControl(db, DELIVERY_QUEUES.executionFinalization, async () => undefined, { companyId, runId: run.id });
await db.transaction(tx => notifyExecutionRunWork(tx, { companyId, runId: run.id }));
await createHeartbeatAgentStatus(db).finalizeAgentStatus(agentId, "failed");
} finally { unsubscribe(); }
const scoped = JSON.stringify({ companyId, issueIds: [issue.id, parent.id].sort() });
expect(hints).toEqual([scoped, scoped, JSON.stringify({ companyId, sourceKind: "agent_error_alert" })]);
});
it("accepts ordinary liveness incident creation and completion with a none leaf sentinel", async () => {
const source = await insertIssue("blocked", { responsibleUserId: "board" });
const service = issueService(db);
+31 -17
View File
@@ -1,8 +1,8 @@
import { and, eq, getTableColumns, inArray, isNull, sql, type SQL } from "drizzle-orm";
import { and, eq, inArray, isNull, sql, type SQL } from "drizzle-orm";
import { agents, decisionRetention, decisions, heartbeatRuns,
issueApprovals, issueRecoveryActions, issueThreadInteractions, issues, approvals, type Db } from "@paperclipai/db";
import { evaluateAgentInvokability } from "./agent-invokability.js";
import { HUMAN_RECOVERY_OWNER_TYPES, ATTENTION_BUDGET_WARNING_PERCENT, ATTENTION_AGENT_ERROR_STATUS, collapsePendingConfirmationsToNewest, listFailedRunsWithNewerRun, loadNewestSettledConfirmationAt, readRunIssueId, requiresReviewAttention, terminalBlockerRequiresAttention } from "./attention-source-eligibility.js";
import { HUMAN_RECOVERY_OWNER_TYPES, ATTENTION_BUDGET_WARNING_PERCENT, ATTENTION_AGENT_ERROR_STATUS, collapsePendingConfirmationsToNewest, listFailedRunsWithNewerRun, loadNewestSettledConfirmationAt, requiresReviewAttention, terminalBlockerRequiresAttention } from "./attention-source-eligibility.js";
import { decisionRetentionService, DEFAULT_DECISION_ARCHIVE_DAYS } from "./decision-retention.js";
import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js";
import { DELIVERY_QUEUES } from "./delivery-work-notifications.js";
@@ -23,12 +23,22 @@ type Candidate = { sourceKind: Kind; sourceId: string; companyId: string; activi
type Entry = Candidate & { id: string; dueAt: number; rootId: string; variant?: "owner" | "terminal" };
type Cursor = { sourceKind: string; sourceId: string };
type PageIssue = typeof issues.$inferSelect & { activityAt: string; deadlineAt: number;
const pageIssueColumns = { id: issues.id, companyId: issues.companyId, parentId: issues.parentId, identifier: issues.identifier,
title: issues.title, status: issues.status, assigneeAgentId: issues.assigneeAgentId, assigneeUserId: issues.assigneeUserId,
executionRunId: issues.executionRunId, unblockDescriptor: issues.unblockDescriptor, blockedTransitionAt: issues.blockedTransitionAt };
const contextSnapshot = heartbeatRuns.contextSnapshot;
const runProbeColumns = { id: heartbeatRuns.id, agentId: heartbeatRuns.agentId, createdAt: heartbeatRuns.createdAt,
issueId: sql<string | null>`case when jsonb_typeof(${contextSnapshot}->'issueId') is not null and jsonb_typeof(${contextSnapshot}->'issueId') <> 'null'
then case when jsonb_typeof(${contextSnapshot}->'issueId') = 'string' then nullif(${contextSnapshot}->>'issueId', '') end
else case when jsonb_typeof(${contextSnapshot}->'taskId') = 'string' then nullif(${contextSnapshot}->>'taskId', '') end end` };
type VisibleIssue = { [K in keyof typeof pageIssueColumns]: (typeof issues.$inferSelect)[K] };
type PageIssue = VisibleIssue & { activityAt: string; deadlineAt: number;
blockedActivityAt: string | null; blockedDeadlineAt: number | null };
type RunProbe = { id: string; agentId: string; createdAt: Date; issueId: string | null };
type PageEvidence = {
decisions: Map<string, { originIssueId: string }>;
recoveries: Map<string, typeof issueRecoveryActions.$inferSelect>;
runs: Map<string, typeof heartbeatRuns.$inferSelect>;
recoveries: Map<string, { id: string; sourceIssueId: string }>;
runs: Map<string, RunProbe>;
agents: Set<string>;
visibleIssues: Map<string, PageIssue>;
newerRuns: Set<string>;
@@ -127,7 +137,7 @@ export function attentionRetentionService(db: Db) {
const candidates = listAttentionRetentionCandidates;
async function visibleIssue(database: Db, companyId: string, issueId: string, conversations = false) {
const [row] = await database.select().from(issues).where(and(eq(issues.companyId, companyId), eq(issues.id, issueId),
const [row] = await database.select(pageIssueColumns).from(issues).where(and(eq(issues.companyId, companyId), eq(issues.id, issueId),
attentionIssueCondition(conversations)));
return row;
}
@@ -149,7 +159,8 @@ export function attentionRetentionService(db: Db) {
const visible = await database.select({ id: issues.id }).from(issues).where(and(eq(issues.companyId, companyId),
inArray(issues.id, rows.map(row => row.issueId)), attentionIssueCondition(true)));
const visibleIds = new Set(visible.map(row => row.id));
const roster = await database.select().from(agents).where(eq(agents.companyId, companyId));
const roster = await database.select({ id: agents.id, companyId: agents.companyId, name: agents.name,
reportsTo: agents.reportsTo, status: agents.status }).from(agents).where(eq(agents.companyId, companyId));
const boardAddressed = (row: typeof issueThreadInteractions.$inferSelect) => !row.addresseeAgentId
|| !evaluateAgentInvokability(roster.find(agent => agent.id === row.addresseeAgentId), roster).invokable;
const confirmationIssueIds = rows.filter(row => row.kind === "request_confirmation").map(row => row.issueId);
@@ -175,7 +186,8 @@ export function attentionRetentionService(db: Db) {
byCompany.set(candidate.companyId, ids);
}
for (const [companyId, ids] of byCompany) {
const rows = await database.select().from(issues).where(and(eq(issues.companyId, companyId),
const rows = await database.select({ id: issues.id, companyId: issues.companyId, status: issues.status,
executionState: issues.executionState, assigneeUserId: issues.assigneeUserId }).from(issues).where(and(eq(issues.companyId, companyId),
inArray(issues.id, ids), attentionIssueCondition(false)));
const pendingApprovals = await database.select({ issueId: issueApprovals.issueId }).from(issueApprovals)
.innerJoin(approvals, eq(approvals.id, issueApprovals.approvalId)).where(and(
@@ -209,14 +221,15 @@ export function attentionRetentionService(db: Db) {
const decisionIds = ids("decision"), recoveryIds = ids("recovery_action"), runIds = ids("failed_run"), blockerIds = ids("blocker_attention");
const decisionRows = decisionIds.length ? await database.select({ id: decisions.id, originIssueId: decisions.originIssueId })
.from(decisions).where(and(eq(decisions.companyId, companyId), inArray(decisions.id, decisionIds), eq(decisions.status, "open"))) : [];
const recoveryRows = recoveryIds.length ? await database.select().from(issueRecoveryActions).where(and(
const recoveryRows = recoveryIds.length ? await database.select({ id: issueRecoveryActions.id,
sourceIssueId: issueRecoveryActions.sourceIssueId }).from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, companyId), inArray(issueRecoveryActions.id, recoveryIds))) : [];
const runRows = runIds.length ? await database.select().from(heartbeatRuns).where(and(
const runRows: RunProbe[] = runIds.length ? await database.select(runProbeColumns).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, companyId), inArray(heartbeatRuns.id, runIds))) : [];
const runAgents = runRows.length ? await database.select({ id: agents.id }).from(agents).where(and(
eq(agents.companyId, companyId), inArray(agents.id, [...new Set(runRows.map(row => row.agentId))]),
sql`${agents.status} <> 'terminated'`)) : [];
const readIssues = async (issueIds: string[]) => issueIds.length ? database.select({ ...getTableColumns(issues),
const readIssues = async (issueIds: string[]) => issueIds.length ? database.select({ ...pageIssueColumns,
activityAt: sql<string>`${issues.updatedAt}::text`,
deadlineAt: sql<number>`ceil(extract(epoch from ${issues.updatedAt}) * 1000 + ${archiveDelay})`.mapWith(Number),
blockedActivityAt: sql<string | null>`${issues.blockedTransitionAt}::text`,
@@ -224,7 +237,7 @@ export function attentionRetentionService(db: Db) {
}).from(issues).where(and(eq(issues.companyId, companyId), inArray(issues.id, [...new Set(issueIds)]),
attentionIssueCondition(false))) : [];
const issueIds = [...blockerIds, ...decisionRows.map(row => row.originIssueId), ...recoveryRows.map(row => row.sourceIssueId),
...runRows.map(row => readRunIssueId(row.contextSnapshot))].filter((id): id is string => Boolean(id));
...runRows.map(row => row.issueId)].filter((id): id is string => Boolean(id));
const visibleRows = await readIssues(issueIds);
const visible = new Map(visibleRows.map(row => [row.id, row]));
const blockerRoots = visibleRows.filter(row => blockerIds.includes(row.id));
@@ -238,7 +251,7 @@ export function attentionRetentionService(db: Db) {
decisions: new Map(decisionRows.map(row => [row.id, row])), recoveries: new Map(recoveryRows.map(row => [row.id, row])),
runs: new Map(runRows.map(row => [row.id, row])), agents: new Set(runAgents.map(row => row.id)), visibleIssues: visible,
newerRuns: await listFailedRunsWithNewerRun(database, companyId,
runRows.map(run => ({ ...run, issueId: readRunIssueId(run.contextSnapshot) }))), blockers, deferredRoots: blockerBatch.deferredRootIds,
runRows), blockers, deferredRoots: blockerBatch.deferredRootIds,
});
}
return result;
@@ -261,7 +274,8 @@ export function attentionRetentionService(db: Db) {
const attention = interactionAttention ?? await interactionEligibility(database, [candidate]);
if (!attention.get(sourceId)) return null;
} else if (candidate.sourceKind === "recovery_action") {
const row = evidence ? evidence.recoveries.get(sourceId) : (await database.select().from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, candidate.companyId), eq(issueRecoveryActions.id, sourceId))))[0];
const row = evidence ? evidence.recoveries.get(sourceId) : (await database.select({ id: issueRecoveryActions.id,
sourceIssueId: issueRecoveryActions.sourceIssueId }).from(issueRecoveryActions).where(and(eq(issueRecoveryActions.companyId, candidate.companyId), eq(issueRecoveryActions.id, sourceId))))[0];
if (!row || !(evidence ? evidence.visibleIssues.has(row.sourceIssueId)
: await visibleIssue(database, candidate.companyId, row.sourceIssueId))) return null;
} else if (candidate.sourceKind === "review") {
@@ -297,14 +311,14 @@ export function attentionRetentionService(db: Db) {
eq(decisionRetention.sourceKind, candidate.sourceKind), eq(decisionRetention.sourceId, sourceId)));
if (retention?.keep || retention?.archivedAt) return null;
} else if (candidate.sourceKind === "failed_run") {
const run = evidence ? evidence.runs.get(sourceId) : (await database.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, candidate.companyId), eq(heartbeatRuns.id, sourceId))))[0];
const run: RunProbe | undefined = evidence ? evidence.runs.get(sourceId) : (await database.select(runProbeColumns).from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, candidate.companyId), eq(heartbeatRuns.id, sourceId))))[0];
if (!run) return null;
const agent = evidence ? evidence.agents.has(run.agentId) : (await database.select({ id: agents.id }).from(agents).where(and(eq(agents.companyId, candidate.companyId),
eq(agents.id, run.agentId), sql`${agents.status} <> 'terminated'`)))[0];
const issueId = readRunIssueId(run.contextSnapshot);
const issueId = run.issueId;
if (!agent || issueId && !(evidence ? evidence.visibleIssues.has(issueId)
: await visibleIssue(database, candidate.companyId, issueId))) return null;
if ((evidence ? evidence.newerRuns : await listFailedRunsWithNewerRun(database, candidate.companyId, [{ ...run, issueId }])).has(run.id)) return null;
if ((evidence ? evidence.newerRuns : await listFailedRunsWithNewerRun(database, candidate.companyId, [run])).has(run.id)) return null;
}
return { ...candidate, sourceId, activityAt, rootId: candidate.sourceId, variant,
id: `${candidate.sourceKind}:${candidate.sourceId}:${variant ?? "source"}`, dueAt };
@@ -10,7 +10,7 @@ import { DELIVERY_QUEUES, notifyDeliveryWork, type DeliveryQueue } from "./deliv
export type ExecutionWorkScope = { companyId: string; agentId?: string; runId?: string; issueId?: string };
async function notifyExecutionAttentionRetention(transaction: object, scope: ExecutionWorkScope) {
if (scope.issueId) await notifyIssueAttentionRetentionWork(transaction, { companyId: scope.companyId, issueId: scope.issueId });
if (scope.issueId || scope.runId) await notifyIssueAttentionRetentionWork(transaction, { companyId: scope.companyId, issueId: scope.issueId, runId: scope.runId });
else await notifyAttentionRetentionWork(transaction, scope);
}
@@ -94,7 +94,7 @@ export function createHeartbeatAgentStatus(db: Db) {
const nextStatus = running ? "running" : settledStatus(outcome, options?.keepIdleOnFailure);
const now = new Date();
await notifyStrandedAgentStatus(tx, { companyId: existing.companyId, agentId });
await notifyAttentionRetentionWork(tx, { companyId: existing.companyId });
await notifyAttentionRetentionWork(tx, { companyId: existing.companyId, sourceKind: "agent_error_alert" });
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
const [updated] = await tx.update(agents).set({
status: nextStatus,
@@ -152,7 +152,7 @@ export function createHeartbeatAgentStatus(db: Db) {
eq(heartbeatRuns.agentId, candidate.id)))
.orderBy(desc(heartbeatRuns.finishedAt), desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)).limit(1);
const nextStatus = settledStatus(lastRun?.status);
await notifyAttentionRetentionWork(tx, { companyId: candidate.companyId });
await notifyAttentionRetentionWork(tx, { companyId: candidate.companyId, sourceKind: "agent_error_alert" });
const [agent] = await tx.update(agents).set({ status: nextStatus,
errorReason: nextStatus === "error" ? errorReason(lastRun?.error) : null, updatedAt: now,
}).where(and(eq(agents.id, candidate.id), eq(agents.companyId, candidate.companyId),
+2 -2
View File
@@ -3,7 +3,7 @@ import { runOwnerFenceHeldSql } from "../remote-workspace-writer-hold.js";
import { AGENT_DIRECTORY_BUSY_ERROR_CODE, AGENT_DIRECTORY_BUSY_MAX_DEFERRALS, AGENT_DIRECTORY_BUSY_RETRY_REASON, AGENT_DIRECTORY_BUSY_RETRY_WAKE_REASON, agentDirectoryBusyRetryDelayMs, type AgentDirectoryPreparationDeferral } from "./agent-directory-busy.js";
import { isInterruptedNativePreparation } from "../cancelled-native-startup.js";
import { isPreDispatchExecutionWaitVerified } from "../pre-dispatch-execution-wait.js";
import { notifyAttentionRetentionWork } from "../retention-work-notifications.js";
import { notifyIssueAttentionRetentionWork } from "../retention-work-notifications.js";
import { normalizeAgentNameKey } from "../agent-name-key.js";
import { notifyStatusCardIssueChange } from "../status-card-notifications.js";
import { notifyScheduledRetryDispatch } from "../run-dispatch-notifications.js";
@@ -1209,7 +1209,7 @@ export function createHeartbeatRetries(db: Db, dependencies: HeartbeatRetryDepen
}
}
await notifyAttentionRetentionWork(tx, { companyId: run.companyId });
await notifyIssueAttentionRetentionWork(tx, { companyId: run.companyId, runId: run.id });
await notifyScheduledRetryDispatch(tx, { companyId: run.companyId, runId: run.id });
const wakeupRequest = await tx
.insert(agentWakeupRequests)
@@ -10,7 +10,7 @@ import { createCostEventInTransaction } from "../costs.js";
import { withOperationAccountingTransaction } from "../accounting-transaction.js";
import { notifyStatusCardIssueChange } from "../status-card-notifications.js";
import { nativeRunKeyScope, nativeSkillTestToolAllowed, assertNativeSkillTestTool } from "./native-skill-test-scope.js";
import { notifyAttentionRetentionWork } from "../retention-work-notifications.js";
import { notifyIssueAttentionRetentionWork } from "../retention-work-notifications.js";
import { requiredQualityReviewSchema, type RequestCheckboxConfirmationOption } from "@paperclipai/shared";
import { priorRecoveryApiReceipt } from "../recovery-effect-receipts.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js";
@@ -2560,7 +2560,7 @@ export class PaperclipRunnerToolAuthority {
replayMonitor = summarizeIssueMonitor(target, normalizeIssueExecutionPolicy(target.executionPolicy));
return { taskId: target.id, status: target.status, monitor: replayMonitor };
}
await notifyAttentionRetentionWork(tx, { companyId: this.binding.companyId });
await notifyIssueAttentionRetentionWork(tx, { companyId: this.binding.companyId, issueId: target.id });
const patch = prepareIssueMonitorUpdate(target, input.monitor, actor);
await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors);
await notifyStatusCardIssueChange(tx, this.binding.companyId);
@@ -6,6 +6,7 @@ import type { AttentionSourceKind } from "@paperclipai/shared";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
export const ATTENTION_RETENTION_MAX_ISSUE_IDS = 100;
const UUID_PATTERN = "^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$";
export type AttentionRetentionScope = { companyId?: string; sourceKind?: AttentionSourceKind;
issueId?: string; issueIds?: string[]; sourceId?: string };
@@ -27,9 +28,10 @@ export async function notifyIssueAttentionRetentionWork(transaction: object,
where company_id = ${scope.companyId}::uuid and id = ${scope.runId}::uuid and issue_id is not null
union select native_issue_id::text from heartbeat_runs
where company_id = ${scope.companyId}::uuid and id = ${scope.runId}::uuid and native_issue_id is not null
union select issue.id::text from heartbeat_runs run inner join issues issue
on issue.company_id = run.company_id
and issue.id::text = coalesce(run.context_snapshot->>'issueId', run.context_snapshot->>'taskId')
union select issue.id::text from heartbeat_runs run
cross join lateral (select coalesce(run.context_snapshot->>'issueId', run.context_snapshot->>'taskId') as saved) context
inner join issues issue on issue.company_id = run.company_id
and issue.id = case when context.saved ~* ${UUID_PATTERN} then context.saved::uuid end
where run.company_id = ${scope.companyId}::uuid and run.id = ${scope.runId}::uuid`);
for (const row of rows) seeds.push(row.id as string);
}