Files
PaperClipAI/server/src/lib/agent-reporting-chain.ts
T
DottaandClaude Opus 5.5 c7efca1eaf perf(recovery): read an agent's own reporting chain and each pass's review waits once (#21724)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server runs its background recovery queues on one Node main
thread. Each database statement and each byte returned costs main-thread
time. The target is 1,000 concurrent agents per server.
> - On the idle hosted staging instance, the stranded-task check is the
top remaining cost. Each check read the full agent row (with its
configuration blobs) and then every agent of the company, only to decide
whether this one agent can run. Per check it also opened a
repeatable-read budget transaction for the same company and agent.
> - The org-chain evaluation only looks at the agent's own managers.
Reading the whole company is work proportional to the size of the
company, for an answer that depends on a few rows.
> - The same check also read the external review wait of each in-review
task one task at a time.
> - This pull request reads only the agent's reporting chain (one
recursive query up `reports_to`), gives the stranded check a lean agent
read, and reads the review waits of a pass's tasks in one call.
> - The benefit is half the bytes per check (17.2 KB to 8.6 KB), 87
percent fewer rows, and no per-task review-wait chain for tasks that
wait, with the same answers.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

The per-task reads of the stranded-task recovery check
(`reconcileStrandedAssignedIssues`) and of every caller of
`evaluateAgentInvokabilityFromDb` and `agentService.getById`.

**Subsystem affected**

Server: issue recovery and agent invokability.

**Current behavior**

`evaluateAgentInvokabilityFromDb` reads `id, company_id, name,
reports_to, status` for every agent of the company on each call.
`getById` does the same for `orgChainHealth`. The stranded check calls
it once per task, after a full-row agent read with the configuration
blobs. On staging at idle these are the `agents where company_id = $1`
read at 4.7/s and the full `agents where id = $1` read at 4.2/s. The
check also calls `listExternalReviewWaits` for one task at a time (the
`chat_message_links` / `chat_github_reviews` reads at about 3/s on
staging).

**Proposed behavior**

Read only the agent's managers, with the same cycle, missing-manager,
cross-company and terminated rules. Read nothing when the agent has no
manager or when its own status already decides the answer. Read the
stranded check's agent with only the fields it uses. Read the review
waits of all tasks of a pass in one call, and use that answer only to
skip a task that waits.

**Reason and benefit**

At idle, the average result size on staging is about 17 KB per
statement. The stranded check is about 22 statements and 17 KB per task,
and it runs about 3 times per second there.

## What Changed

- `server/src/lib/agent-reporting-chain.ts` (new):
`readAgentReportingChain` reads the agent's managers with one recursive
CTE up `reports_to`, limited to the agent's company. `UNION` stops the
walk at a cycle. A root agent costs no statement.
- `server/src/services/agent-invokability.ts`:
`evaluateAgentInvokabilityFromDb` evaluates against the chain, not the
company list. It reads nothing when the agent's own status is not
invokable, because the status then decides the answer and the chain is
not part of it. Every caller benefits with no change: wake enqueue,
queued-run claim, retries, run dispatch, wake-queue promotion,
continuation delivery, task watchdogs, thread interactions, native
completion feedback, native session resumption, native review
participants, agent conversations and recovery.
- `server/src/lib/agent-records.ts`: `getById` computes `orgChainHealth`
from the chain, not from the company list.
- `server/src/services/recovery/service.ts`, stranded check:
- The agent read selects only `id, company_id, name, reports_to, status,
lifecycle_state, lifecycle_holds`, not the full row.
- `hasPersistedDurableWaitPath` takes the pass. When the task is in the
pass, the first in-review task reads the external review waits of every
task of the pass with one `listExternalReviewWaits` call. A task in that
answer waits and is skipped. A task not in that answer is read again
alone, exactly as before, so a missing answer never lets a check go
further than a fresh read would.
- A company-wide call without a pass gets a pass of its in-review
candidates.
- `server/src/services/scoped-deadline-work.ts`: an optional
`beforeChecks(entries)` hook gets the entries of the batch before their
checks run.
- `server/src/services/recovery/pass-reads.ts` (new) and
`server/src/services/recovery/work.ts`: the task queues build a pass
from the batch in `beforeChecks` and give it to each check of that
batch.
- Tests:
- `server/src/__tests__/agent-reporting-chain.test.ts` (new): property
test, see Verification.
- `server/src/__tests__/issue-recovery-pass-reads.test.ts` (new):
statements and bytes per check, per pass and per full worker drain.
- `server/src/__tests__/issue-recovery-review-wait-pass.test.ts` (new):
review-wait answers and reads, see Verification.
- `server/src/__tests__/monthly-spend-service.test.ts`: the
select-sequence stub no longer serves a company agent list to `getById`.
- `server/src/__tests__/helpers/recovery-population.ts` (new): seeds a
company with an org chart, review tasks with succeeded native runs, and
optional paused, terminated and recovery-action tasks.
- `server/src/__tests__/issue-recovery.test-fixture.ts`: an optional
statement meter, installed before the first connection.
- `server/src/__tests__/agent-record-read-projections.test.ts`:
`getById` now makes one typed `agents` select; the chain is a raw query.

### Why the shared review-wait read cannot start or escalate work that a
fresh read would not

- A pass is one coordinator batch: at most 25 checks, one after the
other. Its read is never older than the start of the batch. Nothing is
kept across batches.
- Only the positive answer is used: "this task waits on an external
review or reply", which leads to a skip. A skip only delays the task to
its next check, and the end of a wait sends its own hint.
- A negative answer is not used. The task is read again alone, so every
decision that goes further (an enqueue or an escalation) uses a fresh
read, as today.
- A first version also kept the agent's invokability and the budget
answer for the pass. Review found that a kept permissive answer can
reach an escalation, which does not re-check those facts the way an
enqueue does. That reuse is removed: those reads stay per task. No
authorization, pause, approval or budget check is removed or reordered.

## Verification

- Equality proof (`agent-reporting-chain.test.ts`, embedded PostgreSQL):
- 60 generated org charts, 2 companies each, up to 45 agents. Shapes:
deep chains (depth 44), trees, full cycles and mixed graphs.
- Managers are missing (inserted with foreign keys off), in another
company, terminated, paused or pending approval at random depths. Agent
statuses include an unknown value.
- For each agent, the test compares the old list-based result with the
new result. `evaluateAgentInvokabilityFromDb` deep-equals
`evaluateAgentInvokability(agent, companyRows)`.
`getAgentWorkEligibility` on the chain deep-equals it on the company
list (this is the `getById` path). The test also asserts that every
block reason and every chain-health reason occurred.
- The list-based evaluation has no depth cap (`reporting_chain_too_deep`
is declared but never returned), so the chain read has none either. A
cycle ends the walk in both.
- Mutation check: when the recursive step does not walk up, both tests
fail.
- Review-wait equality (`issue-recovery-review-wait-pass.test.ts`,
embedded PostgreSQL): two tasks with a direct chat reply wait, one task
with an inherited GitHub review wait, one task with none, and one task
with an inherited GitHub review wait in another company. One call for
all of them gives the same per-task answer as one call per task, for
both companies.
- Review-wait reads per pass (same file): the pass makes 2 bound-run
message reads (one shared, one for the task whose candidate is not a
wait) where per-task reads make 3, and 89 statements instead of 91. The
check results are identical. On master the pass is ignored, so the test
fails (3 is not less than 3).
- Statements and bytes (`issue-recovery-pass-reads.test.ts`, 50 agents,
200 review tasks with succeeded native runs and no review waits,
embedded PostgreSQL, statement meter on the client sockets):

  | | master | this branch |
  | --- | --- | --- |
  | One check: statements | 22 | 22 |
  | One check: bytes received | 17,170 | 8,617 |
  | One check: rows | 57 | 8 |
  | 25 checks: bytes | 429,331 | 215,505 |
| Real worker, full company drain (200 checks): statements | 4,414 |
4,422 |
  | Real worker, full company drain: bytes | 3,455,957 | 1,749,701 |
  | Real worker, full company drain: rows | 11,600 | 1,800 |
  | Company agent-list reads per drain | 200 | 0 |

Here no task waits, so each batch pays one extra review-wait statement
(8 per drain). Where tasks wait, the shared read replaces the per-task
review, publication and authorization reads.
- Regression after the review change, embedded PostgreSQL (664 tests):
scoped-deadline-work, heartbeat-process-recovery,
issue-recovery-deadlines, assigned-work-invariant,
recovery-deadline-acceptance, heartbeat-issue-liveness-escalation,
slack-conversation-lifecycle. Also issue-list-review-wait-authorization,
external-review-wait, monthly-spend-service.
- Regression of the first version, embedded PostgreSQL (757 tests):
issue-assignee-invokability-routes, agent-record-read-projections,
issue-graph-liveness-equivalence, heartbeat-paused-assignee-new-task,
heartbeat-process-recovery, heartbeat-issue-liveness-escalation,
assigned-work-invariant, recovery-deadline-acceptance,
issue-recovery-actions, heartbeat-refused-assignment-wake,
failed-continuation-continues-again, agent-invokability,
agent-cross-tenant-authz-routes, heartbeat-stop-queued-task,
issue-tree-pause-resume-run.
- `tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- A test double that gives `evaluateAgentInvokabilityFromDb` an object
with only `select` now needs `execute`, and a select-sequence stub of
`getById` gets one select fewer. CI found one (`monthly-spend-service`),
fixed here.
- A batch with no waiting task pays one extra cheap review-wait
statement.
- This branch and #21729 both touch
`server/src/services/recovery/work.ts` in nearby lines. The second one
to merge may need a small rebase.
- `server/src/services/recovery/service.ts` was already past 1,000
lines. This change adds about 25 lines to it, beside the check it
changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local PostgreSQL test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 01:34:08 -06:00

23 lines
1.2 KiB
TypeScript

import { sql } from "drizzle-orm";
import type { agents, Db } from "@paperclipai/db";
export type AgentReportingChainRow = Pick<typeof agents.$inferSelect, "id" | "companyId" | "name" | "reportsTo" | "status">;
export async function readAgentReportingChain(
db: Pick<Db, "execute">,
agent: { companyId: string; reportsTo: string | null },
): Promise<AgentReportingChainRow[]> {
if (!agent.reportsTo) return [];
const rows = await db.execute<{ id: string; company_id: string; name: string; reports_to: string | null; status: string }>(sql`
with recursive chain(id, company_id, name, reports_to, status) as (
select id, company_id, name, reports_to, status from agents
where company_id = ${agent.companyId}::uuid and id = ${agent.reportsTo}::uuid
union
select manager.id, manager.company_id, manager.name, manager.reports_to, manager.status
from agents manager inner join chain on manager.id = chain.reports_to
where manager.company_id = ${agent.companyId}::uuid
) select id::text, company_id::text, name, reports_to::text, status from chain`);
return rows.map(row => ({ id: row.id, companyId: row.company_id, name: row.name, reportsTo: row.reports_to,
status: row.status as AgentReportingChainRow["status"] }));
}