Files
PaperClipAI/server/scripts
DottaandClaude Fable 5.2 4e932fd178 perf(attention): index the agents and review paths once for every review of a request (#21132)
<!-- Write all pull request text in Simplified Technical English
(ASD-STE100): short sentences, one instruction per sentence, simple
approved vocabulary, and the active voice. -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A task in review needs a path to its next action: a reviewer, a
pending interaction or approval, a monitor, a run, a wake, an external
reply or a recovery. A review without a path is stalled, and the
attention feed tells a person about it.
> - The server decides this for each task in review
(`listIssueReviewAttentionMap` in `server/src/services/issues.ts`, with
the classification in
`server/src/services/recovery/issue-graph-liveness.ts`). The task list,
the blocked inbox and the attention feed call it. `GET
/attention/summary`, which the board sidebar reads on most pages, runs
it for every task in review of the company.
> - The classification did the company-wide work again for each task.
Each check of an agent copied the agent list, indexed it again and
walked the reporting chain. Each task scanned every run, wake,
interaction, approval, external wait and recovery of the request. Each
stalled review also filtered and sorted every agent again, with one such
check for each agent.
> - Thus the work grew with reviews × agents for covered reviews and
with stalled reviews × agents² for stalled reviews. It also allocated a
new list and a new index for each check, and the garbage collector of
the server paid for that.
> - On a hosted instance, an allocation sample of the server put 4.7% of
all allocated bytes under this function, and garbage collection used 25%
of the main thread. The two slowest board requests there are the
attention summary and the task list, which both call it.
> - This pull request builds one index for one classification: the
agents by id, the eligibility of each agent, the invokable agents of
each company in id order, and the paths of each task. Each task then
costs its own paths.
> - The benefit is that the review classification of a summary request
falls from 62 ms to 5 ms for 900 reviews and 60 agents, and from 123 ms
to 8 ms for 300 reviews and 180 agents. The request allocates 57% to 91%
fewer bytes. Every finding, every review path and every count is the
same.

## Linked Issues or Issue Description

Refs #20851 and #20858 (they batched the review-wait provenance check
and the task read decision of the same requests; this change is the next
cost on that path).
Refs #19194 and #13729 (open; they change the failed-run lookup of the
attention feed, which is a different statement and a different
mechanism; this pull request does not touch it).

**What existing behavior does this improve?**

The time and the memory that the server uses to decide the review
attention of the tasks in review, for `GET
/api/companies/:companyId/attention/summary`, `GET
/api/companies/:companyId/attention`, `GET
/api/companies/:companyId/issues` and the blocked inbox.

**Subsystem affected**

Server: task-graph liveness classification
(`server/src/services/recovery/issue-graph-liveness.ts`) and the
review-attention map in `server/src/services/issues.ts`.

**Current behavior**

- `classifyIssueReviewPaths(input, issue)` built `new
Map(input.agents…)` for each task and scanned six path lists of the
request for each task.
- `isInvokableAgent` copied the agent list for each check.
`getAgentOrgChainHealth` then indexed that copy again.
- `ownerCandidatesForRecoveryIssue` ran for each stalled review. It
filtered and sorted all agents, and it checked each agent again in two
more loops. Each check did the copy and the index above.
- `hasActiveExecutionPath` built a joined list of the runs and the wakes
for each blocker it looked at.
- `listIssueReviewAttentionMap` searched the review list for each open
recovery task (`reviewIds.includes`) and for each review
(`issues.find`).

Evidence from a hosted instance (sanitized):

- Server main thread at 93% of one core. Garbage collection: 25% of the
main thread (387 pauses in 32 s, the longest 224 ms).
- Allocation sample, 30 s: 77.7 MB/s in total.
`listIssueReviewAttentionMap` and its callees: 4.7% of all bytes
(`classifyIssueReviewPaths` 4.4%, `classifyIssueGraphLiveness` 2.3%,
`reviewFinding` 2.3%).
- `GET /attention/summary` took 16.3 s of server time and `GET /issues`
up to 15.3 s. Most of that time is waiting behind other work on the same
thread (see "Not verified").

**Proposed behavior**

One classification indexes its input one time. The cost of one task is
the number of its own paths. The cost of the agents is one eligibility
decision for each agent. The results do not change.

**Reason and benefit**

The summary request runs on most board pages. Its CPU time and its
allocation also delay every other request, because the server has one
main thread and one garbage collector.

**Breaking changes**

None. `classifyIssueGraphLiveness` and `classifyIssueReviewPaths` keep
their signatures and results. `packages/shared` does not change.

## What Changed

- `server/src/services/recovery/issue-graph-liveness.ts`
- New internal index for one input (`createIssueGraphLivenessIndex`):
agents by id, a memo of `getAgentWorkEligibility` for each agent record,
the invokable agents of each company in id order, and the paths of each
task for the six path lists. The path lists are indexed on first use and
keep the order of the input. The company check of each path stays.
- `classifyIssueGraphLiveness` builds the index one time and uses it for
the review paths, the waiting paths of blockers, the owner candidates
and the assignee eligibility of blockers.
- New export `createIssueReviewPathClassifier(input)`. It returns the
review paths of many tasks from one index.
`classifyIssueReviewPaths(input, issue)` stays for single tasks.
- The memo uses the agent record as its key, not the agent id. Thus an
input that lists one id two times gets the same answers as before.
- The eligibility decision itself is still `getAgentWorkEligibility` of
`@paperclipai/shared`, with the same agent list. The index calls it one
time for each agent, not one time for each check.
- `server/src/services/issues.ts` (`listIssueReviewAttentionMap` only):
one classifier for all reviews of the request, a set for the review ids,
and the review by position instead of a search. No query changes.
- Tests and the reproduction (see Verification):
- `server/src/__tests__/helpers/issue-graph-liveness-reference.ts`: the
classification as it was before this change. It is the oracle of the
equivalence tests.
- `server/src/__tests__/issue-graph-liveness-equivalence.test.ts`,
`server/src/__tests__/review-attention-classification-equivalence.test.ts`,
`server/src/__tests__/helpers/review-attention-fixture.ts`.
- `server/scripts/benchmark-review-attention.ts`: the reproduction
through the real routes.

## Verification

Reproduction and before/after measurement. The script starts a local
PostgreSQL, seeds a synthetic company, mounts the real `attentionRoutes`
and `issueRoutes` with a board user, and sends the two requests over
HTTP. With `--baseline-ref` it runs the pre-fix modules in a second
process against the same database. It fails unless both versions return
the same summary, the same full attention feed, the same task list and
the same review attention for every task in review.

```sh
pnpm --filter @paperclipai/server exec tsx scripts/benchmark-review-attention.ts --baseline-ref=ef2b53b675feebb1bfbd6701627f6c302f8c6916 --tasks=3000,9000 --agents=60
pnpm --filter @paperclipai/server exec tsx scripts/benchmark-review-attention.ts --baseline-ref=ef2b53b675feebb1bfbd6701627f6c302f8c6916 --tasks=3000 --agents=180
```

Fixture: one company with tasks of every status (10% in review, 10%
blocked), four of five tasks are children, a reporting tree of agents
with paused, terminated and pending agents. The reviews cover every kind
of path and three ways to stall (31% stalled). The blocked tasks wait
behind chains, cycles, cancelled, done, backlog and unassigned blockers.
Open recovery tasks (5% of the tasks) name reviews, blocked tasks and
done tasks. 6% of the reviews wait for a GitHub reply. A second company
with the same shapes is in the same database.

Embedded PostgreSQL 18.1, Node v26.4.0, `VACUUM ANALYZE` after seeding.
Each version ran in a new process, three rounds in alternating order, 2
warm-up requests and 5 measured requests in each round. A CPU profile of
3 more requests gives the last two columns.

| Tasks / agents / reviews | Request | Version | Statements | Server ms,
median (min) | CPU ms, median (min) | Main thread busy ms, median (min)
| Allocated MB | Review classification ms | GC ms |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 3000 / 60 / 300 (94 stalled) | attention/summary | before | 69 | 121.4
(112.7) | 131.5 (114.4) | 80.2 (73.2) | 170.4 | 16.8 | 5.0 |
| 3000 / 60 / 300 (94 stalled) | attention/summary | after | 69 | 105.9
(94.7) | 113.3 (96.3) | 67.3 (58.6) | 73.3 | 2.4 | 3.4 |
| 9000 / 60 / 900 (283 stalled) | attention/summary | before | 85 |
280.9 (263.9) | 281.4 (253.4) | 203.7 (186.8) | 509.8 | 62.1 | 30.0 |
| 9000 / 60 / 900 (283 stalled) | attention/summary | after | 85 | 224.8
(213.4) | 220.1 (205.8) | 149.9 (142.3) | 219.1 | 4.8 | 6.6 |
| 3000 / 180 / 300 (94 stalled) | attention/summary | before | 69 |
241.2 (203.9) | 248.7 (227.4) | 197.1 (171.1) | 950.2 | 123.2 | 21.2 |
| 3000 / 180 / 300 (94 stalled) | attention/summary | after | 69 | 127.8
(108.5) | 128.5 (102.0) | 76.7 (64.8) | 87.3 | 7.8 | 5.1 |
| 3000 / 60 / 300 | task list (500 rows, 48 reviews) | before | 59 |
102.7 (96.7) | 92.4 (83.6) | 54.7 (48.9) | 73.2 | 2.6 | 2.4 |
| 3000 / 60 / 300 | task list (500 rows, 48 reviews) | after | 59 |
106.2 (91.2) | 89.9 (78.8) | 55.2 (44.8) | 63.4 | 0.9 | 1.6 |
| 9000 / 60 / 900 | task list (500 rows, 50 reviews) | before | 42 |
194.3 (186.0) | 77.5 (69.1) | 43.3 (38.4) | 57.3 | 1.0 | 1.8 |
| 9000 / 60 / 900 | task list (500 rows, 50 reviews) | after | 42 |
189.6 (182.3) | 72.1 (69.5) | 41.9 (39.4) | 57.8 | 1.2 | 2.5 |
| 3000 / 180 / 300 | task list (500 rows, 48 reviews) | before | 59 |
124.6 (95.7) | 110.2 (99.4) | 67.8 (58.1) | 165.4 | 15.4 | 3.8 |
| 3000 / 180 / 300 | task list (500 rows, 48 reviews) | after | 59 |
104.1 (91.9) | 90.7 (79.9) | 56.2 (47.1) | 78.2 | 2.9 | 5.3 |

How to read the table:

- "Review classification ms" is the time in
`listIssueReviewAttentionMap` and everything it calls on the main
thread, for one request, from the CPU profile. It is the direct measure
of this change. Before the fix it was the largest block of application
code in the summary handler: 22% of the busy time at 3000 tasks, 20% at
9000 tasks, and 61% with 180 agents. `getAgentOrgChainHealth` under
`isInvokableAgent` under `ownerCandidatesForRecoveryIssue` had most of
it. After the fix it is 4% to 9%.
- The scaling: three times the tasks gave 3.7 times the classification
time before and 2.0 times after. Three times the agents gave 7.3 times
before and 3.3 times after.
- "Allocated MB" is all bytes that the server process allocated for one
request (`total_allocated_bytes`), including the HTTP client of the
script.
- Other agents used the same machine during the measurement (load
average 8 to 16 on 18 CPUs), so the medians of server time have noise.
The profile columns and the allocation column are the stable ones.
- The task list changes little with 60 agents, because one page has
about 50 reviews. With 180 agents its classification falls from 15 ms to
3 ms and its allocation by half.
- Statement counts do not change: this change removes no statement.

Red/green regression test for the mechanism
(`server/src/__tests__/issue-graph-liveness-equivalence.test.ts`):

- `reads each agent and each path a number of times that does not grow
with the reviews`. The test counts the reads of the agent status and of
the path task id for 20 and for 400 stalled reviews with 40 agents and
360 paths. On the code before this change: `expected { agentStatus:
200800, pathIssueId: 144000 } to deeply equal { agentStatus: 10040,
pathIssueId: 7200 }`. After: 316 agent reads and 360 path reads for both
sizes (one read of each path). The test also runs the oracle and asserts
that its reads grow, so it shows what it catches.
- `classifies the review paths of a page from one index`. The same count
for `createIssueReviewPathClassifier`.

Equivalence tests (old code and new code on the same input):

- `returns the findings and review paths of the pre-fix classification
for random task graphs`. 400 seeded random graphs of two companies. Each
graph has reviews with and without paths, blocked chains and cycles,
terminal blockers, relations and paths that name the other company or a
missing task, reporting chains with cycles, missing managers and
managers of the other company, and ids that the input lists two times.
The test compares the findings of the graph and the review paths of
every task, for both `classifyIssueReviewPaths` and the new classifier.
It asserts that the graphs reach all six finding states and all nine
kinds of path, each at least 20 times.
- `returns the pre-fix findings for blocked chains, cycles and terminal
blockers`. One named graph with the expected state of each finding, so
the comparison cannot pass on two wrong answers.
-
`server/src/__tests__/review-attention-classification-equivalence.test.ts`,
on real PostgreSQL through the real routes. The test wraps the
classification module: every classification that a request runs is also
run by the oracle on the same input, and a difference fails the test.
- `gives every review the paths the fixture was built with, and the
pre-fix classification of every request`. `GET
/issues?status=in_review`, `GET /issues` and `GET
/issues?attention=blocked` (the blocked inbox, whose graph has the
chains, cycles and terminal blockers). Each of the 64 reviews has the
path kinds and the stalled reason that the fixture states for it,
including reviews that wait for an external reply and reviews that open
recovery tasks cover.
- `counts the reviews that need a person, leaves out the ones a person
dismissed, and reads one company`. `GET /attention/summary` and `GET
/attention`: the review count and the review cards are the reviews that
the fixture built to need a person. The second company has its own
counts and none of the first company's tasks. Three dismissed cards
leave the count and stay stalled in the task list.
- The benchmark above compares the complete responses of both versions
through the routes at each size (summary, full feed, task list, review
attention of every review).

Commands run locally:

- `pnpm --filter @paperclipai/server exec vitest run` for
`issue-graph-liveness-equivalence`,
`review-attention-classification-equivalence`, `issue-liveness`,
`recovery-classifiers`, `issue-review-attention`,
`issue-blocker-attention`, `blocked-inbox-review-wait-scope`,
`heartbeat-issue-liveness-escalation`, `attention-retention-work` and
`attention-service`: 210 tests passed. In that run the `afterAll` hook
of `attention-retention-work` and of `attention-service` (the stop of
the test database) timed out at 30 s while other agents loaded the
machine. No assertion failed. CI runs both files.
- `tsc --noEmit` for the server package: no errors. The new test files
and the script are outside the server `tsconfig`; a focused `tsc`
program for them reports no error in these files.
- `pnpm check:module-boundaries`, `node scripts/check-no-git-push.mjs`
and `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs`:
pass.
- The first CI run of this pull request failed `ci / Runner support
suite`. An earlier revision also let the eligibility functions of
`packages/shared` accept a prebuilt index, and the source of
`packages/shared` is pinned for the runner SDK source proof
(`tests/runner-e2e/source-sdk-public-inputs.json`). The fix does not
need that change, so the current revision leaves `packages/shared` as it
is on `master`: no pin moves, and the UI, the CLI and the runner use the
same eligibility code as before. `source-sdk-public-inputs.test.mjs`,
`source-sdk-evidence.test.ts` and `source-sdk-public-type.test.ts` pass
locally on this revision (58 tests), and the drift guard reports no
stale pin.
- `pnpm verify` returns exit 2 (broader verification required) because
the change adds a file under `server/scripts/`. CI supplies the broad
coverage. The repo-wide typecheck, build and test suite were not run
locally (shared machine).

Not verified:

- The effect on the hosted instance. This pull request does not deploy.
The local requests take 0.1 s to 0.4 s, and the hosted requests took 15
s. Thus most of the hosted time is not this code: the hosted main thread
is saturated by other work, and each of the 69 to 85 statements of a
summary request waits for its turn on that thread. This change removes
CPU time and allocation from that thread. It does not remove a
statement.
- The number of agents and of stalled reviews of the hosted company. The
hosted allocation sample has most of the bytes in
`classifyIssueReviewPaths`, which is the cost for each review (reviews ×
agents). The local fixture has more stalled reviews, where the owner
candidates cost more.
- A remaining cost inside the classification: each eligibility decision
of `@paperclipai/shared` indexes the agent list again. One
classification now does this one time for each agent (agents² map
entries for one request, 3 ms with 180 agents), not one time for each
check. The helper can accept a prebuilt index in a later change; that
change moves the SDK source pins, so it needs its own review.
- A second, independent cost that this pull request does not change:
`GET /attention/summary` returns counts only, but it builds the full
feed. After this change, the largest application blocks in its profile
at 9000 tasks are the full task list of the 900 blocked tasks (15 ms on
the main thread, of which 9 ms decode description previews that the
summary never returns), the blocker attention of those tasks (9 ms), and
the text of its 85 statements. A later change can give the summary a
count-only path.

## Risks

- The attention feed decides what a person is told needs them. The risk
is a classification that differs from the previous one. Mitigation: the
rules did not move. Only the lookups changed (a map lookup instead of a
scan, a stored eligibility instead of a new one). The oracle tests
compare the findings and the paths for random graphs, for a named graph
and for every classification of the route tests. The benchmark compares
the full responses.
- The index is a snapshot of the input.
`createIssueReviewPathClassifier` documents that the input must not
change while the classifier is in use. The two callers build the input
and use it in the same synchronous block.
- When the input has no `now`, one classification now reads the clock
one time. Before, `classifyIssueGraphLiveness` read it one time for the
blocker checks and one more time for each review. All callers in the
server that classify many tasks pass `now` or run in one synchronous
call, so the values differ by less than the run time of the call.
- The oracle file is a copy of the previous classification (about 640
lines, test-only). A later change of a classification rule must change
the oracle in the same way. The header of the file says so.
- The repository rule in `CONTRIBUTING.md` says that a pull request has
no code comments. This pull request has comments where the code has a
constraint that the names do not show (the memo key, the snapshot rule,
the purpose of the oracle, the usage of the script), as the merged
changes around this code do.
- No migration. No query change. No change to company scope or to actor
permissions: the change is after the reads, on rows that the same
queries return.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude
Code with tool use (code reading, editing, shell, tests, CPU profiles).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 10:26:24 -06:00
..