mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
DottaandClaude Opus 5.5
765ccd8691
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Chat connections save each external message as a `chat_actions` row
before the server admits it as an agent wakeup.
> - The server checks if an issue holds chat input that it did not
admit. Recovery skips such an issue. Manual task start refuses it.
> - On a hosted instance, the recovery read with this check used the
most total database time: 488 calls at about 1 s each. The manual-start
check ran 155 times at about 0.9 s.
> - The check matches `chat_actions.payload->>'issueId'` against the
issue. No index covers this value, so each call reads the full
`chat_actions` table and its payloads.
> - This pull request adds one concurrent partial index for the check.
The check SQL does not change.
> - The benefit is less background database load, so request reads wait
less. The results stay the same.
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The chat-admission check `unadmittedChatWakeupCondition`. Recovery uses
it in the per-issue candidate read. Manual task start uses it in the
held-input check.
**Subsystem affected**
Server chat admission, issue recovery, manual task start, and the
database schema.
**Current behavior**
The check finds `inbound_wakeup` actions by `company_id` and
`payload->>'issueId'`. Two nested checks find `failed_run_retry` actions
and later `inbound_wakeup` actions on the same company and payload issue
ID. `chat_actions` has no index on `company_id` or on the payload issue
ID. Each call reads the full table and reads each payload. On a hosted
instance, the two statements used about 630 s of database time in one
statistics window.
**Proposed behavior**
The check and its nested checks read a small set of rows from one index.
The admission result does not change.
**Reason and benefit**
The hosted database had many concurrent active queries. Request reads
waited behind background work such as this check.
**Breaking changes**
None. The migration adds an index only.
## What Changed
- Migration `0458_chat_admission_issue_index` builds
`chat_actions_company_issue_kind_idx` with `CREATE INDEX CONCURRENTLY IF
NOT EXISTS`. It runs through the existing top-level statement path,
which repairs an invalid index on retry.
- Keys: `(company_id, payload->>'issueId', kind, status)`.
- Partial: `kind IN ('inbound_wakeup', 'failed_run_retry')`.
- One index serves the outer input lookup, the failed-run retry lookup,
and the later-input lookup. All three match on the same company and
payload issue ID.
- The Drizzle schema declares the index. The snapshot chain moves to
`0458` and keeps five snapshots.
- New regression test
`server/src/__tests__/chat-admission-condition.test.ts` pins the
admission semantics on real PostgreSQL:
- Pending and failed input without a receipt holds the issue. Received,
processed and cancelled input does not. Other action kinds do not.
- A skipped receipt, a receipt for another agent, or a receipt in
another company does not admit input. A failed or cancelled run still
counts as admission.
- The issue ID is compared as company-scoped text. Uppercase, array,
null, other-issue and other-company values do not match.
- Only an admitted retry in the same conversation and issue that lists
the action releases skipped input.
- Only later admitted input in the same conversation, issue and session
generation releases a failed input. Pending input stays held.
- The check reads `chat_actions` only through the new index.
- `scripts/benchmark-chat-admission-index.ts` measures both caller
statements with and without the index and fails if the rows differ.
- `scripts/benchmark-chat-admission-index-migration.ts` times the
migration through the normal migration runner.
- `doc/DATABASE.md` documents the index and the benchmark commands.
## Verification
Local SQL measurements on synthetic data. PostgreSQL 18.1 (embedded),
default settings. The fixture has 100,995 `chat_actions` rows (662 MB):
60,000 GitHub actions with 6 KiB incompressible payloads, 2,000 issues
with 20 admitted inbound messages each (3 KiB payloads), and 1,000
failed-run retries. Each case has 2 warmups and 9 samples of `EXPLAIN
ANALYZE` execution time. Runs alternate: no index, index, no index,
index.
| Caller statement | Case | No index, median (range) | Index, median
(range) |
| --- | --- | --- | --- |
| Manual-start held check | quiet issue | 3,093 ms (2,150–3,519) / 3,586
ms (2,375–11,235) | 0.036 ms (0.034–0.052) / 0.040 ms (0.033–0.047) |
| Manual-start held check | admitted input | 3,555 ms (3,310–8,080) /
5,205 ms (3,353–9,525) | 0.049 ms / 0.052 ms |
| Manual-start held check | unadmitted input | 3,002 ms (2,864–3,221) /
4,179 ms (3,200–6,585) | 0.051 ms / 0.044 ms |
| Manual-start held check | failed, later admitted | 3,586 ms / 3,722 ms
| 0.066 ms / 0.054 ms |
| Manual-start held check | admitted retry | 2,733 ms / 4,953 ms | 0.049
ms / 0.053 ms |
| Recovery candidate read | quiet issue | 43 ms (41–48) / 96 ms (44–219)
| 0.047 ms / 0.055 ms |
| Recovery candidate read | admitted input | 203 ms (48–575) / 174 ms
(44–314) | 0.055 ms / 0.058 ms |
| Recovery candidate read | unadmitted input | 66 ms / 144 ms | 0.053 ms
/ 0.054 ms |
- Buffers per call fall from about 40,000 shared hits to 5–23.
- Locally, the recovery read without the index uses a hashed full scan
of `chat_actions`. The held check uses a full scan per call. On the
hosted instance both took about 1 s per call. The local fixture is
smaller than the hosted table.
- Correctness: complete positional rows match across all four runs for
all six cases (quiet, admitted, unadmitted, failed then admitted,
retried, other-company decoy). The script asserts the expected held
result for each case.
- Migration timing: 600,001 `chat_actions` rows (1,759 MB), 300,000 of
them in the index scope. First run through `applyPendingMigrations`: 4.2
s. Rerun after the history row is removed: 0.5 s. Concurrent build
alone: 3.4 s. Index size: 30 MB.
Commands:
```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-chat-admission-index.ts
OTHER_PAYLOAD_KIB=2 pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-chat-admission-index-migration.ts
cd server && npx vitest run src/__tests__/chat-admission-condition.test.ts # 6 passed
pnpm --filter @paperclipai/db check:migrations # passed
pnpm --filter @paperclipai/server typecheck # passed
```
## Risks
- The migration builds one index concurrently. It does not block writes.
On the 1.76 GB local fixture it took about 4 s. A larger hosted table
takes longer, but the build does not lock out reads or writes.
- Writes to `inbound_wakeup` and `failed_run_retry` actions maintain one
more index. Status updates on these rows were already not HOT updates,
because other partial indexes use `status`.
- The planner has no expression statistics for a partial index. The new
test pins that the check uses the index on an analyzed table with
unrelated rows.
- The check SQL does not change, so admission behavior does not change.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>