perf(native): index heartbeat runs by company and native session (#20560)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs share a native session. Retention, recovery, resume and
archive code reads all runs of one session from `heartbeat_runs`.
> - On a hosted instance, the native retention lookup ran about 67,000
times in one day at about 22 ms, and about 10,000 times at about 11 ms
in a later window.
> - No index has `native_session_id`. Each read walks the agent's or the
company's run history and filters the rows.
> - This pull request adds one concurrent partial index on `(company_id,
native_session_id)`. The SQL does not change.
> - The benefit is less background database load. Request reads wait
less, and the results stay the same.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The native session-run lookups. The largest caller is
`sessionRetentionBlocker` in native harness backup and checkpoint
retention. It selects `id, status, finished_at`, an incomplete-recovery
flag and the issue ID for one company, agent and native session.

**Subsystem affected**
Server native runtime (retention, recovery, resume and archive reads)
and the database schema.

**Current behavior**
`heartbeat_runs` has no index on `native_session_id`. PostgreSQL reads
the whole run history of the company (or of the agent through
`heartbeat_runs_company_agent_started_idx`) and filters by session. The
cost grows with the run history, not with the size of the session.

**Proposed behavior**
The index finds the few runs of one session directly. Every caller
already filters by `company_id` and `native_session_id`, so one index
serves the agent-scoped and the issue-scoped reads.

**Reason and benefit**
This statement family used about 1,500 s of database time in one day on
a busy hosted database. The local proof below shows 12.3 ms to 0.04 ms
per call.

## What Changed

- `packages/db`: add `heartbeat_runs_company_native_session_idx` on
`(company_id, native_session_id) WHERE native_session_id IS NOT NULL`.
Migration `0457_native_session_run_index` builds it with `CREATE INDEX
CONCURRENTLY IF NOT EXISTS`. The migration is registered as a
top-level-statement migration, the same way as 0454–0456.
- Add a regression test in `native-harness-backup-expiration.test.ts`.
It captures the real retention lookup and proves that its plan can use
the index. It also pins the company and session scope: a live run of
another company with the same session ID does not protect a checkpoint.
A live run of the same agent in another session does not protect it. A
live run of the same session does protect it.
- Add `scripts/benchmark-native-session-run-index.ts`. It runs the real
checkpoint retention pass (dry run), captures the exact SQL, and replays
it with and without the index. It also times the migration through the
normal migration runner.
- `doc/DATABASE.md`: describe the index and the benchmark command.

## Verification

Local SQL measurements, synthetic data. PostgreSQL 18.1 (embedded test
server). Fixture: 100,000 runs (50,000 of the measured agent, 50,000 of
a second agent), 4 runs per session, 4 KiB incompressible
`runner_profile_json` per run, 638 MB table. Order: no index, index, no
index, index. 15 `EXPLAIN (ANALYZE, BUFFERS)` samples after 3 warmups.

| Pass | Plan | Median execution | Min / max | Shared buffers |
| --- | --- | --- | --- | --- |
| No index (1) | Seq Scan | 12.30 ms | 10.33 / 35.27 ms | 3,607 |
| Index (2) | Index Scan using
`heartbeat_runs_company_native_session_idx` | 0.036 ms | 0.021 / 0.115
ms | 38 |
| No index (3) | Seq Scan | 12.38 ms | 10.24 / 26.14 ms | 3,607 |
| Index (4) | Index Scan using
`heartbeat_runs_company_native_session_idx` | 0.043 ms | 0.022 / 0.098
ms | 38 |

- Correctness: the service SQL is the same in every pass. The returned
rows (all columns) and the retention result are equal in every pass. The
script fails on any difference.
- Migration: the first run through the migration runner took 525 ms on
the 638 MB fixture. The rerun took 273 ms and skipped the existing
index. A direct concurrent build took 145–221 ms. Index size: 1.9 MB.
- `pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-native-session-run-index.ts`
- `npx vitest run
src/__tests__/native-harness-backup-expiration.test.ts` (server): 108
passed.
- `pnpm --filter @paperclipai/db check:migrations`: passed. Five
snapshots remain.
- `pnpm verify` exits 2 (a new script under `scripts/` is a global CI
input). Required CI supplies the broad coverage.

## Risks

- Low risk. The migration adds one index concurrently and does not lock
writes. The SQL of every caller is unchanged.
- The index adds a small write cost to `heartbeat_runs` inserts and to
updates of `native_session_id`. Rows with a null session are not
indexed.
- Persisted run state: no saved run, session or checkpoint shape
changes. State saved by the previous release carries forward unchanged.
- Hosted latency is not measured yet. The numbers above are local SQL
measurements.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edits, local PostgreSQL).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

packages/db/src/client.ts

@@ -619,6 +619,7 @@ const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([
   "0454_lumpy_ronan.sql",
   "0455_pending_budget_cancellation_index.sql",
   "0456_chat_review_lookup_indexes.sql",
+  "0457_native_session_run_index.sql",
 
 
   "0384_funny_adam_warlock.sql",

packages/db/src/migrations/0457_native_session_run_index.sql

@@ -0,0 +1 @@
+CREATE INDEX CONCURRENTLY IF NOT EXISTS "heartbeat_runs_company_native_session_idx" ON "heartbeat_runs" USING btree ("company_id","native_session_id") WHERE "heartbeat_runs"."native_session_id" is not null;

packages/db/src/migrations/meta/0457_snapshot.json

@@ -1,6 +1,6 @@
 {
-  "id": "64e8be1e-e0cb-4cd8-85e0-0fdcce5eafde",
-  "prevId": "b5699366-77c2-48c8-b7dc-658a75de7658",
+  "id": "109f5f96-1597-40da-9e08-6ed488f8ad27",
+  "prevId": "a2606b41-36c2-4eb5-a59d-28cc730bc455",
   "version": "7",
   "dialect": "postgresql",
   "tables": {
@@ -10955,6 +10955,28 @@
           "concurrently": false,
           "method": "btree",
           "with": {}
+        },
+        "chat_deliveries_github_automatic_delivery_idx": {
+          "name": "chat_deliveries_github_automatic_delivery_idx",
+          "columns": [
+            {
+              "expression": "endpoint_id",
+              "isExpression": false,
+              "asc": true,
+              "nulls": "last"
+            },
+            {
+              "expression": "(\"normalized_event\"->'githubAutomatic'->'context'->>'deliveryId')",
+              "asc": true,
+              "isExpression": true,
+              "nulls": "last"
+            }
+          ],
+          "isUnique": false,
+          "where": "(\"chat_deliveries\".\"normalized_event\"->'githubAutomatic'->'context'->>'deliveryId') is not null",
+          "concurrently": true,
+          "method": "btree",
+          "with": {}
         }
       },
       "foreignKeys": {
@@ -12393,6 +12415,44 @@
           "concurrently": false,
           "method": "btree",
           "with": {}
+        },
+        "chat_message_links_company_comment_idx": {
+          "name": "chat_message_links_company_comment_idx",
+          "columns": [
+            {
+              "expression": "company_id",
+              "isExpression": false,
+              "asc": true,
+              "nulls": "last"
+            },
+            {
+              "expression": "comment_id",
+              "isExpression": false,
+              "asc": true,
+              "nulls": "last"
+            }
+          ],
+          "isUnique": false,
+          "where": "\"chat_message_links\".\"comment_id\" is not null",
+          "concurrently": true,
+          "method": "btree",
+          "with": {}
+        },
+        "chat_message_links_delivery_idx": {
+          "name": "chat_message_links_delivery_idx",
+          "columns": [
+            {
+              "expression": "delivery_id",
+              "isExpression": false,
+              "asc": true,
+              "nulls": "last"
+            }
+          ],
+          "isUnique": false,
+          "where": "\"chat_message_links\".\"delivery_id\" is not null",
+          "concurrently": true,
+          "method": "btree",
+          "with": {}
         }
       },
       "foreignKeys": {
@@ -34552,6 +34612,34 @@
           "concurrently": false,
           "method": "btree",
           "with": {}
+        },
+        "heartbeat_run_events_retry_exhausted_idx": {
+          "name": "heartbeat_run_events_retry_exhausted_idx",
+          "columns": [
+            {
+              "expression": "company_id",
+              "isExpression": false,
+              "asc": true,
+              "nulls": "last"
+            },
+            {
+              "expression": "run_id",
+              "isExpression": false,
+              "asc": true,
+              "nulls": "last"
+            },
+            {
+              "expression": "id",
+              "isExpression": false,
+              "asc": false,
+              "nulls": "last"
+            }
+          ],
+          "isUnique": false,
+          "where": "\"heartbeat_run_events\".\"event_type\" = 'lifecycle' and \"heartbeat_run_events\".\"message\" like 'Bounded retry exhausted%'",
+          "concurrently": true,
+          "method": "btree",
+          "with": {}
         }
       },
       "foreignKeys": {
@@ -35393,6 +35481,22 @@
           "method": "btree",
           "with": {}
         },
+        "heartbeat_runs_pending_budget_cancellation_idx": {
+          "name": "heartbeat_runs_pending_budget_cancellation_idx",
+          "columns": [
+            {
+              "expression": "company_id",
+              "isExpression": false,
+              "asc": true,
+              "nulls": "last"
+            }
+          ],
+          "isUnique": false,
+          "where": "jsonb_typeof(\"heartbeat_runs\".\"result_json\"->'startupCancellation'->'budgetEnforcement') = 'object'\n        and \"heartbeat_runs\".\"result_json\"->'startupCancellation'->'budgetEnforcement'->>'deliveredAt' is null",
+          "concurrently": true,
+          "method": "btree",
+          "with": {}
+        },
         "heartbeat_runs_tree_hold_suppression_idx": {
           "name": "heartbeat_runs_tree_hold_suppression_idx",
           "columns": [
@@ -35431,6 +35535,27 @@
           "method": "btree",
           "with": {}
         },
+        "heartbeat_runs_company_wake_comment_idx": {
+          "name": "heartbeat_runs_company_wake_comment_idx",
+          "columns": [
+            {
+              "expression": "company_id",
+              "isExpression": false,
+              "asc": true,
+              "nulls": "last"
+            },
+            {
+              "expression": "(\"context_snapshot\"->>'wakeCommentId')",
+              "asc": true,
+              "isExpression": true,
+              "nulls": "last"
+            }
+          ],
+          "isUnique": false,
+          "concurrently": true,
+          "method": "btree",
+          "with": {}
+        },
         "heartbeat_runs_cost_accounting_pending_idx": {
           "name": "heartbeat_runs_cost_accounting_pending_idx",
           "columns": [
@@ -35562,6 +35687,49 @@
           "method": "btree",
           "with": {}
         },
+        "heartbeat_runs_company_native_workspace_idx": {
+          "name": "heartbeat_runs_company_native_workspace_idx",
+          "columns": [
+            {
+              "expression": "company_id",
+              "isExpression": false,
+              "asc": true,
+              "nulls": "last"
+            },
+            {
+              "expression": "(\"runner_profile_json\"->'nativeWorkspaceSync'->>'workspaceId')",
+              "asc": true,
+              "isExpression": true,
+              "nulls": "last"
+            }
+          ],
+          "isUnique": false,
+          "concurrently": true,
+          "method": "btree",
+          "with": {}
+        },
+        "heartbeat_runs_company_native_session_idx": {
+          "name": "heartbeat_runs_company_native_session_idx",
+          "columns": [
+            {
+              "expression": "company_id",
+              "isExpression": false,
+              "asc": true,
+              "nulls": "last"
+            },
+            {
+              "expression": "native_session_id",
+              "isExpression": false,
+              "asc": true,
+              "nulls": "last"
+            }
+          ],
+          "isUnique": false,
+          "where": "\"heartbeat_runs\".\"native_session_id\" is not null",
+          "concurrently": true,
+          "method": "btree",
+          "with": {}
+        },
         "heartbeat_runs_company_issue_created_idx": {
           "name": "heartbeat_runs_company_issue_created_idx",
           "columns": [

packages/db/src/migrations/meta/_journal.json

@@ -3186,6 +3186,13 @@
       "when": 1794270045155,
       "tag": "0456_chat_review_lookup_indexes",
       "breakpoints": true
+    },
+    {
+      "idx": 457,
+      "version": "7",
+      "when": 1794323383634,
+      "tag": "0457_native_session_run_index",
+      "breakpoints": true
     }
   ]
 }

packages/db/src/schema/heartbeat_runs.ts

@@ -208,6 +208,12 @@ export const heartbeatRuns = pgTable(
     companyNativeWorkspaceIdx: index("heartbeat_runs_company_native_workspace_idx")
       .on(table.companyId, sql`(${table.runnerProfileJson}->'nativeWorkspaceSync'->>'workspaceId')`)
       .concurrently(),
+    // Runs of one native session. Retention, recovery and resume reads filter
+    // by company and session; without this they walk the agent's run history.
+    companyNativeSessionIdx: index("heartbeat_runs_company_native_session_idx")
+      .on(table.companyId, table.nativeSessionId)
+      .where(sql`${table.nativeSessionId} is not null`)
+      .concurrently(),
     companyIssueCreatedIdx: index("heartbeat_runs_company_issue_created_idx").on(
       table.companyId,
       table.issueId,