perf(workspaces): index legacy workspace companion lookups (#20372)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Online workspace migration must retain prepared sources and recovery
holds.
> - Companion discovery reads saved workspace references from run
profiles.
> - The selector can scan large saved profiles even when it returns few
rows.
> - This pull request adds one company and workspace expression index.
> - Companion discovery uses the same expression as existing migration
lookups.
> - The result preserves each source check and reduces history scan
work.

## Linked Issues or Issue Description

**What happened?**

The companion selector used a JSON path expression with no matching
index. A read-only plan on about 22,000 run rows showed a sequential
scan over a table with about 1.3 GB of table and TOAST storage. A
repeated inspection reached its statement timeout. Existing migration
lookups use a different JSON operator form, so one expression index must
match both callers.

**Expected behavior**

Use a company-scoped indexed lookup. Keep prepared and finalized
references, the exact text comparison, native issue references, and run
order. Keep all source and recovery validation.

**Steps to reproduce**

1. Retain many run profiles with saved workspace references and wide
JSON fields.
2. Inspect a workspace with few matching companion runs.
3. Explain the company and saved-workspace selector. Without the new
index, it can scan run history and read unrelated JSON values.

**Paperclip version or commit**

Baseline: `0e3fdebea8257bdff11573081c76b306f12db230`.

**Deployment mode**

Server with PostgreSQL. The local proof uses disposable PostgreSQL 18.1.
The observed server plan used PostgreSQL 17.11.

Related: [batched companion discovery in
#19735](https://github.com/paperclipai/paperclip/pull/19735). The
existing batching remains in place. The duplicate search found no open
workspace index change. This branch also carries the isolated API
route-summary correction shared with
[#20370](https://github.com/paperclipai/paperclip/pull/20370).

## What Changed

- Add one concurrent index on company ID and the saved native workspace
ID expression. The index has no state, runtime, or status predicate.
- Use nested JSON operators in companion discovery. Existing migration
lookups already use this expression. Keep both accepted states and the
same ordering.
- Register the migration on the existing top-level migration path.
Retain its advisory lock, interrupted index repair, and completion
journal behavior.
- Add a real PostgreSQL migration and ordered-result test. Update the
existing companion query counter for the equivalent SQL form.
- Generate the schema snapshot and retain the five newest snapshots.
- Describe the task case-list route as “Find the cases for a task.” This
keeps the route on the first discovery search page. Search logic and
test assertions are unchanged.

## Verification

- `pnpm --filter @paperclipai/db exec vitest run
src/heartbeat-workspace-companion-index.test.ts --maxWorkers=1`: **1
passed**. It builds the concurrent index through the canonical migrator,
checks index validity, preserves prepared/finalized and
cross-issue/null-issue results, excludes foreign and malformed
references, and checks one journal entry after a second migration call.
- `pnpm --filter @paperclipai/server exec vitest run
src/services/remote-workspace-online-source.test.ts -t 'discovers all
Git companions' --maxWorkers=1`: **1 passed; 86 not selected**. This
uses the real source owner and retains finalized recovery holds, company
isolation, and a fresh obligation between inspections.
- `pnpm --filter @paperclipai/db typecheck`, migration numbering/safety
checks, and `git diff --check`: **passed**.
- One small local SQL comparison used the captured baseline selector,
the changed selector, 3,012 synthetic rows, about 37.8 MB of table
storage, and the new expression index. Ordered results were identical.
One client timing changed from **8.70 ms to 0.86 ms**. This is one local
warm-cache comparison, not a server latency claim. Hosted improvement is
**unverified**.
- Initial setup attempts failed before the affected test ran: registry
access, local tsx IPC, and a missing SDK build. The canonical Node
24.22.2/pnpm 9.15.4 setup and SDK prerequisite resolved these setup
failures. The local SQL timing client used Node 26.4.0; the focused
application tests used Node 24.22.2.
- [The first CI
run](https://github.com/paperclipai/paperclip/actions/runs/42705129724/job/129460702073)
failed the inherited case-history API first-page search assertion. The
same failure was reproduced on the unchanged base: the route ranked
sixth for “case for task.” The isolated route-summary correction from
#20370 passed the same three assertions. The search algorithm and
assertions were not changed.
- No public command changes require a documentation update. The schema
and migration record the added index. No hosted DDL or deployment was
performed for this PR.

- Final head `c50101dabf46ade42b545d2051d724d93f413ccd`: [full
CI](https://github.com/paperclipai/paperclip/actions/runs/42707623920)
passed with 60 successful checks and three configured skips. [Fresh
Greptile
review](https://github.com/paperclipai/paperclip/pull/20372#issuecomment-6752305627)
is 5/5 on this head with no open threads.

## Risks

- The index adds storage and write cost when a run profile changes.
- A concurrent build reads the table and can wait for old transactions.
The existing migration path runs it outside a transaction and can repair
an interrupted invalid index.
- Existing nested-operator lookups can use the index. The companion
selector needs this code change to use it. Other JSON path queries are
unchanged.
- The small local result does not establish end-to-end import speed or
PostgreSQL 17 hosted latency. Fresh deployed measurement remains
necessary.

## Model Used

OpenAI Codex, GPT-6 family. The exact runtime model ID and context
window are not exposed in this session. Used reasoning, code editing,
shell tools, and local PostgreSQL tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-11-09 16:17:13 -06:00
1 parent 6583ffa3b4
commit facd409073
8 files changed
+376 -8

No files matched your search

+1
View File
@@ -615,6 +615,7 @@ const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([
"0436_legal_reavers.sql",
"0447_legacy_controller_process.sql",
"0450_petite_frightful_four.sql",
"0453_curious_ben_parker.sql",
"0384_funny_adam_warlock.sql",
@@ -0,0 +1,63 @@
import { createHash, randomUUID } from "node:crypto";
import { readFile } from "node:fs/promises";
import postgres from "postgres";
import { describe, expect, it } from "vitest";
import { applyPendingMigrations } from "./client.js";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./test-embedded-postgres.js";
const support = await getEmbeddedPostgresTestSupport();
const indexName = "heartbeat_runs_company_native_workspace_idx";
const migrationFile = "0453_curious_ben_parker.sql";
(support.supported ? describe : describe.skip)("workspace companion expression index", () => {
it("builds outside a transaction and preserves ordered prepared/finalized references", async () => {
const database = await startEmbeddedPostgresTestDatabase("workspace-companion-index-");
const sql = postgres(database.connectionString, { max: 1, onnotice: () => {} });
try {
const migration = await readFile(new URL(`./migrations/${migrationFile}`, import.meta.url), "utf8");
const hash = createHash("sha256").update(migration).digest("hex");
await sql`DELETE FROM drizzle.__drizzle_migrations WHERE hash=${hash}`;
await sql.unsafe(`DROP INDEX ${indexName}`);
const companies = await sql`INSERT INTO companies(name,issue_prefix) VALUES ('Index fixture','CIX1'),('Foreign fixture','CIX2') RETURNING id`;
const company = companies[0]!.id, foreignCompany = companies[1]!.id;
const [agent] = await sql`INSERT INTO agents(company_id,name) VALUES (${company},'Local fixture') RETURNING id`;
const [foreignAgent] = await sql`INSERT INTO agents(company_id,name) VALUES (${foreignCompany},'Foreign fixture') RETURNING id`;
const issues = await sql`INSERT INTO issues(company_id,title) VALUES (${company},'Source issue'),(${company},'Other source issue') RETURNING id`;
const workspace = randomUUID(), expected: string[] = [];
for (const [state, issue] of [["prepared", issues[0]!.id], ["finalized", issues[1]!.id], ["prepared", null], ["finalized", null]] as const) {
const id = randomUUID(); expected.push(id);
await sql`INSERT INTO heartbeat_runs(id,company_id,agent_id,native_issue_id,runner_profile_json)
VALUES (${id},${company},${agent!.id},${issue},${sql.json({ nativeWorkspaceSync: { workspaceId: workspace, state } })})`;
}
await sql`INSERT INTO heartbeat_runs(company_id,agent_id,runner_profile_json)
VALUES (${foreignCompany},${foreignAgent!.id},${sql.json({ nativeWorkspaceSync: { workspaceId: workspace, state: "prepared" } })})`;
for (const sync of [{ workspaceId: { malformed: true }, state: "prepared" }, { workspaceId: workspace.toUpperCase(), state: "prepared" }, { workspaceId: workspace, state: "other" }, { workspaceId: workspace }, "malformed", null]) {
await sql`INSERT INTO heartbeat_runs(company_id,agent_id,runner_profile_json)
VALUES (${company},${agent!.id},${sql.json({ nativeWorkspaceSync: sync })})`;
}
const before = await sql`SELECT id,native_issue_id,runner_profile_json#>>'{nativeWorkspaceSync,workspaceId}' workspace_id,
runner_profile_json#>>'{nativeWorkspaceSync,state}' state FROM heartbeat_runs
WHERE company_id=${company} AND runner_profile_json#>>'{nativeWorkspaceSync,workspaceId}'=${workspace}
AND runner_profile_json#>>'{nativeWorkspaceSync,state}' IN ('prepared','finalized') ORDER BY id`;
expect(before.map(row => row.id)).toEqual(expected.sort());
await applyPendingMigrations(database.connectionString);
const after = await sql`SELECT id,native_issue_id,runner_profile_json->'nativeWorkspaceSync'->>'workspaceId' workspace_id,
runner_profile_json->'nativeWorkspaceSync'->>'state' state FROM heartbeat_runs
WHERE company_id=${company} AND runner_profile_json->'nativeWorkspaceSync'->>'workspaceId'=${workspace}
AND runner_profile_json->'nativeWorkspaceSync'->>'state' IN ('prepared','finalized') ORDER BY id`;
expect(after).toEqual(before);
const [index] = await sql`SELECT indisvalid,indisready,pg_get_expr(indpred,indrelid) predicate,pg_get_indexdef(indexrelid) definition
FROM pg_index WHERE indexrelid=${indexName}::regclass`;
expect(index).toMatchObject({ indisvalid: true, indisready: true, predicate: null });
expect(index!.definition).toContain("company_id");
expect(index!.definition).toContain("nativeWorkspaceSync");
expect(index!.definition).toContain("workspaceId");
await applyPendingMigrations(database.connectionString);
const [history] = await sql`SELECT count(*)::int count FROM drizzle.__drizzle_migrations WHERE hash=${hash}`;
expect(history!.count).toBe(1);
} finally {
await sql.end();
await database.cleanup();
}
}, 90_000);
});
@@ -0,0 +1 @@
CREATE INDEX CONCURRENTLY IF NOT EXISTS "heartbeat_runs_company_native_workspace_idx" ON "heartbeat_runs" USING btree ("company_id",("runner_profile_json"->'nativeWorkspaceSync'->>'workspaceId'));
@@ -1,6 +1,6 @@
{
"id": "360fef2e-1ac3-4ae8-92a2-245e544b61da",
"prevId": "76f06e29-53a8-4ecc-809a-d52b6bbb6520",
"id": "d54bfeb4-dedb-4f8b-9195-a4d4aa67d71f",
"prevId": "64e8be1e-e0cb-4cd8-85e0-0fdcce5eafde",
"version": "7",
"dialect": "postgresql",
"tables": {
@@ -4102,6 +4102,28 @@
"method": "btree",
"with": {}
},
"agent_wakeup_requests_issue_blocker_stalled_uq": {
"name": "agent_wakeup_requests_issue_blocker_stalled_uq",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "idempotency_key",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"where": "\"agent_wakeup_requests\".\"idempotency_key\" LIKE 'issue_blocker_stalled:%' AND \"agent_wakeup_requests\".\"status\" IN ('queued', 'deferred_issue_execution', 'claimed', 'coalesced', 'completed')",
"concurrently": true,
"method": "btree",
"with": {}
},
"agent_wakeup_requests_company_payload_issue_idx": {
"name": "agent_wakeup_requests_company_payload_issue_idx",
"columns": [
@@ -7860,6 +7882,150 @@
},
"isRLSEnabled": false
},
"public.budget_resume_claims": {
"name": "budget_resume_claims",
"schema": "",
"columns": {
"run_id": {
"name": "run_id",
"type": "uuid",
"primaryKey": true,
"notNull": true
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"incident_id": {
"name": "incident_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"actor_user_id": {
"name": "actor_user_id",
"type": "text",
"primaryKey": false,
"notNull": true
},
"actor_source": {
"name": "actor_source",
"type": "text",
"primaryKey": false,
"notNull": false
},
"state": {
"name": "state",
"type": "text",
"primaryKey": false,
"notNull": true,
"default": "'pending'"
},
"claimed_at": {
"name": "claimed_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"settled_at": {
"name": "settled_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"reason": {
"name": "reason",
"type": "text",
"primaryKey": false,
"notNull": false
},
"request_id": {
"name": "request_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"budget_resume_claims_pending_idx": {
"name": "budget_resume_claims_pending_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "run_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"budget_resume_claims\".\"state\" = 'pending'",
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"budget_resume_claims_run_id_heartbeat_runs_id_fk": {
"name": "budget_resume_claims_run_id_heartbeat_runs_id_fk",
"tableFrom": "budget_resume_claims",
"tableTo": "heartbeat_runs",
"columnsFrom": [
"run_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"budget_resume_claims_company_id_companies_id_fk": {
"name": "budget_resume_claims_company_id_companies_id_fk",
"tableFrom": "budget_resume_claims",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"budget_resume_claims_incident_id_budget_incidents_id_fk": {
"name": "budget_resume_claims_incident_id_budget_incidents_id_fk",
"tableFrom": "budget_resume_claims",
"tableTo": "budget_incidents",
"columnsFrom": [
"incident_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.built_in_managed_resources": {
"name": "built_in_managed_resources",
"schema": "",
@@ -17233,7 +17399,7 @@
"columnsTo": [
"id"
],
"onDelete": "set null",
"onDelete": "cascade",
"onUpdate": "no action"
},
"company_secret_proposals_secret_proposal_id_company_secret_proposals_id_fk": {
@@ -21977,6 +22143,112 @@
"checkConstraints": {},
"isRLSEnabled": false
},
"public.controller_instances": {
"name": "controller_instances",
"schema": "",
"columns": {
"boot_id": {
"name": "boot_id",
"type": "text",
"primaryKey": true,
"notNull": true
},
"epoch": {
"name": "epoch",
"type": "bigserial",
"primaryKey": false,
"notNull": true
},
"process_space": {
"name": "process_space",
"type": "text",
"primaryKey": false,
"notNull": true
},
"pid": {
"name": "pid",
"type": "integer",
"primaryKey": false,
"notNull": true
},
"started_at": {
"name": "started_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"renewed_at": {
"name": "renewed_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"lease_expires_at": {
"name": "lease_expires_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true
},
"fenced_at": {
"name": "fenced_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"fenced_by_epoch": {
"name": "fenced_by_epoch",
"type": "bigint",
"primaryKey": false,
"notNull": false
},
"stopped_at": {
"name": "stopped_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
}
},
"indexes": {
"controller_instances_epoch_idx": {
"name": "controller_instances_epoch_idx",
"columns": [
{
"expression": "epoch",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
},
"controller_instances_lease_idx": {
"name": "controller_instances_lease_idx",
"columns": [
{
"expression": "lease_expires_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.cost_events": {
"name": "cost_events",
"schema": "",
@@ -35290,6 +35562,27 @@
"method": "btree",
"with": {}
},
"heartbeat_runs_company_native_workspace_idx": {
"name": "heartbeat_runs_company_native_workspace_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "(\"runner_profile_json\"->'nativeWorkspaceSync'->>'workspaceId')",
"asc": true,
"isExpression": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": true,
"method": "btree",
"with": {}
},
"heartbeat_runs_company_issue_created_idx": {
"name": "heartbeat_runs_company_issue_created_idx",
"columns": [
+7
View File
@@ -3158,6 +3158,13 @@
"when": 1794205579544,
"tag": "0452_secret_proposal_secret_cascade",
"breakpoints": true
},
{
"idx": 453,
"version": "7",
"when": 1794260223713,
"tag": "0453_curious_ben_parker",
"breakpoints": true
}
]
}
+3
View File
@@ -190,6 +190,9 @@ export const heartbeatRuns = pgTable(
table.agentId,
table.startedAt,
),
companyNativeWorkspaceIdx: index("heartbeat_runs_company_native_workspace_idx")
.on(table.companyId, sql`(${table.runnerProfileJson}->'nativeWorkspaceSync'->>'workspaceId')`)
.concurrently(),
companyIssueCreatedIdx: index("heartbeat_runs_company_issue_created_idx").on(
table.companyId,
table.issueId,
@@ -202,11 +202,11 @@ export async function inspectOnlineWorkspaceSource(db: ReadDb, workspace: Worksp
if (recipe && (!recipe.restoredAt || recipe.contents && !recipe.contentsRestoredAt
|| recipe.restorePoint && !recipe.restorePoint.expiredAt && !recipe.restorePoint.appliedAt)) return null;
const runs = await db.select({ id: heartbeatRuns.id, issueId: heartbeatRuns.nativeIssueId,
workspaceId: sql<string>`${heartbeatRuns.runnerProfileJson}#>>'{nativeWorkspaceSync,workspaceId}'`,
state: sql<string>`${heartbeatRuns.runnerProfileJson}#>>'{nativeWorkspaceSync,state}'` }).from(heartbeatRuns).where(and(
workspaceId: sql<string>`${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync'->>'workspaceId'`,
state: sql<string>`${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync'->>'state'` }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, workspace.companyId),
inArray(sql<string>`${heartbeatRuns.runnerProfileJson}#>>'{nativeWorkspaceSync,workspaceId}'`, included.map(node => node.row.id)),
sql`${heartbeatRuns.runnerProfileJson}#>>'{nativeWorkspaceSync,state}' in ('prepared','finalized')`,
inArray(sql<string>`${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync'->>'workspaceId'`, included.map(node => node.row.id)),
sql`${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync'->>'state' in ('prepared','finalized')`,
)).orderBy(heartbeatRuns.id);
const runsByWorkspace = new Map<string, CompanionRun[]>();
for (const run of runs) {
@@ -797,7 +797,7 @@ describe.skipIf(!support.supported)("online ordinary physical source fence", ()
const selected = await stored(f.child.id), debug = db.$client.options.debug;
const scans: string[] = [];
db.$client.options.debug = (_connection, query) => {
if (query.includes('from "heartbeat_runs"') && query.includes("nativeWorkspaceSync,workspaceId")) scans.push(query);
if (query.includes('from "heartbeat_runs"') && query.includes("->'nativeWorkspaceSync'->>'workspaceId'")) scans.push(query);
};
try {
const scope = await inspectOnlineWorkspaceSource(db, selected);