Files
PaperClipAI/packages/db/src/table-size-estimates.ts
T
Nicky LeachandPaperclip 22001bbd2f feat(db): add migration safety lint
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The `packages/db` module owns all database migrations via a
sequential numbering system already validated at build time
> - A recent migration introduced an O(n²) batch backfill over a large,
unindexed table — it caused the server's `listen` to block for ~5
minutes on databases with millions of rows
> - Nothing in the current CI pipeline catches large-table migration
risk patterns (DO-loop mutations, batched LIMIT mutations without
support indexes, full-table mutations, non-concurrent index creation)
before they land
> - This PR wires a new static migration-safety checker
(`check-migration-safety.ts`) into the existing `check:migrations` gate
in `packages/db/package.json`, so risky patterns fail CI before reaching
production
> - The checker baselines all historical findings already present in the
codebase, so the gate fails only on *new* unbaselined risky patterns
> - The benefit is that the specific O(n²) backfill shape (and related
patterns) will be caught at author time rather than at incident time

## Linked Issues or Issue Description

**Feature — static migration safety lint**

**Problem or motivation**

Migrations against large tables (millions of rows) have caused
production startup blocks. The root pattern is a batched `LIMIT`-based
backfill iterating via an unindexed column, making each batch a
sequential scan — O(n²) overall. No CI gate exists to flag this class of
problem before merge.

**Proposed solution**

A static SQL-level checker that scans new migration files for known
dangerous patterns against known-large tables, producing structured
findings that are either baselined (suppressed) or fail the build.
Patterns detected: `DO $$ loop` mutations on large tables without a
same-migration support index, batched `LIMIT` mutations on large tables
missing a same-migration support index, unbounded full-table mutations
(no `WHERE` clause), and `CREATE INDEX` without `CONCURRENTLY` on large
tables.

**Alternatives considered**

Runtime instrumentation (only catches issues in production), advisory
locking in migrations (doesn't prevent the pattern), per-migration code
review (doesn't scale consistently).

**Roadmap alignment**

Defensive infrastructure / operational reliability — keeps migrations
from blocking production startups. Not a user-facing feature.

## What Changed

- **`packages/db/package.json`** — extended `check:migrations` script to
run `check-migration-safety.ts` after the existing numbering check
- **`packages/db/src/check-migration-safety.ts`** — new static checker:
SQL pattern matching, rule detection for four dangerous patterns,
baseline diffing, and structured exit with findings summary
- **`packages/db/src/migration-safety-baseline.ts`** — baseline of all
existing historical findings (suppressed from failing the gate); new
migrations matching these patterns without a baseline entry will fail
- **`packages/db/src/table-size-estimates.ts`** — rough table size
estimates from the local dev database; drives `isKnownLargeTable()` used
by the safety rules
- **`packages/db/src/check-migration-safety.test.ts`** — Vitest coverage
for the O(n²) backfill failure mode, suppression via baseline, and each
rule type

## Verification

```bash
# Run the migration safety checker directly
cd packages/db
tsx src/check-migration-safety.ts

# Run tests
cd packages/db
npx vitest run src/check-migration-safety.test.ts

# Run the full migration check gate (numbering + safety)
cd packages/db
pnpm run check:migrations
```

- Tests cover the core O(n²) backfill pattern (the motivating incident),
baseline suppression, and all four rule types
- `check:migrations` now exits non-zero for any new unbaselined
large-table migration risk pattern

## Risks

- **False positives:** Table-size estimates are from a local dev
database snapshot — a table small in dev but large in production would
be missed. Best-effort heuristic.
- **Baseline drift:** If a baselined finding's SQL changes
significantly, the baseline ID (content-hash-based) will no longer match
and the finding will re-surface. Intentional but may surprise authors
doing incremental fixes.
- **SQL parsing limitations:** Regex-based pattern matching rather than
a full AST parser — complex SQL may not be detected. Acceptable for an
initial gate.
- **Low risk to existing behavior:** The gate only fails on *new*
findings not present in the baseline. All existing migrations are
baselined.

## Model Used

- **Provider:** Anthropic
- **Model ID:** `claude-sonnet-4-6`
- **Context window:** 200K
- **Tool use:** yes (file reading, bash, git operations)
- **Reasoning mode:** standard (no extended thinking)

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-07-07 01:43:12 +00:00

179 lines
7.5 KiB
TypeScript

export type TableSizeBucket = "large" | "medium" | "small";
export type LocalTableRowCount = {
readonly table: string;
readonly localRows: number;
};
export type TableSizeEstimate = LocalTableRowCount & {
readonly estimateFactor: number;
readonly estimatedRows: number;
readonly bucket: TableSizeBucket;
};
export const TABLE_SIZE_ESTIMATE_SOURCE = {
collectedAt: "2026-07-06",
database: "default local dev embedded Postgres",
method: "SELECT count(*) in a read-only transaction",
} as const;
export const TABLE_SIZE_ESTIMATE_FACTOR = 250;
export const TABLE_SIZE_BUCKET_THRESHOLDS = {
largeRows: 1_000_000,
mediumRows: 100_000,
} as const;
export const LOCAL_TABLE_ROW_COUNTS = [
{ table: "agent_wakeup_requests", localRows: 52_791 },
{ table: "activity_log", localRows: 22_930 },
{ table: "issue_reference_mentions", localRows: 13_218 },
{ table: "heartbeat_run_events", localRows: 10_833 },
{ table: "issue_comments", localRows: 5_034 },
{ table: "document_revisions", localRows: 3_906 },
{ table: "heartbeat_runs", localRows: 3_620 },
{ table: "workspace_operations", localRows: 3_600 },
{ table: "environment_leases", localRows: 3_492 },
{ table: "cost_events", localRows: 3_330 },
{ table: "agent_task_sessions", localRows: 1_743 },
{ table: "documents", localRows: 1_692 },
{ table: "issue_documents", localRows: 1_678 },
{ table: "issues", localRows: 1_609 },
{ table: "secret_access_events", localRows: 933 },
{ table: "issue_read_states", localRows: 608 },
{ table: "issue_thread_interactions", localRows: 455 },
{ table: "issue_relations", localRows: 445 },
{ table: "execution_workspaces", localRows: 414 },
{ table: "agent_config_revisions", localRows: 145 },
{ table: "document_annotation_anchor_snapshots", localRows: 132 },
{ table: "document_annotation_comments", localRows: 122 },
{ table: "document_annotation_threads", localRows: 80 },
{ table: "routine_revisions", localRows: 64 },
{ table: "issue_tree_hold_members", localRows: 62 },
{ table: "routine_runs", localRows: 52 },
{ table: "issue_recovery_actions", localRows: 50 },
{ table: "assets", localRows: 46 },
{ table: "issue_attachments", localRows: 44 },
{ table: "principal_permission_grants", localRows: 25 },
{ table: "company_memberships", localRows: 19 },
{ table: "agent_runtime_state", localRows: 18 },
{ table: "agents", localRows: 18 },
{ table: "routine_triggers", localRows: 16 },
{ table: "company_skills", localRows: 15 },
{ table: "routine_documents", localRows: 14 },
{ table: "routines", localRows: 14 },
{ table: "issue_inbox_archives", localRows: 13 },
{ table: "issue_approvals", localRows: 12 },
{ table: "company_secret_bindings", localRows: 11 },
{ table: "issue_tree_holds", localRows: 10 },
{ table: "approvals", localRows: 9 },
{ table: "session", localRows: 9 },
{ table: "issue_work_products", localRows: 7 },
{ table: "company_skill_versions", localRows: 6 },
{ table: "feedback_exports", localRows: 6 },
{ table: "feedback_votes", localRows: 6 },
{ table: "issue_execution_decisions", localRows: 6 },
{ table: "company_secret_versions", localRows: 3 },
{ table: "company_secrets", localRows: 3 },
{ table: "projects", localRows: 3 },
{ table: "approval_comments", localRows: 2 },
{ table: "goals", localRows: 2 },
{ table: "project_goals", localRows: 2 },
{ table: "project_workspaces", localRows: 2 },
{ table: "account", localRows: 1 },
{ table: "companies", localRows: 1 },
{ table: "environments", localRows: 1 },
{ table: "instance_settings", localRows: 1 },
{ table: "instance_user_roles", localRows: 1 },
{ table: "user", localRows: 1 },
{ table: "agent_api_keys", localRows: 0 },
{ table: "agent_memberships", localRows: 0 },
{ table: "board_api_keys", localRows: 0 },
{ table: "budget_incidents", localRows: 0 },
{ table: "budget_policies", localRows: 0 },
{ table: "cli_auth_challenges", localRows: 0 },
{ table: "cloud_upstream_connections", localRows: 0 },
{ table: "cloud_upstream_runs", localRows: 0 },
{ table: "company_logos", localRows: 0 },
{ table: "company_secret_provider_configs", localRows: 0 },
{ table: "company_skill_comments", localRows: 0 },
{ table: "company_skill_stars", localRows: 0 },
{ table: "company_user_sidebar_preferences", localRows: 0 },
{ table: "environment_custom_image_setup_sessions", localRows: 0 },
{ table: "environment_custom_image_templates", localRows: 0 },
{ table: "external_object_mentions", localRows: 0 },
{ table: "external_objects", localRows: 0 },
{ table: "finance_events", localRows: 0 },
{ table: "heartbeat_run_watchdog_decisions", localRows: 0 },
{ table: "inbox_dismissals", localRows: 0 },
{ table: "invites", localRows: 0 },
{ table: "issue_labels", localRows: 0 },
{ table: "issue_plan_decompositions", localRows: 0 },
{ table: "issue_watchdogs", localRows: 0 },
{ table: "join_requests", localRows: 0 },
{ table: "labels", localRows: 0 },
{ table: "pipeline_automation_executions", localRows: 0 },
{ table: "pipeline_case_blockers", localRows: 0 },
{ table: "pipeline_case_documents", localRows: 0 },
{ table: "pipeline_case_events", localRows: 0 },
{ table: "pipeline_case_issue_links", localRows: 0 },
{ table: "pipeline_cases", localRows: 0 },
{ table: "pipeline_documents", localRows: 0 },
{ table: "pipeline_stages", localRows: 0 },
{ table: "pipeline_transitions", localRows: 0 },
{ table: "pipelines", localRows: 0 },
{ table: "plugin_company_settings", localRows: 0 },
{ table: "plugin_config", localRows: 0 },
{ table: "plugin_database_namespaces", localRows: 0 },
{ table: "plugin_entities", localRows: 0 },
{ table: "plugin_job_runs", localRows: 0 },
{ table: "plugin_jobs", localRows: 0 },
{ table: "plugin_logs", localRows: 0 },
{ table: "plugin_managed_resources", localRows: 0 },
{ table: "plugin_migrations", localRows: 0 },
{ table: "plugin_state", localRows: 0 },
{ table: "plugin_webhook_deliveries", localRows: 0 },
{ table: "plugins", localRows: 0 },
{ table: "project_memberships", localRows: 0 },
{ table: "user_secret_declarations", localRows: 0 },
{ table: "user_secret_definitions", localRows: 0 },
{ table: "user_sidebar_preferences", localRows: 0 },
{ table: "verification", localRows: 0 },
{ table: "workspace_runtime_services", localRows: 0 },
] as const satisfies readonly LocalTableRowCount[];
function bucketForEstimatedRows(estimatedRows: number): TableSizeBucket {
if (estimatedRows >= TABLE_SIZE_BUCKET_THRESHOLDS.largeRows) return "large";
if (estimatedRows >= TABLE_SIZE_BUCKET_THRESHOLDS.mediumRows) return "medium";
return "small";
}
export const TABLE_SIZE_ESTIMATES: readonly TableSizeEstimate[] = LOCAL_TABLE_ROW_COUNTS.map(
({ table, localRows }) => {
const estimatedRows = localRows * TABLE_SIZE_ESTIMATE_FACTOR;
return {
table,
localRows,
estimateFactor: TABLE_SIZE_ESTIMATE_FACTOR,
estimatedRows,
bucket: bucketForEstimatedRows(estimatedRows),
};
},
);
export const TABLE_SIZE_ESTIMATES_BY_TABLE: ReadonlyMap<string, TableSizeEstimate> = new Map(
TABLE_SIZE_ESTIMATES.map((estimate) => [estimate.table, estimate]),
);
export function getTableSizeEstimate(table: string): TableSizeEstimate | undefined {
return TABLE_SIZE_ESTIMATES_BY_TABLE.get(table);
}
export function getTableSizeBucket(table: string): TableSizeBucket {
return getTableSizeEstimate(table)?.bucket ?? "small";
}
export function isKnownLargeTable(table: string): boolean {
return getTableSizeBucket(table) === "large";
}