mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-27 11:20:42 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The `packages/db` module owns all database migrations via a sequential numbering system already validated at build time > - A recent migration introduced an O(n²) batch backfill over a large, unindexed table — it caused the server's `listen` to block for ~5 minutes on databases with millions of rows > - Nothing in the current CI pipeline catches large-table migration risk patterns (DO-loop mutations, batched LIMIT mutations without support indexes, full-table mutations, non-concurrent index creation) before they land > - This PR wires a new static migration-safety checker (`check-migration-safety.ts`) into the existing `check:migrations` gate in `packages/db/package.json`, so risky patterns fail CI before reaching production > - The checker baselines all historical findings already present in the codebase, so the gate fails only on *new* unbaselined risky patterns > - The benefit is that the specific O(n²) backfill shape (and related patterns) will be caught at author time rather than at incident time ## Linked Issues or Issue Description **Feature — static migration safety lint** **Problem or motivation** Migrations against large tables (millions of rows) have caused production startup blocks. The root pattern is a batched `LIMIT`-based backfill iterating via an unindexed column, making each batch a sequential scan — O(n²) overall. No CI gate exists to flag this class of problem before merge. **Proposed solution** A static SQL-level checker that scans new migration files for known dangerous patterns against known-large tables, producing structured findings that are either baselined (suppressed) or fail the build. Patterns detected: `DO $$ loop` mutations on large tables without a same-migration support index, batched `LIMIT` mutations on large tables missing a same-migration support index, unbounded full-table mutations (no `WHERE` clause), and `CREATE INDEX` without `CONCURRENTLY` on large tables. **Alternatives considered** Runtime instrumentation (only catches issues in production), advisory locking in migrations (doesn't prevent the pattern), per-migration code review (doesn't scale consistently). **Roadmap alignment** Defensive infrastructure / operational reliability — keeps migrations from blocking production startups. Not a user-facing feature. ## What Changed - **`packages/db/package.json`** — extended `check:migrations` script to run `check-migration-safety.ts` after the existing numbering check - **`packages/db/src/check-migration-safety.ts`** — new static checker: SQL pattern matching, rule detection for four dangerous patterns, baseline diffing, and structured exit with findings summary - **`packages/db/src/migration-safety-baseline.ts`** — baseline of all existing historical findings (suppressed from failing the gate); new migrations matching these patterns without a baseline entry will fail - **`packages/db/src/table-size-estimates.ts`** — rough table size estimates from the local dev database; drives `isKnownLargeTable()` used by the safety rules - **`packages/db/src/check-migration-safety.test.ts`** — Vitest coverage for the O(n²) backfill failure mode, suppression via baseline, and each rule type ## Verification ```bash # Run the migration safety checker directly cd packages/db tsx src/check-migration-safety.ts # Run tests cd packages/db npx vitest run src/check-migration-safety.test.ts # Run the full migration check gate (numbering + safety) cd packages/db pnpm run check:migrations ``` - Tests cover the core O(n²) backfill pattern (the motivating incident), baseline suppression, and all four rule types - `check:migrations` now exits non-zero for any new unbaselined large-table migration risk pattern ## Risks - **False positives:** Table-size estimates are from a local dev database snapshot — a table small in dev but large in production would be missed. Best-effort heuristic. - **Baseline drift:** If a baselined finding's SQL changes significantly, the baseline ID (content-hash-based) will no longer match and the finding will re-surface. Intentional but may surprise authors doing incremental fixes. - **SQL parsing limitations:** Regex-based pattern matching rather than a full AST parser — complex SQL may not be detected. Acceptable for an initial gate. - **Low risk to existing behavior:** The gate only fails on *new* findings not present in the baseline. All existing migrations are baselined. ## Model Used - **Provider:** Anthropic - **Model ID:** `claude-sonnet-4-6` - **Context window:** 200K - **Tool use:** yes (file reading, bash, git operations) - **Reasoning mode:** standard (no extended thinking) ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
179 lines
7.5 KiB
TypeScript
179 lines
7.5 KiB
TypeScript
export type TableSizeBucket = "large" | "medium" | "small";
|
|
|
|
export type LocalTableRowCount = {
|
|
readonly table: string;
|
|
readonly localRows: number;
|
|
};
|
|
|
|
export type TableSizeEstimate = LocalTableRowCount & {
|
|
readonly estimateFactor: number;
|
|
readonly estimatedRows: number;
|
|
readonly bucket: TableSizeBucket;
|
|
};
|
|
|
|
export const TABLE_SIZE_ESTIMATE_SOURCE = {
|
|
collectedAt: "2026-07-06",
|
|
database: "default local dev embedded Postgres",
|
|
method: "SELECT count(*) in a read-only transaction",
|
|
} as const;
|
|
|
|
export const TABLE_SIZE_ESTIMATE_FACTOR = 250;
|
|
|
|
export const TABLE_SIZE_BUCKET_THRESHOLDS = {
|
|
largeRows: 1_000_000,
|
|
mediumRows: 100_000,
|
|
} as const;
|
|
|
|
export const LOCAL_TABLE_ROW_COUNTS = [
|
|
{ table: "agent_wakeup_requests", localRows: 52_791 },
|
|
{ table: "activity_log", localRows: 22_930 },
|
|
{ table: "issue_reference_mentions", localRows: 13_218 },
|
|
{ table: "heartbeat_run_events", localRows: 10_833 },
|
|
{ table: "issue_comments", localRows: 5_034 },
|
|
{ table: "document_revisions", localRows: 3_906 },
|
|
{ table: "heartbeat_runs", localRows: 3_620 },
|
|
{ table: "workspace_operations", localRows: 3_600 },
|
|
{ table: "environment_leases", localRows: 3_492 },
|
|
{ table: "cost_events", localRows: 3_330 },
|
|
{ table: "agent_task_sessions", localRows: 1_743 },
|
|
{ table: "documents", localRows: 1_692 },
|
|
{ table: "issue_documents", localRows: 1_678 },
|
|
{ table: "issues", localRows: 1_609 },
|
|
{ table: "secret_access_events", localRows: 933 },
|
|
{ table: "issue_read_states", localRows: 608 },
|
|
{ table: "issue_thread_interactions", localRows: 455 },
|
|
{ table: "issue_relations", localRows: 445 },
|
|
{ table: "execution_workspaces", localRows: 414 },
|
|
{ table: "agent_config_revisions", localRows: 145 },
|
|
{ table: "document_annotation_anchor_snapshots", localRows: 132 },
|
|
{ table: "document_annotation_comments", localRows: 122 },
|
|
{ table: "document_annotation_threads", localRows: 80 },
|
|
{ table: "routine_revisions", localRows: 64 },
|
|
{ table: "issue_tree_hold_members", localRows: 62 },
|
|
{ table: "routine_runs", localRows: 52 },
|
|
{ table: "issue_recovery_actions", localRows: 50 },
|
|
{ table: "assets", localRows: 46 },
|
|
{ table: "issue_attachments", localRows: 44 },
|
|
{ table: "principal_permission_grants", localRows: 25 },
|
|
{ table: "company_memberships", localRows: 19 },
|
|
{ table: "agent_runtime_state", localRows: 18 },
|
|
{ table: "agents", localRows: 18 },
|
|
{ table: "routine_triggers", localRows: 16 },
|
|
{ table: "company_skills", localRows: 15 },
|
|
{ table: "routine_documents", localRows: 14 },
|
|
{ table: "routines", localRows: 14 },
|
|
{ table: "issue_inbox_archives", localRows: 13 },
|
|
{ table: "issue_approvals", localRows: 12 },
|
|
{ table: "company_secret_bindings", localRows: 11 },
|
|
{ table: "issue_tree_holds", localRows: 10 },
|
|
{ table: "approvals", localRows: 9 },
|
|
{ table: "session", localRows: 9 },
|
|
{ table: "issue_work_products", localRows: 7 },
|
|
{ table: "company_skill_versions", localRows: 6 },
|
|
{ table: "feedback_exports", localRows: 6 },
|
|
{ table: "feedback_votes", localRows: 6 },
|
|
{ table: "issue_execution_decisions", localRows: 6 },
|
|
{ table: "company_secret_versions", localRows: 3 },
|
|
{ table: "company_secrets", localRows: 3 },
|
|
{ table: "projects", localRows: 3 },
|
|
{ table: "approval_comments", localRows: 2 },
|
|
{ table: "goals", localRows: 2 },
|
|
{ table: "project_goals", localRows: 2 },
|
|
{ table: "project_workspaces", localRows: 2 },
|
|
{ table: "account", localRows: 1 },
|
|
{ table: "companies", localRows: 1 },
|
|
{ table: "environments", localRows: 1 },
|
|
{ table: "instance_settings", localRows: 1 },
|
|
{ table: "instance_user_roles", localRows: 1 },
|
|
{ table: "user", localRows: 1 },
|
|
{ table: "agent_api_keys", localRows: 0 },
|
|
{ table: "agent_memberships", localRows: 0 },
|
|
{ table: "board_api_keys", localRows: 0 },
|
|
{ table: "budget_incidents", localRows: 0 },
|
|
{ table: "budget_policies", localRows: 0 },
|
|
{ table: "cli_auth_challenges", localRows: 0 },
|
|
{ table: "cloud_upstream_connections", localRows: 0 },
|
|
{ table: "cloud_upstream_runs", localRows: 0 },
|
|
{ table: "company_logos", localRows: 0 },
|
|
{ table: "company_secret_provider_configs", localRows: 0 },
|
|
{ table: "company_skill_comments", localRows: 0 },
|
|
{ table: "company_skill_stars", localRows: 0 },
|
|
{ table: "company_user_sidebar_preferences", localRows: 0 },
|
|
{ table: "environment_custom_image_setup_sessions", localRows: 0 },
|
|
{ table: "environment_custom_image_templates", localRows: 0 },
|
|
{ table: "external_object_mentions", localRows: 0 },
|
|
{ table: "external_objects", localRows: 0 },
|
|
{ table: "finance_events", localRows: 0 },
|
|
{ table: "heartbeat_run_watchdog_decisions", localRows: 0 },
|
|
{ table: "inbox_dismissals", localRows: 0 },
|
|
{ table: "invites", localRows: 0 },
|
|
{ table: "issue_labels", localRows: 0 },
|
|
{ table: "issue_plan_decompositions", localRows: 0 },
|
|
{ table: "issue_watchdogs", localRows: 0 },
|
|
{ table: "join_requests", localRows: 0 },
|
|
{ table: "labels", localRows: 0 },
|
|
{ table: "pipeline_automation_executions", localRows: 0 },
|
|
{ table: "pipeline_case_blockers", localRows: 0 },
|
|
{ table: "pipeline_case_documents", localRows: 0 },
|
|
{ table: "pipeline_case_events", localRows: 0 },
|
|
{ table: "pipeline_case_issue_links", localRows: 0 },
|
|
{ table: "pipeline_cases", localRows: 0 },
|
|
{ table: "pipeline_documents", localRows: 0 },
|
|
{ table: "pipeline_stages", localRows: 0 },
|
|
{ table: "pipeline_transitions", localRows: 0 },
|
|
{ table: "pipelines", localRows: 0 },
|
|
{ table: "plugin_company_settings", localRows: 0 },
|
|
{ table: "plugin_config", localRows: 0 },
|
|
{ table: "plugin_database_namespaces", localRows: 0 },
|
|
{ table: "plugin_entities", localRows: 0 },
|
|
{ table: "plugin_job_runs", localRows: 0 },
|
|
{ table: "plugin_jobs", localRows: 0 },
|
|
{ table: "plugin_logs", localRows: 0 },
|
|
{ table: "plugin_managed_resources", localRows: 0 },
|
|
{ table: "plugin_migrations", localRows: 0 },
|
|
{ table: "plugin_state", localRows: 0 },
|
|
{ table: "plugin_webhook_deliveries", localRows: 0 },
|
|
{ table: "plugins", localRows: 0 },
|
|
{ table: "project_memberships", localRows: 0 },
|
|
{ table: "user_secret_declarations", localRows: 0 },
|
|
{ table: "user_secret_definitions", localRows: 0 },
|
|
{ table: "user_sidebar_preferences", localRows: 0 },
|
|
{ table: "verification", localRows: 0 },
|
|
{ table: "workspace_runtime_services", localRows: 0 },
|
|
] as const satisfies readonly LocalTableRowCount[];
|
|
|
|
function bucketForEstimatedRows(estimatedRows: number): TableSizeBucket {
|
|
if (estimatedRows >= TABLE_SIZE_BUCKET_THRESHOLDS.largeRows) return "large";
|
|
if (estimatedRows >= TABLE_SIZE_BUCKET_THRESHOLDS.mediumRows) return "medium";
|
|
return "small";
|
|
}
|
|
|
|
export const TABLE_SIZE_ESTIMATES: readonly TableSizeEstimate[] = LOCAL_TABLE_ROW_COUNTS.map(
|
|
({ table, localRows }) => {
|
|
const estimatedRows = localRows * TABLE_SIZE_ESTIMATE_FACTOR;
|
|
return {
|
|
table,
|
|
localRows,
|
|
estimateFactor: TABLE_SIZE_ESTIMATE_FACTOR,
|
|
estimatedRows,
|
|
bucket: bucketForEstimatedRows(estimatedRows),
|
|
};
|
|
},
|
|
);
|
|
|
|
export const TABLE_SIZE_ESTIMATES_BY_TABLE: ReadonlyMap<string, TableSizeEstimate> = new Map(
|
|
TABLE_SIZE_ESTIMATES.map((estimate) => [estimate.table, estimate]),
|
|
);
|
|
|
|
export function getTableSizeEstimate(table: string): TableSizeEstimate | undefined {
|
|
return TABLE_SIZE_ESTIMATES_BY_TABLE.get(table);
|
|
}
|
|
|
|
export function getTableSizeBucket(table: string): TableSizeBucket {
|
|
return getTableSizeEstimate(table)?.bucket ?? "small";
|
|
}
|
|
|
|
export function isKnownLargeTable(table: string): boolean {
|
|
return getTableSizeBucket(table) === "large";
|
|
}
|