From 739f9e40a461833560a5ecd6b711ae3da2295e7a Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 14 Oct 2026 11:03:16 -0500 Subject: [PATCH] docs(skills): add slow-query diagnosis and fix skill (#16303) ## Thinking Path > - Paperclip helps people manage agents that work across tasks and services. > - Slow pages and background work can come from expensive or repeated database queries. > - The query often starts in the Paperclip app while its evidence lives in Paperclip Cloud, Neon, and application logs. > - A useful fix must reproduce the slow SQL locally and preserve its complete results and user behavior. > - The procedure also needs to measure any exceptional migration on a database of realistic size. > - This pull request gives contributors a repeatable skill for that work and for carrying the fix through PR checks. > - The benefit is a clear, measured handoff with fewer query regressions and fewer risky migration proposals. ## Linked Issues or Issue Description **Issue type** Missing documentation. **Where is the issue?** `.agents/skills/` has no shared procedure for diagnosing and fixing slow Paperclip queries across Cloud and Neon. **What's wrong?** A contributor must reconstruct the path from a slow workspace to its database, capture the service's actual SQL, reproduce it locally, compare complete results, and track the PR through review and CI. That makes it easy to mistake cumulative statistics for a recent request trace or a faster but changed result for a valid fix. **Suggested fix** Add a project skill that covers discovery, baseline and treatment benchmarks, regression checks, migration constraints, and PR follow-through. Related work: #16278, #16286, and #16291 supplied the query cases and local benchmark lessons. No existing slow-query skill PR or issue appeared in the duplicate search. ## What Changed - Add `fix-slow-queries` with a bounded workflow from Cloud discovery to a reviewed PR. - Add references for locating the right Neon database and for measuring full service SQL against complete output comparisons on disposable PostgreSQL. - Require a serious reason for migrations, prohibit boot-time migration workarounds, and require production-size timing and escalation for slow migrations. ## Verification - Run the skill-creator `quick_validate.py` against `.agents/skills/fix-slow-queries`: passed. - Run `pnpm --filter @paperclipai/skills-catalog test`: 5 files, 20 tests passed, including the repo-wide skill-description budget check. - Check all five local Markdown links, metadata length, final newlines, and whitespace: passed. - Run `git diff --check`: passed. - Review the documented procedure against the cases in #16278, #16286, and #16291: baseline helper isolation, large JSON, older deep links, and latest-commit PR checks are covered. - This change adds instructions only. It changes no database schema or runtime behavior. ## Risks - Provider menus and private Cloud file locations may change. The skill requires contributors to verify the current stack mapping and owning repository before using them. - Synthetic benchmarks do not establish deployed latency. The skill requires a fresh deployed measurement when rollout is authorized. ## Model Used OpenAI GPT-6 via Codex, with tool use and code execution. The served model variant and context window were not exposed to this task. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .agents/skills/fix-slow-queries/SKILL.md | 189 ++++++++++++++++++ .../fix-slow-queries/agents/openai.yaml | 3 + .../references/cloud-discovery.md | 119 +++++++++++ .../references/local-proof.md | 183 +++++++++++++++++ 4 files changed, 494 insertions(+) create mode 100644 .agents/skills/fix-slow-queries/SKILL.md create mode 100644 .agents/skills/fix-slow-queries/agents/openai.yaml create mode 100644 .agents/skills/fix-slow-queries/references/cloud-discovery.md create mode 100644 .agents/skills/fix-slow-queries/references/local-proof.md diff --git a/.agents/skills/fix-slow-queries/SKILL.md b/.agents/skills/fix-slow-queries/SKILL.md new file mode 100644 index 00000000000..b268fd36471 --- /dev/null +++ b/.agents/skills/fix-slow-queries/SKILL.md @@ -0,0 +1,189 @@ +--- +name: fix-slow-queries +description: >- + Diagnose and fix slow Paperclip PostgreSQL queries using Cloud and Neon evidence, + local reproduction, performance and result comparisons, regression tests, and + a PR with passing checks. Use for costly queries, repeated database work, or + slow requests with a suspected database cause. +--- + +# Fix Slow Queries + +Deliver a measured improvement with preserved behavior and a reviewable PR whose +latest commit passes checks. A list of suspicious SQL or an unverified rewrite +is an intermediate result. Respect a narrower request for diagnosis only. + +## 1. Establish the target and finish line + +- Resolve the reported URL, environment, company/workspace, deployed revision, + owning repository, and local checkout. Read its `AGENTS.md` and scoped rules. + Paperclip and the private `paperclip-cloud` control plane have different rules. +- Work in a dedicated branch/worktree. Reuse suitable existing work; keep the + checkout running a live server out of branch switching and benchmark setup. +- Select a bounded batch, usually the top two or three query families. Record + deferred candidates so discovery does not turn into an unbounded rewrite. +- The finish line is: reproducible baseline, faster treatment, output/behavior + proof, relevant regression coverage, and green checks on each final PR head. + A concrete access or infrastructure blocker must be reported as incomplete. + +## 2. Find the actual database and rank the work + +Read [Cloud discovery](references/cloud-discovery.md) when starting from a hosted +workspace. Follow the chain from Paperclip Cloud fleet admin to the stack's Neon +project, branch, endpoint, database, and runtime role. Record the corresponding +application deployment and time window. Do not reuse IDs from a previous run. + +Capture query text/fingerprint, calls, total execution time, mean time, rows, +and available I/O, temporary-file, and lock-wait evidence. Rank both expensive +individual calls and frequent work with high aggregate cost. Group variants by +their actual caller; do not add unrelated query means into an endpoint latency. + +Keep three claims separate: + +| Claim | Evidence needed | +| --- | --- | +| SQL is expensive | Query statistics and a reproducible execution plan | +| The database path dominates a slow request | Correlated SQL, connection wait, and application timings | +| More database capacity would help | Resource/queue pressure in the same window and a capacity-specific experiment | + +Cumulative query statistics are not a recent-window trace. Capture reset times +or two snapshots and calculate deltas. A large database-time total alone does +not justify resizing Neon. Application CPU headroom also does not rule out a +busy Node main thread. + +## 3. Trace each candidate to its behavior + +Use `rg` on distinctive SQL fragments, table names, JSON keys, and predicates. +Trace the full path: route or scheduler → service → generated SQL → indexes. +Record frequency, fan-out, page limits, redaction, authorization, and any second +query the caller performs. Compare the deployed implementation with local code. + +State a falsifiable cause and the contract the fix must preserve. Typical causes +include casts on indexed columns, repeated detoasting of large JSON, broad +candidate scans, unbounded result sets, duplicate reads, and unnecessary polling. +Use plans and caller evidence to choose among them. + +Inspect existing indexes and recent related fixes before adding another index. +For advisory locks, identify the caller, key, transaction duration, and wait +chain first. Preserve checkout, leases, ownership, recovery, and no-replay rules; +removing a lock or polling loop is not a query optimization by itself. + +## 4. Reproduce the baseline locally + +Read [Local proof](references/local-proof.md) before building or changing the +benchmark. Use disposable real PostgreSQL and synthetic data with representative +row counts, selectivity, and payload sizes. Record the PostgreSQL version and +any difference from the hosted server. + +Pin the pre-fix commit. Capture the complete SQL and parameters the real service +generates, including projections, joins, ordering, limits, and relevant imported +query helpers. Prove the baseline exhibits the suspected cost before editing. +If it does not, improve the fixture or revise the hypothesis. + +Save baseline results, timings, and `EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON)`. +Define the performance acceptance criterion for this fixture before measuring +the treatment. Avoid a brittle universal millisecond threshold. + +## 5. Implement and prove both kinds of green + +Make the smallest change supported by the measurements. Author the fix on our +branch from source and evidence. Related community reports can supply leads; +respect any instruction to keep the implementation independent of their code. + +Run baseline and treatment against equivalent fixtures under the same settings: + +- **Performance:** use warmups and repeated samples, report medians and spread, + and examine plan/buffer changes. Alternate order or repeat in reverse order + when caching could explain the gain. Report neutral or slower paths honestly. +- **Correctness:** compare every returned SQL column and the externally visible + behavior. Preserve ordering, duplicates, nulls, tenant scope, and redaction. + A faster wrong result fails even when row counts match. + +When intentionally bounding history, compare query speed at the same result +limit separately from the contract change. Test default/max limits and older +record access. Do not call truncated output equivalent to the former full list. + +Reject or revise the change if either proof fails. Keep the reproducible harness +with the source where repository conventions allow it; keep bulky evidence out +of git. Exhaust viable query/caller changes and existing indexes before proposing +an exceptional migration. Document the serious reason and why migration-free +alternatives are insufficient. Do not add startup scripts, boot-time DDL, or +recurring backfills to bypass the migration constraint. + +If a migration remains necessary, follow the database migration rules and the +production-size timing procedure in [Local proof](references/local-proof.md). +Measure the complete migration, including index builds, backfills, and lock waits. +First optimize a migration exceeding the roughly one-to-two-minute budget; if +it cannot be brought within an acceptable duration, flag it for human review +with the measured timings and alternatives before presenting it as deployable. + +## 6. Verify the blast radius + +Choose checks from the caller graph and affected contracts, not only the edited +function. Relevant cases from this workflow include: + +| Change | Regression evidence | +| --- | --- | +| UUID/JSON joins | Canonical, uppercase, malformed, missing and non-string IDs; native/legacy precedence; company scope | +| JSON projections | Scalar/array/object/null inputs, text conversion, all fields, redaction | +| History bounds | Service and route callers, UI/API defaults, older deep links, empty recent history, retry and navigation races | +| Ownership/recovery reads | Retained processes, active leases, blocked replay, workspace failure, receipts and delivery | +| Polling/caching | Idle work reduction, committed-change wakes, missed events, restart/sleep recovery, fresh destructive checks | + +Use real database tests for SQL semantics. Exercise assembled services/routes +and actual UI components when those are affected. Update query mocks when the +builder shape changes; a broken mock is not evidence that runtime SQL is wrong. + +Run focused checks first, then the owning repository's required pre-PR checks. +For this repo, consult `AGENTS.md` for typecheck, full tests, build, and conditional +map/token/migration gates. Capture command, commit, exit status, and log location. +Do not call an interrupted suite a pass. Preserve its partial evidence and state +what completed locally versus on the final commit in CI. + +## 7. Create the PR and carry it through checks + +Use [prepare-paperclip-pr](../prepare-paperclip-pr/SKILL.md) for public Paperclip +changes, [check-pr](../check-pr/SKILL.md) for feedback, and +[prcheckloop](../prcheckloop/SKILL.md) for checks. Follow the owning repository's +workflow instead when the fix belongs in `paperclip-cloud`. + +- Use the full PR template. Include cause, contract, benchmark command, baseline + revision, fixture, before/after timings, equality evidence, regression results, + and remaining risks. Label synthetic local timings as local SQL measurements. + For an exceptional migration, include its justification, production-size fixture, + complete duration, blocking time, optimization attempts, and any human decision + still needed. Query speedup does not establish acceptable migration cost. +- Split independent fixes into independently reviewable PRs; disclose actual + dependencies. Attach every created PR to the current chat/task where supported. +- After each push, refresh the head SHA. Address actionable reviews and failures, + rerun affected local proof, and wait for fresh review and all expected checks + on that SHA. Follow this repo's Greptile 5/5/no-unresolved-findings requirement. +- Distinguish code failures from runner shutdowns or external outages. Rerun a + demonstrated transient failure once. Cancel obsolete runs only after verifying + they belong to this work and an older head; never terminate unrelated jobs. +- Do not finish with "CI pending." Continue the loop within its repair bounds, + or report the exact blocker, check URL, attempts, and next action. An empty + check list, an old review score, or a draft-only subset of checks is not green. + +Thanking contributors and closing their PRs requires explicit authorization. +When authorized, explain which report our verified PR covers, acknowledge any +uncovered scope, and leave unrelated PRs open. This skill does not itself grant +permission to comment, close, merge, deploy, resize, or mutate a hosted database. + +## Constraints + +- your solution should not require a migration script to be run unless there is a serious reason - in almost all cases a migration script is a non starter +- similarly you should not add some new boot script - e.g. dont' work around migration scripts by running the migration on every boot +- if you add migrations, you should also have a sense on the time it will take on a large paperclip instance with thousands of tasks. the best option is to test on a clone of a production-size database and time it. if you don't have that you should make one. migrations that take an unacceptably large time to run (e.g. more than 1 or 2 minutes) must be flagged - first try to optimize the migration and then flag for a human if that's impossible + +## 8. Hand off the evidence + +Report each PR URL and final head, measured gains and neutral paths, correctness +comparison, completed local checks, latest-head CI/review status, and limitations. +If working from a Paperclip issue, attach evidence/work products using the +repository's artifact workflow. Keep private identifiers and tenant data out of +public PRs. State whether anything was deployed. + +If rollout is authorized, verify the deployed revision and compare a fresh, +comparable query-statistics window and request timing. Until then, production +latency improvement remains unverified even when local SQL is much faster. diff --git a/.agents/skills/fix-slow-queries/agents/openai.yaml b/.agents/skills/fix-slow-queries/agents/openai.yaml new file mode 100644 index 00000000000..98064d747fc --- /dev/null +++ b/.agents/skills/fix-slow-queries/agents/openai.yaml @@ -0,0 +1,3 @@ +interface: + display_name: "Fix Slow Queries" + short_description: "Trace cloud SQL hotspots through local proof and green PRs" diff --git a/.agents/skills/fix-slow-queries/references/cloud-discovery.md b/.agents/skills/fix-slow-queries/references/cloud-discovery.md new file mode 100644 index 00000000000..05a2d6e5a71 --- /dev/null +++ b/.agents/skills/fix-slow-queries/references/cloud-discovery.md @@ -0,0 +1,119 @@ +# From a slow workspace to its queries + +Use this reference for hosted Paperclip investigations. Resolve current resources +and credentials from the authorized environment; the examples describe lookup +shapes, not fixed account, stack, or project IDs. + +## Resolve the cross-service identity + +1. Start with the reported workspace URL and affected user action. Record UTC + observation times, environment, and a representative slow request. A fast + health endpoint does not establish that task/run endpoints are fast. +2. Open the authorized Paperclip Cloud fleet-admin stack record matching that + URL. Match its domain and workspace identity, not only a friendly name. +3. In stack details, inspect the database provider resources and **Open in Neon** + link. Record project, branch, endpoint/compute, database, and runtime role. + Provider refs may be CloudOps-only. Do not infer them when they are redacted. +4. Record the application provider's project, environment, service, active + deployment/revision, and region. On Railway these identify the logs and + metrics to compare with Neon. A stack can move after reprovisioning. +5. Confirm the database connection's actual database and role before querying. + Keep private mapping details in restricted evidence, not public PR bodies. + +Use available provider connectors or CLIs first; the authenticated browser is +useful for fleet-admin resource links and Neon monitoring. If tool access is +missing, inspect the authorized `paperclip-cloud` checkout and its instructions. +Relevant files, relative to that repository: + +- `docs/cloud-harness-app-paperclip-cloud-runbook.md`: provider credentials and + secret-store rules. +- `src/providers/neon.ts`: project/branch/role mapping and connection URI lookup. +- `src/providers/railway.ts`: provider API and metrics/log access. +- `web/src/admin/routes/StackDetail.tsx`: displayed provider resource mapping. + +Discover its path from the current project context; do not assume a developer's +home directory or copy their AWS profile. Verify the account/environment against +the current runbook. Staging and production secret stores are distinct. + +## Open a bounded read-only diagnostic connection + +Prefer the scoped runtime connection secret already recorded for the stack. +When authorized operator access is needed, the provider's connection lookup uses +`GET /projects/{projectId}/connection_uri` with `branch_id`, `endpoint_id`, +`database_name`, `role_name`, and `pooled=true`; verify against the current provider +implementation. Pass returned credentials directly in memory to the client. + +Load only necessary secret fields. Never print connection URIs, secret values, +or credential-bearing error objects; never put them in command arguments, +repository `.env` files, evidence, or PRs. Query text and parameters can also +contain tenant data. Keep full SQL privately only when necessary and sanitize +before publishing; simple literal replacement is not complete redaction. + +Set a recognizable diagnostic `application_name`, connection timeout, short +statement/lock timeouts, and read-only transactions. Example session: + +```sql +BEGIN READ ONLY; +SET LOCAL statement_timeout = '10s'; +SET LOCAL lock_timeout = '1s'; +SELECT current_database(), current_user, version(); +SELECT extname FROM pg_extension WHERE extname = 'pg_stat_statements'; +COMMIT; +``` + +Do not install extensions, reset shared statistics, change capacity, or run +schema/data changes as a side effect of diagnosis. `EXPLAIN ANALYZE` executes +the query; keep that work on the local fixture unless the live execution has +been separately justified and authorized. Plain `EXPLAIN` is the lighter live +option when permitted. Read-only SQL can still impose load; bound each probe. + +## Obtain historical and current evidence + +In Neon, select the confirmed branch, endpoint, and database, then inspect +**Monitoring → Query performance** and **Monitoring → Metrics**. UI labels can +change; use the observed navigation. Preserve the source link and capture time. + +When SQL access to `pg_stat_statements` is available, this supplies a first cut: + +```sql +SELECT userid, dbid, queryid, calls, total_exec_time, mean_exec_time, + max_exec_time, rows, shared_blks_hit, shared_blks_read, + temp_blks_written, query +FROM pg_stat_statements +WHERE dbid = (SELECT oid FROM pg_database WHERE datname = current_database()) +ORDER BY total_exec_time DESC +LIMIT 20; +``` + +Do not interpret an absent/inaccessible SQL extension as proof that the Neon +console has no query history. During this investigation the console supplied +history when the runtime connection could not query `pg_stat_statements`. +Without either source, take bounded samples of `pg_stat_activity` and scoped +application timings; label them samples, not complete historical attribution. + +For interval statistics, save two snapshots in separate transactions. Match +database, user, query ID and, where exposed, top-level status. Compute +`delta_total_exec_time / delta_calls` only for positive calls and compatible +reset epochs. Counter decreases, resets, or evicted/reappearing entries invalidate +that interval comparison. Record `pg_stat_statements_info.stats_reset` when +available. Mean/max values do not provide p95; p95 needs individual samples. + +Useful supporting observations: + +| Surface | Read | Interpretation | +| --- | --- | --- | +| Neon metrics | CPU allocation/usage, RAM, cache/working set, pooler wait, connection count | Correlate with the slow-request window before attributing capacity pressure | +| `pg_stat_activity`, `pg_locks` | Active query age, wait events, blockers, transaction age | Lock waits and long transactions need caller attribution | +| `pg_stat_database` | Transaction, block, temporary-file and deadlock deltas; stats reset | Cumulative counters need an interval; reported I/O time depends on tracking settings | +| `pg_stat_user_tables` | Scans, rows visited, live/dead estimates, analyze/vacuum times | Investigate broad reads and stale statistics; estimates are not exact row counts | +| Railway/app logs | Endpoint duration, background-job frequency, deployment | Separate request work from recurring maintenance | +| Traces/error system | SQL spans, pool/connect waits, errors scoped to the stack | Missing traces are not proof of zero database time | + +Measure connection establishment separately from several `SELECT 1` round trips. +These show network/pool overhead; they do not measure a business query. Likewise, +low total container CPU does not exclude Node event-loop saturation, and a busy +background query does not prove it caused a particular slow page. + +Produce a short ranked table: caller/query family, calls or call rate, mean SQL +time, total SQL time, observation window/reset caveat, suspected cause, and next +local experiment. Choose the bounded batch from this table. diff --git a/.agents/skills/fix-slow-queries/references/local-proof.md b/.agents/skills/fix-slow-queries/references/local-proof.md new file mode 100644 index 00000000000..bc9d217f62a --- /dev/null +++ b/.agents/skills/fix-slow-queries/references/local-proof.md @@ -0,0 +1,183 @@ +# Local performance and correctness proof + +The test must answer both questions: does the changed query do less work, and +does the surrounding feature still behave correctly? + +## Start from the existing harnesses + +Search `scripts/benchmark-*.ts` and `packages/db/src` before creating a harness. +This work produced these examples; use those present in the current checkout: + +- `scripts/benchmark-slow-queries.ts`: chat notices, provenance, redaction, + candidate indexes, and an unchanged history control. +- `scripts/benchmark-run-history.ts`: baseline service loading, captured history + SQL, wide JSON projection, complete positional-row comparison. +- `scripts/benchmark-recovery-queries.ts`: ownership/blocker/delivery queries and + baseline helper dependency isolation. +- `startEmbeddedPostgresTestDatabase` from `@paperclipai/db`: disposable real + PostgreSQL lifecycle, used by these benchmarks and service integration tests. + +Inspect a harness before running it. Use disposable synthetic fixtures by default; +for exceptional migration timing, use an authorized isolated production-size clone +or create an equivalent production-size database as described below. Do not reuse +the developer's persistent database or a live cloud `DATABASE_URL`. Planner, +index, UUID, and TOAST claims need real PostgreSQL, not mocked query builders. +Prefer the hosted major version when practical and disclose version differences. + +Set `SLOW_QUERY_BASELINE` to a verified pre-fix commit and pass it explicitly: + +```sh +pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-run-history.ts --baseline-ref="$SLOW_QUERY_BASELINE" +``` + +Resolve the ref to a SHA in the report. `origin/master` may already contain the +fix after merge, so it is not a durable historical baseline. Capture treatment +SHA plus any uncommitted diff, dependency versions, DB settings and fixture size. + +## Capture the query actually used + +Invoke the real service/query builder and capture `.toSQL()` plus parameters at +execution. Stop before dispatching notifications or starting runs. If the +service reads an issue before an expensive action query, capture the correct +subsequent query and include the first read in an endpoint benchmark if claimed. + +Do not benchmark only an exported eligibility predicate and label it a complete +blocker query. Projections, joins, ordering, limit, and intermediate result sets +can materially change the result. Measure application post-processing separately +if the claim includes redaction or end-to-end request latency. + +Load baseline source with `git show` or a separate checkout. When dynamically +loading old modules against current dependencies, include every changed helper +that can affect SQL. An old recovery module importing the new conversation +helper contaminates the baseline. Record which dependencies were held constant. + +For index-only changes, use the same SQL in both treatments. Isolate index sets +in disposable databases or explicitly toggle only candidate indexes there; +record the definitions. Never drop indexes on a live database for a benchmark. + +## Construct a fixture that exercises the cost and the contract + +Match the mechanism observed in the cloud: candidate counts, match selectivity, +status distribution, tenant distribution, history depth, and payload width. +Include a small control and a representative stress case when size changes the +plan. Run `ANALYZE` after loading equivalent data. + +For large JSON, repeated filler compresses unrealistically well. Use synthetic +incompressible content so TOAST reads resemble the actual workload. The original +fixture used 1,400 runs with approximately 192 KiB contexts; these are examples, +not universal sizes. Historical resolved recovery actions also matter: an empty +or tiny action table can hide the expensive join. + +Fix timestamps and distinguish ordering ties. Include cross-company decoys, +empty results, duplicates, missing JSON keys, null/scalar/array/object contexts, +malformed/noncanonical IDs, and conflicting native/legacy bindings where the +affected contract permits them. Seed only relevant cases in each test. + +## Keep complete results + +Compare positional rows, for example `postgres.js` `.values()`, or give every +projection a unique alias. Object rows can overwrite duplicate column names, +silently hiding changed values. Matching row counts or IDs alone is insufficient. + +Compare exact order where the API promises it. Otherwise compare multisets of +complete rows, preserving duplicates; do not sort away a pagination bug. Keep +parameter values, company/user scope, limits, and stable clock inputs equivalent. +Assert equality outside the timing interval and fail the harness on mismatch. + +For intentional behavior changes, specify an explicit oracle. A bounded history +API may return fewer rows by design; prove those are the correct ordered page, +that requested limits are honored/clamped, and that older by-ID navigation still +works. Benchmark projection changes with the same limit on both versions. + +## Measure and decide + +1. Confirm baseline results and the suspected slow plan before applying the fix. +2. Warm both versions. Use several measured iterations; start with two warmups + and at least five timed samples, expanding if the result is noisy. +3. Record all samples, median and range or quartiles. Alternate baseline/treatment + or repeat in reverse order to check cache bias. Keep noisy full-suite/build + work out of the timed interval. Do not claim cold-cache results from warm runs. +4. Capture `EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON)` on the local fixture outside + ordinary timing samples. Inspect actual rows/loops, filtering, sorts/spills, + buffer accesses, and index use. Explain why the plan supports the measured gain. +5. Report baseline/treatment medians, speedup `before / after`, row equality, + sample count, and limits of the claim. If spread overlaps materially, repeat + or label the result inconclusive rather than selling a small fluctuation. + +Keep semantic assertions in deterministic regression tests. Keep noisy timing +thresholds in the explicit benchmark unless the CI performance environment is +controlled. A test proving an index exists is weaker than real SQL results plus +plan/performance evidence. A proposed optimization can be neutral or slower: +our initial status-delivery rewrite repeated JSON reads and regressed until it +was simplified; the final path was reported as neutral. + +## Time an exceptional migration at production size + +Migrations are normally a non-starter for this workflow. First establish the +serious reason one is unavoidable and why query rewrites, existing indexes, or +caller changes cannot solve the problem. New boot scripts or startup migrations +are not an alternative to that requirement. + +Prefer an authorized, isolated clone of a production-size database. If none is +available, create a production-size test database rather than timing a tiny +fixture: thousands of tasks plus representative related runs, events, comments, +JSON payloads, indexes, and history depth for the tables the migration touches. +Use observed aggregate sizes/distributions to size synthetic data when a real +clone is unavailable. Keep copied tenant data in its approved environment and +out of logs, fixtures committed to git, and public PRs. + +Run the actual migration through its normal runner against the pre-migration +schema. Record total wall-clock time and per-step duration, including index +builds, backfills, lock waits, and commit. Record table/row sizes, PostgreSQL +version, compute/storage characteristics, and blocking time under representative +concurrent reads/writes where relevant. A concurrent index build can still take +too long even when it does not block writes. A fast laptop run is not proof of +an equally fast deployment on smaller compute. + +Restore the starting state before timing an optimized version; an already-applied +or resumable no-op is not a first-run measurement. Also test interrupted execution +and rerun safety where applicable. Report first-run and rerun timings separately. + +Treat durations above roughly one to two minutes as a flag, not an acceptable +cost to hide behind a query speedup. First optimize and repeat the production-size +timing. If an acceptable duration cannot be achieved, flag it for a human with +the serious reason, measured duration/blocking, attempted improvements, and +alternatives. Do not move the work into boot, run it against the live database, +or label the migration ready to deploy while that decision remains unresolved. + +## Anticipate the compatibility traps + +- Casting JSON text to UUID can reject malformed values or broaden matches for + uppercase/noncanonical text. Preserve the former text-comparison semantics + deliberately; guard casts and keep the indexed UUID side unwrapped. +- A native first-class ID may take precedence over legacy JSON. Do not change + fallback order, cross-company access, or ownership holds while improving joins. +- A single JSON expansion must preserve legacy `->>` conversions and non-object + inputs. Returning fewer projected keys can break redaction or downstream UI. +- History limits can break older deep links. Verify the actual run panel with + an older selected run, including an empty recent list, fetch failure/retry, + company/agent mismatch, and navigation while a lookup is pending. Check all + shipped page variants and relevant desktop/mobile branches. +- Query-builder mocks may need new joins, but mock updates do not replace a real + database test or justify removing a retry/disconnect regression test. + +If PostgreSQL startup or a broad suite hits local shared-memory/resource limits, +save the real error, reduce concurrency, and clean up only this task's resources. +Re-run the affected proof. Report remaining limits honestly and distinguish +completed CI from incomplete local commands. Do not skip a failing assertion as +an environment issue or terminate another worktree's tests. + +## Evidence to carry into the PR + +Retain the reproducible command, baseline/treatment identity, synthetic fixture +definition, database version/settings, full samples and plans, equality result, +focused regression commands/results, and broader gate outcomes. Summarize: + +| Caller/query | Before median | After median | Samples/spread | Complete output/contract check | Scope | +| --- | --- | --- | --- | --- | --- | +| Measured query | ms | ms | recorded | pass/fail and rows | SQL only, service, or request | + +Preserve raw evidence in the task's artifact location and put the useful summary +in the PR. Re-run affected proof after a review fix that changes the query or +contract. Source correctness, local speed, final-head CI, and deployed behavior +are separate conclusions; report each only as far as the evidence supports it.