diff --git a/.agents/skills/fix-slow-queries/SKILL.md b/.agents/skills/fix-slow-queries/SKILL.md new file mode 100644 index 00000000000..b268fd36471 --- /dev/null +++ b/.agents/skills/fix-slow-queries/SKILL.md @@ -0,0 +1,189 @@ +--- +name: fix-slow-queries +description: >- + Diagnose and fix slow Paperclip PostgreSQL queries using Cloud and Neon evidence, + local reproduction, performance and result comparisons, regression tests, and + a PR with passing checks. Use for costly queries, repeated database work, or + slow requests with a suspected database cause. +--- + +# Fix Slow Queries + +Deliver a measured improvement with preserved behavior and a reviewable PR whose +latest commit passes checks. A list of suspicious SQL or an unverified rewrite +is an intermediate result. Respect a narrower request for diagnosis only. + +## 1. Establish the target and finish line + +- Resolve the reported URL, environment, company/workspace, deployed revision, + owning repository, and local checkout. Read its `AGENTS.md` and scoped rules. + Paperclip and the private `paperclip-cloud` control plane have different rules. +- Work in a dedicated branch/worktree. Reuse suitable existing work; keep the + checkout running a live server out of branch switching and benchmark setup. +- Select a bounded batch, usually the top two or three query families. Record + deferred candidates so discovery does not turn into an unbounded rewrite. +- The finish line is: reproducible baseline, faster treatment, output/behavior + proof, relevant regression coverage, and green checks on each final PR head. + A concrete access or infrastructure blocker must be reported as incomplete. + +## 2. Find the actual database and rank the work + +Read [Cloud discovery](references/cloud-discovery.md) when starting from a hosted +workspace. Follow the chain from Paperclip Cloud fleet admin to the stack's Neon +project, branch, endpoint, database, and runtime role. Record the corresponding +application deployment and time window. Do not reuse IDs from a previous run. + +Capture query text/fingerprint, calls, total execution time, mean time, rows, +and available I/O, temporary-file, and lock-wait evidence. Rank both expensive +individual calls and frequent work with high aggregate cost. Group variants by +their actual caller; do not add unrelated query means into an endpoint latency. + +Keep three claims separate: + +| Claim | Evidence needed | +| --- | --- | +| SQL is expensive | Query statistics and a reproducible execution plan | +| The database path dominates a slow request | Correlated SQL, connection wait, and application timings | +| More database capacity would help | Resource/queue pressure in the same window and a capacity-specific experiment | + +Cumulative query statistics are not a recent-window trace. Capture reset times +or two snapshots and calculate deltas. A large database-time total alone does +not justify resizing Neon. Application CPU headroom also does not rule out a +busy Node main thread. + +## 3. Trace each candidate to its behavior + +Use `rg` on distinctive SQL fragments, table names, JSON keys, and predicates. +Trace the full path: route or scheduler → service → generated SQL → indexes. +Record frequency, fan-out, page limits, redaction, authorization, and any second +query the caller performs. Compare the deployed implementation with local code. + +State a falsifiable cause and the contract the fix must preserve. Typical causes +include casts on indexed columns, repeated detoasting of large JSON, broad +candidate scans, unbounded result sets, duplicate reads, and unnecessary polling. +Use plans and caller evidence to choose among them. + +Inspect existing indexes and recent related fixes before adding another index. +For advisory locks, identify the caller, key, transaction duration, and wait +chain first. Preserve checkout, leases, ownership, recovery, and no-replay rules; +removing a lock or polling loop is not a query optimization by itself. + +## 4. Reproduce the baseline locally + +Read [Local proof](references/local-proof.md) before building or changing the +benchmark. Use disposable real PostgreSQL and synthetic data with representative +row counts, selectivity, and payload sizes. Record the PostgreSQL version and +any difference from the hosted server. + +Pin the pre-fix commit. Capture the complete SQL and parameters the real service +generates, including projections, joins, ordering, limits, and relevant imported +query helpers. Prove the baseline exhibits the suspected cost before editing. +If it does not, improve the fixture or revise the hypothesis. + +Save baseline results, timings, and `EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON)`. +Define the performance acceptance criterion for this fixture before measuring +the treatment. Avoid a brittle universal millisecond threshold. + +## 5. Implement and prove both kinds of green + +Make the smallest change supported by the measurements. Author the fix on our +branch from source and evidence. Related community reports can supply leads; +respect any instruction to keep the implementation independent of their code. + +Run baseline and treatment against equivalent fixtures under the same settings: + +- **Performance:** use warmups and repeated samples, report medians and spread, + and examine plan/buffer changes. Alternate order or repeat in reverse order + when caching could explain the gain. Report neutral or slower paths honestly. +- **Correctness:** compare every returned SQL column and the externally visible + behavior. Preserve ordering, duplicates, nulls, tenant scope, and redaction. + A faster wrong result fails even when row counts match. + +When intentionally bounding history, compare query speed at the same result +limit separately from the contract change. Test default/max limits and older +record access. Do not call truncated output equivalent to the former full list. + +Reject or revise the change if either proof fails. Keep the reproducible harness +with the source where repository conventions allow it; keep bulky evidence out +of git. Exhaust viable query/caller changes and existing indexes before proposing +an exceptional migration. Document the serious reason and why migration-free +alternatives are insufficient. Do not add startup scripts, boot-time DDL, or +recurring backfills to bypass the migration constraint. + +If a migration remains necessary, follow the database migration rules and the +production-size timing procedure in [Local proof](references/local-proof.md). +Measure the complete migration, including index builds, backfills, and lock waits. +First optimize a migration exceeding the roughly one-to-two-minute budget; if +it cannot be brought within an acceptable duration, flag it for human review +with the measured timings and alternatives before presenting it as deployable. + +## 6. Verify the blast radius + +Choose checks from the caller graph and affected contracts, not only the edited +function. Relevant cases from this workflow include: + +| Change | Regression evidence | +| --- | --- | +| UUID/JSON joins | Canonical, uppercase, malformed, missing and non-string IDs; native/legacy precedence; company scope | +| JSON projections | Scalar/array/object/null inputs, text conversion, all fields, redaction | +| History bounds | Service and route callers, UI/API defaults, older deep links, empty recent history, retry and navigation races | +| Ownership/recovery reads | Retained processes, active leases, blocked replay, workspace failure, receipts and delivery | +| Polling/caching | Idle work reduction, committed-change wakes, missed events, restart/sleep recovery, fresh destructive checks | + +Use real database tests for SQL semantics. Exercise assembled services/routes +and actual UI components when those are affected. Update query mocks when the +builder shape changes; a broken mock is not evidence that runtime SQL is wrong. + +Run focused checks first, then the owning repository's required pre-PR checks. +For this repo, consult `AGENTS.md` for typecheck, full tests, build, and conditional +map/token/migration gates. Capture command, commit, exit status, and log location. +Do not call an interrupted suite a pass. Preserve its partial evidence and state +what completed locally versus on the final commit in CI. + +## 7. Create the PR and carry it through checks + +Use [prepare-paperclip-pr](../prepare-paperclip-pr/SKILL.md) for public Paperclip +changes, [check-pr](../check-pr/SKILL.md) for feedback, and +[prcheckloop](../prcheckloop/SKILL.md) for checks. Follow the owning repository's +workflow instead when the fix belongs in `paperclip-cloud`. + +- Use the full PR template. Include cause, contract, benchmark command, baseline + revision, fixture, before/after timings, equality evidence, regression results, + and remaining risks. Label synthetic local timings as local SQL measurements. + For an exceptional migration, include its justification, production-size fixture, + complete duration, blocking time, optimization attempts, and any human decision + still needed. Query speedup does not establish acceptable migration cost. +- Split independent fixes into independently reviewable PRs; disclose actual + dependencies. Attach every created PR to the current chat/task where supported. +- After each push, refresh the head SHA. Address actionable reviews and failures, + rerun affected local proof, and wait for fresh review and all expected checks + on that SHA. Follow this repo's Greptile 5/5/no-unresolved-findings requirement. +- Distinguish code failures from runner shutdowns or external outages. Rerun a + demonstrated transient failure once. Cancel obsolete runs only after verifying + they belong to this work and an older head; never terminate unrelated jobs. +- Do not finish with "CI pending." Continue the loop within its repair bounds, + or report the exact blocker, check URL, attempts, and next action. An empty + check list, an old review score, or a draft-only subset of checks is not green. + +Thanking contributors and closing their PRs requires explicit authorization. +When authorized, explain which report our verified PR covers, acknowledge any +uncovered scope, and leave unrelated PRs open. This skill does not itself grant +permission to comment, close, merge, deploy, resize, or mutate a hosted database. + +## Constraints + +- your solution should not require a migration script to be run unless there is a serious reason - in almost all cases a migration script is a non starter +- similarly you should not add some new boot script - e.g. dont' work around migration scripts by running the migration on every boot +- if you add migrations, you should also have a sense on the time it will take on a large paperclip instance with thousands of tasks. the best option is to test on a clone of a production-size database and time it. if you don't have that you should make one. migrations that take an unacceptably large time to run (e.g. more than 1 or 2 minutes) must be flagged - first try to optimize the migration and then flag for a human if that's impossible + +## 8. Hand off the evidence + +Report each PR URL and final head, measured gains and neutral paths, correctness +comparison, completed local checks, latest-head CI/review status, and limitations. +If working from a Paperclip issue, attach evidence/work products using the +repository's artifact workflow. Keep private identifiers and tenant data out of +public PRs. State whether anything was deployed. + +If rollout is authorized, verify the deployed revision and compare a fresh, +comparable query-statistics window and request timing. Until then, production +latency improvement remains unverified even when local SQL is much faster. diff --git a/.agents/skills/fix-slow-queries/agents/openai.yaml b/.agents/skills/fix-slow-queries/agents/openai.yaml new file mode 100644 index 00000000000..98064d747fc --- /dev/null +++ b/.agents/skills/fix-slow-queries/agents/openai.yaml @@ -0,0 +1,3 @@ +interface: + display_name: "Fix Slow Queries" + short_description: "Trace cloud SQL hotspots through local proof and green PRs" diff --git a/.agents/skills/fix-slow-queries/references/cloud-discovery.md b/.agents/skills/fix-slow-queries/references/cloud-discovery.md new file mode 100644 index 00000000000..05a2d6e5a71 --- /dev/null +++ b/.agents/skills/fix-slow-queries/references/cloud-discovery.md @@ -0,0 +1,119 @@ +# From a slow workspace to its queries + +Use this reference for hosted Paperclip investigations. Resolve current resources +and credentials from the authorized environment; the examples describe lookup +shapes, not fixed account, stack, or project IDs. + +## Resolve the cross-service identity + +1. Start with the reported workspace URL and affected user action. Record UTC + observation times, environment, and a representative slow request. A fast + health endpoint does not establish that task/run endpoints are fast. +2. Open the authorized Paperclip Cloud fleet-admin stack record matching that + URL. Match its domain and workspace identity, not only a friendly name. +3. In stack details, inspect the database provider resources and **Open in Neon** + link. Record project, branch, endpoint/compute, database, and runtime role. + Provider refs may be CloudOps-only. Do not infer them when they are redacted. +4. Record the application provider's project, environment, service, active + deployment/revision, and region. On Railway these identify the logs and + metrics to compare with Neon. A stack can move after reprovisioning. +5. Confirm the database connection's actual database and role before querying. + Keep private mapping details in restricted evidence, not public PR bodies. + +Use available provider connectors or CLIs first; the authenticated browser is +useful for fleet-admin resource links and Neon monitoring. If tool access is +missing, inspect the authorized `paperclip-cloud` checkout and its instructions. +Relevant files, relative to that repository: + +- `docs/cloud-harness-app-paperclip-cloud-runbook.md`: provider credentials and + secret-store rules. +- `src/providers/neon.ts`: project/branch/role mapping and connection URI lookup. +- `src/providers/railway.ts`: provider API and metrics/log access. +- `web/src/admin/routes/StackDetail.tsx`: displayed provider resource mapping. + +Discover its path from the current project context; do not assume a developer's +home directory or copy their AWS profile. Verify the account/environment against +the current runbook. Staging and production secret stores are distinct. + +## Open a bounded read-only diagnostic connection + +Prefer the scoped runtime connection secret already recorded for the stack. +When authorized operator access is needed, the provider's connection lookup uses +`GET /projects/{projectId}/connection_uri` with `branch_id`, `endpoint_id`, +`database_name`, `role_name`, and `pooled=true`; verify against the current provider +implementation. Pass returned credentials directly in memory to the client. + +Load only necessary secret fields. Never print connection URIs, secret values, +or credential-bearing error objects; never put them in command arguments, +repository `.env` files, evidence, or PRs. Query text and parameters can also +contain tenant data. Keep full SQL privately only when necessary and sanitize +before publishing; simple literal replacement is not complete redaction. + +Set a recognizable diagnostic `application_name`, connection timeout, short +statement/lock timeouts, and read-only transactions. Example session: + +```sql +BEGIN READ ONLY; +SET LOCAL statement_timeout = '10s'; +SET LOCAL lock_timeout = '1s'; +SELECT current_database(), current_user, version(); +SELECT extname FROM pg_extension WHERE extname = 'pg_stat_statements'; +COMMIT; +``` + +Do not install extensions, reset shared statistics, change capacity, or run +schema/data changes as a side effect of diagnosis. `EXPLAIN ANALYZE` executes +the query; keep that work on the local fixture unless the live execution has +been separately justified and authorized. Plain `EXPLAIN` is the lighter live +option when permitted. Read-only SQL can still impose load; bound each probe. + +## Obtain historical and current evidence + +In Neon, select the confirmed branch, endpoint, and database, then inspect +**Monitoring → Query performance** and **Monitoring → Metrics**. UI labels can +change; use the observed navigation. Preserve the source link and capture time. + +When SQL access to `pg_stat_statements` is available, this supplies a first cut: + +```sql +SELECT userid, dbid, queryid, calls, total_exec_time, mean_exec_time, + max_exec_time, rows, shared_blks_hit, shared_blks_read, + temp_blks_written, query +FROM pg_stat_statements +WHERE dbid = (SELECT oid FROM pg_database WHERE datname = current_database()) +ORDER BY total_exec_time DESC +LIMIT 20; +``` + +Do not interpret an absent/inaccessible SQL extension as proof that the Neon +console has no query history. During this investigation the console supplied +history when the runtime connection could not query `pg_stat_statements`. +Without either source, take bounded samples of `pg_stat_activity` and scoped +application timings; label them samples, not complete historical attribution. + +For interval statistics, save two snapshots in separate transactions. Match +database, user, query ID and, where exposed, top-level status. Compute +`delta_total_exec_time / delta_calls` only for positive calls and compatible +reset epochs. Counter decreases, resets, or evicted/reappearing entries invalidate +that interval comparison. Record `pg_stat_statements_info.stats_reset` when +available. Mean/max values do not provide p95; p95 needs individual samples. + +Useful supporting observations: + +| Surface | Read | Interpretation | +| --- | --- | --- | +| Neon metrics | CPU allocation/usage, RAM, cache/working set, pooler wait, connection count | Correlate with the slow-request window before attributing capacity pressure | +| `pg_stat_activity`, `pg_locks` | Active query age, wait events, blockers, transaction age | Lock waits and long transactions need caller attribution | +| `pg_stat_database` | Transaction, block, temporary-file and deadlock deltas; stats reset | Cumulative counters need an interval; reported I/O time depends on tracking settings | +| `pg_stat_user_tables` | Scans, rows visited, live/dead estimates, analyze/vacuum times | Investigate broad reads and stale statistics; estimates are not exact row counts | +| Railway/app logs | Endpoint duration, background-job frequency, deployment | Separate request work from recurring maintenance | +| Traces/error system | SQL spans, pool/connect waits, errors scoped to the stack | Missing traces are not proof of zero database time | + +Measure connection establishment separately from several `SELECT 1` round trips. +These show network/pool overhead; they do not measure a business query. Likewise, +low total container CPU does not exclude Node event-loop saturation, and a busy +background query does not prove it caused a particular slow page. + +Produce a short ranked table: caller/query family, calls or call rate, mean SQL +time, total SQL time, observation window/reset caveat, suspected cause, and next +local experiment. Choose the bounded batch from this table. diff --git a/.agents/skills/fix-slow-queries/references/local-proof.md b/.agents/skills/fix-slow-queries/references/local-proof.md new file mode 100644 index 00000000000..bc9d217f62a --- /dev/null +++ b/.agents/skills/fix-slow-queries/references/local-proof.md @@ -0,0 +1,183 @@ +# Local performance and correctness proof + +The test must answer both questions: does the changed query do less work, and +does the surrounding feature still behave correctly? + +## Start from the existing harnesses + +Search `scripts/benchmark-*.ts` and `packages/db/src` before creating a harness. +This work produced these examples; use those present in the current checkout: + +- `scripts/benchmark-slow-queries.ts`: chat notices, provenance, redaction, + candidate indexes, and an unchanged history control. +- `scripts/benchmark-run-history.ts`: baseline service loading, captured history + SQL, wide JSON projection, complete positional-row comparison. +- `scripts/benchmark-recovery-queries.ts`: ownership/blocker/delivery queries and + baseline helper dependency isolation. +- `startEmbeddedPostgresTestDatabase` from `@paperclipai/db`: disposable real + PostgreSQL lifecycle, used by these benchmarks and service integration tests. + +Inspect a harness before running it. Use disposable synthetic fixtures by default; +for exceptional migration timing, use an authorized isolated production-size clone +or create an equivalent production-size database as described below. Do not reuse +the developer's persistent database or a live cloud `DATABASE_URL`. Planner, +index, UUID, and TOAST claims need real PostgreSQL, not mocked query builders. +Prefer the hosted major version when practical and disclose version differences. + +Set `SLOW_QUERY_BASELINE` to a verified pre-fix commit and pass it explicitly: + +```sh +pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-run-history.ts --baseline-ref="$SLOW_QUERY_BASELINE" +``` + +Resolve the ref to a SHA in the report. `origin/master` may already contain the +fix after merge, so it is not a durable historical baseline. Capture treatment +SHA plus any uncommitted diff, dependency versions, DB settings and fixture size. + +## Capture the query actually used + +Invoke the real service/query builder and capture `.toSQL()` plus parameters at +execution. Stop before dispatching notifications or starting runs. If the +service reads an issue before an expensive action query, capture the correct +subsequent query and include the first read in an endpoint benchmark if claimed. + +Do not benchmark only an exported eligibility predicate and label it a complete +blocker query. Projections, joins, ordering, limit, and intermediate result sets +can materially change the result. Measure application post-processing separately +if the claim includes redaction or end-to-end request latency. + +Load baseline source with `git show` or a separate checkout. When dynamically +loading old modules against current dependencies, include every changed helper +that can affect SQL. An old recovery module importing the new conversation +helper contaminates the baseline. Record which dependencies were held constant. + +For index-only changes, use the same SQL in both treatments. Isolate index sets +in disposable databases or explicitly toggle only candidate indexes there; +record the definitions. Never drop indexes on a live database for a benchmark. + +## Construct a fixture that exercises the cost and the contract + +Match the mechanism observed in the cloud: candidate counts, match selectivity, +status distribution, tenant distribution, history depth, and payload width. +Include a small control and a representative stress case when size changes the +plan. Run `ANALYZE` after loading equivalent data. + +For large JSON, repeated filler compresses unrealistically well. Use synthetic +incompressible content so TOAST reads resemble the actual workload. The original +fixture used 1,400 runs with approximately 192 KiB contexts; these are examples, +not universal sizes. Historical resolved recovery actions also matter: an empty +or tiny action table can hide the expensive join. + +Fix timestamps and distinguish ordering ties. Include cross-company decoys, +empty results, duplicates, missing JSON keys, null/scalar/array/object contexts, +malformed/noncanonical IDs, and conflicting native/legacy bindings where the +affected contract permits them. Seed only relevant cases in each test. + +## Keep complete results + +Compare positional rows, for example `postgres.js` `.values()`, or give every +projection a unique alias. Object rows can overwrite duplicate column names, +silently hiding changed values. Matching row counts or IDs alone is insufficient. + +Compare exact order where the API promises it. Otherwise compare multisets of +complete rows, preserving duplicates; do not sort away a pagination bug. Keep +parameter values, company/user scope, limits, and stable clock inputs equivalent. +Assert equality outside the timing interval and fail the harness on mismatch. + +For intentional behavior changes, specify an explicit oracle. A bounded history +API may return fewer rows by design; prove those are the correct ordered page, +that requested limits are honored/clamped, and that older by-ID navigation still +works. Benchmark projection changes with the same limit on both versions. + +## Measure and decide + +1. Confirm baseline results and the suspected slow plan before applying the fix. +2. Warm both versions. Use several measured iterations; start with two warmups + and at least five timed samples, expanding if the result is noisy. +3. Record all samples, median and range or quartiles. Alternate baseline/treatment + or repeat in reverse order to check cache bias. Keep noisy full-suite/build + work out of the timed interval. Do not claim cold-cache results from warm runs. +4. Capture `EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON)` on the local fixture outside + ordinary timing samples. Inspect actual rows/loops, filtering, sorts/spills, + buffer accesses, and index use. Explain why the plan supports the measured gain. +5. Report baseline/treatment medians, speedup `before / after`, row equality, + sample count, and limits of the claim. If spread overlaps materially, repeat + or label the result inconclusive rather than selling a small fluctuation. + +Keep semantic assertions in deterministic regression tests. Keep noisy timing +thresholds in the explicit benchmark unless the CI performance environment is +controlled. A test proving an index exists is weaker than real SQL results plus +plan/performance evidence. A proposed optimization can be neutral or slower: +our initial status-delivery rewrite repeated JSON reads and regressed until it +was simplified; the final path was reported as neutral. + +## Time an exceptional migration at production size + +Migrations are normally a non-starter for this workflow. First establish the +serious reason one is unavoidable and why query rewrites, existing indexes, or +caller changes cannot solve the problem. New boot scripts or startup migrations +are not an alternative to that requirement. + +Prefer an authorized, isolated clone of a production-size database. If none is +available, create a production-size test database rather than timing a tiny +fixture: thousands of tasks plus representative related runs, events, comments, +JSON payloads, indexes, and history depth for the tables the migration touches. +Use observed aggregate sizes/distributions to size synthetic data when a real +clone is unavailable. Keep copied tenant data in its approved environment and +out of logs, fixtures committed to git, and public PRs. + +Run the actual migration through its normal runner against the pre-migration +schema. Record total wall-clock time and per-step duration, including index +builds, backfills, lock waits, and commit. Record table/row sizes, PostgreSQL +version, compute/storage characteristics, and blocking time under representative +concurrent reads/writes where relevant. A concurrent index build can still take +too long even when it does not block writes. A fast laptop run is not proof of +an equally fast deployment on smaller compute. + +Restore the starting state before timing an optimized version; an already-applied +or resumable no-op is not a first-run measurement. Also test interrupted execution +and rerun safety where applicable. Report first-run and rerun timings separately. + +Treat durations above roughly one to two minutes as a flag, not an acceptable +cost to hide behind a query speedup. First optimize and repeat the production-size +timing. If an acceptable duration cannot be achieved, flag it for a human with +the serious reason, measured duration/blocking, attempted improvements, and +alternatives. Do not move the work into boot, run it against the live database, +or label the migration ready to deploy while that decision remains unresolved. + +## Anticipate the compatibility traps + +- Casting JSON text to UUID can reject malformed values or broaden matches for + uppercase/noncanonical text. Preserve the former text-comparison semantics + deliberately; guard casts and keep the indexed UUID side unwrapped. +- A native first-class ID may take precedence over legacy JSON. Do not change + fallback order, cross-company access, or ownership holds while improving joins. +- A single JSON expansion must preserve legacy `->>` conversions and non-object + inputs. Returning fewer projected keys can break redaction or downstream UI. +- History limits can break older deep links. Verify the actual run panel with + an older selected run, including an empty recent list, fetch failure/retry, + company/agent mismatch, and navigation while a lookup is pending. Check all + shipped page variants and relevant desktop/mobile branches. +- Query-builder mocks may need new joins, but mock updates do not replace a real + database test or justify removing a retry/disconnect regression test. + +If PostgreSQL startup or a broad suite hits local shared-memory/resource limits, +save the real error, reduce concurrency, and clean up only this task's resources. +Re-run the affected proof. Report remaining limits honestly and distinguish +completed CI from incomplete local commands. Do not skip a failing assertion as +an environment issue or terminate another worktree's tests. + +## Evidence to carry into the PR + +Retain the reproducible command, baseline/treatment identity, synthetic fixture +definition, database version/settings, full samples and plans, equality result, +focused regression commands/results, and broader gate outcomes. Summarize: + +| Caller/query | Before median | After median | Samples/spread | Complete output/contract check | Scope | +| --- | --- | --- | --- | --- | --- | +| Measured query | ms | ms | recorded | pass/fail and rows | SQL only, service, or request | + +Preserve raw evidence in the task's artifact location and put the useful summary +in the PR. Re-run affected proof after a review fix that changes the query or +contract. Source correctness, local speed, final-head CI, and deployed behavior +are separate conclusions; report each only as far as the evidence supports it.