From 393d1c410862aa93561548e0a39feec3b89df6ff Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 13 Oct 2026 14:08:51 -0500 Subject: [PATCH] docs(skills): add standalone test-audit workflow (#16211) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Repository skills guide agents through engineering work. > - Test gardening must preserve useful regression coverage while reducing repeated work. > - Contributors need a local procedure for evidence, validation, and delivery. > - This pull request adds a standalone test-audit skill with a short entry point and focused guides. > - The procedure supports focused cleanups and coordinated audits without requiring an external skill checkout. ## Linked Issues or Issue Description **Issue type** Missing documentation. **Where is the issue?** The repository has no standalone `.agents/skills/test-audit/` procedure on master. **What's wrong?** Test audits need consistent rules for retaining contracts, proving removal safety, recording cost evidence, and tracking stacked delivery. **Suggested fix** Add a self-contained skill with candidate evidence, validation, campaign, and Paperclip delivery guides. Preserve upstream attribution. Related audit work: Refs #15978, Refs #16022, Refs #16197. The duplicate search found no standalone skill PR. This docs change does not duplicate planned core features in ROADMAP.md. ## What Changed - Add a short test-audit entry point and agent metadata. - Require contract and keeper evidence before removing tests. Apply an authoring gate to added or changed tests. - Define targeted and exhaustive campaign modes, a second review of redundant layers, and full keeper dependency checks. - Document matched baselines, intended-failure controls, selection checks, honest cost claims, and current-head delivery gates. - Add Paperclip commands and invariants. Retain OpenClaw attribution and its MIT notice. ## Verification - The skill frontmatter validator passed with cached PyYAML. - Agent metadata YAML parsed. The default prompt names `$test-audit`. - All eight relative Markdown links resolve. All seven delivery files matched the reviewed source. - `git diff --check` passed. The PR contains only seven Markdown/YAML skill files. - An independent scenario review checked the campaign completion and coverage rules. - Local application typecheck, full Vitest, and build were not run for this documentation-only change. Fresh current-head CI passed: 55 successful checks and two expected skips on `1a9af8ffacd532882f0a01bac8ab3f5063ab9cd9`. - E2E shard 8/8 was interrupted after its assertions passed. A single same-head job retry passed. The interruption cause was not exposed. - Greptile reviewed the current head at 5/5 with no actionable findings. There are no review threads. The PR has no merge conflicts and remains open as requested. ## Risks - Low runtime risk. This PR changes contributor instructions only. - Agents can misapply audit evidence. The skill requires valid keepers and preservation proof, rejects keeper cycles, and makes incomplete audits explicit. - CI routing can change. The guide requires readers to inspect current repository files. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex based on GPT-6 wrote and tightened the skill. Its exact deployment model ID and context-window size are unavailable. Independent scenario review and PR delivery used OpenAI `gpt-6.1-sol`, with reasoning, repository inspection, and tool execution. The context-window size was not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- .agents/skills/test-audit/CAMPAIGN.md | 98 +++++++++++++++++++ .agents/skills/test-audit/NOTICE.md | 34 +++++++ .agents/skills/test-audit/SKILL.md | 78 +++++++++++++++ .agents/skills/test-audit/agents/openai.yaml | 4 + .agents/skills/test-audit/references/audit.md | 72 ++++++++++++++ .../skills/test-audit/references/paperclip.md | 71 ++++++++++++++ .../test-audit/references/validation.md | 87 ++++++++++++++++ 7 files changed, 444 insertions(+) create mode 100644 .agents/skills/test-audit/CAMPAIGN.md create mode 100644 .agents/skills/test-audit/NOTICE.md create mode 100644 .agents/skills/test-audit/SKILL.md create mode 100644 .agents/skills/test-audit/agents/openai.yaml create mode 100644 .agents/skills/test-audit/references/audit.md create mode 100644 .agents/skills/test-audit/references/paperclip.md create mode 100644 .agents/skills/test-audit/references/validation.md diff --git a/.agents/skills/test-audit/CAMPAIGN.md b/.agents/skills/test-audit/CAMPAIGN.md new file mode 100644 index 00000000000..d761f82ad65 --- /dev/null +++ b/.agents/skills/test-audit/CAMPAIGN.md @@ -0,0 +1,98 @@ +# Coordinating a test-gardening campaign + +Use this guide for a broad sweep or multiple PRs. Apply the decision and +preservation rules in [SKILL.md](SKILL.md) to every batch. + +## 1. Define the audit boundary + +From the request, establish scope, desired benefit, time budget, target branch, +and delivery authority. Pin the base revision and inspect CI routing and cost +data, including setup/import overhead. Choose one mode: + +- **Targeted sweep:** investigate promising owners or expensive repeated work. + Report the examined area and exclusions; do not claim subsystem completeness. +- **Exhaustive audit:** inventory every in-scope test file and scenario, including + colocated, shared-package, browser, and dedicated-lane coverage. Assign each to + one owner and record its baseline as passed, failed, blocked, or unrun. + +For an exhaustive claim, every declaration needs a decision and evidence; +classify table rows separately when their contracts differ. Missing baseline +results or unread tests leave the audit partial. Existing matching-revision CI +can supply results; a targeted sweep does not require a full-repository census. +Reserve time for preservation checks and reporting; an unfinished validation +leaves an unvalidated local patch, not a PR-ready result. + +## 2. Assign owners and record decisions + +Split work by protected production behavior, including tests across directories. +When delegation is authorized, assign each worker an isolated checkout, owner, +prior findings, model if specified, and discovery/edit/publication authority. +Tell workers to preserve others' changes. One coordinator owns shared fixtures, +routing, and integration. Do not edit a checkout with active test runs. + +Keep one ledger: candidate, contract, retain/repair/consolidate/delete/defer, +reason, keeper, baseline, validation, and branch/PR status. Record reviewed areas +with no safe removals too. Discovery stays read-only until ownership is clear; +qualified edits need no additional user-approval round within existing authority. + +## 3. Plan removal of redundant layers + +Before editing a broad batch, review the ledger a second time by contract and +boundary. Look for suites that replay the same behavior through wrappers, +mocked collaborators, or provider-specific copies of a shared implementation. +Individual cases can look reasonable while the entire layer is redundant. + +Produce a compact layer plan: retired suites, retained owner per contract, +unique assertions to transfer, affected routing/commands, validation controls, +and any unused fixtures or production seams. Prefer a boundary that reaches +the responsible code; preserve distinct transport, lifecycle, and permission +risks. If no whole layer qualifies, record that and proceed with justified +case-level changes. Large deletions are not a requirement. + +The coordinator reconciles keeper dependencies across all batches before +cutover. Follow each chain to surviving proof; reject cycles and recheck keepers +another batch modifies or excludes from routing. A batch is ready when every +retired contract has retained proof or retirement evidence. Production-seam +removal still needs separate caller evidence. + +## 4. Cut over and verify preservation + +Move unique assertions first, apply one coherent batch, and update selection +and ownership references. Follow [validation.md](references/validation.md) for +baselines, controls, restoration, and cost claims. Serialize heavy database or +process runs when host capacity is shared; clean up only owned resources. + +Review the combined diff against the layer plan. For broad or consequential +batches, use independent preservation review when delegation is available and +authorized; otherwise disclose that limitation. Resolve each finding with +source evidence or restored coverage and validate repaired assertions with an +intended-failure control. No batch is complete with an unresolved coverage gap. + +An exhaustive audit requires a decision for every declaration, a completed and +classified baseline for every file, and passing preservation proof for each +removal. Diagnosed pre-existing defects may remain as explicit follow-ups; +unresolved setup failures or unread tests leave the audit partial. If time +expires first, report the remaining inventory and validation status. + +## 5. Deliver and integrate + +Prefer a few coherent PRs to one PR per trivial deletion. For a stack, record +each PR's head SHA, base branch/SHA, unique diff, and intended final target. +After a parent is squash-merged, discover its actual merge SHA and replay only +the child's unique change if necessary. Inspect the resulting PR diff for +accidentally reintroduced parent commits or files. + +Resolve integration conflicts by reading new contracts on the target branch. +If a deleted file gained a regression, preserve that regression at an appropriate +owner; do not blindly accept the deletion or restore the entire old suite. + +For authorized merges, use checks and reviews tied to the current PR head, +inspect unresolved findings, and respect the repository's merge policy. A +previous green head does not validate a changed head. + +A PR marked “merged” into an intermediate stack branch is not necessarily +landed in the intended integration branch. Verify the full parent chain and +final diff inclusion before reporting the campaign landed. Preserve or clean +up branches/worktrees according to the user's instructions and retain needed +evidence before cleanup. Report delivery separately from audit completeness; +stop new discovery when the user asks to wrap up. diff --git a/.agents/skills/test-audit/NOTICE.md b/.agents/skills/test-audit/NOTICE.md new file mode 100644 index 00000000000..f38cbe74b94 --- /dev/null +++ b/.agents/skills/test-audit/NOTICE.md @@ -0,0 +1,34 @@ +# Attribution + +Paperclip's test-audit skill is inspired by the organization and audit principles +of OpenClaw's [test-audit skill](https://github.com/openclaw/openclaw/tree/main/.agents/skills/test-audit), +consulted on 2026-10-13. It uses the short entry point, separate campaign guide, +and focused references pattern, with independently written Paperclip procedures +and lessons from our test-gardening work. It has no execution or documentation +dependency on an OpenClaw checkout or another installed skill. + +The upstream license is retained below for the adapted material. + +```text +MIT License + +Copyright (c) 2026 OpenClaw Foundation + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. +``` diff --git a/.agents/skills/test-audit/SKILL.md b/.agents/skills/test-audit/SKILL.md new file mode 100644 index 00000000000..36288974712 --- /dev/null +++ b/.agents/skills/test-audit/SKILL.md @@ -0,0 +1,78 @@ +--- +name: test-audit +description: > + Garden the test suite by finding and removing redundant, vacuous, or + implementation-coupled tests while preserving useful regression coverage. + Use for test audits, test pruning, duplicate coverage, or reducing CI test + cost. Supports focused cleanups and coordinated audit campaigns. +--- + +# Test Audit + +Reduce execution and maintenance cost while preserving meaningful failure +detection. Investigate aggressively; delete only with evidence. Retaining an +area after review is a valid result. This skill is self-contained; upstream +attribution is in [NOTICE.md](NOTICE.md). + +## Choose the scope + +- Focused cleanup: follow this entry point and read the + [candidate evidence guide](references/audit.md) before editing. +- Multiple owners, agents, or PRs: also read [CAMPAIGN.md](CAMPAIGN.md). Declare + whether the work is a targeted sweep or an exhaustive subsystem audit. +- Validation and cost claims: read [references/validation.md](references/validation.md). +- Paperclip commands, invariants, and delivery: read + [references/paperclip.md](references/paperclip.md) alongside repository instructions. + +## Decision gate + +For every proposed removal, answer: + +1. Which observable contract and plausible regression do the assertions cover? +2. Which retained file/case detects that regression through the responsible + production code? Otherwise, why is the assertion vacuous or contract retired? +3. Does this candidate uniquely cover a transport, permission, lifecycle, + storage boundary, platform, default, or other independent failure mode? +4. What improves: executions, fixtures, processes, maintenance, or runtime? + +Prefer the smallest test that reaches the code responsible for the contract. +A broader suite is not automatically stronger proof. Similarity, size, slowness, +or suspected AI authorship alone never justify deletion. + +## Prevent new duplication + +Apply this gate to tests added or changed during gardening: + +- Name the regression that existing proof misses. Extend an appropriate owner + or table before adding another suite; another layer needs a distinct risk. +- Exercise the responsible production behavior. Do not add a mock that supplies + the result being asserted or a production hook solely for convenient testing. +- Check observable outcomes that survive behavior-preserving refactors. A + negative case must reach its named guard with otherwise valid inputs. + +## Focused workflow + +1. **Inspect:** pin the base; read candidates, keepers, fixtures, production + callers, history, and CI routing. Preserve unrelated work. +2. **Plan:** mark retain, repair, consolidate, delete, or defer. Name remaining + proof and assertions to move; look for redundant suites as well as cases. +3. **Prove:** establish the affected baseline locally or from matching-revision + CI. Diagnose failures before treating them as cleanup candidates. +4. **Edit:** move unique assertions first, then retire duplicates and unused + support. Production-seam deletion needs its own caller/contract evidence. +5. **Validate:** run keepers and affected siblings, verify selection, and apply + the validation guide's intended-failure controls where needed. +6. **Review:** reconcile deleted contracts with retained proof. Publish or merge + only within the task's authorization. + +## Completion and reporting + +Report removals and keepers, retained/deferred candidates, checks and limitations, +distinct cases versus repeated executions, and test/support versus production +LOC. Count documentation separately. Label cost evidence as measured, estimated, +or historical; a proven simplification does not require a measurable speedup. + +State the actual endpoint: findings, validated local patch, published PR, +intermediate merge, or landed in the intended target. Stop at the requested +scope/time limit or completed delivery, not a deletion quota. For another pass, +refresh the base and prior findings; revisit retained tests only with new evidence. diff --git a/.agents/skills/test-audit/agents/openai.yaml b/.agents/skills/test-audit/agents/openai.yaml new file mode 100644 index 00000000000..9987fcc761e --- /dev/null +++ b/.agents/skills/test-audit/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Test Audit" + short_description: "Prune redundant tests while preserving useful coverage" + default_prompt: "Use $test-audit to garden this test suite and remove well-supported redundancy while preserving meaningful regression coverage." diff --git a/.agents/skills/test-audit/references/audit.md b/.agents/skills/test-audit/references/audit.md new file mode 100644 index 00000000000..e821e2f4ec1 --- /dev/null +++ b/.agents/skills/test-audit/references/audit.md @@ -0,0 +1,72 @@ +# Candidate evidence and retention + +Use this reference before deleting or consolidating tests. + +## Search for waste, then inspect the contract + +| Candidate pattern | What to establish before changing it | +| --- | --- | +| Several tests repeat one assertion through the same production path | A retained case has equivalent inputs, failure sensitivity, and relevant setup. | +| A test file only imports another test file | The real owner is selected in the same required runs; the alias adds no setup, configuration, or separate entry point. Check both CI and documented commands. | +| A large table repeats equivalent rows | Preserve rows that exercise different branches, boundary values, error classes, platforms, or defaults. | +| A bulk fixture creates many interchangeable records | Establish that the count does not cross batching, pagination, concurrency, or transaction thresholds. Retain mixed outcomes and persisted state assertions. | +| Assertions copy lists, exports, strings, or source shapes | Determine whether the list or shape is an independent public, packaging, generated, or architecture contract. | +| Mocks implement the behavior being asserted | Find a test that exercises the responsible production behavior; otherwise repair or relocate this proof before removing it. | +| Fixtures fabricate receipts, ordering, or persisted state | Verify that the responsible production path creates that evidence; fixture-supplied outcomes cannot prove it does. | +| A negative case passes for an unrelated reason | Reach the named guard with otherwise valid inputs. Fix useful coverage rather than discarding it as waste. | +| A test-only export, reset hook, or wrapper becomes unused | Search runtime callers, package exports, generated callers, supported integrations, and history. File location alone does not make it dead production code. | + +Also inspect self-comparisons, assertions with expected values derived from the +subject, probes with no checks, and names that promise more than the body tests. +Explicitly assertion-free smoke tests can still protect module loading or +execution; identify that purpose before deleting them. + +## Reasons to retain apparently similar tests + +Two tests can use the same payload while protecting different failure modes: + +- Authorization and tenant isolation at separate exposed entry points. +- Direct API versus SDK/MCP transport, including serialization and errors. +- Initial execution versus replay, recovery, restart, or concurrent access. +- In-memory return values versus committed database state or rollback. +- Default-off behavior versus explicit opt-in; missing versus invalid input. +- Platform behavior, published package exports, migrations, generated-language + compatibility, or intentional prompt/protocol bytes. +- Browser interaction, reconnect, pagination, lost responses, attachment + persistence, or visible ordering absent from lower-level coverage. + +Keep a static check when it is an independent and appropriate guard of a +contract. Implementation-shaped checks that fail on a harmless identifier +rename deserve scrutiny; this does not make all source inspection disposable. +Likewise, a slow integration test can be the only meaningful proof of a +transaction or lifecycle. Cost helps rank investigation, not determine truth. + +## Evidence before editing + +Read the candidate's full body and parameter rows, setup/teardown, production +owner, real callers, proposed keeper, configuration, and relevant history. +Look for why the test was introduced and regressions added since. Inspect the +dependency's actual contract when a claim depends on library behavior. + +Use a compact table or PR notes: + +| Candidate | Decision and reason | Contract/regression | Keeper or retirement evidence | Validation and benefit | +| --- | --- | --- | --- | --- | +| File and case/rows | Retain / repair / consolidate / delete / defer | Observable failure detected | Exact file/case, or evidence the contract no longer exists | Commands/control, cost or maintenance reduction, limitations | + +For consolidation, list unique assertions to move. The keeper must actually +exercise them with valid fixtures; adding an assertion that cannot fail is +not coverage preservation. If setup or the selected test project differs, +explain why it does not change the protected contract. + +## Two useful distinctions + +- An import-only alias and its directly selected owner may duplicate execution. + An identical-looking alias importing a package suite absent from the default + run may be the only entry point and must stay until routing is repaired. +- Repeated successful records in a bulk route fixture may be reducible while + retaining mixed failures and persisted outcomes. The same reduction would + invalidate a test whose record count exercises a batch or pagination limit. + +If evidence is missing, defer that candidate and keep working on independent +ones. Audit-only findings do not become permission to delete tests elsewhere. diff --git a/.agents/skills/test-audit/references/paperclip.md b/.agents/skills/test-audit/references/paperclip.md new file mode 100644 index 00000000000..56d916fe26f --- /dev/null +++ b/.agents/skills/test-audit/references/paperclip.md @@ -0,0 +1,71 @@ +# Paperclip integration + +This reference adapts the standalone workflow to this repository. Read root +and scoped `AGENTS.md` and their required product/development documents. Resolve +paths below from the repository root and inspect current contents; counts and +shard layouts evolve. + +## Discovery and test selection + +- `package.json`, `vitest.config.ts`, and package-local Vitest configurations + describe the test projects and commands. +- `scripts/run-vitest-stable.mjs` selects and runs the default suites. Inspect + its supported dry-run/mode/shard flags before constructing routing checks. +- `.github/workflows/pr.yml` identifies the PR workflow; follow its reusable + workflow references to the actual lanes and shard matrix. +- `scripts/general-server-shard.mjs` and the general-server, serialized, and + E2E duration JSON files under `scripts/` provide routing and historical cost + clues. Read each manifest's provenance; a file's weight is not per-case time. +- Search tests colocated with production modules as well as `__tests__/`. + Some acceptance files import other suites. Verify both direct selection and + aliases, including package suites outside the default project list. + +Example focused command, replacing the example path with the actual owner: + +```sh +pnpm --filter @paperclipai/server exec vitest run src/services/OWNER.test.ts +``` + +`pnpm test` is the normal Vitest entry point. Browser and release-smoke suites +are opt-in; run them when affected or explicitly verifying those flows. +Follow repository requirements for a PR-ready handoff, including the full +typecheck/test/build checks when required; identify local versus CI execution +and report anything unrun. Do not rerun the entire repository on every audit +heartbeat. Test removals do not excuse required checks. + +## Contracts deserving special care + +Preserve company isolation, board/agent authorization, atomic checkout, +single-assignee rules, approval gates, budget hard stops, and mutation activity +logging. Persistence, idempotency, cancellation, concurrency, recovery, and +ordering often have separate owners even when test names look similar. + +Distinguish first-party Telemetry, opt-in OpenTelemetry observability, and the +local run log by their paths as defined in `AGENTS.md`. A pruning task is not +permission to remove privacy, endpoint gating, or event-contract coverage. + +## Environment and delivery + +Database-backed tests may start real embedded Postgres even though development +can use PGlite. Check the actual fixture. An isolated install with scripts +disabled may need documented dependency setup before its binaries work. +Resource exhaustion must not be interpreted as a tested product failure or a +passing control. Coordinate heavy runs and clean up only owned resources. + +Follow existing branch/worktree conventions and protect unrelated working-tree +edits. New repository audit plans/receipts belong in +`doc/plans/YYYY-MM-DD--test-audit.md`. Prefer one concise receipt per +coherent batch or campaign. For Paperclip issue work, follow the issue's artifact +and work-product requirements rather than relying only on local paths. + +When publishing is authorized, fill every section of +`.github/PULL_REQUEST_TEMPLATE.md`, including exact model information that is +actually known; do not fabricate context-window details. Use an accurate PR +title consistent with current repository policy. A deletion-only cleanup may +be a refactor and does not need a token added test to satisfy a title heuristic. + +When merging is authorized, verify current-head CI and fresh repository review +results, inspect review threads and edited bot summaries, and resolve genuine +findings. After a stack merge, verify inclusion in the intended final branch +as described in [CAMPAIGN.md](../CAMPAIGN.md). No fixed check count or old +review score substitutes for inspecting the current requirements. diff --git a/.agents/skills/test-audit/references/validation.md b/.agents/skills/test-audit/references/validation.md new file mode 100644 index 00000000000..8a86e6452ea --- /dev/null +++ b/.agents/skills/test-audit/references/validation.md @@ -0,0 +1,87 @@ +# Preservation, cost, and evidence + +## Prove the retained behavior + +Pin baseline and candidate revisions and record the actual command, project, +filters, environment differences, and pass/fail/skip counts. Prefer focused +owner and affected sibling runs. A zero-test run, setup error, or fully skipped +suite is not a successful baseline. Use recent CI only when its revision and +test selection support the claim being made. + +For straightforward exact duplicates, compare all assertions, fixtures, +configuration, and selection. For assertion migration, weaker-to-stronger +coverage substitution, security/persistence boundaries, or uncertain failure +sensitivity, use a deliberate control: + +1. Run the keeper normally and observe it passing. +2. In an isolated, idle checkout, temporarily introduce the specific plausible + production regression the keeper is meant to detect, or revert the relevant + fix. Keep unrelated guards and fixtures valid. +3. Verify the expected case fails at the intended assertion. A timeout, missing + dependency, unrelated rejection, or initialization failure is not proof. +4. Restore the exact pre-control bytes in a guaranteed cleanup step. Check the + diff/hash and run the same selected case successfully again. +5. Run the affected candidate owners without that narrow name filter before + calling the complete owner suite green. + +Never mutate a checkout with active tests, commit a temporary mutation, or +overwrite someone else's source changes during restoration. Do not add a +permanent replacement test solely to justify removing an obvious duplicate. + +## Verify what actually runs + +Inspect the real runner entry point, include/exclude rules, package projects, +shard routing, dedicated CI lanes, and documented focused commands. A test +existing on disk does not guarantee that it executes in the required run. +The required invocations are the affected CI lanes and supported developer +commands for the audit's scope. Redirecting a documented alias command to its +owner is valid when the configuration and protected contracts remain equivalent; +preservation does not require retaining an obsolete filename indefinitely. + +When removing test-import aliases, prove that the retained owners remain +selected with the intended multiplicity in each required invocation. Update +explicit commands and ownership inventories. Use dry-run routing to inspect +selection and an actual focused run to confirm execution where import caching +or collector behavior makes listing ambiguous. Intentional runs on different +platforms/configurations are separate contracts, not duplicate registrations. + +## Measure the benefit honestly + +Keep these quantities separate: + +| Quantity | Interpretation | +| --- | --- | +| Deleted source declarations / expanded table cases | Change in test code or case inventory. | +| Avoided registrations / suite invocations | Repeated execution eliminated; may remove no unique contract. | +| Test/support LOC | Maintenance reduction, not a runtime estimate. Report additions and net change. | +| Production LOC | Runtime code changed, with its own justification; do not combine it with test LOC. | +| Historical manifest weight | Scheduling estimate at its recorded revision/date, often including setup/import cost. | +| Case time / owner suite time / process time | Different scopes; compare like with like. | +| Shard time / workflow critical path | Scheduling and parallelism effects; summing suite savings does not predict elapsed CI savings. | + +Use bounded matched baseline/candidate runs when a quantified runtime claim is +important. Keep environment and configuration comparable, alternate order when +practical, and disclose sample count and variation. Inspect neighboring suites +for host-wide drift. A faster unrelated shard is not deletion evidence; a slower +whole suite does not establish that removing a duplicate caused a regression. + +Logs that only print slow cases provide censored observations: an absent case +is below the reporting threshold, not zero milliseconds. A passing retry is +validation evidence, not a fair speed comparison with the failed attempt. +Report uncertain speed effects as inconclusive and keep structural benefits +separate. Do not demand significance testing for every safe maintenance edit. + +## Failures and durable reporting + +Classify failures before acting: product assertion, test defect, setup/dependency, +host capacity, or CI infrastructure. Fix relevant defects or report the blocker; +do not weaken assertions, increase budgets, skip suites, or erase a failing +baseline to obtain a green audit. Retry a demonstrated transient problem within +a stated bound; a recurring failure needs investigation. + +Use PR notes for small changes. For a campaign, keep a compact dated receipt +with revisions, decisions, keepers, commands, results, controls, measurements, +and limitations. Link accessible CI runs/artifacts when they support claims. +Temporary paths and hashes alone do not make evidence durably inspectable. +Do not generate large evidence dossiers for trivial deletions or store raw +logs containing secrets. State precisely which broad checks remain unrun.