feat(dev): add incremental local verification and pass reuse (#19345)

## Thinking Path

> - Paperclip coordinates agents that make and verify repository
changes.
> - Local verification currently defaults to the complete Vitest suite.
> - Contributor instructions also require broad local checks at each PR
handoff.
> - CI already owns conservative affected-test selection and broad
final-revision checks.
> - This pull request adds local selection and pass reuse from those
existing contracts.
> - Agents can get focused feedback and use CI to close the remaining
coverage gaps.

## Linked Issues or Issue Description

**Subsystem affected**

Developer verification scripts and contributor guidance. Related
selection work: #18398 and #18464.

**Problem or motivation**

An agent that repeats an unchanged local verification step starts the
complete suite again. The handoff guidance also duplicates broad CI
coverage locally.

**Proposed solution**

Add `pnpm verify` for the actual working tree. Select affected Vitest
specifications with the CI task policy and dependency graph. Reuse
successful checks whose inputs are modeled. Run opaque checks fresh.
Report exit 2 when broader verification is required. Update the
contributor contract to accept applicable final-revision CI coverage.

**Alternatives considered**

Plain `vitest related` omits filesystem and opaque dependencies.
Commit-only cache keys miss dirty files. Running full local suites at
every handoff repeats CI work.

**Roadmap alignment**

This is repository development tooling. It does not change the product
roadmap.

## What Changed

- Add `pnpm verify`, `--explain`, `--base`, `--full`, and `--no-cache`.
- Reuse CI selection and exact runner ownership for dirty, staged,
committed, deleted, renamed, and untracked inputs.
- Keep native server suites and serialized suites in their local owners.
Keep six sequential chat shards.
- Cache selected passes within one worktree. Hash source, fixtures,
installed package contents, dependency bindings and installation
metadata, dotenv files, child environment, toolchains, platform, and
selection.
- Run opaque consumers fresh. Preserve failed, interrupted,
mutated-input, and partial-coverage outcomes.
- Update AGENTS.md, CONTRIBUTING.md, and DEVELOPING.md guidance. Record
the shared `verify` command as the sole exception to the package-script
rule. Register the new tests in the existing CI contract-test command.
- Add optional root/environment inputs to shared graph helpers. Existing
CI defaults stay the same.

## Verification

- Fourteen new integration tests pass locally across focused runs with
real Vitest selection and execution; CI colors are covered.
- 51 existing targeted selector/input, project-isolation, and
stable-runner checks passed. Four relevant runner contracts also pass
after the workspace change.
- Repeated unchanged execution skips reusable tests. Source,
environment, installed code, dependency bindings, and new untracked
tests invalidate results.
- Installed-code mutation with a restored modification time still
invalidates a cached pass. Internal skill-directory links preserve
reuse.
- Workspace B/C tests prove discovery, reusable/fresh partition
execution, pass reuse, and external-input failure. Chat selection
produces exactly six single-worker partitions.
- A deliberate unrelated failing test is omitted by narrow selection and
caught by `--full`.
- Red/green regressions prove fingerprints preserve file boundaries and
reject edits made between diff discovery and snapshotting.
- Opaque external-file changes still fail on a subsequent run. Source
mutation and process interruption cannot publish reusable passes.
- `pnpm verify --explain` works in the checkout and requests broad CI
coverage for this tooling change. No duplicate full local suite was
started.
- [Final-revision
CI](https://github.com/paperclipai/paperclip/actions/runs/42117291789)
passed on `ed85d96ddff82c05c325948680b593b0f48c473d` (attempt 2). The
scoped Canary retry preserved all passing test/build results.
- [Greptile
review](https://github.com/paperclipai/paperclip/pull/19345#issuecomment-6670050175)
is 5/5; all four review threads are resolved.
- The additional Paperclip review concluded neutral because its
execution stopped; it produced no assessment.

## Risks

- Pass reuse depends on conservative dependency classification. Opaque
consumers, external database/live settings, and non-isolated projects
retain fresh or broad coverage.
- Local passes cover selected Vitest checks. Required CI,
build/typecheck, release, browser, and platform acceptance remain
separate obligations.
- Cross-worktree caching and cross-agent scheduling are outside this
change.
- The initial installation scan reads dependency contents;
stat-validated, worktree-local digest memoization avoids rereading
unchanged contents. Subsequent scans still inspect file identity and
nanosecond change/modification times.
- No dependency lockfile or workflow changes.

## Model Used

OpenAI Codex, `gpt-6.1-sol`, max reasoning, with local shell and file
tools. Context window size is not exposed by this host.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

scripts/local-verify/inputs.mjs

@@ -0,0 +1,96 @@
+import { execFileSync } from "node:child_process";
+import { createHash } from "node:crypto";
+import { lstatSync, readFileSync, realpathSync, readlinkSync } from "node:fs";
+import path from "node:path";
+import { sourceFiles } from "../ci/execution-plan.mjs";
+import { productInputs } from "../../.github/scripts/ci-inputs.mjs";
+import { installedInputs } from "./installed-inputs.mjs";
+
+export const protocolVersion = 1;
+
+export function git(root, ...args) {
+  return execFileSync("git", args, { cwd: root, encoding: "utf8", maxBuffer: 32 * 1024 * 1024,
+    stdio: ["ignore", "pipe", "pipe"] });
+}
+
+export function localEnvironment(env = process.env) {
+  const result = { ...env };
+  delete result.GITHUB_WORKFLOW;
+  delete result.PAPERCLIP_CI_TEST_SELECTION;
+  delete result.PAPERCLIP_CI_SELECTION_HASH;
+  return result;
+}
+
+export function comparison(root, requestedBase) {
+  let base;
+  for (const ref of requestedBase ? [requestedBase] : ["origin/master", "master"]) {
+    try {
+      const commit = git(root, "rev-parse", "--verify", "--end-of-options", `${ref}^{commit}`).trim();
+      base = git(root, "merge-base", "HEAD", commit).trim();
+      break;
+    } catch {}
+  }
+  if (!base) throw new Error("Cannot resolve a base. Pass --base <ref>.");
+  const changed = [...new Set([
+    ...git(root, "diff", "--name-only", "--no-renames", "-z", base, "--").split("\0"),
+    ...git(root, "diff", "--cached", "--name-only", "--no-renames", "-z", base, "--").split("\0"),
+    ...git(root, "ls-files", "--others", "--exclude-standard", "-z").split("\0"),
+  ].filter(Boolean))].sort();
+  return { base, head: git(root, "rev-parse", "HEAD").trim(), changed };
+}
+
+function version(root, env, command, args) {
+  try { return execFileSync(command, args, { cwd: root, env, encoding: "utf8", stdio: ["ignore", "pipe", "pipe"] }).trim(); }
+  catch { return "unavailable"; }
+}
+
+export function snapshot(root, env = localEnvironment()) {
+  root = realpathSync(root);
+  const files = sourceFiles(root);
+  const hash = createHash("sha256");
+  const append = record => hash.update(`${JSON.stringify(record)}\n`);
+  const digest = file => createHash("sha256").update(readFileSync(file)).digest("hex");
+  let cacheable = true;
+  const addFile = file => {
+    const absolute = path.join(root, file);
+    try {
+      const stat = lstatSync(absolute);
+      if (stat.isSymbolicLink()) {
+        const target = realpathSync(absolute);
+        if (!target.startsWith(`${root}${path.sep}`)) cacheable = false;
+        const targetStat = lstatSync(target), regular = targetStat.isFile();
+        if (!regular && !targetStat.isDirectory()) cacheable = false;
+        append(["link", file, stat.mode, readlinkSync(absolute), target, regular ? digest(target) : "directory"]);
+      } else if (stat.isFile()) append(["file", file, stat.mode, digest(absolute)]);
+      else { cacheable = false; append(["special", file, stat.mode]); }
+    } catch (error) {
+      if (error.code !== "ENOENT") throw error;
+      append(["missing", file]);
+    }
+  };
+  productInputs(files).forEach(addFile);
+  const directories = new Set(["", ...files.filter(file => path.basename(file) === "package.json").map(path.dirname)]);
+  for (const directory of [...directories].sort()) {
+    for (const name of [".env", ".env.local", ".env.test", ".env.test.local"]) addFile(path.join(directory, name));
+  }
+  installedInputs(root, [...directories].sort(), append);
+  for (const file of ["node_modules/.modules.yaml", "node_modules/.pnpm/lock.yaml"]) addFile(file);
+  const environment = Object.fromEntries(Object.keys(env).sort().map(name => [name, env[name]]));
+  if (Object.entries(env).some(([name, value]) => value?.trim() && (
+    name === "DATABASE_URL" || name === "PAPERCLIP_TEST_POSTGRES_ADMIN_URL"
+    || /^PAPERCLIP_.*_TEST_DATABASE_URL$/.test(name)
+    || /^(?:PAPERCLIP|TEST)_.*(?:LIVE|E2E_LIVE)$/.test(name) && /^(?:1|true)$/i.test(value)
+  ))) cacheable = false;
+  const context = { version: protocolVersion, root, node: process.version, executable: realpathSync(process.execPath),
+    platform: process.platform, arch: process.arch, environment,
+    pnpm: version(root, env, "pnpm", ["--version"]),
+    rust: version(root, env, "rustc", ["--version", "--verbose"]),
+    cargo: version(root, env, "cargo", ["--version"]) };
+  append(["context", context]);
+  return { hash: hash.digest("hex"), cacheable };
+}
+
+export function requestKey(inputs, diff, full) {
+  return createHash("sha256").update(JSON.stringify({ version: protocolVersion, inputs: inputs.hash,
+    base: diff.base, changed: diff.changed, full })).digest("hex");
+}

scripts/local-verify/installed-inputs.mjs

@@ -0,0 +1,59 @@
+import { createHash, randomUUID } from "node:crypto";
+import { lstatSync, mkdirSync, readdirSync, readFileSync, readlinkSync, renameSync, writeFileSync } from "node:fs";
+import path from "node:path";
+
+const caches = new Map();
+
+export function installedInputs(root, directories, append) {
+  const cacheFile = path.join(root, ".turbo/local-verify/installed-digests.json");
+  if (!caches.has(root)) {
+    let entries = [];
+    try {
+      const saved = JSON.parse(readFileSync(cacheFile, "utf8"));
+      if (saved.version === 1) entries = Object.entries(saved.digests).filter(([, hash]) => /^[a-f0-9]{64}$/.test(hash));
+    } catch {}
+    caches.set(root, new Map(entries));
+  }
+  const digests = caches.get(root), current = new Map();
+  let dirty = false;
+  const visited = new Set();
+  const workspaces = new Set(directories.map(directory => path.join(root, directory)));
+  const visit = absolute => {
+    if (visited.has(absolute)) return;
+    visited.add(absolute);
+    let stat;
+    try { stat = lstatSync(absolute, { bigint: true }); }
+    catch (error) {
+      if (error.code !== "ENOENT") throw error;
+      append(["installed-missing", absolute]);
+      return;
+    }
+    if (stat.isSymbolicLink()) {
+      const target = readlinkSync(absolute);
+      append(["installed-link", absolute, String(stat.mode), target]);
+      visit(path.resolve(path.dirname(absolute), target));
+    } else if (stat.isDirectory()) {
+      if (workspaces.has(absolute)) { visit(path.join(absolute, "node_modules")); return; }
+      const names = readdirSync(absolute).sort().filter(name => path.basename(absolute) !== "node_modules"
+        || ![".vite", ".vite-temp", ".cache"].includes(name));
+      append(["installed-directory", absolute, String(stat.mode), names]);
+      for (const name of names) visit(path.join(absolute, name));
+    } else if (stat.isFile()) {
+      const identity = [stat.dev, stat.ino, stat.size, stat.mtimeNs, stat.ctimeNs, stat.mode].join(":");
+      if (!digests.has(identity)) {
+        digests.set(identity, createHash("sha256").update(readFileSync(absolute)).digest("hex"));
+        dirty = true;
+      }
+      current.set(identity, digests.get(identity));
+      append(["installed-file", absolute, String(stat.mode), digests.get(identity)]);
+    } else throw new Error(`Unsupported installed input: ${absolute}`);
+  };
+  for (const directory of directories) visit(path.join(root, directory, "node_modules"));
+  if (dirty || current.size !== digests.size) {
+    mkdirSync(path.dirname(cacheFile), { recursive: true });
+    const temporary = `${cacheFile}.${randomUUID()}.tmp`;
+    writeFileSync(temporary, JSON.stringify({ version: 1, digests: Object.fromEntries(current) }), { mode: 0o600 });
+    renameSync(temporary, cacheFile);
+  }
+  caches.set(root, current);
+}

scripts/local-verify/plan.mjs

@@ -0,0 +1,85 @@
+import { execFileSync } from "node:child_process";
+import { readFileSync } from "node:fs";
+import path from "node:path";
+import { planTasks } from "../../.github/scripts/ci-plan.mjs";
+import { productInputs } from "../../.github/scripts/ci-inputs.mjs";
+import { selectionEligibility } from "../ci/select-tests.mjs";
+
+const chatSuite = "server/src/__tests__/chat-channels.integration.test.ts";
+
+function inventory(root, env, group) {
+  const args = group === "serialized" ? ["--mode", "serialized"] : ["--mode", "general", "--group", group];
+  const shards = ["general-workspaces-b", "general-workspaces-c"].includes(group) ? []
+    : ["--shard-index", "0", "--shard-count", "1"];
+  return JSON.parse(execFileSync(process.execPath, [path.join(root, "scripts/run-vitest-stable.mjs"), ...args,
+    ...shards, "--dry-run"], {
+    cwd: root, env, encoding: "utf8", maxBuffer: 16 * 1024 * 1024,
+  }));
+}
+
+export async function planVerification({ root, env, diff, full = false }) {
+  const config = JSON.parse(readFileSync(path.join(root, "turbo.json"), "utf8"));
+  const lanes = JSON.parse(readFileSync(path.join(root, ".github/ci-task-matrix.json"), "utf8"));
+  const coarse = planTasks({ config, lanes, paths: diff.changed });
+  const requiredCI = diff.changed.length ? Object.entries(coarse.jobs).filter(([, job]) => job.run).map(([name]) => name) : [];
+  if (full) return { kind: "full", jobs: [{ id: "full", args: null, cacheable: false }], requiredCI, selectedFiles: null, reason: "Full Vitest suite explicitly requested; opaque integration coverage runs fresh" };
+  if (!productInputs(diff.changed).length) return { kind: "selected", jobs: [], requiredCI, selectedFiles: [], reason: "No changed product inputs" };
+  const reason = coarse.forceReason ?? selectionEligibility(coarse, root);
+  if (reason) return { kind: "broad", reason, jobs: [], requiredCI };
+  try {
+    const groups = [...new Set(["general_tests", "verify_serialized_server"].flatMap(job =>
+      coarse.jobs[job]?.run ? coarse.jobs[job].matrix.include.map(entry => entry.group ?? "serialized") : []))];
+    const inventories = new Map(groups.filter(group => group !== "general-chat").map(group => [group, inventory(root, env, group)]));
+    const projects = [...new Set([...inventories.values()].flatMap(value => value.workspaceProjects ?? ["@paperclipai/server"]))];
+    const { selectProjects } = await import("../ci/select-projects.mjs");
+    const graphs = await selectProjects({ root, projects, changed: productInputs(diff.changed), concurrency: 1, env });
+    if (graphs.some(graph => !graph.isolated)) throw new Error("Non-isolated projects require broad verification");
+    const covered = new Set(graphs.flatMap(graph => graph.coveredChanges));
+    const uncovered = productInputs(diff.changed).filter(file => !covered.has(file));
+    if (uncovered.length) throw new Error(`Changes outside resolved test graphs: ${uncovered.join(", ")}`);
+    const byProject = new Map(graphs.map(graph => [graph.project, graph]));
+    const manifest = { version: 1, head: diff.head, lanes: {} }, jobs = [], selectedFiles = new Set();
+    for (const group of groups) {
+      if (group === "general-chat") {
+        if (byProject.get("@paperclipai/server")?.selected.includes(chatSuite)) {
+          selectedFiles.add(chatSuite);
+          for (let index = 0; index < 6; index++) jobs.push({ id: `chat-${index + 1}`, cacheable: false, args: ["--mode", "general", "--group", group,
+            "--shard-index", String(index), "--shard-count", "6", "--serial-chat"] });
+        }
+        continue;
+      }
+      const value = inventories.get(group);
+      const owners = value.workspaceProjects
+        ? Object.fromEntries(value.workspaceProjects.map(project => [project, byProject.get(project).inventory]))
+        : { "@paperclipai/server": group === "serialized" ? value.selectedSerializedSuites : value.selectedGeneralServerSuites };
+      const reusable = {}, fresh = {};
+      for (const [project, files] of Object.entries(owners)) {
+        const graph = byProject.get(project), all = new Set(graph.inventory), affected = new Set(graph.selected);
+        if (!Array.isArray(files) || files.some(file => !all.has(file))) throw new Error(`Runner/graph inventory mismatch: ${group}`);
+        const matches = files.filter(file => affected.has(file));
+        const opaque = new Set(graph.alwaysRun);
+        const pure = matches.filter(file => !opaque.has(file)), retained = matches.filter(file => opaque.has(file));
+        if (pure.length) reusable[project] = pure;
+        if (retained.length) fresh[project] = retained;
+        matches.forEach(file => selectedFiles.add(file));
+      }
+      const partitions = [reusable, fresh].filter(files => Object.keys(files).length);
+      if (!partitions.length) continue;
+      manifest.lanes[group] = { shards: partitions };
+      for (const [index, files] of partitions.entries()) jobs.push({ id: `${group}-${files === reusable ? "reusable" : "fresh"}`,
+        cacheable: files === reusable, selected: true, args: ["--mode", group === "serialized" ? "serialized" : "general",
+          ...(group === "serialized" ? [] : ["--group", group]), "--shard-index", String(index), "--shard-count", String(partitions.length)] });
+    }
+    const expected = new Set(graphs.flatMap(graph => graph.selected));
+    if (expected.size !== selectedFiles.size || [...expected].some(file => !selectedFiles.has(file))) {
+      throw new Error("Selected tests are not completely covered by local runner ownership");
+    }
+    const { validateSelection } = await import("../ci/test-selection-manifest.mjs");
+    validateSelection(manifest);
+    return { kind: "selected", jobs, manifest, requiredCI, selectedFiles: [...selectedFiles].sort(),
+      totalFiles: graphs.reduce((count, graph) => count + graph.inventory.length, 0),
+      retainedFiles: graphs.reduce((count, graph) => count + graph.alwaysRun.length, 0) };
+  } catch (error) {
+    return { kind: "broad", reason: `Conservative selection unavailable: ${error.message}`, jobs: [], requiredCI };
+  }
+}

scripts/local-verify/run.mjs

@@ -0,0 +1,86 @@
+import { spawn } from "node:child_process";
+import { mkdirSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs";
+import path from "node:path";
+import { createHash, randomUUID } from "node:crypto";
+import { selectionHash } from "../ci/test-selection-manifest.mjs";
+import { protocolVersion } from "./inputs.mjs";
+
+export const cacheDirectory = root => path.join(root, ".turbo/local-verify");
+const planHash = plan => createHash("sha256").update(JSON.stringify(plan)).digest("hex");
+
+export function readPass(root, key) {
+  try {
+    const result = JSON.parse(readFileSync(path.join(cacheDirectory(root), `${key}.json`), "utf8"));
+    return result.version === protocolVersion && result.key === key && result.status === "passed" && result.planHash === planHash(result.plan)
+      && ["selected", "full"].includes(result.plan?.kind) && Array.isArray(result.plan.jobs) ? result : null;
+  } catch { return null; }
+}
+
+export function readPlan(root, key) {
+  try {
+    const result = JSON.parse(readFileSync(path.join(cacheDirectory(root), `plan-${key}.json`), "utf8"));
+    return result.version === protocolVersion && result.key === key && result.status === "planned" && result.planHash === planHash(result.plan)
+      && result.plan?.kind === "selected" && Array.isArray(result.plan.jobs) ? result.plan : null;
+  } catch { return null; }
+}
+
+export function mayReuseAttempt(root, key) {
+  try {
+    const result = JSON.parse(readFileSync(path.join(cacheDirectory(root), `attempt-${key}.json`), "utf8"));
+    return result.version === protocolVersion && result.key === key && result.status === "passed";
+  } catch (error) { return error.code === "ENOENT"; }
+}
+
+export function writeRecord(root, name, record) {
+  const directory = cacheDirectory(root);
+  mkdirSync(directory, { recursive: true });
+  const temporary = path.join(directory, `${name}.${randomUUID()}.tmp`);
+  writeFileSync(temporary, JSON.stringify({ ...record, planHash: planHash(record.plan) }, null, 2), { mode: 0o600 });
+  renameSync(temporary, path.join(directory, `${name}.json`));
+}
+
+export const invalidatePass = (root, key) => rmSync(path.join(cacheDirectory(root), `${key}.json`), { force: true });
+
+async function execute(command, args, root, env, label) {
+  return new Promise(resolve => {
+    const grouped = process.platform !== "win32";
+    const child = spawn(command, args, { cwd: root, env, stdio: "inherit", detached: grouped });
+    let stopped;
+    let finished = false;
+    const interrupt = signal => {
+      stopped = signal;
+      try {
+        if (grouped && child.pid) process.kill(-child.pid, signal);
+        else child.kill(signal);
+      } catch (error) { if (error.code !== "ESRCH") throw error; }
+    };
+    const onInterrupt = () => interrupt("SIGINT"), onTerminate = () => interrupt("SIGTERM");
+    process.on("SIGINT", onInterrupt); process.on("SIGTERM", onTerminate);
+    const finish = code => {
+      if (finished) return;
+      finished = true;
+      process.off("SIGINT", onInterrupt); process.off("SIGTERM", onTerminate);
+      resolve(stopped ? stopped === "SIGINT" ? 130 : 143 : code);
+    };
+    child.once("error", error => { console.error(error.message); finish(1); });
+    child.once("close", code => finish(code ?? 1));
+    console.log(`[verify] Running ${label}`);
+  });
+}
+
+export async function runPlan(root, env, plan) {
+  const directory = cacheDirectory(root);
+  mkdirSync(directory, { recursive: true });
+  const manifestFile = path.join(directory, `selection-${randomUUID()}.json`);
+  if (plan.manifest) writeFileSync(manifestFile, JSON.stringify(plan.manifest));
+  try {
+    for (const job of plan.jobs) {
+      const code = job.args === null ? await execute("pnpm", ["test:run"], root, env, job.id)
+        : await execute(process.execPath, [path.join(root, "scripts/run-vitest-stable.mjs"), ...job.args], root,
+          job.selected ? { ...env, PAPERCLIP_CI_TEST_SELECTION: manifestFile,
+            PAPERCLIP_CI_SELECTION_HASH: selectionHash(plan.manifest) } : env, job.id);
+      if (code !== 0) return code;
+    }
+    return 0;
+  } finally { rmSync(manifestFile, { force: true }); }
+}