mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 07:10:34 +01:00
feat(dev): add incremental local verification and pass reuse (#19345)
## Thinking Path > - Paperclip coordinates agents that make and verify repository changes. > - Local verification currently defaults to the complete Vitest suite. > - Contributor instructions also require broad local checks at each PR handoff. > - CI already owns conservative affected-test selection and broad final-revision checks. > - This pull request adds local selection and pass reuse from those existing contracts. > - Agents can get focused feedback and use CI to close the remaining coverage gaps. ## Linked Issues or Issue Description **Subsystem affected** Developer verification scripts and contributor guidance. Related selection work: #18398 and #18464. **Problem or motivation** An agent that repeats an unchanged local verification step starts the complete suite again. The handoff guidance also duplicates broad CI coverage locally. **Proposed solution** Add `pnpm verify` for the actual working tree. Select affected Vitest specifications with the CI task policy and dependency graph. Reuse successful checks whose inputs are modeled. Run opaque checks fresh. Report exit 2 when broader verification is required. Update the contributor contract to accept applicable final-revision CI coverage. **Alternatives considered** Plain `vitest related` omits filesystem and opaque dependencies. Commit-only cache keys miss dirty files. Running full local suites at every handoff repeats CI work. **Roadmap alignment** This is repository development tooling. It does not change the product roadmap. ## What Changed - Add `pnpm verify`, `--explain`, `--base`, `--full`, and `--no-cache`. - Reuse CI selection and exact runner ownership for dirty, staged, committed, deleted, renamed, and untracked inputs. - Keep native server suites and serialized suites in their local owners. Keep six sequential chat shards. - Cache selected passes within one worktree. Hash source, fixtures, installed package contents, dependency bindings and installation metadata, dotenv files, child environment, toolchains, platform, and selection. - Run opaque consumers fresh. Preserve failed, interrupted, mutated-input, and partial-coverage outcomes. - Update AGENTS.md, CONTRIBUTING.md, and DEVELOPING.md guidance. Record the shared `verify` command as the sole exception to the package-script rule. Register the new tests in the existing CI contract-test command. - Add optional root/environment inputs to shared graph helpers. Existing CI defaults stay the same. ## Verification - Fourteen new integration tests pass locally across focused runs with real Vitest selection and execution; CI colors are covered. - 51 existing targeted selector/input, project-isolation, and stable-runner checks passed. Four relevant runner contracts also pass after the workspace change. - Repeated unchanged execution skips reusable tests. Source, environment, installed code, dependency bindings, and new untracked tests invalidate results. - Installed-code mutation with a restored modification time still invalidates a cached pass. Internal skill-directory links preserve reuse. - Workspace B/C tests prove discovery, reusable/fresh partition execution, pass reuse, and external-input failure. Chat selection produces exactly six single-worker partitions. - A deliberate unrelated failing test is omitted by narrow selection and caught by `--full`. - Red/green regressions prove fingerprints preserve file boundaries and reject edits made between diff discovery and snapshotting. - Opaque external-file changes still fail on a subsequent run. Source mutation and process interruption cannot publish reusable passes. - `pnpm verify --explain` works in the checkout and requests broad CI coverage for this tooling change. No duplicate full local suite was started. - [Final-revision CI](https://github.com/paperclipai/paperclip/actions/runs/42117291789) passed on `ed85d96ddff82c05c325948680b593b0f48c473d` (attempt 2). The scoped Canary retry preserved all passing test/build results. - [Greptile review](https://github.com/paperclipai/paperclip/pull/19345#issuecomment-6670050175) is 5/5; all four review threads are resolved. - The additional Paperclip review concluded neutral because its execution stopped; it produced no assessment. ## Risks - Pass reuse depends on conservative dependency classification. Opaque consumers, external database/live settings, and non-isolated projects retain fresh or broad coverage. - Local passes cover selected Vitest checks. Required CI, build/typecheck, release, browser, and platform acceptance remain separate obligations. - Cross-worktree caching and cross-agent scheduling are outside this change. - The initial installation scan reads dependency contents; stat-validated, worktree-local digest memoization avoids rereading unchanged contents. Subsequent scans still inspect file identity and nanosecond change/modification times. - No dependency lockfile or workflow changes. ## Model Used OpenAI Codex, `gpt-6.1-sol`, max reasoning, with local shell and file tools. Context window size is not exposed by this host. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
70d23612dd
commit
2f35eb1080
15 files changed
+907
-19
No files matched your search
@@ -0,0 +1,359 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { execFileSync, spawn, spawnSync } from "node:child_process";
|
||||
import { chmodSync, cpSync, lstatSync, mkdirSync, mkdtempSync, readFileSync, realpathSync, rmSync, symlinkSync, utimesSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import test from "node:test";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { stripVTControlCharacters } from "node:util";
|
||||
import { comparison, localEnvironment, snapshot } from "../local-verify/inputs.mjs";
|
||||
import { planVerification } from "../local-verify/plan.mjs";
|
||||
|
||||
const repository = fileURLToPath(new URL("../../", import.meta.url));
|
||||
|
||||
function fixture(t) {
|
||||
const root = realpathSync(mkdtempSync(path.join(tmpdir(), "local-verify-")));
|
||||
t.after(() => rmSync(root, { recursive: true, force: true }));
|
||||
const write = (file, text) => {
|
||||
mkdirSync(path.dirname(path.join(root, file)), { recursive: true });
|
||||
writeFileSync(path.join(root, file), text);
|
||||
};
|
||||
for (const directory of ["scripts/ci", "scripts/local-verify", ".github/scripts"]) {
|
||||
cpSync(path.join(repository, directory), path.join(root, directory), { recursive: true });
|
||||
}
|
||||
for (const file of ["scripts/local-verify.mjs", "scripts/run-vitest-stable.mjs", "scripts/general-server-shard.mjs",
|
||||
"scripts/test-line-shard.mjs", "scripts/ci-profile/record.mjs", "turbo.json", ".github/ci-task-matrix.json"]) {
|
||||
write(file, readFileSync(path.join(repository, file), "utf8"));
|
||||
}
|
||||
for (const directory of ["server/src/__tests__", "server/scripts"]) mkdirSync(path.join(root, directory), { recursive: true });
|
||||
mkdirSync(path.join(root, "node_modules/.pnpm"), { recursive: true });
|
||||
for (const name of ["vitest", "oxc-parser", ".bin"]) {
|
||||
symlinkSync(path.join(repository, "node_modules", name), path.join(root, "node_modules", name), "dir");
|
||||
}
|
||||
for (const file of ["node_modules/.modules.yaml", "node_modules/.pnpm/lock.yaml"]) {
|
||||
write(file, readFileSync(path.join(repository, file), "utf8"));
|
||||
}
|
||||
write(".gitignore", "node_modules\nnode_modules/\n.turbo/\n.env\n");
|
||||
write("package.json", JSON.stringify({ name: "local-verification-fixture", private: true, type: "module", packageManager: "pnpm@9.15.4",
|
||||
scripts: { verify: "node scripts/local-verify.mjs", "test:run": "vitest run" } }));
|
||||
write("vitest.config.mjs", 'export default {test:{projects:["ui","cli"]}};');
|
||||
for (const [directory, name] of [["ui", "@paperclipai/ui"], ["cli", "paperclipai"]]) {
|
||||
write(`${directory}/vitest.config.mjs`, `export default {test:{name:${JSON.stringify(name)},include:["src/**/*.test.ts"]}};`);
|
||||
}
|
||||
write("ui/src/leaf.ts", "export const value = 1;\n");
|
||||
write("ui/src/other.ts", "export const other = 1;\n");
|
||||
write("ui/src/consumer.test.ts", 'import {test,expect} from "vitest"; import {value} from "./leaf"; test("affected consumer",()=>{expect(value).toBeGreaterThan(0);expect(process.env.LOCAL_VERIFY_FLAVOR).not.toBe("fail");});');
|
||||
write("ui/src/unrelated.test.ts", 'import {test,expect} from "vitest"; import {other} from "./other"; test("unrelated full-suite control",()=>expect(other).toBe(123456));');
|
||||
write("cli/src/independent.test.ts", 'import {test,expect} from "vitest"; test("unchanged consumer project",()=>expect(1).toBe(1));');
|
||||
const git = (...args) => execFileSync("git", args, { cwd: root, encoding: "utf8", stdio: ["ignore", "pipe", "pipe"] }).trim();
|
||||
git("init", "-q"); git("add", ".");
|
||||
git("-c", "user.name=Test", "-c", "user.email=test@example.invalid", "commit", "-qm", "baseline");
|
||||
const base = git("rev-parse", "HEAD");
|
||||
write("ui/src/leaf.ts", "export const value = 2;\n");
|
||||
const env = localEnvironment({ ...process.env, FORCE_COLOR: "0" });
|
||||
delete env.NODE_TEST_CONTEXT;
|
||||
delete env.DATABASE_URL;
|
||||
delete env.PAPERCLIP_TEST_POSTGRES_ADMIN_URL;
|
||||
for (const name of Object.keys(env)) if (/^PAPERCLIP_.*_TEST_DATABASE_URL$/.test(name)) delete env[name];
|
||||
const run = (args = [], extra = {}) => {
|
||||
const result = spawnSync(process.execPath, [path.join(root, "scripts/local-verify.mjs"), "--base", base, ...args], {
|
||||
cwd: root, env: { ...env, ...extra }, encoding: "utf8", timeout: 120_000, maxBuffer: 8 * 1024 * 1024,
|
||||
});
|
||||
assert.equal(result.error, undefined);
|
||||
return { ...result, text: stripVTControlCharacters(result.stdout + result.stderr) };
|
||||
};
|
||||
return { root, write, git, base, env, run };
|
||||
}
|
||||
|
||||
test("real selected execution reuses passes and invalidates source, environment, untracked tests and broad inputs", { timeout: 180_000 }, async t => {
|
||||
const f = fixture(t);
|
||||
const first = f.run();
|
||||
assert.equal(first.status, 0, first.text);
|
||||
assert.match(first.text, /Test Files\s+1 passed/);
|
||||
assert.match(first.text, /1\/3 Vitest test files selected/);
|
||||
assert.doesNotMatch(first.text, /unrelated full-suite control/);
|
||||
const reused = f.run();
|
||||
assert.equal(reused.status, 0, reused.text);
|
||||
assert.match(reused.text, /reusing a passing local result/);
|
||||
assert.doesNotMatch(reused.text, /Test Files/);
|
||||
const last = JSON.parse(readFileSync(path.join(f.root, ".turbo/local-verify/last-run.json"), "utf8"));
|
||||
const passFile = path.join(f.root, ".turbo/local-verify", `${last.key}.json`);
|
||||
const corrupt = JSON.parse(readFileSync(passFile, "utf8"));
|
||||
corrupt.plan.selectedFiles = [];
|
||||
writeFileSync(passFile, JSON.stringify(corrupt));
|
||||
const recovered = f.run();
|
||||
assert.equal(recovered.status, 0, recovered.text);
|
||||
assert.doesNotMatch(recovered.text, /reusing a passing local result/);
|
||||
assert.match(recovered.text, /Reusing passed general-workspaces-a-reusable/);
|
||||
f.write("ui/src/leaf.ts", "export const value = 3;\n");
|
||||
const edited = f.run();
|
||||
assert.equal(edited.status, 0, edited.text);
|
||||
assert.match(edited.text, /Test Files\s+1 passed/);
|
||||
const environment = f.run([], { LOCAL_VERIFY_FLAVOR: "different" });
|
||||
assert.equal(environment.status, 0, environment.text);
|
||||
assert.match(environment.text, /Test Files\s+1 passed/);
|
||||
const environmentFailure = f.run([], { LOCAL_VERIFY_FLAVOR: "fail" });
|
||||
assert.equal(environmentFailure.status, 1, environmentFailure.text);
|
||||
assert.match(environmentFailure.text, /affected consumer/);
|
||||
const installed = readFileSync(path.join(f.root, "node_modules/.modules.yaml"), "utf8");
|
||||
f.write("node_modules/.modules.yaml", `${installed}\n# changed installation\n`);
|
||||
const dependencies = f.run();
|
||||
assert.equal(dependencies.status, 0, dependencies.text);
|
||||
assert.match(dependencies.text, /Test Files\s+1 passed/);
|
||||
rmSync(path.join(f.root, "node_modules/vitest"));
|
||||
const brokenInstall = f.run();
|
||||
assert.equal(brokenInstall.status, 2, brokenInstall.text);
|
||||
assert.doesNotMatch(brokenInstall.text, /reusing a passing local result/);
|
||||
symlinkSync(path.join(repository, "node_modules/vitest"), path.join(f.root, "node_modules/vitest"), "dir");
|
||||
f.write("ui/src/new.test.ts", 'import {test,expect} from "vitest"; test("new untracked regression",()=>expect(false).toBe(true));');
|
||||
const untracked = f.run();
|
||||
assert.equal(untracked.status, 1, untracked.text);
|
||||
assert.match(untracked.text, /new untracked regression/);
|
||||
const failedAgain = f.run();
|
||||
assert.equal(failedAgain.status, 1, failedAgain.text);
|
||||
assert.doesNotMatch(failedAgain.text, /reusing a passing local result/);
|
||||
rmSync(path.join(f.root, "ui/src/new.test.ts"));
|
||||
f.write("ui/vitest.config.mjs", 'export default {test:{name:"@paperclipai/ui",include:["src/**/*.test.ts"],testTimeout:10000}};');
|
||||
const config = f.run();
|
||||
assert.equal(config.status, 2, config.text);
|
||||
assert.match(config.text, /Broader verification required/);
|
||||
assert.doesNotMatch(config.text, /Test Files/);
|
||||
assert.match(config.text, /vitest.config.mjs/);
|
||||
});
|
||||
|
||||
test("explicit full execution cannot reuse a passing subset; opaque checks stay fresh", { timeout: 180_000 }, async t => {
|
||||
const f = fixture(t);
|
||||
assert.equal(f.run().status, 0);
|
||||
const full = f.run(["--full"]);
|
||||
assert.equal(full.status, 1, full.text);
|
||||
assert.match(full.text, /unrelated full-suite control/);
|
||||
f.write("ui/src/consumer.test.ts", 'import {test,expect} from "vitest"; import {readFileSync} from "node:fs"; import {value} from "./leaf"; test("opaque fresh check",()=>expect(readFileSync(process.env.LOCAL_VERIFY_INPUT,"utf8")).toBe(String(value)));');
|
||||
f.git("add", "ui/src/consumer.test.ts");
|
||||
f.write(".turbo/external-input", "2");
|
||||
const input = { LOCAL_VERIFY_INPUT: path.join(f.root, ".turbo/external-input") };
|
||||
const opaque = f.run([], input);
|
||||
assert.equal(opaque.status, 0, opaque.text);
|
||||
assert.match(opaque.text, /Test Files\s+1 passed/);
|
||||
const fresh = f.run([], input);
|
||||
assert.equal(fresh.status, 0, fresh.text);
|
||||
assert.match(fresh.text, /Test Files\s+1 passed/);
|
||||
f.write(".turbo/external-input", "changed externally");
|
||||
const external = f.run([], input);
|
||||
assert.equal(external.status, 1, external.text);
|
||||
assert.match(external.text, /opaque fresh check/);
|
||||
});
|
||||
|
||||
test("working tree comparison includes commits, staging, deletion, rename and untracked files", t => {
|
||||
const f = fixture(t);
|
||||
f.git("add", "ui/src/leaf.ts");
|
||||
f.git("-c", "user.name=Test", "-c", "user.email=test@example.invalid", "commit", "-qm", "committed change");
|
||||
f.git("mv", "ui/src/other.ts", "ui/src/renamed.ts");
|
||||
rmSync(path.join(f.root, "cli/src/independent.test.ts"));
|
||||
f.write("ui/src/added.ts", "export const added = true;");
|
||||
assert.deepEqual(comparison(f.root, f.base).changed, ["cli/src/independent.test.ts", "ui/src/added.ts", "ui/src/leaf.ts", "ui/src/other.ts", "ui/src/renamed.ts"]);
|
||||
assert.throws(() => comparison(f.root, "missing-ref"), /Cannot resolve a base/);
|
||||
});
|
||||
|
||||
test("dependency installation, ignored dotenv and child environment enter the cache fingerprint", t => {
|
||||
const f = fixture(t), before = snapshot(f.root, f.env).hash;
|
||||
f.write(".env", "FEATURE=true\n");
|
||||
assert.notEqual(snapshot(f.root, f.env).hash, before);
|
||||
rmSync(path.join(f.root, ".env"));
|
||||
assert.equal(snapshot(f.root, f.env).hash, before);
|
||||
f.write("pnpm-lock.yaml", "lockfileVersion: '9.0'\n");
|
||||
assert.notEqual(snapshot(f.root, f.env).hash, before);
|
||||
assert.notEqual(snapshot(f.root, { ...f.env, NEW_TEST_FLAG: "1" }).hash, before);
|
||||
assert.equal(snapshot(f.root, { ...f.env, PAPERCLIP_TEST_POSTGRES_ADMIN_URL: "postgres://external.invalid/test" }).cacheable, false);
|
||||
});
|
||||
|
||||
test("installed dependency code changes invalidate a real passing result", { timeout: 120_000 }, t => {
|
||||
const f = fixture(t);
|
||||
const installed = "node_modules/.pnpm/clsx@1.0.0/node_modules/clsx";
|
||||
f.write(`${installed}/package.json`, '{"name":"clsx","version":"1.0.0","type":"module","exports":"./index.mjs"}');
|
||||
f.write(`${installed}/index.mjs`, "export default () => 2;\n");
|
||||
symlinkSync(path.join(f.root, installed), path.join(f.root, "node_modules/clsx"), "dir");
|
||||
f.write("ui/src/consumer.test.ts", 'import {test,expect} from "vitest"; import clsx from "clsx"; import {value} from "./leaf"; test("installed dependency",()=>expect(clsx()).toBe(value));');
|
||||
const first = f.run();
|
||||
assert.equal(first.status, 0, first.text);
|
||||
const reused = f.run();
|
||||
assert.equal(reused.status, 0, reused.text);
|
||||
assert.match(reused.text, /reusing a passing local result/);
|
||||
const moduleFile = path.join(f.root, installed, "index.mjs"), original = lstatSync(moduleFile);
|
||||
f.write(`${installed}/index.mjs`, "export default () => 0;\n");
|
||||
utimesSync(moduleFile, original.atime, original.mtime);
|
||||
const changed = f.run();
|
||||
assert.equal(changed.status, 1, changed.text);
|
||||
assert.match(changed.text, /installed dependency/);
|
||||
assert.doesNotMatch(changed.text, /reusing a passing local result/);
|
||||
});
|
||||
|
||||
test("repository skill directory links preserve reuse while target edits invalidate it", t => {
|
||||
const f = fixture(t);
|
||||
mkdirSync(path.join(f.root, ".claude/skills"), { recursive: true });
|
||||
symlinkSync("../../ui", path.join(f.root, ".claude/skills/local"), "dir");
|
||||
f.git("add", ".claude/skills/local");
|
||||
const before = snapshot(f.root, f.env);
|
||||
assert.equal(before.cacheable, true);
|
||||
f.write("ui/src/leaf.ts", "export const value = 3;\n");
|
||||
assert.notEqual(snapshot(f.root, f.env).hash, before.hash);
|
||||
});
|
||||
|
||||
test("local chat selection schedules exactly six single-worker partitions", { timeout: 120_000 }, async t => {
|
||||
const f = fixture(t);
|
||||
f.write("ui/src/leaf.ts", "export const value = 1;\n");
|
||||
f.write("vitest.config.mjs", 'export default {test:{projects:["server"]}};');
|
||||
f.write("server/vitest.config.mjs", 'export default {test:{name:"@paperclipai/server",include:["src/**/*.test.ts"]}};');
|
||||
f.write("server/src/leaf.ts", "export const value = 1;\n");
|
||||
f.write("server/src/__tests__/chat-channels.integration.test.ts", 'import {test,expect} from "vitest"; import {value} from "../leaf"; test("chat selection",()=>expect(value).toBeGreaterThan(0));');
|
||||
f.write(".github/ci-task-matrix.json", JSON.stringify({ general_tests: [
|
||||
{ task: "ci:server", group: "general-server-without-chat" }, { task: "ci:chat", group: "general-chat" },
|
||||
] }));
|
||||
f.git("add", "."); f.git("-c", "user.name=Test", "-c", "user.email=test@example.invalid", "commit", "-qm", "chat baseline");
|
||||
const base = f.git("rev-parse", "HEAD");
|
||||
f.write("server/src/leaf.ts", "export const value = 2;\n");
|
||||
const plan = await planVerification({ root: f.root, env: f.env, diff: comparison(f.root, base) });
|
||||
assert.equal(plan.kind, "selected", plan.reason);
|
||||
const shards = plan.jobs.flatMap(job => JSON.parse(execFileSync(process.execPath,
|
||||
[path.join(f.root, "scripts/run-vitest-stable.mjs"), ...job.args, "--dry-run"],
|
||||
{ cwd: f.root, env: f.env, encoding: "utf8" })).chatVitestShards);
|
||||
assert.deepEqual(shards, Array.from({ length: 6 }, (_, index) => ({ index, count: 6 })));
|
||||
});
|
||||
|
||||
for (const [group, task, primary, directory] of [
|
||||
["general-workspaces-b", "ci:workspaces-b", "@paperclipai/shared", "packages/shared"],
|
||||
["general-workspaces-c", "ci:db", "@paperclipai/db", "packages/db"],
|
||||
]) test(`${group} discovers and executes reusable and fresh selected partitions`, { timeout: 180_000 }, t => {
|
||||
const f = fixture(t);
|
||||
const inventory = JSON.parse(execFileSync(process.execPath, [path.join(f.root, "scripts/run-vitest-stable.mjs"),
|
||||
"--mode", "general", "--group", group, "--dry-run"], { cwd: f.root, env: f.env, encoding: "utf8" }));
|
||||
const projects = inventory.workspaceProjects.map((project, index) => {
|
||||
const folder = project === primary ? directory : `packages/fixture-${index}`;
|
||||
f.write(`${folder}/vitest.config.mjs`, `export default {test:{name:${JSON.stringify(project)},include:["src/**/*.test.ts"]}};`);
|
||||
f.write(`${folder}/src/independent.test.ts`, 'import {test,expect} from "vitest"; test("independent workspace",()=>expect(1).toBe(1));');
|
||||
return folder;
|
||||
});
|
||||
f.write("ui/src/leaf.ts", "export const value = 1;\n");
|
||||
f.write("vitest.config.mjs", `export default {test:{projects:${JSON.stringify(projects)}}};`);
|
||||
f.write(".github/ci-task-matrix.json", JSON.stringify({ general_tests: [{ task, group }] }));
|
||||
f.write(`${directory}/src/leaf.ts`, "export const value = 1;\n");
|
||||
f.write(`${directory}/src/reusable.test.ts`, 'import {test,expect} from "vitest"; import {value} from "./leaf"; test("workspace consumer",()=>expect(value).toBeGreaterThan(0));');
|
||||
f.write(`${directory}/src/fresh.test.ts`, 'import {test,expect} from "vitest"; import {readFileSync} from "node:fs"; import {value} from "./leaf"; test("fresh workspace consumer",()=>expect(readFileSync(process.env.LOCAL_VERIFY_INPUT,"utf8")).toBe(String(value)));');
|
||||
f.git("add", "."); f.git("-c", "user.name=Test", "-c", "user.email=test@example.invalid", "commit", "-qm", "workspace baseline");
|
||||
const base = f.git("rev-parse", "HEAD");
|
||||
f.write(`${directory}/src/leaf.ts`, "export const value = 2;\n");
|
||||
f.write(".turbo/external-input", "2");
|
||||
const env = { LOCAL_VERIFY_INPUT: path.join(f.root, ".turbo/external-input") };
|
||||
const first = f.run(["--base", base], env);
|
||||
assert.equal(first.status, 0, first.text);
|
||||
assert.equal((first.text.match(/Test Files\s+1 passed/g) ?? []).length, 2, first.text);
|
||||
const reused = f.run(["--base", base], env);
|
||||
assert.equal(reused.status, 0, reused.text);
|
||||
assert.ok(reused.text.includes(`Reusing passed ${group}-reusable`), reused.text);
|
||||
assert.equal((reused.text.match(/Test Files\s+1 passed/g) ?? []).length, 1, reused.text);
|
||||
f.write(".turbo/external-input", "changed externally");
|
||||
const changed = f.run(["--base", base], env);
|
||||
assert.equal(changed.status, 1, changed.text);
|
||||
assert.match(changed.text, /fresh workspace consumer/);
|
||||
});
|
||||
|
||||
test("fingerprints preserve file boundaries when content resembles another input record", t => {
|
||||
const f = fixture(t), a = "ui/src/hash-a.ts", b = "ui/src/hash-b.ts";
|
||||
f.write(a, ""); f.write(b, "tail");
|
||||
const boundary = JSON.stringify(b) + String(lstatSync(path.join(f.root, b)).mode);
|
||||
f.write(a, `${boundary}payload`);
|
||||
const first = snapshot(f.root, f.env).hash;
|
||||
f.write(a, ""); f.write(b, `payload${boundary}tail`);
|
||||
assert.notEqual(snapshot(f.root, f.env).hash, first);
|
||||
});
|
||||
|
||||
test("an edit between diff discovery and fingerprinting cannot certify a partial selection", { skip: process.platform === "win32" }, t => {
|
||||
const f = fixture(t);
|
||||
const executable = execFileSync("which", ["git"], { encoding: "utf8" }).trim();
|
||||
f.write(".turbo/bin/package.json", '{"type":"commonjs"}');
|
||||
f.write(".turbo/bin/git", `#!${process.execPath}
|
||||
const {execFileSync}=require('node:child_process');
|
||||
const fs=require('node:fs');
|
||||
const args=process.argv.slice(2);
|
||||
const result=execFileSync(${JSON.stringify(executable)},args,{stdio:['ignore','pipe','pipe']});
|
||||
if(args[0]==='diff'&&!args.includes('--cached')&&!fs.existsSync('.turbo/diff-edited')){
|
||||
fs.writeFileSync('ui/src/other.ts','export const other = 2;\\n');
|
||||
fs.writeFileSync('.turbo/diff-edited','done');
|
||||
}
|
||||
process.stdout.write(result);
|
||||
`);
|
||||
chmodSync(path.join(f.root, ".turbo/bin/git"), 0o755);
|
||||
const result = f.run([], { PATH: `${path.join(f.root, ".turbo/bin")}${path.delimiter}${f.env.PATH}` });
|
||||
assert.equal(result.status, 1, result.text);
|
||||
assert.match(result.text, /Inputs changed/);
|
||||
assert.doesNotMatch(result.text, /Local Vitest verification passed/);
|
||||
});
|
||||
|
||||
test("documentation-only changes and empty diffs do not start a full suite", async t => {
|
||||
const f = fixture(t);
|
||||
f.write("ui/src/leaf.ts", "export const value = 1;\n");
|
||||
f.write("doc/DEVELOPING.md", "Guidance only\n");
|
||||
const result = f.run(["--explain"]);
|
||||
assert.equal(result.status, 0, result.text);
|
||||
assert.match(result.text, /0 Vitest test files selected/);
|
||||
assert.doesNotMatch(result.text, /Test Files/);
|
||||
const plan = await planVerification({ root: f.root, env: f.env,
|
||||
diff: { ...comparison(f.root, f.base), changed: [] } });
|
||||
assert.deepEqual(plan.jobs, []);
|
||||
});
|
||||
|
||||
test("local server selection retains native suites and serialized ownership under a CI caller", { timeout: 120_000 }, t => {
|
||||
const f = fixture(t);
|
||||
f.write("ui/src/leaf.ts", "export const value = 1;\n");
|
||||
f.write("vitest.config.mjs", 'export default {test:{projects:["server"]}};');
|
||||
f.write("server/vitest.config.mjs", 'export default {test:{name:"@paperclipai/server",include:["src/**/*.test.ts"]}};');
|
||||
f.write("server/src/leaf.ts", "export const value = 1;\n");
|
||||
const native = "server/src/services/native-runtime/plugin-runner-probe.test.ts";
|
||||
const serialized = "server/src/__tests__/authz-owner.test.ts";
|
||||
for (const [file, imported] of [[native, "../../leaf"], [serialized, "../leaf"]]) {
|
||||
f.write(file, `import {test,expect} from "vitest"; import {value} from ${JSON.stringify(imported)}; test("local owner",()=>expect(value).toBeGreaterThan(0));`);
|
||||
}
|
||||
f.write(".github/ci-task-matrix.json", JSON.stringify({
|
||||
general_tests: [{ task: "ci:server", group: "general-server-without-chat" }],
|
||||
verify_serialized_server: [{ task: "ci:serialized" }],
|
||||
}));
|
||||
f.git("add", "."); f.git("-c", "user.name=Test", "-c", "user.email=test@example.invalid", "commit", "-qm", "server baseline");
|
||||
const base = f.git("rev-parse", "HEAD");
|
||||
f.write("server/src/leaf.ts", "export const value = 2;\n");
|
||||
const result = f.run(["--base", base], { GITHUB_WORKFLOW: "PR" });
|
||||
assert.equal(result.status, 0, result.text);
|
||||
assert.equal((result.text.match(/Test Files\s+1 passed/g) ?? []).length, 2);
|
||||
const record = JSON.parse(readFileSync(path.join(f.root, ".turbo/local-verify/last-run.json"), "utf8"));
|
||||
assert.deepEqual(record.plan.manifest.lanes["general-server-without-chat"].shards[0]["@paperclipai/server"], [native]);
|
||||
assert.deepEqual(record.plan.manifest.lanes.serialized.shards[0]["@paperclipai/server"], [serialized]);
|
||||
});
|
||||
|
||||
test("source mutation and interruption do not publish reusable passes", { timeout: 120_000 }, async t => {
|
||||
const f = fixture(t);
|
||||
const runControlled = action => new Promise((resolve, reject) => {
|
||||
const child = spawn(process.execPath, [path.join(f.root, "scripts/local-verify.mjs"), "--base", f.base, "--no-cache"], {
|
||||
cwd: f.root, env: f.env, stdio: ["ignore", "pipe", "pipe"],
|
||||
});
|
||||
let text = "", acted = false;
|
||||
const timer = setTimeout(() => { child.kill("SIGTERM"); reject(new Error("Controlled verification did not finish")); }, 60_000);
|
||||
const collect = bytes => {
|
||||
text += bytes;
|
||||
if (!acted && text.includes("[verify] Running")) { acted = true; action(child); }
|
||||
};
|
||||
child.stdout.on("data", collect); child.stderr.on("data", collect);
|
||||
child.once("error", reject);
|
||||
child.once("close", (code, signal) => { clearTimeout(timer); resolve({ code, signal, text, acted }); });
|
||||
});
|
||||
const changed = await runControlled(() => f.write("ui/src/leaf.ts", "export const value = 4;\n"));
|
||||
assert.equal(changed.acted, true);
|
||||
assert.equal(changed.code, 1, changed.text);
|
||||
assert.match(changed.text, /Inputs changed during verification/);
|
||||
const beforeInterrupt = f.run();
|
||||
assert.equal(beforeInterrupt.status, 0, beforeInterrupt.text);
|
||||
const interrupted = await runControlled(child => child.kill("SIGTERM"));
|
||||
assert.equal(interrupted.acted, true);
|
||||
assert.equal(interrupted.code, 143, interrupted.text);
|
||||
const result = f.run();
|
||||
assert.equal(result.status, 0, result.text);
|
||||
assert.match(result.text, /Test Files\s+1 passed/);
|
||||
assert.doesNotMatch(result.text, /reusing a passing local result/);
|
||||
});
|
||||
@@ -4,7 +4,7 @@ import { fileURLToPath } from "node:url";
|
||||
|
||||
// Each Vite context stays in its own process: aliases, plugins, globals and
|
||||
// heaps cannot leak between projects. Two contexts bound memory on hosted CI.
|
||||
export async function selectProjects({ projects, changed, root = process.cwd(), concurrency = Math.min(2, availableParallelism()) }) {
|
||||
export async function selectProjects({ projects, changed, root = process.cwd(), concurrency = Math.min(2, availableParallelism()), env = process.env }) {
|
||||
if (!Number.isInteger(concurrency) || concurrency < 1 || concurrency > 2) throw new Error("Expected one or two graph workers");
|
||||
const results = new Array(projects.length), children = new Set();
|
||||
let next = 0, failure;
|
||||
@@ -13,10 +13,10 @@ export async function selectProjects({ projects, changed, root = process.cwd(),
|
||||
const index = next++;
|
||||
try {
|
||||
results[index] = await new Promise((resolve, reject) => {
|
||||
const env = { ...process.env };
|
||||
delete env.NODE_TEST_CONTEXT;
|
||||
const childEnv = { ...env };
|
||||
delete childEnv.NODE_TEST_CONTEXT;
|
||||
const child = fork(fileURLToPath(import.meta.url), [], {
|
||||
cwd: root, env, execArgv: ["--max-old-space-size=2048"], stdio: ["ignore", "inherit", "inherit", "ipc"],
|
||||
cwd: root, env: childEnv, execArgv: ["--max-old-space-size=2048"], stdio: ["ignore", "inherit", "inherit", "ipc"],
|
||||
});
|
||||
children.add(child);
|
||||
let result;
|
||||
|
||||
@@ -8,7 +8,7 @@ import { productInputs } from "../../.github/scripts/ci-inputs.mjs";
|
||||
import { mergeDiff } from "../../.github/scripts/ci-diff.mjs";
|
||||
import { selectProjects } from "./select-projects.mjs";
|
||||
|
||||
export function selectionEligibility(plan) {
|
||||
export function selectionEligibility(plan, root = process.cwd()) {
|
||||
if (plan?.version !== 1 || plan.mode !== "enabled" || plan.forceReason) return "Full/shadow/forced task plan";
|
||||
if (!Array.isArray(plan.changed) || !plan.changed.length) return "Incomplete comparison";
|
||||
const changed = productInputs(plan.changed);
|
||||
@@ -20,7 +20,7 @@ export function selectionEligibility(plan) {
|
||||
|| /(?:^|\/)(?:vite|vitest)\.(?:config|setup)\.[cm]?[jt]sx?$|\.d\.[cm]?ts$/i.test(file)
|
||||
|| file.split("/").includes("..")) return `Broad input: ${file}`;
|
||||
try {
|
||||
if (!lstatSync(file).isFile()) return `Non-regular source: ${file}`;
|
||||
if (!lstatSync(path.join(root, file)).isFile()) return `Non-regular source: ${file}`;
|
||||
} catch { return `Deleted or missing source: ${file}`; }
|
||||
}
|
||||
return null;
|
||||
|
||||
@@ -313,6 +313,7 @@ export async function selectProjectTests({ root = process.cwd(), project, change
|
||||
const selected = [...new Set([...raw, ...inputRelated, ...retained].map(relative))].sort();
|
||||
return {
|
||||
project,
|
||||
isolated: owner.config.isolate !== false,
|
||||
inventory: inventory.map(relative).sort(),
|
||||
selected,
|
||||
related: raw.map(relative).sort(),
|
||||
|
||||
@@ -0,0 +1,99 @@
|
||||
#!/usr/bin/env node
|
||||
import { realpathSync } from "node:fs";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { parseArgs } from "node:util";
|
||||
import { createHash } from "node:crypto";
|
||||
import { comparison, localEnvironment, protocolVersion, requestKey, snapshot } from "./local-verify/inputs.mjs";
|
||||
import { planVerification } from "./local-verify/plan.mjs";
|
||||
import { invalidatePass, mayReuseAttempt, readPass, readPlan, runPlan, writeRecord } from "./local-verify/run.mjs";
|
||||
|
||||
function describe(plan, cached) {
|
||||
console.log(`[verify] ${plan.kind === "full" ? "Full Vitest suite" : plan.kind === "broad" ? "Broader verification required"
|
||||
: `${plan.selectedFiles.length}${plan.totalFiles === undefined ? "" : `/${plan.totalFiles}`} Vitest test files selected`}${cached ? "; reusing a passing local result" : ""}`);
|
||||
if (plan.reason) console.log(`[verify] ${plan.reason}`);
|
||||
if (plan.retainedFiles) console.log(`[verify] ${plan.retainedFiles} test files retained conservatively`);
|
||||
if (plan.requiredCI.length) console.log(`[verify] Required CI coverage: ${plan.requiredCI.join(", ")}`);
|
||||
console.log("[verify] Local Vitest coverage does not replace required CI, typecheck/build, browser, release, or platform-specific checks.");
|
||||
}
|
||||
|
||||
export async function verify({ root = process.cwd(), base, full = false, explain = false, noCache = false, env = localEnvironment() } = {}) {
|
||||
root = realpathSync(root);
|
||||
const before = snapshot(root, env), diff = comparison(root, base), key = requestKey(before, diff, full);
|
||||
const canReuse = !noCache && before.cacheable && mayReuseAttempt(root, key);
|
||||
const cached = canReuse ? readPass(root, key) : null;
|
||||
console.log(`[verify] Base ${diff.base.slice(0, 12)}; ${diff.changed.length} changed paths, including local edits`);
|
||||
if (cached) {
|
||||
if (snapshot(root, env).hash !== before.hash) throw new Error("Inputs changed during cache lookup. Run verification again.");
|
||||
describe(cached.plan, true);
|
||||
console.log(`[verify] Passed result from ${cached.completedAt}; input fingerprint ${before.hash.slice(0, 12)}`);
|
||||
if (explain) console.log(JSON.stringify({ ...diff, inputFingerprint: before.hash, cacheable: before.cacheable, plan: cached.plan }, null, 2));
|
||||
return 0;
|
||||
}
|
||||
if (!explain) {
|
||||
invalidatePass(root, key);
|
||||
writeRecord(root, `attempt-${key}`, { version: protocolVersion, key, status: "running", plan: null });
|
||||
}
|
||||
const plan = (canReuse ? readPlan(root, key) : null)
|
||||
?? await planVerification({ root, env, diff, full });
|
||||
if (plan.manifest) plan.manifest.head = diff.head;
|
||||
if (snapshot(root, env).hash !== before.hash) throw new Error("Inputs changed during selection. Run verification again.");
|
||||
describe(plan, false);
|
||||
if (explain) {
|
||||
console.log(JSON.stringify({ ...diff, inputFingerprint: before.hash, cacheable: before.cacheable, plan }, null, 2));
|
||||
return plan.kind === "broad" ? 2 : 0;
|
||||
}
|
||||
if (plan.kind === "broad") {
|
||||
console.log("[verify] Run focused checks and use required CI for broad coverage, or explicitly run pnpm verify --full.");
|
||||
writeRecord(root, `attempt-${key}`, { version: protocolVersion, key, status: "broad", plan });
|
||||
return 2;
|
||||
}
|
||||
if (before.cacheable && !noCache && plan.kind === "selected") {
|
||||
writeRecord(root, `plan-${key}`, { version: protocolVersion, key, status: "planned", plan });
|
||||
}
|
||||
invalidatePass(root, key);
|
||||
const startedAt = new Date().toISOString();
|
||||
let code = 0;
|
||||
for (const job of plan.jobs) {
|
||||
const jobKey = createHash("sha256").update(`${key}\0${job.id}`).digest("hex");
|
||||
if (canReuse && job.cacheable && readPass(root, jobKey)) {
|
||||
console.log(`[verify] Reusing passed ${job.id}`);
|
||||
continue;
|
||||
}
|
||||
invalidatePass(root, jobKey);
|
||||
code = await runPlan(root, env, { ...plan, jobs: [job] });
|
||||
if (snapshot(root, env).hash !== before.hash) {
|
||||
console.error("[verify] Inputs changed during verification. The result cannot be reused.");
|
||||
code ||= 1;
|
||||
}
|
||||
if (code !== 0) break;
|
||||
if (!noCache && before.cacheable && job.cacheable) writeRecord(root, jobKey, {
|
||||
version: protocolVersion, key: jobKey, status: "passed", plan, jobId: job.id, completedAt: new Date().toISOString(),
|
||||
});
|
||||
}
|
||||
const after = snapshot(root, env);
|
||||
if (after.hash !== before.hash) {
|
||||
console.error("[verify] Inputs changed during verification. The result cannot be reused.");
|
||||
code ||= 1;
|
||||
}
|
||||
const record = { version: protocolVersion, key, status: code === 0 ? "passed" : "failed", startedAt,
|
||||
completedAt: new Date().toISOString(), head: diff.head, inputFingerprint: before.hash, plan, exitCode: code };
|
||||
writeRecord(root, "last-run", record);
|
||||
writeRecord(root, `attempt-${key}`, record);
|
||||
if (code === 0 && !noCache && before.cacheable && after.cacheable && plan.jobs.every(job => job.cacheable)) writeRecord(root, key, record);
|
||||
console.log(`[verify] ${code === 0 ? "Local Vitest verification passed" : "Verification failed or interrupted"}${code === 0 && !before.cacheable ? "; external inputs disable pass reuse" : ""}`);
|
||||
return code;
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const { values } = parseArgs({ options: { base: { type: "string" }, full: { type: "boolean" },
|
||||
explain: { type: "boolean" }, "no-cache": { type: "boolean" }, help: { type: "boolean" } } });
|
||||
if (values.help) {
|
||||
console.log("Usage: pnpm verify [--base REF] [--explain] [--full] [--no-cache]\nRuns affected local Vitest checks. Exit 2 means broader coverage is required; use CI or --full.");
|
||||
return;
|
||||
}
|
||||
process.exitCode = await verify({ base: values.base, full: values.full, explain: values.explain, noCache: values["no-cache"] });
|
||||
}
|
||||
|
||||
if (process.argv[1] && realpathSync(process.argv[1]) === fileURLToPath(import.meta.url)) {
|
||||
main().catch(error => { console.error(`[verify] ${error.message}`); process.exitCode = 1; });
|
||||
}
|
||||
@@ -0,0 +1,96 @@
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { createHash } from "node:crypto";
|
||||
import { lstatSync, readFileSync, realpathSync, readlinkSync } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { sourceFiles } from "../ci/execution-plan.mjs";
|
||||
import { productInputs } from "../../.github/scripts/ci-inputs.mjs";
|
||||
import { installedInputs } from "./installed-inputs.mjs";
|
||||
|
||||
export const protocolVersion = 1;
|
||||
|
||||
export function git(root, ...args) {
|
||||
return execFileSync("git", args, { cwd: root, encoding: "utf8", maxBuffer: 32 * 1024 * 1024,
|
||||
stdio: ["ignore", "pipe", "pipe"] });
|
||||
}
|
||||
|
||||
export function localEnvironment(env = process.env) {
|
||||
const result = { ...env };
|
||||
delete result.GITHUB_WORKFLOW;
|
||||
delete result.PAPERCLIP_CI_TEST_SELECTION;
|
||||
delete result.PAPERCLIP_CI_SELECTION_HASH;
|
||||
return result;
|
||||
}
|
||||
|
||||
export function comparison(root, requestedBase) {
|
||||
let base;
|
||||
for (const ref of requestedBase ? [requestedBase] : ["origin/master", "master"]) {
|
||||
try {
|
||||
const commit = git(root, "rev-parse", "--verify", "--end-of-options", `${ref}^{commit}`).trim();
|
||||
base = git(root, "merge-base", "HEAD", commit).trim();
|
||||
break;
|
||||
} catch {}
|
||||
}
|
||||
if (!base) throw new Error("Cannot resolve a base. Pass --base <ref>.");
|
||||
const changed = [...new Set([
|
||||
...git(root, "diff", "--name-only", "--no-renames", "-z", base, "--").split("\0"),
|
||||
...git(root, "diff", "--cached", "--name-only", "--no-renames", "-z", base, "--").split("\0"),
|
||||
...git(root, "ls-files", "--others", "--exclude-standard", "-z").split("\0"),
|
||||
].filter(Boolean))].sort();
|
||||
return { base, head: git(root, "rev-parse", "HEAD").trim(), changed };
|
||||
}
|
||||
|
||||
function version(root, env, command, args) {
|
||||
try { return execFileSync(command, args, { cwd: root, env, encoding: "utf8", stdio: ["ignore", "pipe", "pipe"] }).trim(); }
|
||||
catch { return "unavailable"; }
|
||||
}
|
||||
|
||||
export function snapshot(root, env = localEnvironment()) {
|
||||
root = realpathSync(root);
|
||||
const files = sourceFiles(root);
|
||||
const hash = createHash("sha256");
|
||||
const append = record => hash.update(`${JSON.stringify(record)}\n`);
|
||||
const digest = file => createHash("sha256").update(readFileSync(file)).digest("hex");
|
||||
let cacheable = true;
|
||||
const addFile = file => {
|
||||
const absolute = path.join(root, file);
|
||||
try {
|
||||
const stat = lstatSync(absolute);
|
||||
if (stat.isSymbolicLink()) {
|
||||
const target = realpathSync(absolute);
|
||||
if (!target.startsWith(`${root}${path.sep}`)) cacheable = false;
|
||||
const targetStat = lstatSync(target), regular = targetStat.isFile();
|
||||
if (!regular && !targetStat.isDirectory()) cacheable = false;
|
||||
append(["link", file, stat.mode, readlinkSync(absolute), target, regular ? digest(target) : "directory"]);
|
||||
} else if (stat.isFile()) append(["file", file, stat.mode, digest(absolute)]);
|
||||
else { cacheable = false; append(["special", file, stat.mode]); }
|
||||
} catch (error) {
|
||||
if (error.code !== "ENOENT") throw error;
|
||||
append(["missing", file]);
|
||||
}
|
||||
};
|
||||
productInputs(files).forEach(addFile);
|
||||
const directories = new Set(["", ...files.filter(file => path.basename(file) === "package.json").map(path.dirname)]);
|
||||
for (const directory of [...directories].sort()) {
|
||||
for (const name of [".env", ".env.local", ".env.test", ".env.test.local"]) addFile(path.join(directory, name));
|
||||
}
|
||||
installedInputs(root, [...directories].sort(), append);
|
||||
for (const file of ["node_modules/.modules.yaml", "node_modules/.pnpm/lock.yaml"]) addFile(file);
|
||||
const environment = Object.fromEntries(Object.keys(env).sort().map(name => [name, env[name]]));
|
||||
if (Object.entries(env).some(([name, value]) => value?.trim() && (
|
||||
name === "DATABASE_URL" || name === "PAPERCLIP_TEST_POSTGRES_ADMIN_URL"
|
||||
|| /^PAPERCLIP_.*_TEST_DATABASE_URL$/.test(name)
|
||||
|| /^(?:PAPERCLIP|TEST)_.*(?:LIVE|E2E_LIVE)$/.test(name) && /^(?:1|true)$/i.test(value)
|
||||
))) cacheable = false;
|
||||
const context = { version: protocolVersion, root, node: process.version, executable: realpathSync(process.execPath),
|
||||
platform: process.platform, arch: process.arch, environment,
|
||||
pnpm: version(root, env, "pnpm", ["--version"]),
|
||||
rust: version(root, env, "rustc", ["--version", "--verbose"]),
|
||||
cargo: version(root, env, "cargo", ["--version"]) };
|
||||
append(["context", context]);
|
||||
return { hash: hash.digest("hex"), cacheable };
|
||||
}
|
||||
|
||||
export function requestKey(inputs, diff, full) {
|
||||
return createHash("sha256").update(JSON.stringify({ version: protocolVersion, inputs: inputs.hash,
|
||||
base: diff.base, changed: diff.changed, full })).digest("hex");
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
import { lstatSync, mkdirSync, readdirSync, readFileSync, readlinkSync, renameSync, writeFileSync } from "node:fs";
|
||||
import path from "node:path";
|
||||
|
||||
const caches = new Map();
|
||||
|
||||
export function installedInputs(root, directories, append) {
|
||||
const cacheFile = path.join(root, ".turbo/local-verify/installed-digests.json");
|
||||
if (!caches.has(root)) {
|
||||
let entries = [];
|
||||
try {
|
||||
const saved = JSON.parse(readFileSync(cacheFile, "utf8"));
|
||||
if (saved.version === 1) entries = Object.entries(saved.digests).filter(([, hash]) => /^[a-f0-9]{64}$/.test(hash));
|
||||
} catch {}
|
||||
caches.set(root, new Map(entries));
|
||||
}
|
||||
const digests = caches.get(root), current = new Map();
|
||||
let dirty = false;
|
||||
const visited = new Set();
|
||||
const workspaces = new Set(directories.map(directory => path.join(root, directory)));
|
||||
const visit = absolute => {
|
||||
if (visited.has(absolute)) return;
|
||||
visited.add(absolute);
|
||||
let stat;
|
||||
try { stat = lstatSync(absolute, { bigint: true }); }
|
||||
catch (error) {
|
||||
if (error.code !== "ENOENT") throw error;
|
||||
append(["installed-missing", absolute]);
|
||||
return;
|
||||
}
|
||||
if (stat.isSymbolicLink()) {
|
||||
const target = readlinkSync(absolute);
|
||||
append(["installed-link", absolute, String(stat.mode), target]);
|
||||
visit(path.resolve(path.dirname(absolute), target));
|
||||
} else if (stat.isDirectory()) {
|
||||
if (workspaces.has(absolute)) { visit(path.join(absolute, "node_modules")); return; }
|
||||
const names = readdirSync(absolute).sort().filter(name => path.basename(absolute) !== "node_modules"
|
||||
|| ![".vite", ".vite-temp", ".cache"].includes(name));
|
||||
append(["installed-directory", absolute, String(stat.mode), names]);
|
||||
for (const name of names) visit(path.join(absolute, name));
|
||||
} else if (stat.isFile()) {
|
||||
const identity = [stat.dev, stat.ino, stat.size, stat.mtimeNs, stat.ctimeNs, stat.mode].join(":");
|
||||
if (!digests.has(identity)) {
|
||||
digests.set(identity, createHash("sha256").update(readFileSync(absolute)).digest("hex"));
|
||||
dirty = true;
|
||||
}
|
||||
current.set(identity, digests.get(identity));
|
||||
append(["installed-file", absolute, String(stat.mode), digests.get(identity)]);
|
||||
} else throw new Error(`Unsupported installed input: ${absolute}`);
|
||||
};
|
||||
for (const directory of directories) visit(path.join(root, directory, "node_modules"));
|
||||
if (dirty || current.size !== digests.size) {
|
||||
mkdirSync(path.dirname(cacheFile), { recursive: true });
|
||||
const temporary = `${cacheFile}.${randomUUID()}.tmp`;
|
||||
writeFileSync(temporary, JSON.stringify({ version: 1, digests: Object.fromEntries(current) }), { mode: 0o600 });
|
||||
renameSync(temporary, cacheFile);
|
||||
}
|
||||
caches.set(root, current);
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { readFileSync } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { planTasks } from "../../.github/scripts/ci-plan.mjs";
|
||||
import { productInputs } from "../../.github/scripts/ci-inputs.mjs";
|
||||
import { selectionEligibility } from "../ci/select-tests.mjs";
|
||||
|
||||
const chatSuite = "server/src/__tests__/chat-channels.integration.test.ts";
|
||||
|
||||
function inventory(root, env, group) {
|
||||
const args = group === "serialized" ? ["--mode", "serialized"] : ["--mode", "general", "--group", group];
|
||||
const shards = ["general-workspaces-b", "general-workspaces-c"].includes(group) ? []
|
||||
: ["--shard-index", "0", "--shard-count", "1"];
|
||||
return JSON.parse(execFileSync(process.execPath, [path.join(root, "scripts/run-vitest-stable.mjs"), ...args,
|
||||
...shards, "--dry-run"], {
|
||||
cwd: root, env, encoding: "utf8", maxBuffer: 16 * 1024 * 1024,
|
||||
}));
|
||||
}
|
||||
|
||||
export async function planVerification({ root, env, diff, full = false }) {
|
||||
const config = JSON.parse(readFileSync(path.join(root, "turbo.json"), "utf8"));
|
||||
const lanes = JSON.parse(readFileSync(path.join(root, ".github/ci-task-matrix.json"), "utf8"));
|
||||
const coarse = planTasks({ config, lanes, paths: diff.changed });
|
||||
const requiredCI = diff.changed.length ? Object.entries(coarse.jobs).filter(([, job]) => job.run).map(([name]) => name) : [];
|
||||
if (full) return { kind: "full", jobs: [{ id: "full", args: null, cacheable: false }], requiredCI, selectedFiles: null, reason: "Full Vitest suite explicitly requested; opaque integration coverage runs fresh" };
|
||||
if (!productInputs(diff.changed).length) return { kind: "selected", jobs: [], requiredCI, selectedFiles: [], reason: "No changed product inputs" };
|
||||
const reason = coarse.forceReason ?? selectionEligibility(coarse, root);
|
||||
if (reason) return { kind: "broad", reason, jobs: [], requiredCI };
|
||||
try {
|
||||
const groups = [...new Set(["general_tests", "verify_serialized_server"].flatMap(job =>
|
||||
coarse.jobs[job]?.run ? coarse.jobs[job].matrix.include.map(entry => entry.group ?? "serialized") : []))];
|
||||
const inventories = new Map(groups.filter(group => group !== "general-chat").map(group => [group, inventory(root, env, group)]));
|
||||
const projects = [...new Set([...inventories.values()].flatMap(value => value.workspaceProjects ?? ["@paperclipai/server"]))];
|
||||
const { selectProjects } = await import("../ci/select-projects.mjs");
|
||||
const graphs = await selectProjects({ root, projects, changed: productInputs(diff.changed), concurrency: 1, env });
|
||||
if (graphs.some(graph => !graph.isolated)) throw new Error("Non-isolated projects require broad verification");
|
||||
const covered = new Set(graphs.flatMap(graph => graph.coveredChanges));
|
||||
const uncovered = productInputs(diff.changed).filter(file => !covered.has(file));
|
||||
if (uncovered.length) throw new Error(`Changes outside resolved test graphs: ${uncovered.join(", ")}`);
|
||||
const byProject = new Map(graphs.map(graph => [graph.project, graph]));
|
||||
const manifest = { version: 1, head: diff.head, lanes: {} }, jobs = [], selectedFiles = new Set();
|
||||
for (const group of groups) {
|
||||
if (group === "general-chat") {
|
||||
if (byProject.get("@paperclipai/server")?.selected.includes(chatSuite)) {
|
||||
selectedFiles.add(chatSuite);
|
||||
for (let index = 0; index < 6; index++) jobs.push({ id: `chat-${index + 1}`, cacheable: false, args: ["--mode", "general", "--group", group,
|
||||
"--shard-index", String(index), "--shard-count", "6", "--serial-chat"] });
|
||||
}
|
||||
continue;
|
||||
}
|
||||
const value = inventories.get(group);
|
||||
const owners = value.workspaceProjects
|
||||
? Object.fromEntries(value.workspaceProjects.map(project => [project, byProject.get(project).inventory]))
|
||||
: { "@paperclipai/server": group === "serialized" ? value.selectedSerializedSuites : value.selectedGeneralServerSuites };
|
||||
const reusable = {}, fresh = {};
|
||||
for (const [project, files] of Object.entries(owners)) {
|
||||
const graph = byProject.get(project), all = new Set(graph.inventory), affected = new Set(graph.selected);
|
||||
if (!Array.isArray(files) || files.some(file => !all.has(file))) throw new Error(`Runner/graph inventory mismatch: ${group}`);
|
||||
const matches = files.filter(file => affected.has(file));
|
||||
const opaque = new Set(graph.alwaysRun);
|
||||
const pure = matches.filter(file => !opaque.has(file)), retained = matches.filter(file => opaque.has(file));
|
||||
if (pure.length) reusable[project] = pure;
|
||||
if (retained.length) fresh[project] = retained;
|
||||
matches.forEach(file => selectedFiles.add(file));
|
||||
}
|
||||
const partitions = [reusable, fresh].filter(files => Object.keys(files).length);
|
||||
if (!partitions.length) continue;
|
||||
manifest.lanes[group] = { shards: partitions };
|
||||
for (const [index, files] of partitions.entries()) jobs.push({ id: `${group}-${files === reusable ? "reusable" : "fresh"}`,
|
||||
cacheable: files === reusable, selected: true, args: ["--mode", group === "serialized" ? "serialized" : "general",
|
||||
...(group === "serialized" ? [] : ["--group", group]), "--shard-index", String(index), "--shard-count", String(partitions.length)] });
|
||||
}
|
||||
const expected = new Set(graphs.flatMap(graph => graph.selected));
|
||||
if (expected.size !== selectedFiles.size || [...expected].some(file => !selectedFiles.has(file))) {
|
||||
throw new Error("Selected tests are not completely covered by local runner ownership");
|
||||
}
|
||||
const { validateSelection } = await import("../ci/test-selection-manifest.mjs");
|
||||
validateSelection(manifest);
|
||||
return { kind: "selected", jobs, manifest, requiredCI, selectedFiles: [...selectedFiles].sort(),
|
||||
totalFiles: graphs.reduce((count, graph) => count + graph.inventory.length, 0),
|
||||
retainedFiles: graphs.reduce((count, graph) => count + graph.alwaysRun.length, 0) };
|
||||
} catch (error) {
|
||||
return { kind: "broad", reason: `Conservative selection unavailable: ${error.message}`, jobs: [], requiredCI };
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,86 @@
|
||||
import { spawn } from "node:child_process";
|
||||
import { mkdirSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
import { selectionHash } from "../ci/test-selection-manifest.mjs";
|
||||
import { protocolVersion } from "./inputs.mjs";
|
||||
|
||||
export const cacheDirectory = root => path.join(root, ".turbo/local-verify");
|
||||
const planHash = plan => createHash("sha256").update(JSON.stringify(plan)).digest("hex");
|
||||
|
||||
export function readPass(root, key) {
|
||||
try {
|
||||
const result = JSON.parse(readFileSync(path.join(cacheDirectory(root), `${key}.json`), "utf8"));
|
||||
return result.version === protocolVersion && result.key === key && result.status === "passed" && result.planHash === planHash(result.plan)
|
||||
&& ["selected", "full"].includes(result.plan?.kind) && Array.isArray(result.plan.jobs) ? result : null;
|
||||
} catch { return null; }
|
||||
}
|
||||
|
||||
export function readPlan(root, key) {
|
||||
try {
|
||||
const result = JSON.parse(readFileSync(path.join(cacheDirectory(root), `plan-${key}.json`), "utf8"));
|
||||
return result.version === protocolVersion && result.key === key && result.status === "planned" && result.planHash === planHash(result.plan)
|
||||
&& result.plan?.kind === "selected" && Array.isArray(result.plan.jobs) ? result.plan : null;
|
||||
} catch { return null; }
|
||||
}
|
||||
|
||||
export function mayReuseAttempt(root, key) {
|
||||
try {
|
||||
const result = JSON.parse(readFileSync(path.join(cacheDirectory(root), `attempt-${key}.json`), "utf8"));
|
||||
return result.version === protocolVersion && result.key === key && result.status === "passed";
|
||||
} catch (error) { return error.code === "ENOENT"; }
|
||||
}
|
||||
|
||||
export function writeRecord(root, name, record) {
|
||||
const directory = cacheDirectory(root);
|
||||
mkdirSync(directory, { recursive: true });
|
||||
const temporary = path.join(directory, `${name}.${randomUUID()}.tmp`);
|
||||
writeFileSync(temporary, JSON.stringify({ ...record, planHash: planHash(record.plan) }, null, 2), { mode: 0o600 });
|
||||
renameSync(temporary, path.join(directory, `${name}.json`));
|
||||
}
|
||||
|
||||
export const invalidatePass = (root, key) => rmSync(path.join(cacheDirectory(root), `${key}.json`), { force: true });
|
||||
|
||||
async function execute(command, args, root, env, label) {
|
||||
return new Promise(resolve => {
|
||||
const grouped = process.platform !== "win32";
|
||||
const child = spawn(command, args, { cwd: root, env, stdio: "inherit", detached: grouped });
|
||||
let stopped;
|
||||
let finished = false;
|
||||
const interrupt = signal => {
|
||||
stopped = signal;
|
||||
try {
|
||||
if (grouped && child.pid) process.kill(-child.pid, signal);
|
||||
else child.kill(signal);
|
||||
} catch (error) { if (error.code !== "ESRCH") throw error; }
|
||||
};
|
||||
const onInterrupt = () => interrupt("SIGINT"), onTerminate = () => interrupt("SIGTERM");
|
||||
process.on("SIGINT", onInterrupt); process.on("SIGTERM", onTerminate);
|
||||
const finish = code => {
|
||||
if (finished) return;
|
||||
finished = true;
|
||||
process.off("SIGINT", onInterrupt); process.off("SIGTERM", onTerminate);
|
||||
resolve(stopped ? stopped === "SIGINT" ? 130 : 143 : code);
|
||||
};
|
||||
child.once("error", error => { console.error(error.message); finish(1); });
|
||||
child.once("close", code => finish(code ?? 1));
|
||||
console.log(`[verify] Running ${label}`);
|
||||
});
|
||||
}
|
||||
|
||||
export async function runPlan(root, env, plan) {
|
||||
const directory = cacheDirectory(root);
|
||||
mkdirSync(directory, { recursive: true });
|
||||
const manifestFile = path.join(directory, `selection-${randomUUID()}.json`);
|
||||
if (plan.manifest) writeFileSync(manifestFile, JSON.stringify(plan.manifest));
|
||||
try {
|
||||
for (const job of plan.jobs) {
|
||||
const code = job.args === null ? await execute("pnpm", ["test:run"], root, env, job.id)
|
||||
: await execute(process.execPath, [path.join(root, "scripts/run-vitest-stable.mjs"), ...job.args], root,
|
||||
job.selected ? { ...env, PAPERCLIP_CI_TEST_SELECTION: manifestFile,
|
||||
PAPERCLIP_CI_SELECTION_HASH: selectionHash(plan.manifest) } : env, job.id);
|
||||
if (code !== 0) return code;
|
||||
}
|
||||
return 0;
|
||||
} finally { rmSync(manifestFile, { force: true }); }
|
||||
}
|
||||
@@ -267,6 +267,7 @@ function parseCliOptions(argv) {
|
||||
let shardCount = null;
|
||||
let group = null;
|
||||
let dryRun = false;
|
||||
let serialChat = false;
|
||||
|
||||
for (let index = 0; index < argv.length; index += 1) {
|
||||
const arg = argv[index];
|
||||
@@ -312,6 +313,11 @@ function parseCliOptions(argv) {
|
||||
continue;
|
||||
}
|
||||
|
||||
if (arg === "--serial-chat") {
|
||||
serialChat = true;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (arg === "--group") {
|
||||
group = readOptionValue(argv, index, arg);
|
||||
index += 1;
|
||||
@@ -337,10 +343,12 @@ function parseCliOptions(argv) {
|
||||
const shardAllowed =
|
||||
mode === serializedModeName ||
|
||||
(mode === generalModeName &&
|
||||
([...generalServerGroups, generalChatGroupName, generalWorkspacesAGroupName].includes(group)));
|
||||
([...generalServerGroups, generalChatGroupName, generalWorkspacesAGroupName].includes(group)
|
||||
|| Boolean(process.env.PAPERCLIP_CI_TEST_SELECTION)
|
||||
&& [generalWorkspacesBGroupName, generalWorkspacesCGroupName].includes(group)));
|
||||
if (!shardAllowed && shardIndex !== null) {
|
||||
fail(
|
||||
"--shard-index/--shard-count are only valid with serialized mode or a shardable general server/chat/workspaces-a group.",
|
||||
"--shard-index/--shard-count require serialized mode, a shardable general server/chat/workspaces-a group, or a selected workspace group.",
|
||||
);
|
||||
}
|
||||
|
||||
@@ -352,6 +360,10 @@ function parseCliOptions(argv) {
|
||||
fail(`Unknown group "${group}". Expected one of: ${allowedGeneralGroupNames.join(", ")}.`);
|
||||
}
|
||||
|
||||
if (serialChat && (mode !== generalModeName || group !== generalChatGroupName)) {
|
||||
fail("--serial-chat is only valid with --mode general --group general-chat.");
|
||||
}
|
||||
|
||||
if (shardIndex !== null) {
|
||||
if (shardIndex >= shardCount) {
|
||||
fail(`--shard-index must be less than --shard-count. Received ${shardIndex} of ${shardCount}.`);
|
||||
@@ -374,6 +386,7 @@ function parseCliOptions(argv) {
|
||||
shardCount,
|
||||
group,
|
||||
dryRun,
|
||||
serialChat,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -493,7 +506,7 @@ async function runProjectGroup(projects, groupName, shardIndex = null, shardCoun
|
||||
}
|
||||
|
||||
function chatShardWorkers(index, count) {
|
||||
const workers = count > 1 && !hasSharedTestDatabase ? 2 : 1;
|
||||
const workers = count > 1 && !hasSharedTestDatabase && !options.serialChat ? 2 : 1;
|
||||
return Array.from({ length: workers }, (_, offset) => ({
|
||||
index: (index ?? 0) * workers + offset,
|
||||
count: (count ?? 1) * workers,
|
||||
|
||||
Reference in new issue
Block a user