diff --git a/doc/plans/2026-10-02-stock-harness-paperclip-checklist.md b/doc/plans/2026-10-02-stock-harness-paperclip-checklist.md index c333ae5385..1455e78acf 100644 --- a/doc/plans/2026-10-02-stock-harness-paperclip-checklist.md +++ b/doc/plans/2026-10-02-stock-harness-paperclip-checklist.md @@ -3,8 +3,9 @@ Created: 2026-10-02. Status: item 1 merged for native Codex app-server in [PR #14920](https://github.com/paperclipai/paperclip/pull/14920). Item 2's default hire manual is reduced to identity only, and common legacy startup/resume -instructions are reduced locally. Additional carriers and native instructions -remain open. +instructions are in [PR #14948](https://github.com/paperclipai/paperclip/pull/14948). +GitHub live qualification and a matched prior-instruction comparison are in progress. +Additional carriers and native instructions remain open. Goal: keep the agent's stock harness behavior and add only what it needs to work with Paperclip. Apply this across legacy adapters, the new Runner, and their @@ -118,7 +119,8 @@ and existing-test update. specialized wake contracts, custom templates, skills, and auth. - [ ] **2.2 Review additional legacy carriers.** Reduce Hermes local/gateway wrappers, review Pi system delivery, and check OpenClaw fresh-wake framing. - Preserve transport facts and user configuration. + Preserve transport facts and user configuration. Dotta deferred this item on + 2026-10-02 for a later revisit; it remains open. - [ ] **2.3 Reduce native Runner instructions and constraints.** Improve discoverable tool documentation first, then shorten fixed guidance and consolidate completion rules. Verify prompt revisions, digests, and session @@ -140,6 +142,16 @@ custom `promptTemplate`, and `bootstrapPromptTemplate` mechanics are unchanged. Hermes's own wrappers and Pi's system carrier remain follow-up 2.2; native fixed instructions and full-turn constraints remain follow-up 2.3. +The later read-only 2.2 audit found another OpenClaw-owned HTTP identity, +checkout/status, delegation, plan-approval and task-discovery wrapper in +`buildWakeText`. Its short conversation branch is selected when optional task +Markdown is present, including on ordinary task dispatch. Existing dispatch +coverage does not assert the absence of conversation waiting language. This is +a framing concern to verify, not a measured failure. Pi already uses additive +`--append-system-prompt` delivery and suppresses the duplicate user-prompt copy; +its next step is a delivery audit with minimal changes. These findings are +deferred with 2.2 and are not implemented in PR #14948. + The new defaults apply when prompts are assembled after deployment. Existing provider sessions can retain earlier startup instructions in their history until reset; no active session or saved custom template is rewritten here. diff --git a/tests/runner-e2e/STOCK-HARNESS.md b/tests/runner-e2e/STOCK-HARNESS.md index 605ab5b13c..bfa2156939 100644 --- a/tests/runner-e2e/STOCK-HARNESS.md +++ b/tests/runner-e2e/STOCK-HARNESS.md @@ -20,8 +20,17 @@ including the Rust test. It writes JSON reports, the source SHA, a working-sourc fingerprint, selected checks, test counts, and `providerCalls: 0` under `results/stock-harness-preflight-/`. Missing reports or required skipped assertions fail. Unrelated native tests filtered by the name selector remain -explicitly skipped; they are not counted as executed coverage. Run this gate -before qualifying live cells; the live launcher does not run Cargo implicitly. +explicitly skipped; they are not counted as executed coverage. The live launcher +runs the prerequisites automatically before loading local credentials or starting +an isolated provider instance. Its subprocess receives only allowlisted toolchain +and operating-system variables. Disposable GitHub runners may resolve Cargo +dependencies; local prerequisites retain offline Cargo execution. + +The direct Playwright path also requires the retained prerequisite receipt. It +verifies the exact checkout SHA, evaluated-source fingerprint, all requested +Vitest reports and required assertions, and the Rust result before creating a +company. Missing, stale, partial, or failed prerequisites cannot qualify a cell. +Oracle/admission calibration is itself included in the prerequisite gate. ## Live matrix @@ -77,8 +86,9 @@ existing secret sanitizer; screenshots remain the original captured pixels. The oracle's identity is independent of the implementation constant, so changing the shipped manual cannot silently change the expected result. The suite -definition digest incorporates the fixture, journeys, grader, and execution -integration sources. Positive and plausible-negative support tests exercise +definition digest incorporates the evaluated default manual and shared prompt +implementation, fixture, journeys, grader, prerequisite, and execution integration +sources. Positive and plausible-negative support tests exercise missing/malformed receipts, manual regrowth, removed startup/resume procedures, absent connection guidance, and budget drift. Existing calibrated lifecycle graders still own task/chat success. @@ -108,3 +118,41 @@ need a provider-session reset to restore a previously replaced vendor base. Private Runner protocol definitions remain separate: they use mock control-plane operations and cannot substitute for this public hiring and assembled-prompt coverage. + +## GitHub qualification and matched comparison + +The instruction reductions and coverage are under review in +[PR #14948](https://github.com/paperclipai/paperclip/pull/14948). +The first diagnostic native Codex skill cell +[passed on GitHub](https://github.com/paperclipai/paperclip/actions/runs/37034213743) +at `a63437069de58d22ee5adbcb6a6202c007dcf037`. All seven independent skill/task +checks passed; the provider run lasted 34.745 seconds and the cell 56.660 seconds. +Its tiny public hire bundle and budget receipts passed, and cleanup passed. +This diagnostic predates enforced prerequisite admission and the expanded source +digest, so it is retained separately from the final qualification matrix. + +Dispatch the trusted workflow from `master`, with `target_branch` naming the +same-repository candidate and an exact cell selector first. The workflow resolves +that target once to an immutable SHA. Never dispatch target-controlled workflow +definitions with protected credentials. Protected environments, scoped provider +keys, frozen target dependencies, report sanitization, publication, and existing +bounded retry/cleanup policies retain their existing owners. + +A temporary `codex/stock-harness-previous-instructions` branch compares the same +24 cells with the previous default manual and shared startup/resume prompts. +It holds merged native Codex fix #14920 constant. Only those two production +instruction sources differ. Its explicit historical structural oracle expects +the old manual and records old generic procedures; the candidate's reduction +assertions remain mandatory. Historical tests verify that prior contract. +The independent skill/context/chat journeys, behavioral graders, fixtures, +models, effort, tools, permissions, and credentials are identical. The retained +comparison manifest records their hashes and the restored instruction revision. + +One full campaign per variant initially expects 48 provider turns, plus any +existing bounded automatic retries; the earlier one-cell diagnostic remains +separate. Compare behavioral results by profile and journey, with missing +evidence unqualified. Keep structural instruction differences separate from task +success. Report every attempt, failure attribution, provider timing, token usage, +reported costs and unknown spend. A reported zero subtotal is not proof of zero +provider spending. This small single-trial matrix cannot establish general coding +quality or broad performance equivalence, and does not compare #14920 before/after. diff --git a/tests/runner-e2e/launch.ts b/tests/runner-e2e/launch.ts index ceb5927ec7..33d709a71b 100644 --- a/tests/runner-e2e/launch.ts +++ b/tests/runner-e2e/launch.ts @@ -50,6 +50,7 @@ import { type RunnerE2EResult, } from "./types.js"; import { assertRunnerE2EPrerequisites } from "./prerequisites.js"; +import { prepareStockHarnessPreflight, STOCK_PREFLIGHT_ENV } from "./stock-harness-admission.js"; import { reapNewDetachedDarwinSharedMemory, snapshotDarwinSharedMemory, @@ -1098,6 +1099,9 @@ async function main() { // Keep admission before local-env loading and credential checks. Pending // profiles remain discoverable, but cannot reach a provider. assertRunnerE2EPrerequisites(executions); + if (executions.some(execution => execution.suite.id === "stock-harness")) { + process.env[STOCK_PREFLIGHT_ENV] = prepareStockHarnessPreflight(); + } await loadLocalEnvironment(process.env); const missingCredentials = [ diff --git a/tests/runner-e2e/runner.spec.ts b/tests/runner-e2e/runner.spec.ts index 430ed59eec..13d1905018 100644 --- a/tests/runner-e2e/runner.spec.ts +++ b/tests/runner-e2e/runner.spec.ts @@ -15,6 +15,7 @@ import { runEverydayFlow } from "./everyday-flow.js"; import { gradeTaskTitle } from "./task-titles.js"; import { runContextIntegrityFlow } from "./context-integrity-flow.js"; import { captureStockHarness, gradeStockHarness, gradeStockHire } from "./stock-harness.js"; +import { verifyStockHarnessPreflight, STOCK_PREFLIGHT_ENV } from "./stock-harness-admission.js"; import { createTaskThroughUi, submitTaskReply } from "./user-actions.js"; import { runFirstTaskFlow, setupFirstTaskFixtures } from "./first-task-flow.js"; @@ -815,6 +816,10 @@ for (const execution of executions) { }); try { + if (execution.suite.id === "stock-harness") { + const receipt = verifyStockHarnessPreflight(process.env[STOCK_PREFLIGHT_ENV]); + await writeSanitizedJson(snapshotsDir, "stock-harness-preflight.json", receipt, secrets); + } const experimental = await api.patch<{ enableNativeRunner: boolean; }>("/api/instance/settings/experimental", { diff --git a/tests/runner-e2e/stock-harness-admission.test.ts b/tests/runner-e2e/stock-harness-admission.test.ts new file mode 100644 index 0000000000..0872ae55e5 --- /dev/null +++ b/tests/runner-e2e/stock-harness-admission.test.ts @@ -0,0 +1,43 @@ +import { beforeEach, describe, expect, it, vi } from "vitest"; +import { spawnSync } from "node:child_process"; +import { prepareStockHarnessPreflight, stockPreflightEnvironment, verifyStockHarnessPreflight } from "./stock-harness-admission.js"; +import { CREDENTIAL_NAMES } from "./types.js"; + +vi.mock("node:child_process", () => ({ spawnSync: vi.fn() })); +const spawn = vi.mocked(spawnSync); +beforeEach(() => { vi.resetAllMocks(); vi.unstubAllEnvs(); }); + +describe("stock harness credential-free admission", () => { + it("allows toolchain paths and excludes every present or future credential", () => { + const source = { PATH: "/bin", HOME: "/home/fixture", CARGO_HOME: "/cargo", CI: "true", + GH_TOKEN: "secret", FUTURE_PROVIDER_API_KEY: "secret", ...Object.fromEntries(CREDENTIAL_NAMES.map(name => [name, "secret"])) }; + expect(stockPreflightEnvironment(source)).toEqual({ PATH: "/bin", HOME: "/home/fixture", CARGO_HOME: "/cargo", CI: "true" }); + }); + it("rejects missing receipts without invoking a process", () => { + expect(() => verifyStockHarnessPreflight(undefined)).toThrow("no prerequisite receipt"); + expect(spawn).not.toHaveBeenCalled(); + }); + it("fails before provider execution when the prerequisite subprocess fails", () => { + spawn.mockReturnValue({ status: 1 } as ReturnType); + expect(() => prepareStockHarnessPreflight()).toThrow("before provider execution"); + expect(spawn).toHaveBeenCalledTimes(1); + }); + it("verifies retained evidence after a passing prerequisite subprocess", () => { + spawn.mockReturnValueOnce({ status: 0 } as ReturnType); + spawn.mockReturnValueOnce({ status: 0, stdout: '{"passed":true}' } as ReturnType); + const receipt = prepareStockHarnessPreflight(); + expect(receipt).toMatch(/stock-harness-preflight-.*\/preflight.json$/); + expect(spawn.mock.calls[1]?.[1]).toContain(`--verify=${receipt}`); + }); + it("allows Cargo dependency resolution only on the disposable GitHub runner", () => { + vi.stubEnv("GITHUB_ACTIONS", "true"); + spawn.mockReturnValueOnce({ status: 0 } as ReturnType); + spawn.mockReturnValueOnce({ status: 0, stdout: '{}' } as ReturnType); + prepareStockHarnessPreflight(); + expect(spawn.mock.calls[0]?.[1]).toContain("--allow-rust-network"); + }); + it("refuses failed receipt verification", () => { + spawn.mockReturnValue({ status: 1, stderr: "stale source" } as ReturnType); + expect(() => verifyStockHarnessPreflight("/fixture/preflight.json")).toThrow("stale source"); + }); +}); diff --git a/tests/runner-e2e/stock-harness-admission.ts b/tests/runner-e2e/stock-harness-admission.ts new file mode 100644 index 0000000000..09ff72e8fc --- /dev/null +++ b/tests/runner-e2e/stock-harness-admission.ts @@ -0,0 +1,36 @@ +import { spawnSync } from "node:child_process"; +import { randomUUID } from "node:crypto"; +import path from "node:path"; + +const root = path.resolve(import.meta.dirname, "../.."); +export const STOCK_PREFLIGHT_ENV = "PAPERCLIP_RUNNER_E2E_STOCK_PREFLIGHT"; + +// An allowlist keeps every provider credential, ambient auth override, and GH +// token out of the prerequisite subprocess, including secrets added later. +export function stockPreflightEnvironment(source: NodeJS.ProcessEnv): NodeJS.ProcessEnv { + return Object.fromEntries([ + "PATH", "HOME", "TMPDIR", "TMP", "TEMP", "SYSTEMROOT", "LANG", "LC_ALL", + "CARGO_HOME", "RUSTUP_HOME", "CI", "GITHUB_ACTIONS", + ].flatMap(name => source[name] === undefined ? [] : [[name, source[name]]])); +} + +export function verifyStockHarnessPreflight(receiptPath: string | undefined) { + if (!receiptPath) throw new Error("Stock harness has no prerequisite receipt; use the live launcher."); + const run = spawnSync(process.execPath, [path.join(root, "tests/runner-e2e/stock-harness-checks.mjs"), `--verify=${receiptPath}`], { + cwd: root, env: stockPreflightEnvironment(process.env), encoding: "utf8", timeout: 30_000, + }); + if (run.status !== 0) throw new Error(`Stock harness prerequisite verification failed: ${run.stderr || run.error?.message || run.status}`); + return JSON.parse(run.stdout) as Record; +} + +export function prepareStockHarnessPreflight() { + const output = path.join(root, "tests/runner-e2e/results", `stock-harness-preflight-${randomUUID()}`); + const receipt = path.join(output, "preflight.json"); + const run = spawnSync(process.execPath, [path.join(root, "tests/runner-e2e/stock-harness-checks.mjs"), + `--output-dir=${output}`, ...(process.env.GITHUB_ACTIONS === "true" ? ["--allow-rust-network"] : [])], { + cwd: root, env: stockPreflightEnvironment(process.env), stdio: "inherit", timeout: 50 * 60_000, + }); + if (run.status !== 0) throw new Error(`Stock harness prerequisites failed before provider execution. Retained receipt: ${receipt}`); + verifyStockHarnessPreflight(receipt); + return receipt; +} diff --git a/tests/runner-e2e/stock-harness-checks.mjs b/tests/runner-e2e/stock-harness-checks.mjs index d3013151a7..4f8b2d5179 100644 --- a/tests/runner-e2e/stock-harness-checks.mjs +++ b/tests/runner-e2e/stock-harness-checks.mjs @@ -38,6 +38,13 @@ export const stockHarnessGates = [ { id: "SH-3-hermes", name: "Hermes shared-prompt delivery", cwd: "packages/adapters/hermes", files: ["src/server/prompt-rendering.test.ts"], required: ["renders standard assignment wake with task authority", "renders scoped planning wake authority"] }, + { id: "SH-eval", name: "Independent oracle and qualification admission", cwd: ".", + config: "tests/runner-e2e/vitest.config.ts", + files: ["tests/runner-e2e/stock-harness.test.ts", "tests/runner-e2e/stock-harness-checks.test.mjs", + "tests/runner-e2e/stock-harness-admission.test.ts", "tests/runner-e2e/stock-harness-digest.test.ts"], + required: ["rejects old SHA before providers", "allows toolchain paths and excludes every present or future credential", + "changes when the evaluated server/src/onboarding-assets/default/AGENTS.md changes", + "changes when the evaluated packages/adapter-utils/src/server-utils.ts changes"] }, ]; export function gradeGate(gate, report, exitCode) { @@ -54,37 +61,12 @@ export function gradeGate(gate, report, exitCode) { failedTests: report?.numFailedTests ?? 0, pendingTests: report?.numPendingTests ?? 0 }; } -export function main(args = process.argv.slice(2)) { - if (args.includes("--list")) { - console.log(JSON.stringify({ gates: stockHarnessGates, rust: "runtime_instructions_are_additive_for_codex_on_start_and_resume", live: "not invoked" }, null, 2)); - return; - } - if (args.length) throw new Error("Use --list or no arguments; this command never runs paid providers."); - const output = join(root, "tests/runner-e2e/results", `stock-harness-preflight-${new Date().toISOString().replaceAll(":", "-")}`); - mkdirSync(output, { recursive: true }); - const results = []; - for (const gate of stockHarnessGates) { - console.log(`Checking ${gate.id}: ${gate.name}`); - const file = join(output, `${gate.id}.json`); - const run = spawnSync(process.execPath, [join(root, "node_modules/vitest/vitest.mjs"), "run", ...gate.files, - ...(gate.testPattern ? ["--testNamePattern", gate.testPattern] : []), - "--reporter=default", "--reporter=json", `--outputFile.json=${file}`], - { cwd: resolve(root, gate.cwd), stdio: "inherit", timeout: 10 * 60_000 }); - let report; - try { report = JSON.parse(readFileSync(file, "utf8")); } catch { /* missing evidence fails closed below */ } - results.push(gradeGate(gate, report, run.status)); - } - const rust = spawnSync("cargo", ["test", "--offline", "-p", "paperclip-runner-core", "--test", "codex_provider", - "runtime_instructions_are_additive_for_codex_on_start_and_resume", "--", "--exact"], - { cwd: join(root, "packages/paperclip-runner/runner"), encoding: "utf8", timeout: 10 * 60_000 }); - const rustOutput = `${rust.stdout ?? ""}\n${rust.stderr ?? ""}`; - writeFileSync(join(output, "rust.txt"), rustOutput); - results.push({ id: "SH-1-rust", passed: rust.status === 0 && /test runtime_instructions_are_additive_for_codex_on_start_and_resume \.\.\. ok/.test(rustOutput), exitCode: rust.status }); - const git = spawnSync("git", ["rev-parse", "HEAD"], { cwd: root, encoding: "utf8" }); +export function sourceFingerprint() { const hash = createHash("sha256"); const sources = new Set([ ...stockHarnessGates.flatMap(gate => gate.files.map(file => join(gate.cwd, file))), "tests/runner-e2e/stock-harness.ts", "tests/runner-e2e/stock-harness-checks.mjs", "tests/runner-e2e/catalog.ts", + "tests/runner-e2e/stock-harness-admission.ts", "tests/runner-e2e/launch.ts", "tests/runner-e2e/runner.spec.ts", "packages/adapter-utils/src/server-utils.ts", "server/src/onboarding-assets/default/AGENTS.md", "server/src/routes/agents.ts", "packages/paperclip-runner/src/drivers/codex/codex-app-server-driver-impl.ts", "packages/paperclip-runner/src/live/runnerd-codex-transport.ts", @@ -98,9 +80,76 @@ export function main(args = process.argv.slice(2)) { try { hash.update(readFileSync(join(root, source))); } catch { sourceErrors.push(source); } } + return { fingerprint: hash.digest("hex"), sourceErrors }; +} + +export function assertPreflightReceipt(report, current) { + const expected = [...stockHarnessGates.map(gate => gate.id), "SH-1-rust"]; + if (report?.schema !== "paperclip.stock-harness-preflight.v1" || report.passed !== true || + report.providerCalls !== 0 || report.sourceSha !== current.sha || + report.sourceFingerprint !== current.fingerprint || report.sourceErrors?.length !== 0 || + !Array.isArray(report.gates) || report.gates.length !== expected.length || + expected.some(id => report.gates.filter(gate => gate.id === id && gate.passed === true && gate.exitCode === 0).length !== 1)) { + throw new Error("Stock harness requires passing prerequisites for this exact source SHA and fingerprint."); + } + return report; +} + +export function main(args = process.argv.slice(2)) { + if (args.includes("--list")) { + console.log(JSON.stringify({ gates: stockHarnessGates, rust: "runtime_instructions_are_additive_for_codex_on_start_and_resume", live: "not invoked" }, null, 2)); + return; + } + if (args.some(arg => !arg.startsWith("--output-dir=") && !arg.startsWith("--verify=") && arg !== "--allow-rust-network")) + throw new Error("Use --list, --output-dir=, --verify=, or --allow-rust-network; never paid providers."); + const git = spawnSync("git", ["rev-parse", "HEAD"], { cwd: root, encoding: "utf8" }); + const verify = args.find(arg => arg.startsWith("--verify="))?.slice("--verify=".length); + if (verify) { + const source = sourceFingerprint(); + if (git.status !== 0 || source.sourceErrors.length) throw new Error("Cannot verify stock harness source provenance."); + const report = assertPreflightReceipt(JSON.parse(readFileSync(verify, "utf8")), { + sha: git.stdout.trim(), fingerprint: source.fingerprint, + }); + const output = resolve(verify, ".."); + for (const gate of stockHarnessGates) { + const grade = gradeGate(gate, JSON.parse(readFileSync(join(output, `${gate.id}.json`), "utf8")), 0); + if (!grade.passed) throw new Error(`Missing or failed retained prerequisite assertions: ${gate.id}`); + } + if (!/test runtime_instructions_are_additive_for_codex_on_start_and_resume \.\.\. ok/.test(readFileSync(join(output, "rust.txt"), "utf8"))) + throw new Error("Missing passing retained Rust prerequisite."); + console.log(JSON.stringify(report)); + return report; + } + const output = args.find(arg => arg.startsWith("--output-dir="))?.slice("--output-dir=".length) ?? + join(root, "tests/runner-e2e/results", `stock-harness-preflight-${new Date().toISOString().replaceAll(":", "-")}`); + mkdirSync(output, { recursive: true }); + const env = Object.fromEntries([ + "PATH", "HOME", "TMPDIR", "TMP", "TEMP", "SYSTEMROOT", "LANG", "LC_ALL", + "CARGO_HOME", "RUSTUP_HOME", "CI", "GITHUB_ACTIONS", + ].flatMap(name => process.env[name] === undefined ? [] : [[name, process.env[name]]])); + const results = []; + for (const gate of stockHarnessGates) { + console.log(`Checking ${gate.id}: ${gate.name}`); + const file = join(output, `${gate.id}.json`); + const run = spawnSync(process.execPath, [join(root, "node_modules/vitest/vitest.mjs"), "run", ...gate.files, + ...(gate.config ? ["--config", gate.config] : []), + ...(gate.testPattern ? ["--testNamePattern", gate.testPattern] : []), + "--reporter=default", "--reporter=json", `--outputFile.json=${file}`], + { cwd: resolve(root, gate.cwd), env, stdio: "inherit", timeout: 10 * 60_000 }); + let report; + try { report = JSON.parse(readFileSync(file, "utf8")); } catch { /* missing evidence fails closed below */ } + results.push(gradeGate(gate, report, run.status)); + } + const rust = spawnSync("cargo", ["test", ...(args.includes("--allow-rust-network") ? [] : ["--offline"]), "-p", "paperclip-runner-core", "--test", "codex_provider", + "runtime_instructions_are_additive_for_codex_on_start_and_resume", "--", "--exact"], + { cwd: join(root, "packages/paperclip-runner/runner"), env, encoding: "utf8", timeout: 10 * 60_000 }); + const rustOutput = `${rust.stdout ?? ""}\n${rust.stderr ?? ""}`; + writeFileSync(join(output, "rust.txt"), rustOutput); + results.push({ id: "SH-1-rust", passed: rust.status === 0 && /test runtime_instructions_are_additive_for_codex_on_start_and_resume \.\.\. ok/.test(rustOutput), exitCode: rust.status }); + const { fingerprint, sourceErrors } = sourceFingerprint(); const report = { schema: "paperclip.stock-harness-preflight.v1", sourceSha: git.stdout?.trim() || null, - sourceFingerprint: hash.digest("hex"), measuredAt: new Date().toISOString(), providerCalls: 0, - live: "not_run", passed: sourceErrors.length === 0 && results.every(row => row.passed), sourceErrors, gates: results }; + sourceFingerprint: fingerprint, measuredAt: new Date().toISOString(), providerCalls: 0, + live: "not_run", passed: git.status === 0 && sourceErrors.length === 0 && results.every(row => row.passed), sourceErrors, gates: results }; writeFileSync(join(output, "preflight.json"), JSON.stringify(report, null, 2) + "\n"); console.log(`Stock harness prerequisite evidence: ${output}/preflight.json`); if (!report.passed) process.exitCode = 1; diff --git a/tests/runner-e2e/stock-harness-checks.test.mjs b/tests/runner-e2e/stock-harness-checks.test.mjs index 3397cae78c..2af6a8fee6 100644 --- a/tests/runner-e2e/stock-harness-checks.test.mjs +++ b/tests/runner-e2e/stock-harness-checks.test.mjs @@ -1,5 +1,5 @@ import { describe, expect, it } from "vitest"; -import { gradeGate, stockHarnessGates } from "./stock-harness-checks.mjs"; +import { assertPreflightReceipt, gradeGate, stockHarnessGates } from "./stock-harness-checks.mjs"; const gate = { id: "SH-test", name: "Boundary", files: ["boundary.test.ts"], required: ["must preserve instructions"] }; const report = () => ({ testResults: [{ name: "/repo/boundary.test.ts", status: "passed", @@ -7,7 +7,7 @@ const report = () => ({ testResults: [{ name: "/repo/boundary.test.ts", status: numTotalTests: 1, numPassedTests: 1, numFailedTests: 0, numPendingTests: 0 }); describe("stock harness prerequisite coverage", () => { it("maps every implemented change to an executable gate", () => { - expect(stockHarnessGates.map(gate => gate.id)).toEqual(["SH-1", "SH-2", "SH-3", "SH-3-hermes"]); + expect(stockHarnessGates.map(gate => gate.id)).toEqual(["SH-1", "SH-2", "SH-3", "SH-3-hermes", "SH-eval"]); expect(stockHarnessGates.every(gate => gate.files.length > 0 && gate.required.length > 0)).toBe(true); }); it("accepts an executed passing boundary", () => expect(gradeGate(gate, report(), 0).passed).toBe(true)); @@ -38,3 +38,23 @@ describe("stock harness prerequisite coverage", () => { expect(stockHarnessGates.find(gate => gate.id === "SH-3-hermes").cwd).toBe("packages/adapters/hermes"); }); }); + +describe("stock harness prerequisite admission", () => { + const current = { sha: "a".repeat(40), fingerprint: "b".repeat(64) }; + const receipt = () => ({ schema: "paperclip.stock-harness-preflight.v1", passed: true, + providerCalls: 0, sourceSha: current.sha, sourceFingerprint: current.fingerprint, + sourceErrors: [], gates: [...stockHarnessGates.map(g => g.id), "SH-1-rust"].map(id => ({ id, passed: true, exitCode: 0 })) }); + it("admits the same passing source revision", () => expect(assertPreflightReceipt(receipt(), current).passed).toBe(true)); + it.each([ + ["old SHA", r => { r.sourceSha = "c".repeat(40); }], + ["changed source", r => { r.sourceFingerprint = "d".repeat(64); }], + ["failed boundary", r => { r.gates[0].passed = false; }], + ["nonzero exit", r => { r.gates[0].exitCode = 1; }], + ["missing Rust", r => { r.gates.pop(); }], + ["duplicate boundary", r => { r.gates[1] = r.gates[0]; }], + ["provider calls", r => { r.providerCalls = 1; }], + ["missing source", r => { r.sourceErrors.push("missing.ts"); }], + ])("rejects %s before providers", (_name, mutate) => { + const r = receipt(); mutate(r); expect(() => assertPreflightReceipt(r, current)).toThrow("exact source SHA and fingerprint"); + }); +}); diff --git a/tests/runner-e2e/stock-harness-digest.test.ts b/tests/runner-e2e/stock-harness-digest.test.ts new file mode 100644 index 0000000000..05c08b4f5d --- /dev/null +++ b/tests/runner-e2e/stock-harness-digest.test.ts @@ -0,0 +1,15 @@ +import { describe, expect, it, vi } from "vitest"; +import { readFileSync } from "node:fs"; +import { stockHarnessSourceDigest } from "./stock-harness.js"; + +vi.mock("node:fs", async importOriginal => ({ ...await importOriginal(), readFileSync: vi.fn() })); + +describe("stock harness instruction revision", () => { + it.each(["server/src/onboarding-assets/default/AGENTS.md", "packages/adapter-utils/src/server-utils.ts"])( + "changes when the evaluated %s changes", source => { + vi.mocked(readFileSync).mockImplementation(() => Buffer.from("unchanged")); + const original = stockHarnessSourceDigest(); + vi.mocked(readFileSync).mockImplementation(file => Buffer.from(String(file).endsWith(source) ? "changed instructions" : "unchanged")); + expect(stockHarnessSourceDigest()).not.toBe(original); + }); +}); diff --git a/tests/runner-e2e/stock-harness.ts b/tests/runner-e2e/stock-harness.ts index 5993e06cb1..ac8e8dec65 100644 --- a/tests/runner-e2e/stock-harness.ts +++ b/tests/runner-e2e/stock-harness.ts @@ -110,6 +110,9 @@ export function stockHarnessSourceDigest() { for (const source of [ "stock-harness.ts", "context-integrity-cases.ts", "context-integrity-scoring.ts", "context-integrity-flow.ts", "chat-cases.ts", "chat-flow.ts", "live-fixtures.ts", "runner.spec.ts", + "stock-harness-checks.mjs", "stock-harness-admission.ts", + "../../server/src/onboarding-assets/default/AGENTS.md", + "../../packages/adapter-utils/src/server-utils.ts", ]) hash.update(source).update(readFileSync(new URL(source, import.meta.url))); return hash.digest("hex"); }