diff --git a/doc/plans/2026-10-02-stock-harness-paperclip-checklist.md b/doc/plans/2026-10-02-stock-harness-paperclip-checklist.md index 1455e78acf..4a9d35b6b4 100644 --- a/doc/plans/2026-10-02-stock-harness-paperclip-checklist.md +++ b/doc/plans/2026-10-02-stock-harness-paperclip-checklist.md @@ -397,6 +397,8 @@ will address them. Record intentional behavior explicitly rather than as a bug. | 2026-10-02 | Dotta directed an identity-only default manual with no skill or runtime pointers; the harness already handles coordination. | Reduced the default to eight words. Existing hire and onboarding suites passed all 64 tests. Shared prompts and role templates remain pending. | | 2026-10-02 | Dotta requested three explicit shared-prompt follow-ups and selected common legacy startup/resume reduction first. | Follow-ups 2.1–2.3 recorded. 2.1 complete locally: both defaults 113 words; generic resume contract removed. 563 focused tests and shared utility typecheck/build passed. Extra carriers and native instruction reduction remain pending. | | 2026-10-02 | Dotta requested executable eval coverage for everything implemented so far and all subsequent changes before continuing. | Added SH-1–SH-3 coverage map, credential-free prerequisite, and 24 production-default-hire cells. 478 prerequisite and 860 support tests passed; E2E typecheck/discovery passed. Live provider results remain `not_run`; item 2.2/2.3 unchanged. | +| 2026-10-02 | Dotta requested PRs and GitHub-runner before/after qualification while discussing subsequent work separately. | Draft [PR #14948](https://github.com/paperclipai/paperclip/pull/14948); native Codex diagnostic [passed](https://github.com/paperclipai/paperclip/actions/runs/37034213743), 34.745 s provider / 56.660 s cell. Comparison restores only the prior manual/shared prompts and holds #14920 constant; no general coding-quality claim. | +| 2026-10-02 | Full candidate cold setup failed before provider admission; stopped and retained the attempt. | [Run 37037105491](https://github.com/paperclipai/paperclip/actions/runs/37037105491), target `36e987246`: prerequisite imports lacked the plugin SDK build. Added ordinary dependency setup before credential-free prerequisites and exact-source coverage of connection guidance. Cold pilot and full matched campaigns pending. | For each completed item, add the chosen behavior, changed paths, verification results, remaining exceptions, and follow-ups here before checking it off. diff --git a/tests/runner-e2e/STOCK-HARNESS.md b/tests/runner-e2e/STOCK-HARNESS.md index bfa2156939..f7b2e30d86 100644 --- a/tests/runner-e2e/STOCK-HARNESS.md +++ b/tests/runner-e2e/STOCK-HARNESS.md @@ -25,6 +25,8 @@ runs the prerequisites automatically before loading local credentials or startin an isolated provider instance. Its subprocess receives only allowlisted toolchain and operating-system variables. Disposable GitHub runners may resolve Cargo dependencies; local prerequisites retain offline Cargo execution. +The ordinary server SDK dependency builder prepares the shared/SDK outputs +needed by route tests on a cold install with lifecycle scripts disabled. The direct Playwright path also requires the retained prerequisite receipt. It verifies the exact checkout SHA, evaluated-source fingerprint, all requested @@ -86,8 +88,8 @@ existing secret sanitizer; screenshots remain the original captured pixels. The oracle's identity is independent of the implementation constant, so changing the shipped manual cannot silently change the expected result. The suite -definition digest incorporates the evaluated default manual and shared prompt -implementation, fixture, journeys, grader, prerequisite, and execution integration +definition digest incorporates the evaluated default manual, shared prompt +implementation and connection guidance, fixture, journeys, grader, prerequisite, and execution integration sources. Positive and plausible-negative support tests exercise missing/malformed receipts, manual regrowth, removed startup/resume procedures, absent connection guidance, and budget drift. Existing calibrated lifecycle @@ -108,7 +110,7 @@ are not counted as passing coverage. Local evidence is retained under `results/stock-harness-preflight-2026-10-02T16-15-39.064Z/preflight.json`. Earlier interrupted, discovery-failure, and setup/test-timeout attempts remain retained; the final unchanged-assertion retry passed. No live cells or paid -providers were run. This establishes executable coverage, not a live reliability +providers were run during that initial setup. This establishes executable coverage, not a live reliability result or improved coding quality. A quality claim needs comparable tasks, models, effort, tools, and independently graded before/after results. @@ -131,6 +133,15 @@ Its tiny public hire bundle and budget receipts passed, and cleanup passed. This diagnostic predates enforced prerequisite admission and the expanded source digest, so it is retained separately from the final qualification matrix. +The first full candidate attempt at `36e987246b649927e96ce1184cd616c4e490106e` +[was cancelled during prerequisites](https://github.com/paperclipai/paperclip/actions/runs/37037105491). +Cold protected installs disable lifecycle scripts, leaving the plugin SDK unbuilt; +SH-2/SH-3 could not import it. Provider admission was not reached. This setup +failure is retained separately from behavioral results. The prerequisite now runs +the ordinary server dependency builder first, retains its output and exit status, +and refuses admission when setup fails. A cold legacy pilot must pass before the +full matrix retry. + Dispatch the trusted workflow from `master`, with `target_branch` naming the same-repository candidate and an exact cell selector first. The workflow resolves that target once to an immutable SHA. Never dispatch target-controlled workflow diff --git a/tests/runner-e2e/stock-harness-checks.mjs b/tests/runner-e2e/stock-harness-checks.mjs index 4f8b2d5179..3e3368d99f 100644 --- a/tests/runner-e2e/stock-harness-checks.mjs +++ b/tests/runner-e2e/stock-harness-checks.mjs @@ -44,7 +44,8 @@ export const stockHarnessGates = [ "tests/runner-e2e/stock-harness-admission.test.ts", "tests/runner-e2e/stock-harness-digest.test.ts"], required: ["rejects old SHA before providers", "allows toolchain paths and excludes every present or future credential", "changes when the evaluated server/src/onboarding-assets/default/AGENTS.md changes", - "changes when the evaluated packages/adapter-utils/src/server-utils.ts changes"] }, + "changes when the evaluated packages/adapter-utils/src/server-utils.ts changes", + "changes when the evaluated packages/shared/src/connection-intent-guidance.ts changes"] }, ]; export function gradeGate(gate, report, exitCode) { @@ -67,7 +68,8 @@ export function sourceFingerprint() { ...stockHarnessGates.flatMap(gate => gate.files.map(file => join(gate.cwd, file))), "tests/runner-e2e/stock-harness.ts", "tests/runner-e2e/stock-harness-checks.mjs", "tests/runner-e2e/catalog.ts", "tests/runner-e2e/stock-harness-admission.ts", "tests/runner-e2e/launch.ts", "tests/runner-e2e/runner.spec.ts", - "packages/adapter-utils/src/server-utils.ts", "server/src/onboarding-assets/default/AGENTS.md", "server/src/routes/agents.ts", + "packages/adapter-utils/src/server-utils.ts", "packages/shared/src/connection-intent-guidance.ts", + "server/src/onboarding-assets/default/AGENTS.md", "server/src/routes/agents.ts", "scripts/ensure-plugin-build-deps.mjs", "packages/paperclip-runner/src/drivers/codex/codex-app-server-driver-impl.ts", "packages/paperclip-runner/src/live/runnerd-codex-transport.ts", "packages/paperclip-runner/src/live/live-session.ts", @@ -85,7 +87,8 @@ export function sourceFingerprint() { export function assertPreflightReceipt(report, current) { const expected = [...stockHarnessGates.map(gate => gate.id), "SH-1-rust"]; - if (report?.schema !== "paperclip.stock-harness-preflight.v1" || report.passed !== true || + if (report?.schema !== "paperclip.stock-harness-preflight.v2" || report.passed !== true || + report.setup?.passed !== true || report.setup?.exitCode !== 0 || report.providerCalls !== 0 || report.sourceSha !== current.sha || report.sourceFingerprint !== current.fingerprint || report.sourceErrors?.length !== 0 || !Array.isArray(report.gates) || report.gates.length !== expected.length || @@ -127,6 +130,23 @@ export function main(args = process.argv.slice(2)) { "PATH", "HOME", "TMPDIR", "TMP", "TEMP", "SYSTEMROOT", "LANG", "LC_ALL", "CARGO_HOME", "RUSTUP_HOME", "CI", "GITHUB_ACTIONS", ].flatMap(name => process.env[name] === undefined ? [] : [[name, process.env[name]]])); + // A protected cold install disables lifecycle scripts. Use the same ordinary + // SDK dependency builder as server startup, before importing server tests. + const setupRun = spawnSync(process.execPath, [join(root, "scripts/ensure-plugin-build-deps.mjs")], { + cwd: root, env, encoding: "utf8", timeout: 5 * 60_000, + }); + writeFileSync(join(output, "setup.txt"), `${setupRun.stdout ?? ""}\n${setupRun.stderr ?? ""}`); + const setup = { passed: setupRun.status === 0, exitCode: setupRun.status }; + if (!setup.passed) { + const source = sourceFingerprint(); + writeFileSync(join(output, "preflight.json"), JSON.stringify({ + schema: "paperclip.stock-harness-preflight.v2", sourceSha: git.stdout?.trim() || null, + sourceFingerprint: source.fingerprint, measuredAt: new Date().toISOString(), providerCalls: 0, + live: "not_run", passed: false, sourceErrors: source.sourceErrors, setup, gates: [], + }, null, 2) + "\n"); + process.exitCode = 1; + return; + } const results = []; for (const gate of stockHarnessGates) { console.log(`Checking ${gate.id}: ${gate.name}`); @@ -147,9 +167,9 @@ export function main(args = process.argv.slice(2)) { writeFileSync(join(output, "rust.txt"), rustOutput); results.push({ id: "SH-1-rust", passed: rust.status === 0 && /test runtime_instructions_are_additive_for_codex_on_start_and_resume \.\.\. ok/.test(rustOutput), exitCode: rust.status }); const { fingerprint, sourceErrors } = sourceFingerprint(); - const report = { schema: "paperclip.stock-harness-preflight.v1", sourceSha: git.stdout?.trim() || null, + const report = { schema: "paperclip.stock-harness-preflight.v2", sourceSha: git.stdout?.trim() || null, sourceFingerprint: fingerprint, measuredAt: new Date().toISOString(), providerCalls: 0, - live: "not_run", passed: git.status === 0 && sourceErrors.length === 0 && results.every(row => row.passed), sourceErrors, gates: results }; + live: "not_run", passed: git.status === 0 && sourceErrors.length === 0 && results.every(row => row.passed), sourceErrors, setup, gates: results }; writeFileSync(join(output, "preflight.json"), JSON.stringify(report, null, 2) + "\n"); console.log(`Stock harness prerequisite evidence: ${output}/preflight.json`); if (!report.passed) process.exitCode = 1; diff --git a/tests/runner-e2e/stock-harness-checks.test.mjs b/tests/runner-e2e/stock-harness-checks.test.mjs index 2af6a8fee6..b3884139af 100644 --- a/tests/runner-e2e/stock-harness-checks.test.mjs +++ b/tests/runner-e2e/stock-harness-checks.test.mjs @@ -41,7 +41,8 @@ describe("stock harness prerequisite coverage", () => { describe("stock harness prerequisite admission", () => { const current = { sha: "a".repeat(40), fingerprint: "b".repeat(64) }; - const receipt = () => ({ schema: "paperclip.stock-harness-preflight.v1", passed: true, + const receipt = () => ({ schema: "paperclip.stock-harness-preflight.v2", passed: true, + setup: { passed: true, exitCode: 0 }, providerCalls: 0, sourceSha: current.sha, sourceFingerprint: current.fingerprint, sourceErrors: [], gates: [...stockHarnessGates.map(g => g.id), "SH-1-rust"].map(id => ({ id, passed: true, exitCode: 0 })) }); it("admits the same passing source revision", () => expect(assertPreflightReceipt(receipt(), current).passed).toBe(true)); @@ -54,6 +55,7 @@ describe("stock harness prerequisite admission", () => { ["duplicate boundary", r => { r.gates[1] = r.gates[0]; }], ["provider calls", r => { r.providerCalls = 1; }], ["missing source", r => { r.sourceErrors.push("missing.ts"); }], + ["failed cold setup", r => { r.setup.passed = false; r.setup.exitCode = 1; }], ])("rejects %s before providers", (_name, mutate) => { const r = receipt(); mutate(r); expect(() => assertPreflightReceipt(r, current)).toThrow("exact source SHA and fingerprint"); }); diff --git a/tests/runner-e2e/stock-harness-digest.test.ts b/tests/runner-e2e/stock-harness-digest.test.ts index 05c08b4f5d..0f51c70ee6 100644 --- a/tests/runner-e2e/stock-harness-digest.test.ts +++ b/tests/runner-e2e/stock-harness-digest.test.ts @@ -5,7 +5,7 @@ import { stockHarnessSourceDigest } from "./stock-harness.js"; vi.mock("node:fs", async importOriginal => ({ ...await importOriginal(), readFileSync: vi.fn() })); describe("stock harness instruction revision", () => { - it.each(["server/src/onboarding-assets/default/AGENTS.md", "packages/adapter-utils/src/server-utils.ts"])( + it.each(["server/src/onboarding-assets/default/AGENTS.md", "packages/adapter-utils/src/server-utils.ts", "packages/shared/src/connection-intent-guidance.ts"])( "changes when the evaluated %s changes", source => { vi.mocked(readFileSync).mockImplementation(() => Buffer.from("unchanged")); const original = stockHarnessSourceDigest(); diff --git a/tests/runner-e2e/stock-harness.ts b/tests/runner-e2e/stock-harness.ts index ac8e8dec65..017a0972cc 100644 --- a/tests/runner-e2e/stock-harness.ts +++ b/tests/runner-e2e/stock-harness.ts @@ -113,6 +113,7 @@ export function stockHarnessSourceDigest() { "stock-harness-checks.mjs", "stock-harness-admission.ts", "../../server/src/onboarding-assets/default/AGENTS.md", "../../packages/adapter-utils/src/server-utils.ts", + "../../packages/shared/src/connection-intent-guidance.ts", ]) hash.update(source).update(readFileSync(new URL(source, import.meta.url))); return hash.digest("hex"); }