fix(evals): prepare cold prerequisites and fingerprint connection guidance

Co-Authored-By: Paperclip <noreply@paperclip.ing>

doc/plans/2026-10-02-stock-harness-paperclip-checklist.md

@@ -397,6 +397,8 @@ will address them. Record intentional behavior explicitly rather than as a bug.
 | 2026-10-02 | Dotta directed an identity-only default manual with no skill or runtime pointers; the harness already handles coordination. | Reduced the default to eight words. Existing hire and onboarding suites passed all 64 tests. Shared prompts and role templates remain pending. |
 | 2026-10-02 | Dotta requested three explicit shared-prompt follow-ups and selected common legacy startup/resume reduction first. | Follow-ups 2.1–2.3 recorded. 2.1 complete locally: both defaults 113 words; generic resume contract removed. 563 focused tests and shared utility typecheck/build passed. Extra carriers and native instruction reduction remain pending. |
 | 2026-10-02 | Dotta requested executable eval coverage for everything implemented so far and all subsequent changes before continuing. | Added SH-1–SH-3 coverage map, credential-free prerequisite, and 24 production-default-hire cells. 478 prerequisite and 860 support tests passed; E2E typecheck/discovery passed. Live provider results remain `not_run`; item 2.2/2.3 unchanged. |
+| 2026-10-02 | Dotta requested PRs and GitHub-runner before/after qualification while discussing subsequent work separately. | Draft [PR #14948](https://github.com/paperclipai/paperclip/pull/14948); native Codex diagnostic [passed](https://github.com/paperclipai/paperclip/actions/runs/37034213743), 34.745 s provider / 56.660 s cell. Comparison restores only the prior manual/shared prompts and holds #14920 constant; no general coding-quality claim. |
+| 2026-10-02 | Full candidate cold setup failed before provider admission; stopped and retained the attempt. | [Run 37037105491](https://github.com/paperclipai/paperclip/actions/runs/37037105491), target `36e987246`: prerequisite imports lacked the plugin SDK build. Added ordinary dependency setup before credential-free prerequisites and exact-source coverage of connection guidance. Cold pilot and full matched campaigns pending. |
 
 For each completed item, add the chosen behavior, changed paths, verification
 results, remaining exceptions, and follow-ups here before checking it off.

tests/runner-e2e/STOCK-HARNESS.md

@@ -25,6 +25,8 @@ runs the prerequisites automatically before loading local credentials or startin
 an isolated provider instance. Its subprocess receives only allowlisted toolchain
 and operating-system variables. Disposable GitHub runners may resolve Cargo
 dependencies; local prerequisites retain offline Cargo execution.
+The ordinary server SDK dependency builder prepares the shared/SDK outputs
+needed by route tests on a cold install with lifecycle scripts disabled.
 
 The direct Playwright path also requires the retained prerequisite receipt. It
 verifies the exact checkout SHA, evaluated-source fingerprint, all requested
@@ -86,8 +88,8 @@ existing secret sanitizer; screenshots remain the original captured pixels.
 
 The oracle's identity is independent of the implementation constant, so changing
 the shipped manual cannot silently change the expected result. The suite
-definition digest incorporates the evaluated default manual and shared prompt
-implementation, fixture, journeys, grader, prerequisite, and execution integration
+definition digest incorporates the evaluated default manual, shared prompt
+implementation and connection guidance, fixture, journeys, grader, prerequisite, and execution integration
 sources. Positive and plausible-negative support tests exercise
 missing/malformed receipts, manual regrowth, removed startup/resume procedures,
 absent connection guidance, and budget drift. Existing calibrated lifecycle
@@ -108,7 +110,7 @@ are not counted as passing coverage. Local evidence is retained under
 `results/stock-harness-preflight-2026-10-02T16-15-39.064Z/preflight.json`.
 Earlier interrupted, discovery-failure, and setup/test-timeout attempts remain
 retained; the final unchanged-assertion retry passed. No live cells or paid
-providers were run. This establishes executable coverage, not a live reliability
+providers were run during that initial setup. This establishes executable coverage, not a live reliability
 result or improved coding quality. A quality claim needs comparable tasks,
 models, effort, tools, and independently graded before/after results.
 
@@ -131,6 +133,15 @@ Its tiny public hire bundle and budget receipts passed, and cleanup passed.
 This diagnostic predates enforced prerequisite admission and the expanded source
 digest, so it is retained separately from the final qualification matrix.
 
+The first full candidate attempt at `36e987246b649927e96ce1184cd616c4e490106e`
+[was cancelled during prerequisites](https://github.com/paperclipai/paperclip/actions/runs/37037105491).
+Cold protected installs disable lifecycle scripts, leaving the plugin SDK unbuilt;
+SH-2/SH-3 could not import it. Provider admission was not reached. This setup
+failure is retained separately from behavioral results. The prerequisite now runs
+the ordinary server dependency builder first, retains its output and exit status,
+and refuses admission when setup fails. A cold legacy pilot must pass before the
+full matrix retry.
+
 Dispatch the trusted workflow from `master`, with `target_branch` naming the
 same-repository candidate and an exact cell selector first. The workflow resolves
 that target once to an immutable SHA. Never dispatch target-controlled workflow

tests/runner-e2e/stock-harness-checks.mjs

@@ -44,7 +44,8 @@ export const stockHarnessGates = [
       "tests/runner-e2e/stock-harness-admission.test.ts", "tests/runner-e2e/stock-harness-digest.test.ts"],
     required: ["rejects old SHA before providers", "allows toolchain paths and excludes every present or future credential",
       "changes when the evaluated server/src/onboarding-assets/default/AGENTS.md changes",
-      "changes when the evaluated packages/adapter-utils/src/server-utils.ts changes"] },
+      "changes when the evaluated packages/adapter-utils/src/server-utils.ts changes",
+      "changes when the evaluated packages/shared/src/connection-intent-guidance.ts changes"] },
 ];
 
 export function gradeGate(gate, report, exitCode) {
@@ -67,7 +68,8 @@ export function sourceFingerprint() {
     ...stockHarnessGates.flatMap(gate => gate.files.map(file => join(gate.cwd, file))),
     "tests/runner-e2e/stock-harness.ts", "tests/runner-e2e/stock-harness-checks.mjs", "tests/runner-e2e/catalog.ts",
     "tests/runner-e2e/stock-harness-admission.ts", "tests/runner-e2e/launch.ts", "tests/runner-e2e/runner.spec.ts",
-    "packages/adapter-utils/src/server-utils.ts", "server/src/onboarding-assets/default/AGENTS.md", "server/src/routes/agents.ts",
+    "packages/adapter-utils/src/server-utils.ts", "packages/shared/src/connection-intent-guidance.ts",
+    "server/src/onboarding-assets/default/AGENTS.md", "server/src/routes/agents.ts", "scripts/ensure-plugin-build-deps.mjs",
     "packages/paperclip-runner/src/drivers/codex/codex-app-server-driver-impl.ts",
     "packages/paperclip-runner/src/live/runnerd-codex-transport.ts",
     "packages/paperclip-runner/src/live/live-session.ts",
@@ -85,7 +87,8 @@ export function sourceFingerprint() {
 
 export function assertPreflightReceipt(report, current) {
   const expected = [...stockHarnessGates.map(gate => gate.id), "SH-1-rust"];
-  if (report?.schema !== "paperclip.stock-harness-preflight.v1" || report.passed !== true ||
+  if (report?.schema !== "paperclip.stock-harness-preflight.v2" || report.passed !== true ||
+      report.setup?.passed !== true || report.setup?.exitCode !== 0 ||
       report.providerCalls !== 0 || report.sourceSha !== current.sha ||
       report.sourceFingerprint !== current.fingerprint || report.sourceErrors?.length !== 0 ||
       !Array.isArray(report.gates) || report.gates.length !== expected.length ||
@@ -127,6 +130,23 @@ export function main(args = process.argv.slice(2)) {
     "PATH", "HOME", "TMPDIR", "TMP", "TEMP", "SYSTEMROOT", "LANG", "LC_ALL",
     "CARGO_HOME", "RUSTUP_HOME", "CI", "GITHUB_ACTIONS",
   ].flatMap(name => process.env[name] === undefined ? [] : [[name, process.env[name]]]));
+  // A protected cold install disables lifecycle scripts. Use the same ordinary
+  // SDK dependency builder as server startup, before importing server tests.
+  const setupRun = spawnSync(process.execPath, [join(root, "scripts/ensure-plugin-build-deps.mjs")], {
+    cwd: root, env, encoding: "utf8", timeout: 5 * 60_000,
+  });
+  writeFileSync(join(output, "setup.txt"), `${setupRun.stdout ?? ""}\n${setupRun.stderr ?? ""}`);
+  const setup = { passed: setupRun.status === 0, exitCode: setupRun.status };
+  if (!setup.passed) {
+    const source = sourceFingerprint();
+    writeFileSync(join(output, "preflight.json"), JSON.stringify({
+      schema: "paperclip.stock-harness-preflight.v2", sourceSha: git.stdout?.trim() || null,
+      sourceFingerprint: source.fingerprint, measuredAt: new Date().toISOString(), providerCalls: 0,
+      live: "not_run", passed: false, sourceErrors: source.sourceErrors, setup, gates: [],
+    }, null, 2) + "\n");
+    process.exitCode = 1;
+    return;
+  }
   const results = [];
   for (const gate of stockHarnessGates) {
     console.log(`Checking ${gate.id}: ${gate.name}`);
@@ -147,9 +167,9 @@ export function main(args = process.argv.slice(2)) {
   writeFileSync(join(output, "rust.txt"), rustOutput);
   results.push({ id: "SH-1-rust", passed: rust.status === 0 && /test runtime_instructions_are_additive_for_codex_on_start_and_resume \.\.\. ok/.test(rustOutput), exitCode: rust.status });
   const { fingerprint, sourceErrors } = sourceFingerprint();
-  const report = { schema: "paperclip.stock-harness-preflight.v1", sourceSha: git.stdout?.trim() || null,
+  const report = { schema: "paperclip.stock-harness-preflight.v2", sourceSha: git.stdout?.trim() || null,
     sourceFingerprint: fingerprint, measuredAt: new Date().toISOString(), providerCalls: 0,
-    live: "not_run", passed: git.status === 0 && sourceErrors.length === 0 && results.every(row => row.passed), sourceErrors, gates: results };
+    live: "not_run", passed: git.status === 0 && sourceErrors.length === 0 && results.every(row => row.passed), sourceErrors, setup, gates: results };
   writeFileSync(join(output, "preflight.json"), JSON.stringify(report, null, 2) + "\n");
   console.log(`Stock harness prerequisite evidence: ${output}/preflight.json`);
   if (!report.passed) process.exitCode = 1;

tests/runner-e2e/stock-harness-checks.test.mjs

@@ -41,7 +41,8 @@ describe("stock harness prerequisite coverage", () => {
 
 describe("stock harness prerequisite admission", () => {
   const current = { sha: "a".repeat(40), fingerprint: "b".repeat(64) };
-  const receipt = () => ({ schema: "paperclip.stock-harness-preflight.v1", passed: true,
+  const receipt = () => ({ schema: "paperclip.stock-harness-preflight.v2", passed: true,
+    setup: { passed: true, exitCode: 0 },
     providerCalls: 0, sourceSha: current.sha, sourceFingerprint: current.fingerprint,
     sourceErrors: [], gates: [...stockHarnessGates.map(g => g.id), "SH-1-rust"].map(id => ({ id, passed: true, exitCode: 0 })) });
   it("admits the same passing source revision", () => expect(assertPreflightReceipt(receipt(), current).passed).toBe(true));
@@ -54,6 +55,7 @@ describe("stock harness prerequisite admission", () => {
     ["duplicate boundary", r => { r.gates[1] = r.gates[0]; }],
     ["provider calls", r => { r.providerCalls = 1; }],
     ["missing source", r => { r.sourceErrors.push("missing.ts"); }],
+    ["failed cold setup", r => { r.setup.passed = false; r.setup.exitCode = 1; }],
   ])("rejects %s before providers", (_name, mutate) => {
     const r = receipt(); mutate(r); expect(() => assertPreflightReceipt(r, current)).toThrow("exact source SHA and fingerprint");
   });

tests/runner-e2e/stock-harness-digest.test.ts

@@ -5,7 +5,7 @@ import { stockHarnessSourceDigest } from "./stock-harness.js";
 vi.mock("node:fs", async importOriginal => ({ ...await importOriginal<typeof import("node:fs")>(), readFileSync: vi.fn() }));
 
 describe("stock harness instruction revision", () => {
-  it.each(["server/src/onboarding-assets/default/AGENTS.md", "packages/adapter-utils/src/server-utils.ts"])(
+  it.each(["server/src/onboarding-assets/default/AGENTS.md", "packages/adapter-utils/src/server-utils.ts", "packages/shared/src/connection-intent-guidance.ts"])(
     "changes when the evaluated %s changes", source => {
       vi.mocked(readFileSync).mockImplementation(() => Buffer.from("unchanged"));
       const original = stockHarnessSourceDigest();

tests/runner-e2e/stock-harness.ts

@@ -113,6 +113,7 @@ export function stockHarnessSourceDigest() {
     "stock-harness-checks.mjs", "stock-harness-admission.ts",
     "../../server/src/onboarding-assets/default/AGENTS.md",
     "../../packages/adapter-utils/src/server-utils.ts",
+    "../../packages/shared/src/connection-intent-guidance.ts",
   ]) hash.update(source).update(readFileSync(new URL(source, import.meta.url)));
   return hash.digest("hex");
 }