test(evals): cover production default hires across stock harnesses

Co-Authored-By: Paperclip <noreply@paperclip.ing>

tests/runner-e2e/FIXTURES.md

@@ -17,6 +17,17 @@ profile, model qualification, environment, task, or ranking-snapshot change
 must change that fingerprint automatically so the dashboard can annotate the
 boundary instead of silently joining unlike totals.
 
+The explicit [stock-harness suite](STOCK-HARNESS.md) wraps existing profiles with
+`productionDefaultHireProfile`: omit only `instructionsBundle` so the public
+hire route loads the shipped default, while preserving runtime, permissions,
+auth, skills, and managed secret references. Do not replace this with a fixture
+copy of the default manual. Public receipts check the exact independently
+specified bundle before provider execution and again during cleanup, along with
+both budget hard stops and actual legacy invocation prompts. Missing evidence
+fails closed. The definition fingerprint includes the helper, graders, journey
+sources, live fixture, and execution integration; editing those sources changes
+the suite revision automatically.
+
 ## Agent profiles
 
 Add `RunnerProfileFixture` entries in `catalog.ts`. A profile declares:

tests/runner-e2e/README.md

@@ -442,6 +442,14 @@ pnpm test:e2e:runner -- --list --suite context-integrity
 pnpm test:e2e:runner -- --id context-integrity.runner-codex.local.ordered-comment-continuation
 ```
 
+`stock-harness` reuses ordered continuation, assigned-skill invocation, and chat
+restart journeys with production-default hires instead of the custom QA manual.
+Its 24 explicit local cells cover eight legacy/native profiles and are excluded
+from `--all`. Run `pnpm test:e2e:runner:stock-harness` for the credential-free
+instruction-layering, hire, and shared-prompt prerequisites. The
+[suite contract](STOCK-HARNESS.md) maps each change to its graders, budgets,
+evidence, and remaining qualification limits.
+
 Each hardening oracle has positive and plausible-negative calibration tests.
 The review grader parses the worker's saved JSON and compares both source values
 and the consistency verdict. Hiring requires one identity, correct reporting

tests/runner-e2e/STOCK-HARNESS.md

@@ -0,0 +1,110 @@
+# Stock harness with Paperclip
+
+This suite covers the instruction reductions tracked in the
+[working checklist](../../doc/plans/2026-10-02-stock-harness-paperclip-checklist.md).
+Existing context-integrity and chat evals supplied a custom QA instruction
+bundle. Their passing results therefore did not qualify a hire with the tiny
+production default. The new suite omits that fixture bundle and lets the public
+agent-creation route materialize the shipped default.
+
+## Coverage contract
+
+| Change | Required deterministic evidence | Product E2E evidence |
+| --- | --- | --- |
+| SH-1: native Codex preserves vendor base instructions | Serialized start/resume requests from the TypeScript driver, recovery paths, runnerd transport, Runner Lab/live sessions, and Rust provider use additive developer instructions. | Real new native Codex hires execute the three journeys below. Task success alone cannot prove vendor base preservation. |
+| SH-2: identity-only default hire manual | Existing public agent-creation and onboarding-asset tests cover default, custom, and CEO exceptions. | The public bundle is exactly one `AGENTS.md` containing the eight-word shipped identity, checked before provider execution and again during cleanup. |
+| SH-3: reduced shared legacy task/chat defaults and resume delta | Shared prompt tests and ACPX, Codex, OpenCode, Pi, Hermes, and Cursor Cloud adapter regressions retain runtime context and exclude removed generic procedures. | Actual legacy `adapter.invoke` prompts retain fresh identity/connection guidance, omit the removed procedures, and have complete receipts for every observed run. |
+
+`pnpm test:e2e:runner:stock-harness` runs these credential-free prerequisites,
+including the Rust test. It writes JSON reports, the source SHA, a working-source
+fingerprint, selected checks, test counts, and `providerCalls: 0` under
+`results/stock-harness-preflight-<UTC>/`. Missing reports or required skipped
+assertions fail. Unrelated native tests filtered by the name selector remain
+explicitly skipped; they are not counted as executed coverage. Run this gate
+before qualifying live cells; the live launcher does not run Cargo implicitly.
+
+## Live matrix
+
+There are 24 explicit local cells: these eight existing profiles each run three
+existing journeys with independently calibrated graders.
+
+- Legacy: `legacy-codex`, `legacy-claude`, `legacy-opencode`,
+  `legacy-acp-codex`, `legacy-acp-claude`.
+- Native: `runner-codex`, `runner-acpx-claude`, `runner-opencode`.
+
+| Journey | Expected provider turns | Observable outcome | Cell deadline |
+| --- | --- | --- | --- |
+| `assigned-skill-explicit-invocation` | 1 | Public skill creation/pinning, an explicit skill request, and saved output containing the marker available only in the skill body. | 12 minutes |
+| `ordered-comment-continuation` | 2 | Initial report followed by three ordered public comments, including repeated wording and a changed scope; final saved report preserves the ledger and requested scope. | 12 minutes |
+| `continuity-restart` | 3 | Task-backed chat retains the requested context across a server restart and subsequent replies. | 15 minutes |
+
+A full matrix expects 48 provider turns. Models, credentials, effort,
+permissions, assigned skills, environment, and managed secret references are
+inherited from the existing profile; only its QA manual is omitted. Both company
+and agent receive a 1,000-cent monthly hard stop before provider execution,
+verified through public records. These limits do not predict final spend:
+attempts, retries, partial runs, unknown billing, and cleanup remain in the
+existing campaign accounting and qualification rules.
+
+`stock-harness` is excluded from `--all`; select it explicitly. It has no Daytona
+cells. Pending or unrepresented harnesses are not live-qualified by this matrix.
+Pi/Hermes/Cursor Cloud rendering coverage is deterministic here. The separate
+Codex-through-ACP base-instruction patch is still an open checklist item: its
+legacy cells qualify the common prompt reduction, not vendor base preservation.
+
+## Run and inspect
+
+```sh
+# No credentials or paid providers:
+pnpm test:e2e:runner:stock-harness --list
+pnpm test:e2e:runner:stock-harness
+pnpm test:e2e:runner:typecheck
+pnpm test:e2e:runner:unit
+pnpm test:e2e:runner -- --list --suite stock-harness
+
+# With authorization and the selected profile's required provider credential:
+pnpm test:e2e:runner -- --id stock-harness.runner-codex.local.assigned-skill-explicit-invocation
+```
+
+Use an exact cell first, then expand profile/journey selections when its evidence
+is understood. The ordinary launcher, isolated instance, fixture cleanup,
+screenshots, attempt history, usage/cost reporting, and dashboard publisher are
+unchanged. `snapshots/stock-harness-hire.json` captures the public bundle and
+budgets before paid execution. `snapshots/stock-harness.json` captures the final
+bundle, budgets, run IDs, actual legacy prompts, and matcher verdicts. Evidence
+API failures retain an error snapshot and cannot pass. All snapshots use the
+existing secret sanitizer; screenshots remain the original captured pixels.
+
+The oracle's identity is independent of the implementation constant, so changing
+the shipped manual cannot silently change the expected result. The suite
+definition digest incorporates the fixture, journeys, grader, and execution
+integration sources. Positive and plausible-negative support tests exercise
+missing/malformed receipts, manual regrowth, removed startup/resume procedures,
+absent connection guidance, and budget drift. Existing calibrated lifecycle
+graders still own task/chat success.
+
+The skill oracle proves the requested pinned skill's output marker reached the
+saved result without appearing in the task request or follow-up comments. Skill
+tool/read event detection is retained as supporting evidence; it is not a
+required cross-provider tool-trace assertion.
+
+## Qualification status
+
+Setup was validated locally on 2026-10-02 with the deterministic prerequisites,
+Product E2E support tests, typecheck, and discovery: 478 executed prerequisite
+tests passed (477 TypeScript plus one Rust), along with 860 support tests and
+discovery of all 24 cells. The 313 unrelated native tests filtered by the gate
+are not counted as passing coverage. Local evidence is retained under
+`results/stock-harness-preflight-2026-10-02T16-15-39.064Z/preflight.json`.
+Earlier interrupted, discovery-failure, and setup/test-timeout attempts remain
+retained; the final unchanged-assertion retry passed. No live cells or paid
+providers were run. This establishes executable coverage, not a live reliability
+result or improved coding quality. A quality claim needs comparable tasks,
+models, effort, tools, and independently graded before/after results.
+
+The suite exercises fresh isolated hires and their continuations. Existing saved
+manuals and old Codex sessions are not automatically migrated. The latter still
+need a provider-session reset to restore a previously replaced vendor base.
+Private Runner protocol definitions remain separate: they use mock control-plane
+operations and cannot substitute for this public hiring and assembled-prompt
+coverage.

tests/runner-e2e/catalog.test.ts

@@ -136,10 +136,10 @@ describe("runner E2E catalog", () => {
     expect(localIntegrityTasks).toHaveLength(2);
     expect(openRouterBreadthTasks).toHaveLength(3);
     expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
-      6, 30, 3, 16, 16, 2, 6, 8, 46, 23, 50, 20, 52, 28, 18, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1,
+      6, 30, 3, 16, 16, 2, 6, 8, 46, 23, 50, 20, 24, 52, 28, 18, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1,
     ]);
-    expect(validateRunnerCatalog()).toHaveLength(436);
-    expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(436);
+    expect(validateRunnerCatalog()).toHaveLength(460);
+    expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(460);
     expect(
       runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
     ).toHaveLength(48);

tests/runner-e2e/catalog.ts

@@ -6,6 +6,7 @@ import { blockerTasks, blockerProfile } from "./blocker-cases.js";
 import { accountingTasks } from "./accounting-cases.js";
 import { continuationTasks } from "./continuation-cases.js";
 import { contextIntegrityTasks } from "./context-integrity-cases.js";
+import { productionDefaultHireProfile, stockHarnessSourceDigest } from "./stock-harness.js";
 import { lifecycleLiveTasks, lifecycleLiveDefinitionDigest } from "./lifecycle-live-cases.js";
 import { everydayTasks, productionStoryProfile } from "./everyday-cases.js";
 
@@ -51,6 +52,7 @@ const SELECTABLE_GROUPS = [
   "chat",
   "onboarding",
   "context-integrity",
+  "stock-harness",
 ] as const;
 const SAMPLE_UUID = "11111111-1111-4111-8111-111111111111";
 
@@ -1211,6 +1213,25 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
       prerequisiteGate: PENDING_PROFILE_PREREQUISITES,
     },
   },
+  {
+    id: "stock-harness",
+    label: "Stock harness with Paperclip",
+    manualOnly: true,
+    description: "Production-default hires, reduced shared prompts, assigned skills, continuation, and persistent chat.",
+    groups: ["stock-harness", "native", "legacy"],
+    profiles: [...contextIntegrityProfiles.filter(profile => !pendingContextIntegrityProfiles.includes(profile)),
+      runnerProfiles.find(profile => profile.id === "legacy-opencode")!].map(productionDefaultHireProfile),
+    environments: [localEnvironment],
+    tasks: [...contextIntegrityTasks, chatTasks.find(task => task.id === "continuity-restart")!],
+    expectedMatrixSize: 24,
+    definitionMetadata: {
+      version: 1, instructions: "production-default-hire", scheduling: "explicit-only",
+      sourceDigest: stockHarnessSourceDigest(),
+      grading: "public-default-bundle-and-delivered-prompts-plus-independent-lifecycle-oracles",
+      vendorBaseEvidence: "required deterministic Codex driver/runnerd/Rust gate; task success is not vendor-base proof",
+      paidCalls: "one skill turn, two ordered-comment turns, three chat turns per profile",
+    },
+  },
   {
     id: "first-task", label: "First-task onboarding",
     description: "Production onboarding, first replies, approval, and durable task execution.",

tests/runner-e2e/live-fixtures.test.ts

@@ -4,6 +4,30 @@ import { runnerMatrix } from "./catalog.js";
 import { setupLiveFixtures } from "./live-fixtures.js";
 
 describe("live runner fixtures", () => {
+  it.each(["runner-codex", "legacy-codex", "runner-acpx-claude", "legacy-opencode"])(
+    "creates a production-default %s hire with company and agent budget stops", async (profile) => {
+      const execution = runnerMatrix.find(row => row.suite.id === "stock-harness" && row.profile.id === profile)!;
+      let agentBody: any;
+      const api = {
+        async get() { return [{ id: "local", driver: "local" }]; },
+        async postSensitive() { return { id: "secret" }; },
+        async post(url: string, data: any) {
+          if (url === "/api/companies") {
+            expect(data.budgetMonthlyCents).toBe(1_000);
+            return { id: "company", name: "Fixture" };
+          }
+          if (url.endsWith("/agents")) { agentBody = data; return { id: "agent", ...data }; }
+          throw new Error(`Unexpected POST ${url}`);
+        },
+      } as unknown as RunnerApi;
+      const fixtures = await setupLiveFixtures({ api, execution, executionNonce: "nonce", workspacePath: "/tmp/fixture",
+        credentials: { [execution.profile.credential]: "fixture-key" } });
+      expect(agentBody).not.toHaveProperty("instructionsBundle");
+      expect(agentBody.budgetMonthlyCents).toBe(1_000);
+      expect(agentBody.adapterConfig.env[execution.profile.credential]).toEqual({ type: "secret_ref", secretId: "secret", version: "latest" });
+      await fixtures.teardown();
+    },
+  );
   it("anchors extended file validation to a public project workspace for local and remote copy-back", async () => {
     const execution = runnerMatrix.find(e => e.id === "extended-harnesses.runner-acpx-pi.local.file-edit-validate")!;
     let projectBody: any;

tests/runner-e2e/live-fixtures.ts

@@ -135,7 +135,8 @@ export async function setupLiveFixtures(input: {
       return api.post<CompanyRecord>("/api/companies", {
         name: `Runner E2E ${execution.id} ${input.executionNonce}`,
         description: "Ephemeral paid full-stack runner acceptance fixture",
-        budgetMonthlyCents: execution.suite.id === "task-titles" ? TASK_TITLE_BUDGET_CENTS : 0,
+        budgetMonthlyCents: execution.suite.id === "task-titles" ? TASK_TITLE_BUDGET_CENTS
+          : execution.suite.id === "stock-harness" ? 1_000 : 0,
       });
     },
     async teardown() {
@@ -287,6 +288,7 @@ export async function setupLiveFixtures(input: {
         secretRefs,
         executionId: input.executionNonce,
       });
+      if (execution.suite.id === "stock-harness") agent.budgetMonthlyCents = 1_000;
       if (managedHiring) {
         const account = value<ManagedAccountFixture>(resolved, "ai-connection");
         const config = agent.adapterConfig as Record<string, unknown>;

tests/runner-e2e/runner.spec.ts

@@ -14,6 +14,7 @@ import { runContinuationFlow } from "./continuation-flow.js";
 import { runEverydayFlow } from "./everyday-flow.js";
 import { gradeTaskTitle } from "./task-titles.js";
 import { runContextIntegrityFlow } from "./context-integrity-flow.js";
+import { captureStockHarness, gradeStockHarness, gradeStockHire } from "./stock-harness.js";
 import { createTaskThroughUi, submitTaskReply } from "./user-actions.js";
 
 import { runFirstTaskFlow, setupFirstTaskFixtures } from "./first-task-flow.js";
@@ -842,6 +843,17 @@ for (const execution of executions) {
         daytonaImage: process.env.PAPERCLIP_E2E_DAYTONA_IMAGE,
       });
 
+      if (execution.suite.id === "stock-harness") {
+        const hire = await captureStockHarness({ api, companyId: fixtures.company.id,
+          agentId: fixtures.agent.id, generation: execution.profile.generation, runIds: [] });
+        const checks = gradeStockHire(hire);
+        await writeSanitizedJson(snapshotsDir, "stock-harness-hire.json", {
+          capturePhase: "before-provider", ...hire, checks,
+        }, secrets);
+        const failed = checks.filter(check => !check.passed);
+        if (failed.length) throw new Error(`Stock hire matcher failures: ${failed.map(check => check.id).join(", ")}`);
+      }
+
       if (execution.suite.id === "api-response-reading") {
         const source = await api.post<{ id: string }>(`/api/companies/${fixtures.company.id}/issues`, {
           title: `Synthetic diagnostic evidence ${nonce}`,
@@ -2714,6 +2726,24 @@ for (const execution of executions) {
             selectedRuns = await Promise.all(companyRuns.map(run => api.get<RunRecord>(`/api/heartbeat-runs/${run.id}`)));
             await writeSanitizedJson(snapshotsDir, execution.task.flow === "first_task" ? "first-task-final-run-ledger.json" : "chat-final-run-ledger.json", selectedRuns, secrets);
           }
+          if (execution.suite.id === "stock-harness") {
+            try {
+              const stock = await captureStockHarness({ api, companyId: fixtures.company.id,
+                agentId: fixtures.agent.id, generation: execution.profile.generation,
+                runIds: selectedRuns.map(run => run.id) });
+              const checks = gradeStockHarness(stock);
+              matcherResults.push(...checks.map(check => ({ matcher: { kind: "json_path" as const,
+                path: `stockHarness.${check.id}`, expected: true }, passed: check.passed, detail: check.detail })));
+              await writeSanitizedJson(snapshotsDir, "stock-harness.json", { ...stock, checks }, secrets);
+              const failures = checks.filter(check => !check.passed);
+              if (failures.length && !primaryError) primaryError = new Error(`Stock harness matcher failures: ${failures.map(check => check.id).join(", ")}`);
+            } catch (error) {
+              await writeSanitizedJson(snapshotsDir, "stock-harness-evidence-error.json", {
+                error: error instanceof Error ? error.message : String(error),
+              }, secrets);
+              if (!primaryError) primaryError = error;
+            }
+          }
           await fixtures.teardown();
           cleanup = "passed";
         } catch (error) {

tests/runner-e2e/stock-harness-checks.mjs

@@ -0,0 +1,109 @@
+// Deterministic prerequisite for the stock-harness Product E2E suite. No model
+// calls or credentials. Reuse the existing protocol assertions, not model
+// self-reports about what its system instructions contain.
+import { spawnSync } from "node:child_process";
+import { createHash } from "node:crypto";
+import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
+import { join, resolve } from "node:path";
+
+const root = resolve(import.meta.dirname, "../..");
+export const stockHarnessGates = [
+  { id: "SH-1", name: "Native Codex additive instructions", cwd: "packages/paperclip-runner", files: [
+    "src/drivers/codex/codex-app-server-driver.test.ts",
+    "src/drivers/codex/codex-app-server-driver.lifecycle.test.ts",
+    "src/live/runnerd-codex-transport.test.ts",
+    "src/live/live-session.test.ts",
+    "src/cli/eval-provider-runtime.test.ts",
+  ], testPattern: "adds Paperclip developer instructions|preserves stock Codex instructions|captures exact provider frames and correlates|direct eval provider runtime|exposes native completion consistently on fresh and resumed sessions|passes caller-supplied native system instructions",
+  required: ["adds Paperclip developer instructions", "preserves stock Codex instructions on task recovery",
+    "preserves stock Codex instructions on prepared recovery", "preserves stock Codex instructions on direct recovery",
+    "captures exact provider frames and correlates", "exposes native completion consistently on fresh and resumed sessions",
+    "passes caller-supplied native system instructions"] },
+  { id: "SH-2", name: "Production-default hire bundle", cwd: ".", files: [
+    "server/src/__tests__/agent-skills-routes.test.ts",
+    "server/src/services/onboarding-first-task-assets.test.ts",
+  ], required: ["materializes minimal default instructions for non-CEO agents with no prompt template"] },
+  { id: "SH-3", name: "Shared legacy startup and continuation", cwd: ".", files: [
+    "packages/adapter-utils/src/server-utils.test.ts",
+    "packages/adapter-utils/src/prompt-sections.test.ts",
+    "packages/adapter-utils/src/acpx-engine/execute.test.ts",
+    "packages/adapters/pi-local/src/server/execute.remote.test.ts",
+    "packages/adapters/opencode-local/src/server/execute.test.ts",
+    "packages/adapters/cursor-cloud/src/server/execute.test.ts",
+    "server/src/__tests__/codex-local-execute.test.ts",
+  ], required: ["keeps task and chat defaults to identity and connection guidance",
+    "does not restore generic procedures on resume or with the legacy opt-in"] },
+  // Hermes is not in the root Vitest project list. Run its package config so
+  // the requested file cannot silently disappear from discovery.
+  { id: "SH-3-hermes", name: "Hermes shared-prompt delivery", cwd: "packages/adapters/hermes",
+    files: ["src/server/prompt-rendering.test.ts"],
+    required: ["renders standard assignment wake with task authority", "renders scoped planning wake authority"] },
+];
+
+export function gradeGate(gate, report, exitCode) {
+  const assertions = (report?.testResults ?? []).flatMap(file => file.assertionResults ?? []);
+  const requirements = (gate.required ?? []).map(name => ({ name,
+    passed: assertions.some(assertion => assertion.fullName?.includes(name) && assertion.status === "passed") }));
+  const files = gate.files.map(file => ({ file,
+    passed: (report?.testResults ?? []).some(result => result.name?.endsWith(file) && result.status === "passed" &&
+      result.assertionResults?.some(assertion => assertion.status === "passed") &&
+      result.assertionResults.every(assertion => assertion.status === "passed" ||
+        (gate.testPattern && assertion.status === "skipped" && !(gate.required ?? []).some(name => assertion.fullName?.includes(name))))) }));
+  return { id: gate.id, name: gate.name, passed: exitCode === 0 && requirements.every(row => row.passed) && files.every(row => row.passed),
+    exitCode, files, requirements, total: report?.numTotalTests ?? 0, passedTests: report?.numPassedTests ?? 0,
+    failedTests: report?.numFailedTests ?? 0, pendingTests: report?.numPendingTests ?? 0 };
+}
+
+export function main(args = process.argv.slice(2)) {
+  if (args.includes("--list")) {
+    console.log(JSON.stringify({ gates: stockHarnessGates, rust: "runtime_instructions_are_additive_for_codex_on_start_and_resume", live: "not invoked" }, null, 2));
+    return;
+  }
+  if (args.length) throw new Error("Use --list or no arguments; this command never runs paid providers.");
+  const output = join(root, "tests/runner-e2e/results", `stock-harness-preflight-${new Date().toISOString().replaceAll(":", "-")}`);
+  mkdirSync(output, { recursive: true });
+  const results = [];
+  for (const gate of stockHarnessGates) {
+    console.log(`Checking ${gate.id}: ${gate.name}`);
+    const file = join(output, `${gate.id}.json`);
+    const run = spawnSync(process.execPath, [join(root, "node_modules/vitest/vitest.mjs"), "run", ...gate.files,
+      ...(gate.testPattern ? ["--testNamePattern", gate.testPattern] : []),
+      "--reporter=default", "--reporter=json", `--outputFile.json=${file}`],
+    { cwd: resolve(root, gate.cwd), stdio: "inherit", timeout: 10 * 60_000 });
+    let report;
+    try { report = JSON.parse(readFileSync(file, "utf8")); } catch { /* missing evidence fails closed below */ }
+    results.push(gradeGate(gate, report, run.status));
+  }
+  const rust = spawnSync("cargo", ["test", "--offline", "-p", "paperclip-runner-core", "--test", "codex_provider",
+    "runtime_instructions_are_additive_for_codex_on_start_and_resume", "--", "--exact"],
+  { cwd: join(root, "packages/paperclip-runner/runner"), encoding: "utf8", timeout: 10 * 60_000 });
+  const rustOutput = `${rust.stdout ?? ""}\n${rust.stderr ?? ""}`;
+  writeFileSync(join(output, "rust.txt"), rustOutput);
+  results.push({ id: "SH-1-rust", passed: rust.status === 0 && /test runtime_instructions_are_additive_for_codex_on_start_and_resume \.\.\. ok/.test(rustOutput), exitCode: rust.status });
+  const git = spawnSync("git", ["rev-parse", "HEAD"], { cwd: root, encoding: "utf8" });
+  const hash = createHash("sha256");
+  const sources = new Set([
+    ...stockHarnessGates.flatMap(gate => gate.files.map(file => join(gate.cwd, file))),
+    "tests/runner-e2e/stock-harness.ts", "tests/runner-e2e/stock-harness-checks.mjs", "tests/runner-e2e/catalog.ts",
+    "packages/adapter-utils/src/server-utils.ts", "server/src/onboarding-assets/default/AGENTS.md", "server/src/routes/agents.ts",
+    "packages/paperclip-runner/src/drivers/codex/codex-app-server-driver-impl.ts",
+    "packages/paperclip-runner/src/live/runnerd-codex-transport.ts",
+    "packages/paperclip-runner/src/live/live-session.ts",
+    "packages/paperclip-runner/runner/crates/runner-core/src/codex_provider.rs",
+    "packages/paperclip-runner/runner/crates/runner-core/tests/codex_provider.rs",
+  ]);
+  const sourceErrors = [];
+  for (const source of [...sources].sort()) {
+    hash.update(source);
+    try { hash.update(readFileSync(join(root, source))); }
+    catch { sourceErrors.push(source); }
+  }
+  const report = { schema: "paperclip.stock-harness-preflight.v1", sourceSha: git.stdout?.trim() || null,
+    sourceFingerprint: hash.digest("hex"), measuredAt: new Date().toISOString(), providerCalls: 0,
+    live: "not_run", passed: sourceErrors.length === 0 && results.every(row => row.passed), sourceErrors, gates: results };
+  writeFileSync(join(output, "preflight.json"), JSON.stringify(report, null, 2) + "\n");
+  console.log(`Stock harness prerequisite evidence: ${output}/preflight.json`);
+  if (!report.passed) process.exitCode = 1;
+}
+
+if (process.argv[1] && resolve(process.argv[1]) === resolve(import.meta.filename)) main();

tests/runner-e2e/stock-harness-checks.test.mjs

@@ -0,0 +1,40 @@
+import { describe, expect, it } from "vitest";
+import { gradeGate, stockHarnessGates } from "./stock-harness-checks.mjs";
+
+const gate = { id: "SH-test", name: "Boundary", files: ["boundary.test.ts"], required: ["must preserve instructions"] };
+const report = () => ({ testResults: [{ name: "/repo/boundary.test.ts", status: "passed",
+  assertionResults: [{ fullName: "Boundary must preserve instructions", status: "passed" }] }],
+numTotalTests: 1, numPassedTests: 1, numFailedTests: 0, numPendingTests: 0 });
+describe("stock harness prerequisite coverage", () => {
+  it("maps every implemented change to an executable gate", () => {
+    expect(stockHarnessGates.map(gate => gate.id)).toEqual(["SH-1", "SH-2", "SH-3", "SH-3-hermes"]);
+    expect(stockHarnessGates.every(gate => gate.files.length > 0 && gate.required.length > 0)).toBe(true);
+  });
+  it("accepts an executed passing boundary", () => expect(gradeGate(gate, report(), 0).passed).toBe(true));
+  it.each([null, undefined, { testResults: [] }])("rejects unavailable evidence %s", (report) => {
+    expect(gradeGate(gate, report, 0).passed).toBe(false);
+  });
+  it.each(["pending", "skipped", "failed", "todo"])("rejects a %s required assertion", (status) => {
+    const observation = report();
+    observation.testResults[0].assertionResults[0].status = status;
+    expect(gradeGate(gate, observation, 0).passed).toBe(false);
+  });
+  it("rejects a passing different assertion and a nonzero launcher exit", () => {
+    const different = report();
+    different.testResults[0].assertionResults[0].fullName = "Unrelated pass";
+    expect(gradeGate(gate, different, 0).passed).toBe(false);
+    expect(gradeGate(gate, report(), 1).passed).toBe(false);
+  });
+  it("allows explicitly filtered unrelated tests while rejecting a skipped required test", () => {
+    const filtered = { ...gate, testPattern: "must preserve instructions" };
+    const observation = report();
+    observation.testResults[0].assertionResults.push({ fullName: "Unrelated", status: "skipped" });
+    expect(gradeGate(filtered, observation, 0).passed).toBe(true);
+    observation.testResults[0].assertionResults[0].status = "skipped";
+    expect(gradeGate(filtered, observation, 0).passed).toBe(false);
+  });
+  it("rejects a requested file absent from Vitest discovery even when required names pass", () => {
+    expect(gradeGate({ ...gate, files: [...gate.files, "undiscovered.test.ts"] }, report(), 0).passed).toBe(false);
+    expect(stockHarnessGates.find(gate => gate.id === "SH-3-hermes").cwd).toBe("packages/adapters/hermes");
+  });
+});

tests/runner-e2e/stock-harness.test.ts

@@ -0,0 +1,111 @@
+import { describe, expect, it } from "vitest";
+import { runnerMatrix, runnerProfiles, suiteDefinitionHash } from "./catalog.js";
+import { captureStockHarness, gradeStockHarness, gradeStockHire, STOCK_HIRE_IDENTITY, type StockHarnessEvidence } from "./stock-harness.js";
+
+function recording(generation: "legacy" | "native" = "legacy"): StockHarnessEvidence {
+  return {
+    schema: "paperclip.stock-harness.v1", generation, agentId: "agent",
+    budgets: { companyMonthlyCents: 1_000, agentMonthlyCents: 1_000 },
+    bundle: { entryFile: "AGENTS.md", files: [{ path: "AGENTS.md", content: STOCK_HIRE_IDENTITY }] },
+    runIds: ["fresh", "resumed"],
+    invocations: generation === "native" ? [] : [
+      { runId: "fresh", prompt: "You are agent agent (QA).\nConnection tools:\nUse connections_search.\nCurrent assignment.", promptMetrics: { heartbeatPromptChars: 113 } },
+      { runId: "resumed", prompt: "Paperclip Resume Delta\nCurrent ordered comments.", promptMetrics: { heartbeatPromptChars: 0 } },
+    ],
+  };
+}
+
+describe("stock harness Product E2E", () => {
+  it("registers 24 explicit local cells with independent skill, continuation and chat oracles", () => {
+    const cells = runnerMatrix.filter(row => row.suite.id === "stock-harness");
+    expect(cells).toHaveLength(24);
+    expect(new Set(cells.map(row => row.profile.id))).toEqual(new Set([
+      "legacy-codex", "legacy-claude", "legacy-opencode", "legacy-acp-codex", "legacy-acp-claude",
+      "runner-codex", "runner-acpx-claude", "runner-opencode",
+    ]));
+    expect(new Set(cells.map(row => row.task.id))).toEqual(new Set([
+      "ordered-comment-continuation", "assigned-skill-explicit-invocation", "continuity-restart",
+    ]));
+    expect(cells.every(row => row.suite.manualOnly && row.environment.id === "local")).toBe(true);
+    expect(cells.reduce((turns, row) => turns + row.task.expectedRunCount, 0)).toBe(48);
+    expect(cells[0]!.suite.definitionMetadata?.sourceDigest).toMatch(/^[a-f0-9]{64}$/);
+  });
+
+  it.each(["legacy-codex", "legacy-claude", "legacy-opencode", "runner-codex", "runner-acpx-claude", "runner-opencode"])(
+    "omits the custom QA bundle for %s while preserving runtime, auth and skills configuration", (profileId) => {
+      const profile = runnerMatrix.find(row => row.suite.id === "stock-harness" && row.profile.id === profileId)!.profile;
+      const original = runnerProfiles.find(row => row.id === profileId)!;
+      const input = { executionId: "test", workspacePath: "/workspace", environmentId: "environment", environmentFixtureId: "local" as const,
+        secretRefs: { [profile.credential]: { type: "secret_ref" as const, secretId: "secret", version: "latest" as const } } };
+      const expected = original.buildAgent(input);
+      const { instructionsBundle: _manual, ...withoutManual } = expected;
+      expect(profile.buildAgent(input)).toEqual(withoutManual);
+      expect(profile.buildAgent(input)).not.toHaveProperty("instructionsBundle");
+    },
+  );
+
+  it.each(["legacy", "native"] as const)("accepts complete %s evidence without interpreting task success as vendor-base proof", (generation) => {
+    expect(gradeStockHarness(recording(generation)).every(check => check.passed)).toBe(true);
+    expect(gradeStockHarness(recording(generation)).some(check => /vendor|base/.test(check.id))).toBe(false);
+  });
+
+  it("checks the hire before paid execution without treating that receipt as provider evidence", () => {
+    const hire = recording();
+    hire.runIds = [];
+    hire.invocations = [];
+    expect(gradeStockHire(hire).every(check => check.passed)).toBe(true);
+    expect(gradeStockHarness(hire)).toContainEqual(expect.objectContaining({ id: "provider-runs-present", passed: false }));
+    hire.bundle.files[0]!.content += "Old operating manual.";
+    expect(gradeStockHire(hire)).toContainEqual(expect.objectContaining({ id: "default-hire-bundle", passed: false }));
+  });
+
+  it.each([
+    ["manual-regrew", (e: StockHarnessEvidence) => { e.bundle.files[0]!.content += "Always comment and delegate."; }, "default-hire-bundle"],
+    ["extra-file", (e: StockHarnessEvidence) => { e.bundle.files.push({ path: "HEARTBEAT.md", content: "Do everything again." }); }, "default-hire-bundle"],
+    ["wrong-entry", (e: StockHarnessEvidence) => { e.bundle.entryFile = "OTHER.md"; }, "default-hire-bundle"],
+    ["missing-runs", (e: StockHarnessEvidence) => { e.runIds = []; }, "provider-runs-present"],
+    ["missing-invocation", (e: StockHarnessEvidence) => { e.invocations.pop(); }, "invocation-evidence-complete"],
+    ["empty-prompt", (e: StockHarnessEvidence) => { e.invocations[1]!.prompt = ""; }, "invocation-evidence-complete"],
+    ["malformed-prompt", (e: StockHarnessEvidence) => { e.invocations[1]!.prompt = { text: "hidden" }; }, "invocation-evidence-complete"],
+    ["old-startup", (e: StockHarnessEvidence) => { e.invocations[0]!.prompt += "\nExecution contract:"; }, "generic-procedures-absent"],
+    ["old-resume", (e: StockHarnessEvidence) => { e.invocations[1]!.prompt += "\na successful process exit or final response is not sufficient"; }, "generic-procedures-absent"],
+    ["missing-connections", (e: StockHarnessEvidence) => { e.invocations[0]!.prompt = "You are agent agent."; }, "fresh-default-delivered"],
+    ["missing-fresh-evidence", (e: StockHarnessEvidence) => { e.invocations[0]!.promptMetrics = {}; }, "fresh-default-delivered"],
+    ["wrong-budget", (e: StockHarnessEvidence) => { e.budgets.agentMonthlyCents = 0; }, "budget-hard-stops"],
+  ] as const)("fails closed for %s", (_name, mutate, failedId) => {
+    const evidence = recording();
+    mutate(evidence);
+    expect(gradeStockHarness(evidence)).toContainEqual(expect.objectContaining({ id: failedId, passed: false }));
+  });
+
+  it("captures actual public bundle and invocation receipts, ignoring unrelated events", async () => {
+    const paths: string[] = [];
+    const api = { async get<T>(url: string): Promise<T> {
+      paths.push(url);
+      if (url.endsWith("/instructions-bundle")) return { entryFile: "AGENTS.md", files: [{ path: "AGENTS.md" }] } as T;
+      if (url.includes("/file?")) return { content: STOCK_HIRE_IDENTITY } as T;
+      if (url.includes("/events?")) return [
+        { eventType: "adapter.invoke", payload: { prompt: "Actual prompt", promptMetrics: { heartbeatPromptChars: 10 } } },
+        { eventType: "assistant", payload: { prompt: "Model claims about instructions" } },
+      ] as T;
+      return { budgetMonthlyCents: 1_000 } as T;
+    } };
+    const evidence = await captureStockHarness({ api, companyId: "company", agentId: "agent", generation: "legacy", runIds: ["run"] });
+    expect(evidence.bundle.files[0]?.content).toBe(STOCK_HIRE_IDENTITY);
+    expect(evidence.invocations).toHaveLength(1);
+    expect(evidence.invocations[0]).toMatchObject({ runId: "run", prompt: "Actual prompt" });
+    expect(paths).toContain("/api/heartbeat-runs/run/events?limit=1000");
+  });
+
+  it("does not hide evidence API failures", async () => {
+    const api = { async get<T>(): Promise<T> { throw new Error("Public API unavailable"); } };
+    await expect(captureStockHarness({ api, companyId: "company", agentId: "agent", generation: "legacy", runIds: ["run"] }))
+      .rejects.toThrow("Public API unavailable");
+  });
+
+  it("changes the definition hash when the grader digest changes", () => {
+    const suite = runnerMatrix.find(row => row.suite.id === "stock-harness")!.suite;
+    expect(suiteDefinitionHash({ ...suite, definitionMetadata: { ...suite.definitionMetadata, sourceDigest: "changed" } }))
+      .not.toBe(suiteDefinitionHash(suite));
+  });
+});

tests/runner-e2e/stock-harness.ts

@@ -0,0 +1,115 @@
+import { createHash } from "node:crypto";
+import { readFileSync } from "node:fs";
+import type { RunnerProfileFixture } from "./types.js";
+
+// These are an independent product contract, not imports of the implementation
+// constants. Otherwise a larger shipped manual could silently update the oracle.
+export const STOCK_HIRE_IDENTITY = "You are an agent in a Paperclip company.\n";
+export const STOCK_TEMPLATE_IDENTITY = "You are agent";
+const REMOVED_PROCEDURES = [
+  "Execution contract:",
+  "Start actionable work in this heartbeat",
+  "clear final disposition",
+  "After 2 consecutive failures of the same control-plane write",
+  "Use child issues for parallel or long delegated work",
+  "a successful process exit or final response is not sufficient",
+];
+
+export function productionDefaultHireProfile(profile: RunnerProfileFixture): RunnerProfileFixture {
+  return {
+    ...profile,
+    buildAgent(input) {
+      const { instructionsBundle: _fixtureManual, ...agent } = profile.buildAgent(input);
+      return agent;
+    },
+  };
+}
+
+export interface StockHarnessEvidence {
+  schema: "paperclip.stock-harness.v1";
+  generation: "legacy" | "native";
+  agentId: string;
+  budgets: { companyMonthlyCents: unknown; agentMonthlyCents: unknown };
+  bundle: { entryFile?: string; files: Array<{ path: string; content: string }> };
+  invocations: Array<{ runId: string; prompt: unknown; promptMetrics?: Record<string, unknown> }>;
+  runIds: string[];
+}
+
+export function gradeStockHire(evidence: Pick<StockHarnessEvidence, "bundle" | "budgets">) {
+  const checks: Array<{ id: string; passed: boolean; detail: string }> = [];
+  const check = (id: string, passed: boolean, detail: string) => checks.push({ id, passed, detail });
+  check("budget-hard-stops", evidence.budgets.companyMonthlyCents === 1_000 && evidence.budgets.agentMonthlyCents === 1_000,
+    "Public company and agent records must both retain the 1,000-cent monthly hard stops.");
+  check("default-hire-bundle", evidence.bundle.entryFile === "AGENTS.md" &&
+    evidence.bundle.files.length === 1 && evidence.bundle.files[0]?.path === "AGENTS.md" &&
+    evidence.bundle.files[0]?.content === STOCK_HIRE_IDENTITY,
+  "The public managed bundle must contain only the eight-word shipped identity, without an injected QA manual.");
+  return checks;
+}
+
+export function gradeStockHarness(evidence: StockHarnessEvidence) {
+  const checks = gradeStockHire(evidence);
+  const check = (id: string, passed: boolean, detail: string) => checks.push({ id, passed, detail });
+  check("provider-runs-present", evidence.runIds.length > 0 && evidence.runIds.every(Boolean),
+    "The lifecycle oracle must reach actual provider runs; missing runs cannot pass instruction delivery.");
+  if (evidence.generation === "legacy") {
+    const prompts = evidence.invocations.filter(row => typeof row.prompt === "string" && row.prompt.length > 0);
+    check("invocation-evidence-complete", evidence.runIds.length > 0 && evidence.runIds.every(runId =>
+      prompts.some(row => row.runId === runId)),
+    "Every legacy run must have a public adapter.invoke event with its actual nonempty prompt.");
+    check("generic-procedures-absent", prompts.length > 0 && prompts.every(row =>
+      REMOVED_PROCEDURES.every(procedure => !String(row.prompt).includes(procedure))),
+    "Neither startup nor continuation may reintroduce the removed generic manual.");
+    const fresh = prompts.filter(row => Number(row.promptMetrics?.heartbeatPromptChars) > 0);
+    check("fresh-default-delivered", fresh.length > 0 && fresh.every(row =>
+      String(row.prompt).includes(STOCK_TEMPLATE_IDENTITY) && String(row.prompt).includes("Connection tools:") &&
+      String(row.prompt).includes("connections_search")),
+    "At least one fresh invocation must carry default identity and connection guidance.");
+  }
+  return checks;
+}
+
+/** Public API receipts only; no provider-memory claim or private runner hook. */
+export async function captureStockHarness(input: {
+  api: { get<T>(path: string): Promise<T> };
+  agentId: string;
+  companyId: string;
+  generation: "legacy" | "native";
+  runIds: string[];
+}): Promise<StockHarnessEvidence> {
+  const bundle = await input.api.get<{ entryFile: string; files: Array<{ path: string }> }>(
+    `/api/agents/${input.agentId}/instructions-bundle`,
+  );
+  const [company, agent] = await Promise.all([
+    input.api.get<{ budgetMonthlyCents: unknown }>(`/api/companies/${input.companyId}`),
+    input.api.get<{ budgetMonthlyCents: unknown }>(`/api/agents/${input.agentId}`),
+  ]);
+  const files = await Promise.all(bundle.files.map(async file => {
+    const detail = await input.api.get<{ content: string }>(
+      `/api/agents/${input.agentId}/instructions-bundle/file?path=${encodeURIComponent(file.path)}`,
+    );
+    return { path: file.path, content: detail.content };
+  }));
+  const events = await Promise.all(input.runIds.map(async runId => ({
+    runId,
+    rows: await input.api.get<Array<{ eventType?: string; payload?: Record<string, unknown> }>>(
+      `/api/heartbeat-runs/${runId}/events?limit=1000`,
+    ),
+  })));
+  return {
+    schema: "paperclip.stock-harness.v1", generation: input.generation,
+    budgets: { companyMonthlyCents: company.budgetMonthlyCents, agentMonthlyCents: agent.budgetMonthlyCents },
+    agentId: input.agentId, bundle: { entryFile: bundle.entryFile, files }, runIds: input.runIds,
+    invocations: events.flatMap(({ runId, rows }) => rows.filter(row => row.eventType === "adapter.invoke")
+      .map(row => ({ runId, prompt: row.payload?.prompt, promptMetrics: row.payload?.promptMetrics as Record<string, unknown> | undefined }))),
+  };
+}
+
+export function stockHarnessSourceDigest() {
+  const hash = createHash("sha256");
+  for (const source of [
+    "stock-harness.ts", "context-integrity-cases.ts", "context-integrity-scoring.ts",
+    "context-integrity-flow.ts", "chat-cases.ts", "chat-flow.ts", "live-fixtures.ts", "runner.spec.ts",
+  ]) hash.update(source).update(readFileSync(new URL(source, import.meta.url)));
+  return hash.digest("hex");
+}

tests/runner-e2e/vitest.config.ts

@@ -4,7 +4,7 @@ export default defineConfig({
   root: import.meta.dirname,
   test: {
     name: "runner-e2e-support",
-    include: ["**/*.test.ts"],
+    include: ["**/*.test.ts", "**/*.test.mjs"],
     testTimeout: 30_000,
   },
 });