test(evals): qualify native blocked reports with persisted outcomes

Co-Authored-By: Paperclip <noreply@paperclip.ing>

doc/plans/2026-10-02-stock-harness-paperclip-checklist.md

@@ -154,6 +154,12 @@ not inherit instructions to call unavailable native finish/block tools.
   and all 900 E2E support tests. Runner core/surface compilation and E2E typecheck
   passed. Both exact SH-4 gates passed (14 selected assertions; unrelated cases
   explicitly skipped). Repository-wide and paid qualification remain pending.
+  Qualification now includes a native-only concrete blocker case on Codex,
+  ACPX Claude, and OpenCode. It checks persisted blocker disposition, owner,
+  action, scope, and the visible explanation. Positive and seven plausible
+  negative oracle calibrations passed. The current suite has 27 cells and 51
+  expected turns; the earlier manual/shared comparison remains 24 cells per
+  variant. No native paid qualification has run at this point.
 - [ ] **2.3.2 Move document/file delivery procedures into native tool docs.**
   Include the live server override and preserve prepared versus delivered receipts.
 - [ ] **2.3.3 Move hiring/dependency recipes into native discovery/tool docs.**

tests/runner-e2e/README.md

@@ -444,7 +444,8 @@ pnpm test:e2e:runner -- --id context-integrity.runner-codex.local.ordered-commen
 
 `stock-harness` reuses ordered continuation, assigned-skill invocation, and chat
 restart journeys with production-default hires instead of the custom QA manual.
-Its 24 explicit local cells cover eight legacy/native profiles and are excluded
+Its 27 explicit local cells cover eight legacy/native profiles, including a
+native-only concrete blocker report, and are excluded
 from `--all`. Run `pnpm test:e2e:runner:stock-harness` for the credential-free
 instruction-layering, hire, and shared-prompt prerequisites. The
 [suite contract](STOCK-HARNESS.md) maps each change to its graders, budgets,

tests/runner-e2e/STOCK-HARNESS.md

@@ -14,7 +14,7 @@ agent-creation route materialize the shipped default.
 | SH-1: native Codex preserves vendor base instructions | Serialized start/resume requests from the TypeScript driver, recovery paths, runnerd transport, Runner Lab/live sessions, and Rust provider use additive developer instructions. | Real new native Codex hires execute the three journeys below. Task success alone cannot prove vendor base preservation. |
 | SH-2: identity-only default hire manual | Existing public agent-creation and onboarding-asset tests cover default, custom, and CEO exceptions. | The public bundle is exactly one `AGENTS.md` containing the eight-word shipped identity, checked before provider execution and again during cleanup. |
 | SH-3: reduced shared legacy task/chat defaults and resume delta | Shared prompt tests and ACPX, Codex, OpenCode, Pi, Hermes, and Cursor Cloud adapter regressions retain runtime context and exclude removed generic procedures. | Actual legacy `adapter.invoke` prompts retain fresh identity/connection guidance, omit the removed procedures, and have complete receipts for every observed run. |
-| SH-4: shared native finish/block tool guidance | Actual authenticated MCP catalogs, Codex start/resume frames, serialized daemon catalogs for all five native provider kinds, and rotation of a retained v13 thread to the new tool contract. | The three native profiles run the existing skill, continuation, and chat journeys. Configured coverage is not a live qualification result. |
+| SH-4: shared native finish/block tool guidance | Actual authenticated MCP catalogs, Codex start/resume frames, serialized daemon catalogs for all five native provider kinds, and rotation of a retained v13 thread to the new tool contract. | The three native profiles run the existing completion journeys and a concrete blocker case. The blocker oracle checks persisted disposition, owner, action, scope, and the visible final explanation. Configured coverage is not a live qualification result. |
 
 SH-4 changes native Runner tools only. Legacy adapters do not receive
 `paperclip_finish` or `paperclip_block` from this change. Their skill/API
@@ -43,8 +43,9 @@ Oracle/admission calibration is itself included in the prerequisite gate.
 
 ## Live matrix
 
-There are 24 explicit local cells: these eight existing profiles each run three
-existing journeys with independently calibrated graders.
+There are 27 explicit local cells: these eight existing profiles each run three
+existing journeys. The three native profiles also run one concrete blocker case.
+All journeys have independently calibrated graders.
 
 - Legacy: `legacy-codex`, `legacy-claude`, `legacy-opencode`,
   `legacy-acp-codex`, `legacy-acp-claude`.
@@ -55,8 +56,10 @@ existing journeys with independently calibrated graders.
 | `assigned-skill-explicit-invocation` | 1 | Public skill creation/pinning, an explicit skill request, and saved output containing the marker available only in the skill body. | 12 minutes |
 | `ordered-comment-continuation` | 2 | Initial report followed by three ordered public comments, including repeated wording and a changed scope; final saved report preserves the ledger and requested scope. | 12 minutes |
 | `continuity-restart` | 3 | Task-backed chat retains the requested context across a server restart and subsequent replies. | 15 minutes |
+| `native-blocked-report` (native profiles only) | 1 | A persisted whole-task blocker identifies Release Owner and the exact requested unblock action; its visible explanation includes the marker. | 5 minutes |
 
-A full matrix expects 48 provider turns. Models, credentials, effort,
+A full current matrix expects 51 provider turns. The earlier manual/shared-prompt
+comparison used the original 24 cells and expected 48 turns. Models, credentials, effort,
 permissions, assigned skills, environment, and managed secret references are
 inherited from the existing profile; only its QA manual is omitted. Both company
 and agent receive a 1,000-cent monthly hard stop before provider execution,
@@ -107,6 +110,12 @@ saved result without appearing in the task request or follow-up comments. Skill
 tool/read event detection is retained as supporting evidence; it is not a
 required cross-provider tool-trace assertion.
 
+The native blocker request supplies the blocking facts and desired user outcome;
+it does not prescribe a tool name or completion payload. Its grader rejects a
+visible claim without a persisted native report, the wrong owner/action/scope,
+a completed-work report, a legacy runtime, and a failed run. It uses the ordinary
+public task-creation and completion flow, screenshots, cleanup, and billing rules.
+
 ## Qualification status
 
 Setup was validated locally on 2026-10-02 with the deterministic prerequisites,

tests/runner-e2e/catalog.test.ts

@@ -136,10 +136,10 @@ describe("runner E2E catalog", () => {
     expect(localIntegrityTasks).toHaveLength(2);
     expect(openRouterBreadthTasks).toHaveLength(3);
     expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
-      6, 30, 3, 16, 16, 2, 6, 8, 46, 23, 50, 20, 26, 52, 28, 18, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1,
+      6, 30, 3, 16, 16, 2, 6, 8, 46, 23, 50, 20, 29, 52, 28, 18, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1,
     ]);
-    expect(validateRunnerCatalog()).toHaveLength(462);
-    expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(462);
+    expect(validateRunnerCatalog()).toHaveLength(465);
+    expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(465);
     expect(
       runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
     ).toHaveLength(48);

tests/runner-e2e/catalog.ts

@@ -7,6 +7,7 @@ import { accountingTasks } from "./accounting-cases.js";
 import { continuationTasks } from "./continuation-cases.js";
 import { contextIntegrityTasks, paperclipDocumentTask } from "./context-integrity-cases.js";
 import { productionDefaultHireProfile, stockHarnessSourceDigest, stockHarnessSkillSources } from "./stock-harness.js";
+import { nativeBlockedReportTask } from "./native-completion-case.js";
 import { lifecycleLiveTasks, lifecycleLiveDefinitionDigest } from "./lifecycle-live-cases.js";
 import { everydayTasks, productionStoryProfile } from "./everyday-cases.js";
 
@@ -1217,22 +1218,26 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
     id: "stock-harness",
     label: "Stock harness with Paperclip",
     manualOnly: true,
-    description: "Production-default hires, reduced shared prompts, assigned skills, continuation, and persistent chat.",
+    description: "Production-default hires, reduced shared prompts, assigned skills, continuation, persistent chat, and native blocker reporting.",
     groups: ["stock-harness", "native", "legacy"],
     profiles: [...contextIntegrityProfiles.filter(profile => !pendingContextIntegrityProfiles.includes(profile)),
       runnerProfiles.find(profile => profile.id === "legacy-opencode")!].map(productionDefaultHireProfile),
     environments: [localEnvironment],
-    tasks: [...contextIntegrityTasks, chatTasks.find(task => task.id === "continuity-restart")!, paperclipDocumentTask],
-    expectedMatrixSize: 26,
-    excludedExecutionIds: ["legacy-codex", "legacy-acp-codex", "legacy-acp-claude", "runner-codex", "runner-acpx-claude", "runner-opencode"]
-      .map(profile => `stock-harness.${profile}.local.${paperclipDocumentTask.id}`),
+    tasks: [...contextIntegrityTasks, chatTasks.find(task => task.id === "continuity-restart")!, paperclipDocumentTask, nativeBlockedReportTask],
+    expectedMatrixSize: 29,
+    excludedExecutionIds: [
+      ...["legacy-codex", "legacy-acp-codex", "legacy-acp-claude", "runner-codex", "runner-acpx-claude", "runner-opencode"]
+        .map(profile => `stock-harness.${profile}.local.${paperclipDocumentTask.id}`),
+      ...["legacy-codex", "legacy-claude", "legacy-opencode", "legacy-acp-codex", "legacy-acp-claude"]
+        .map(profile => `stock-harness.${profile}.local.native-blocked-report`),
+    ],
     definitionMetadata: {
       version: 2, instructions: "production-default-hire", scheduling: "explicit-only",
       sourceDigest: stockHarnessSourceDigest(),
       operationalSkillSources: stockHarnessSkillSources(),
       grading: "public-default-bundle-and-delivered-prompts-plus-independent-lifecycle-oracles",
       vendorBaseEvidence: "required deterministic Codex driver/runnerd/Rust gate; task success is not vendor-base proof",
-      paidCalls: "one skill turn, two ordered-comment turns, three chat turns per profile",
+      paidCalls: "one skill turn, two ordered-comment turns, three chat turns per profile; one native-only blocker turn",
     },
   },
   {

tests/runner-e2e/native-completion-case.test.ts

@@ -0,0 +1,33 @@
+import { describe, expect, it } from "vitest";
+import { evaluateMatchers, type MatcherObservation } from "./matchers.js";
+import { nativeBlockedReportTask } from "./native-completion-case.js";
+import type { MatrixExecution } from "./types.js";
+
+function observation(): MatcherObservation {
+  return { message: "BLOCKED_probe. Release Owner must Grant deployment access.",
+    issueStatus: "blocked", runStatus: "succeeded", runtimeMode: "native",
+    json: { run: { resultJson: { nativeResult: { reportedWorkDisposition: "blocked",
+      blocker: { owner: { name: "Release Owner" }, unblockAction: "Grant deployment access", scope: "task_wide" } } } } } };
+}
+
+describe("native completion blocker oracle", () => {
+  const matchers = nativeBlockedReportTask.buildMatchers("probe", {} as MatrixExecution);
+  it("accepts the persisted blocker, owner and exact requested unblock action", async () => {
+    expect((await evaluateMatchers(matchers, observation())).every(row => row.passed)).toBe(true);
+    expect(nativeBlockedReportTask.buildPrompt("probe")).not.toMatch(/paperclip_(?:finish|block)|reportedWorkDisposition|json/i);
+  });
+  it.each(["visible-claim-only", "wrong-owner", "wrong-action", "wrong-scope", "done-report", "legacy-runtime", "run-failed"])(
+    "rejects %s without trusting the visible completion claim", async variant => {
+      const current = observation();
+      const result = (current.json as { run: { resultJson: { nativeResult?: { reportedWorkDisposition: string;
+        blocker: { owner: { name: string }; unblockAction: string; scope: string } } } } }).run.resultJson;
+      if (variant === "visible-claim-only") { current.issueStatus = "in_progress"; delete result.nativeResult; }
+      if (variant === "wrong-owner") result.nativeResult!.blocker.owner.name = "Someone else";
+      if (variant === "wrong-action") result.nativeResult!.blocker.unblockAction = "Pretend deployment succeeded";
+      if (variant === "wrong-scope") result.nativeResult!.blocker.scope = "current_track";
+      if (variant === "done-report") result.nativeResult!.reportedWorkDisposition = "done";
+      if (variant === "legacy-runtime") current.runtimeMode = "legacy";
+      if (variant === "run-failed") current.runStatus = "failed";
+      expect((await evaluateMatchers(matchers, current)).some(row => !row.passed)).toBe(true);
+    });
+});

tests/runner-e2e/native-completion-case.ts

@@ -0,0 +1,31 @@
+import type { RunnerTaskFixture } from "./types.js";
+
+/** A native-only blocker with facts, not tool names or a completion payload recipe. */
+export const nativeBlockedReportTask: RunnerTaskFixture = {
+  id: "native-blocked-report",
+  label: "Native concrete blocker report",
+  groups: ["native-completion"],
+  workMode: "standard",
+  flow: "single_turn",
+  expectedRunCount: 1,
+  attemptTimeoutMs: { local: 5 * 60_000, daytona: 5 * 60_000 },
+  expectedTerminalState: { issue: "blocked", run: "succeeded" },
+  buildTitle: nonce => `Native blocker report ${nonce}`,
+  buildVisibleMarker: nonce => `BLOCKED_${nonce.replace(/[^a-z0-9]/gi, "")}`,
+  buildPrompt: nonce => [
+    "Deployment cannot proceed because required release access has not been granted.",
+    "Release Owner is the external owner of that grant. Do not attempt deployment, create subtasks, or claim the access exists.",
+    "Record the whole task as blocked, identify the owner as Release Owner, and state the unblock action exactly: Grant deployment access.",
+    `Include BLOCKED_${nonce.replace(/[^a-z0-9]/gi, "")} in your final explanation.`,
+  ].join("\n"),
+  buildMatchers: nonce => [
+    { kind: "issue_status", expected: "blocked" },
+    { kind: "run_status", expected: "succeeded" },
+    { kind: "runtime_mode", expected: "native" },
+    { kind: "message_contains", expected: `BLOCKED_${nonce.replace(/[^a-z0-9]/gi, "")}` },
+    { kind: "json_path", path: "run.resultJson.nativeResult.reportedWorkDisposition", expected: "blocked" },
+    { kind: "json_path", path: "run.resultJson.nativeResult.blocker.owner.name", expected: "Release Owner" },
+    { kind: "json_path", path: "run.resultJson.nativeResult.blocker.unblockAction", expected: "Grant deployment access" },
+    { kind: "json_path", path: "run.resultJson.nativeResult.blocker.scope", expected: "task_wide" },
+  ],
+};

tests/runner-e2e/stock-harness-checks.mjs

@@ -56,11 +56,13 @@ export const stockHarnessGates = [
     config: "tests/runner-e2e/vitest.config.ts",
     files: ["tests/runner-e2e/stock-harness-manifest.test.ts", "tests/runner-e2e/paperclip-document.test.ts", "tests/runner-e2e/stock-harness.test.ts", "tests/runner-e2e/stock-harness-checks.test.mjs",
       "tests/runner-e2e/stock-harness-admission.test.ts", "tests/runner-e2e/stock-harness-digest.test.ts",
-      "tests/runner-e2e/select-rerun-artifacts.test.ts"],
+      "tests/runner-e2e/select-rerun-artifacts.test.ts", "tests/runner-e2e/native-completion-case.test.ts"],
     required: ["requires generated capability manifests for the current skill sources", "the shipped recipe delivers the current", "the shipped recipe supports an unnumbered issue", "rejects old SHA before providers", "allows toolchain paths and excludes every present or future credential",
       "changes when the evaluated server/src/onboarding-assets/default/AGENTS.md changes",
       "changes when the evaluated packages/adapter-utils/src/server-utils.ts changes",
       "changes when the evaluated packages/shared/src/connection-intent-guidance.ts changes",
+      "changes when the evaluated tests/runner-e2e/native-completion-case.ts changes",
+      "accepts the persisted blocker, owner and exact requested unblock action",
       "retains credential-free prerequisites inside the exact campaign root"] },
 ];
 
@@ -93,6 +95,7 @@ export function sourceFingerprint() {
     "packages/paperclip-runner/src/generated/capability-contract.ts", "packages/paperclip-runner/docs/capability-contract.md",
     ...["capabilities.yaml", "mcp-tool-map.yaml", "eval-traceability.yaml", "capability-contract.md", "downstream-handoff.md"]
       .map(file => `packages/paperclip-runner/generated/capability/${file}`),
+    "tests/runner-e2e/native-completion-case.ts",
     "packages/adapter-utils/src/server-utils.ts", "packages/shared/src/connection-intent-guidance.ts",
     "server/src/onboarding-assets/default/AGENTS.md", "server/src/routes/agents.ts", "scripts/ensure-plugin-build-deps.mjs",
     "packages/paperclip-runner/src/drivers/codex/codex-app-server-driver-impl.ts",

tests/runner-e2e/stock-harness-digest.test.ts

@@ -13,6 +13,7 @@ describe("stock harness instruction revision", () => {
     "packages/paperclip-runner/src/contracts/completion-result.ts", "packages/paperclip-runner/src/drivers/codex/codex-driver-values.ts",
     "packages/paperclip-runner/src/drivers/runner-tool-bridge.ts", "packages/paperclip-runner/src/drivers/opencode/mcp-bridge.ts",
     "server/src/services/native-runtime/native-session-resume.ts",
+    "tests/runner-e2e/native-completion-case.ts",
   ])(
     "changes when the evaluated %s changes", source => {
       vi.mocked(readFileSync).mockImplementation(() => Buffer.from("unchanged"));

tests/runner-e2e/stock-harness.test.ts

@@ -16,20 +16,21 @@ function recording(generation: "legacy" | "native" = "legacy"): StockHarnessEvid
 }
 
 describe("stock harness Product E2E", () => {
-  it("registers 26 explicit local cells with the original cases and two focused delivery repair cells", () => {
+  it("registers 29 explicit local cells with the original cases and two focused delivery repair cells", () => {
     const cells = runnerMatrix.filter(row => row.suite.id === "stock-harness");
-    expect(cells).toHaveLength(26);
+    expect(cells).toHaveLength(29);
     expect(new Set(cells.map(row => row.profile.id))).toEqual(new Set([
       "legacy-codex", "legacy-claude", "legacy-opencode", "legacy-acp-codex", "legacy-acp-claude",
       "runner-codex", "runner-acpx-claude", "runner-opencode",
     ]));
     expect(new Set(cells.map(row => row.task.id))).toEqual(new Set([
       "ordered-comment-continuation", "assigned-skill-explicit-invocation", "continuity-restart",
-      "assigned-skill-paperclip-document",
+      "assigned-skill-paperclip-document", "native-blocked-report",
     ]));
     expect(cells.every(row => row.suite.manualOnly && row.environment.id === "local")).toBe(true);
-    expect(cells.reduce((turns, row) => turns + row.task.expectedRunCount, 0)).toBe(50);
+    expect(cells.reduce((turns, row) => turns + row.task.expectedRunCount, 0)).toBe(53);
     expect(cells.filter(row => row.task.id === "assigned-skill-paperclip-document").map(row => row.profile.id).sort()).toEqual(["legacy-claude", "legacy-opencode"]);
+    expect(cells.filter(row => row.task.id === "native-blocked-report").every(row => row.profile.generation === "native")).toBe(true);
     expect(cells[0]!.suite.definitionMetadata?.sourceDigest).toMatch(/^[a-f0-9]{64}$/);
   });
 

tests/runner-e2e/stock-harness.ts

@@ -121,7 +121,7 @@ export async function captureStockHarness(input: {
 export function stockHarnessSourceDigest() {
   const hash = createHash("sha256");
   for (const source of [
-    "stock-harness.ts", "context-integrity-cases.ts", "context-integrity-scoring.ts",
+    "stock-harness.ts", "native-completion-case.ts", "context-integrity-cases.ts", "context-integrity-scoring.ts",
     "context-integrity-flow.ts", "chat-cases.ts", "chat-flow.ts", "live-fixtures.ts", "runner.spec.ts",
     "stock-harness-checks.mjs", "stock-harness-admission.ts", "stock-harness-manifest.ts",
     "../../packages/paperclip-runner/scripts/generate-capability-contract.mjs",