fix(evals): load source chat domains and retain canonical report IDs

doc/plans/2026-10-26-chat-catalog-eval-progress.md

@@ -4,7 +4,7 @@ Updated: 2026-10-26. This is the single campaign progress record. Exact per-scen
 
 - Baseline: `8373d7867963674ff960d2345d0d28086f799bc7`, branch `codex/chat-catalog-evals`. Integrator confirmed PR #17794 merged via `2ee36a4`.
 - Frozen denominator: **244 scenarios / 35 areas / 191 capabilities**; **196 passes required**. Catalog digest: `9dc9cf6a5426cf1b56bc30a3571851f233f9dd5e93a1292eaee414d33b52df77`. Full query/context/outcome and YAML hashes are preserved. Defaults are hashed as well.
-- Campaign state: **22 implemented, 9 partial, 213 unimplemented; 0 attempted, 0 passes; first/final rates 0/244 (0%)**. The two original task journeys and 29 domain definitions are integrated. Six identity definitions have no known missing catalog conditions: foreign-company and private-resource denial, company rename, project navigation, wrong reviewer, and human budget cap. Seven other identity definitions explicitly retain missing variants. Registration and calibration remain separate from live acceptance.
+- Campaign state: **22 implemented, 9 partial, 213 unimplemented; 1 attempted (failed), 0 passes; first/final rates 0/244 (0%)**. The two original task journeys and 29 domain definitions are integrated. Six identity definitions have no known missing catalog conditions: foreign-company and private-resource denial, company rename, project navigation, wrong reviewer, and human budget cap. Seven other identity definitions explicitly retain missing variants. Registration and calibration remain separate from live acceptance.
 - Domain implementation update: 16 internal domain definitions are integrated across projects, tasks, goals, documents and activity. Goal retry, cross-company rejection, missing/inaccessible purpose chains, task clarification, document annotations and scoped company overview have explicit phases and case-specific checks. Three additional task-creation/delegation definitions from `0c67cb39be` cover website work with/without an existing project and project work; the lost-send case replays the exact captured public request and independently rejects duplicate request-bound runs. The worker owns 45 catalog scenarios across tasks/projects/goals/documents-artifacts/inbox-search/activity; two legacy task implementations retain their original shared driver.
 - Owners: manager `/root`; baseline/shared framework `eval_integrator`; internal domains `eval_domains`; identity domains and this finite shared boundary/query integration checkpoint `eval_identity`; external domains `eval_external`; sole live execution/ledger/report owner `eval_campaign`. The six-area-assignment partition below is unchanged. The manifest retains the two legacy task implementations under `eval_integrator`, so its row counts are 43 internal + 29 integrator while the implementation responsibility partition is 45 + 27.
 - Boundaries: one live instance at a time initially; zero automatic retries; every selected-case attempt counts, including setup/infrastructure failures. Maximum one initial and one coherent fix/rerun per initially failing scenario. Existing measurements are historical, not new campaign attempts.
@@ -20,7 +20,7 @@ Updated: 2026-10-26. This is the single campaign progress record. Exact per-scen
 
 ## Results and delivery
 
-No campaign execution or PR yet. Per-scenario qualifying definition/grader digests remain unset until the immutable integration candidate is declared. Final pass counts require matching both per-scenario fingerprints; a later source-closure change invalidates earlier passes for that numerator while preserving their original attempts. Evidence is retained under ignored `tests/runner-e2e/results/`; no private artifact publication. Core fixes, if demonstrated, require separate focused PRs that remain unmerged; their dependent passes do not count toward the eval PR threshold. No eval merge is eligible until all 244 scenarios are genuinely implemented and processed and at least 196 pass on the declared production baseline with current definition provenance.
+The first canonical attempt failed before the query: `goals.create-goal-tree`, ordinal 1, snapshot `10d0ce63610dd85ae338b0cc4d9addcba2cd8f80`, local native Codex subscription `gpt-5.6-sol`, one live instance and zero automatic retries. Launcher PID `16472`, exec session `59846` exited 1. Packaged result: `tests/runner-e2e/results/local-2026-10-26T22-19-23-608Z/chat-features/runner-codex-subscription/local/goals.create-goal-tree/attempt-1/result.json`. Cleanup passed; accounted heartbeat runs are empty; setup hello probes remain outside heartbeat accounting. Original machine class `candidate_failure` is retained; causal diagnosis is an eval harness defect, not product/model behavior. No PR yet. Per-scenario qualifying definition/grader digests remain unset until the immutable integration candidate is declared. Final pass counts require matching both per-scenario fingerprints; a later source-closure change invalidates earlier passes for that numerator while preserving their original attempts. Evidence is retained under ignored `tests/runner-e2e/results/`; no private artifact publication. Core fixes, if demonstrated, require separate focused PRs that remain unmerged; their dependent passes do not count toward the eval PR threshold. No eval merge is eligible until all 244 scenarios are genuinely implemented and processed and at least 196 pass on the declared production baseline with current definition provenance.
 
 ## Complete implementation ownership across waves
 
@@ -36,3 +36,15 @@ The six bounded assignments cover all 244 scenarios. Only three worker slots run
 | `eval_campaign` | 16 | instance-operations, cli-operations, developer-labs | Implement before sole-coordinator campaign execution |
 
 Baseline is unchanged at `8373d7867963674ff960d2345d0d28086f799bc7`; integration work is on `codex/chat-catalog-evals`. All 35 areas retain their frozen requests, contexts, outcomes and denominator membership. No core-fix PR or eval PR has been created.
+
+## Frozen goals execution slice
+
+Candidate `10d0ce63610dd85ae338b0cc4d9addcba2cd8f80` was exported into `/private/tmp/paperclip-chat-candidate-10d0ce63610`. All 12,160 tracked source files were checked against Git blobs. Dependencies and built outputs were copied, 5,191 symlinks audited, one absolute workspace link rebound, and generated package shims rebound to the snapshot; no dependency symlink resolves to the mutable checkout. Eighteen required workspace TypeScript packages were rebuilt without incremental reuse; the runner surface outputs and provider entrypoints were rebuilt. Snapshot full E2E TypeScript checking passed.
+
+Native binary was rebuilt with offline/locked Cargo against the snapshot; Rust sources have zero diff from production baseline `8373d7867963674ff960d2345d0d28086f799bc7`. Binary SHA-256: `9d8af6b85b836a03092ade2273be6f29f5c2f6ec67ee891d83ee9693ba39d3b4`. Build tool identities and workspace-output digest are retained in `tests/runner-e2e/results/chat-catalog-campaign/snapshot-10d0ce63610.json`. Launcher source override independently resolved the declared candidate SHA. Read-only preflight measured 658 free semaphores and 19 shared-memory slots; no host IPC cleanup or kernel changes.
+
+Each goals case has a 497-file selected execution/oracle closure and zero unresolved dynamic import sites. The first definition/grader fingerprint is `b7213eef67c005adcd7d62c6ce0300f7da13adca226afbfbb4a1f997988c14d5`; global suite hash is `afa382de81e8fd69fcb2da74c6ed06a5ddfbcded24cdf120b2e28ca4096a2ef0`. The full source closure receipt and preflight are referenced from its attempt row.
+
+### First failure and allowed correction
+
+The actual Playwright worker failed to resolve the selected domain `.js` file URL to source `.ts`; no human query or requested goal mutation executed. The launcher then exposed a second harness defect: report parsing rejected the canonical dotted scenario ID. The single coherent correction uses the actual existing source file URL while preserving lazy selected-module import, and admits exactly the bounded `chat-features/<area>.<scenario>` identity form without weakening path validation. Two provider-free regressions passed in an actual Playwright worker: importing the real selected goals implementation, and preserving the dotted case identity while rejecting traversal/extra-dot/foreign-suite forms. The original failed measurement remains unchanged. Exactly one corrected rerun is allowed.

tests/runner-e2e/chat-campaign-manifest.json

@@ -2442,9 +2442,41 @@
       "readiness": "implemented",
       "executionId": "chat-features.runner-codex-subscription.local.goals.create-goal-tree",
       "blocker": null,
-      "attempts": [],
-      "qualifyingDefinitionDigest": null,
-      "qualifyingGraderDigest": null
+      "attempts": [
+        {
+          "ordinal": 1,
+          "outcome": "failed",
+          "executionId": "chat-features.runner-codex-subscription.local.goals.create-goal-tree",
+          "startedAt": "2026-10-26T22:19:12.393Z",
+          "finishedAt": "2026-10-26T22:19:57.928Z",
+          "sourceRevision": "10d0ce63610dd85ae338b0cc4d9addcba2cd8f80",
+          "workingTreeDigest": "612cddbe7dee08bf913c475d01604b920cf60d0267e6f5a3e24ef105a6726d2c",
+          "definitionDigest": "b7213eef67c005adcd7d62c6ce0300f7da13adca226afbfbb4a1f997988c14d5",
+          "suiteDefinitionDigest": "afa382de81e8fd69fcb2da74c6ed06a5ddfbcded24cdf120b2e28ca4096a2ef0",
+          "graderDigest": "b7213eef67c005adcd7d62c6ce0300f7da13adca226afbfbb4a1f997988c14d5",
+          "model": "gpt-5.6-sol",
+          "profile": "runner-codex-subscription",
+          "environment": "local",
+          "automaticRetries": 0,
+          "correction": null,
+          "classification": "eval_defect; machine=candidate_failure",
+          "observedFailure": "Selected domain .js dynamic import does not resolve source .ts in Playwright; report also rejects canonical dotted task IDs. No human query or goal action executed.",
+          "evidencePaths": [
+            "tests/runner-e2e/results/chat-catalog-campaign/snapshot-10d0ce63610.json",
+            "tests/runner-e2e/results/chat-catalog-campaign/preflight-10d0ce63610.json",
+            "tests/runner-e2e/results/local-2026-10-26T22-19-23-608Z/chat-features/runner-codex-subscription/local/goals.create-goal-tree/attempt-1/result.json"
+          ],
+          "usage": {
+            "costUsd": null,
+            "coverage": "No accounted heartbeat runs; setup hello probes outside heartbeat accounting",
+            "tokens": null
+          },
+          "cleanup": "passed",
+          "corePrDependencies": []
+        }
+      ],
+      "qualifyingDefinitionDigest": "b7213eef67c005adcd7d62c6ce0300f7da13adca226afbfbb4a1f997988c14d5",
+      "qualifyingGraderDigest": "b7213eef67c005adcd7d62c6ce0300f7da13adca226afbfbb4a1f997988c14d5"
     },
     {
       "id": "goals.explain-purpose",

tests/runner-e2e/chat-catalog-dispatch.spec.ts

@@ -0,0 +1,18 @@
+import { expect, test } from "@playwright/test";
+import { findChatCatalogCase, loadChatCatalogImplementation } from "./chat-catalog-registry.js";
+import { parseReportExecutionId } from "./report-catalog.js";
+
+test("chat catalog loads its selected source module in the actual Playwright worker", async () => {
+  const descriptor = findChatCatalogCase("goals.create-goal-tree")!;
+  const implementation = await loadChatCatalogImplementation(descriptor);
+  expect(implementation.scenarioId).toBe(descriptor.scenarioId);
+  for (const method of [implementation.setup, implementation.observe, implementation.grade, implementation.verifyResult]) expect(typeof method).toBe("function");
+  // Import-only regression: no fixture, database, browser, provider or user query is invoked.
+});
+
+test("chat catalog report preserves its canonical dotted case ID and rejects forged paths", () => {
+  const prefix = "chat-features.runner-codex-subscription.local.";
+  expect(parseReportExecutionId(`${prefix}goals.create-goal-tree`)).toMatchObject({ suiteId: "chat-features", caseId: "goals.create-goal-tree" });
+  for (const suffix of ["../private", "goals/secret", "goals..create", "goals.create.extra"]) expect(() => parseReportExecutionId(prefix + suffix)).toThrow();
+  expect(() => parseReportExecutionId("other-suite.runner.local.goals.create")).toThrow();
+});

tests/runner-e2e/chat-catalog-registry.ts

@@ -1,4 +1,4 @@
-import { readFileSync } from "node:fs";
+import { existsSync, readFileSync } from "node:fs";
 import type { ChatCatalogCase } from "./chat-catalog-contract.js";
 import { chatCatalogScenarios } from "./chat-catalog-inventory.js";
 import type { IdentityBoundaryCase } from "./chat-catalog-domains/identity-boundary-contract.js";
@@ -26,13 +26,18 @@ export function findChatCatalogCase(scenarioId: string): ChatCatalogDescriptor |
     throw new Error(`Incomplete bounds/provenance for ${scenarioId}`);
   return descriptor;
 }
+/** Source-local Playwright imports need the real .ts URL; compiled consumers retain .js. */
+export function chatCatalogModuleUrl(modulePath: string): string {
+  const source = new URL(modulePath.replace(/\.js$/, ".ts"), import.meta.url);
+  return existsSync(source) ? source.href : new URL(modulePath, import.meta.url).href;
+}
 export async function loadChatCatalogImplementation(
   descriptor: ChatCatalogDescriptor,
   importer: (specifier: string) => Promise<Record<string, unknown>> = specifier => import(specifier),
 ): Promise<ChatCatalogCase> {
   if (descriptor.journeyKind === "browser-boundary") throw new Error("Browser boundary cannot enter ordinary Agent Chat dispatch");
   // The sole dynamic import boundary: only the selected descriptor's module executes.
-  const module = await importer(new URL(descriptor.modulePath, import.meta.url).href);
+  const module = await importer(chatCatalogModuleUrl(descriptor.modulePath));
   const implementations = module[descriptor.exportName];
   if (!Array.isArray(implementations)) throw new Error(`Missing domain export: ${descriptor.exportName}`);
   const selected = implementations.filter((row: ChatCatalogCase) => row.scenarioId === descriptor.scenarioId);
@@ -58,7 +63,7 @@ export async function loadIdentityBoundaryImplementation(descriptor: ChatCatalog
     throw new Error("Only explicit catalog browser boundaries may bypass Agent Chat prerequisites");
   if (descriptor.expectedRunCount !== 0 || (descriptor.minimumRunCount ?? 0) !== 0 || descriptor.expectedHumanMessageCount !== 0)
     throw new Error("Pre-chat boundaries require zero agent admissions and zero persisted chat queries");
-  const module = await importer(new URL(descriptor.modulePath, import.meta.url).href);
+  const module = await importer(chatCatalogModuleUrl(descriptor.modulePath));
   const rows = module[descriptor.exportName];
   const selected = Array.isArray(rows) ? rows.filter(row => row.scenarioId === descriptor.scenarioId) : [];
   if (selected.length !== 1) throw new Error("Missing or duplicate browser boundary implementation");

tests/runner-e2e/playwright-support.config.ts

@@ -3,7 +3,7 @@ import { defineConfig } from "@playwright/test";
 /** Browser-only harness regressions: no Paperclip instance or provider credentials. */
 export default defineConfig({
   testDir: ".",
-  testMatch: ["screenshot-readiness.spec.ts", "service-worker-reload.spec.ts", "lost-send.spec.ts", "chat-restart.spec.ts", "settings-toggle.spec.ts", "browser-bootstrap-diagnostics.spec.ts", "service-worker-module-loading.spec.ts", "browser-bootstrap-recovery.spec.ts", "blocker-input.spec.ts", "native-blocker-visible.spec.ts", "copilot-permission-card.spec.ts", "github-review-check.spec.ts"],
+  testMatch: ["chat-catalog-dispatch.spec.ts", "screenshot-readiness.spec.ts", "service-worker-reload.spec.ts", "lost-send.spec.ts", "chat-restart.spec.ts", "settings-toggle.spec.ts", "browser-bootstrap-diagnostics.spec.ts", "service-worker-module-loading.spec.ts", "browser-bootstrap-recovery.spec.ts", "blocker-input.spec.ts", "native-blocker-visible.spec.ts", "copilot-permission-card.spec.ts", "github-review-check.spec.ts"],
   workers: 1,
   retries: 0,
   timeout: 10_000,

tests/runner-e2e/report-catalog.ts

@@ -26,6 +26,8 @@ export function parseReportExecutionId(id: string) {
     throw new Error("Report execution identity must be a string");
   const journey = /^(product-journeys)\.([a-z0-9][a-z0-9_-]*)\.([a-z0-9][a-z0-9_-]*)\.(local|daytona)\.([a-z0-9][a-z0-9_-]*)$/.exec(id);
   if (journey && id.length <= 512) return { suiteId: journey[1]!, targetId: journey[2]!, profileId: journey[3]!, environmentId: journey[4]! as "local" | "daytona", caseId: journey[5]! };
+  const chat = /^(chat-features)\.([a-z0-9][a-z0-9_-]*)\.(local|daytona)\.([a-z0-9][a-z0-9_-]*\.[a-z0-9][a-z0-9_-]*)$/.exec(id);
+  if (chat && id.length <= 512) return { suiteId: chat[1]!, profileId: chat[2]!, environmentId: chat[3]! as "local" | "daytona", caseId: chat[4]!, targetId: undefined };
   const match =
     /^([a-z0-9][a-z0-9_-]*)\.([a-z0-9][a-z0-9_-]*)\.(local|daytona)\.([a-z0-9][a-z0-9_-]*)$/.exec(
       id,