Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-05 21:02:34 +01:00
Code Issues Packages Projects Releases Wiki Activity

eval: checkpoint catalog readiness and bind discovered repositories

Browse Source
This commit is contained in:
Dotta committed 2026-10-26 18:05:30 -05:00
1 parent 9920727210
commit 5578c71a61
6 files changed
+1402 -541
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

doc/plans/2026-10-26-chat-catalog-eval-progress.md
+43 -35
View File
@@ -1,50 +1,58 @@
# Chat catalog eval campaign progress
Updated: 2026-10-26. This is the single campaign progress record. Exact per-scenario content, ownership, implementation status, attempts and outcomes live in [the machine-readable manifest](../../tests/runner-e2e/chat-campaign-manifest.json). The goal remains [the saved execution contract](2026-10-26-chat-catalog-eval-goal.md).
Updated: 2026-10-26. This is the single progress record for [the saved goal](2026-10-26-chat-catalog-eval-goal.md). Exact frozen requests, ownership, readiness, attempt histories and provenance live in [the campaign manifest](../../tests/runner-e2e/chat-campaign-manifest.json). This record distinguishes authored code from live acceptance.
- Baseline: `8373d7867963674ff960d2345d0d28086f799bc7`, branch `codex/chat-catalog-evals`. Integrator confirmed PR #17794 merged via `2ee36a4`.
- Frozen denominator: **244 scenarios / 35 areas / 191 capabilities**; **196 passes required**. Catalog digest: `9dc9cf6a5426cf1b56bc30a3571851f233f9dd5e93a1292eaee414d33b52df77`. Full query/context/outcome and YAML hashes are preserved. Defaults are hashed as well.
- Campaign state: **22 implemented, 9 partial, 213 unimplemented; 1 attempted (failed), 0 passes; first/final rates 0/244 (0%)**. The two original task journeys and 29 domain definitions are integrated. Six identity definitions have no known missing catalog conditions: foreign-company and private-resource denial, company rename, project navigation, wrong reviewer, and human budget cap. Seven other identity definitions explicitly retain missing variants. Registration and calibration remain separate from live acceptance.
- Domain implementation update: 16 internal domain definitions are integrated across projects, tasks, goals, documents and activity. Goal retry, cross-company rejection, missing/inaccessible purpose chains, task clarification, document annotations and scoped company overview have explicit phases and case-specific checks. Three additional task-creation/delegation definitions from `0c67cb39be` cover website work with/without an existing project and project work; the lost-send case replays the exact captured public request and independently rejects duplicate request-bound runs. The worker owns 45 catalog scenarios across tasks/projects/goals/documents-artifacts/inbox-search/activity; two legacy task implementations retain their original shared driver.
- Owners: manager `/root`; baseline/shared framework `eval_integrator`; internal domains `eval_domains`; identity domains and this finite shared boundary/query integration checkpoint `eval_identity`; external domains `eval_external`; sole live execution/ledger/report owner `eval_campaign`. The six-area-assignment partition below is unchanged. The manifest retains the two legacy task implementations under `eval_integrator`, so its row counts are 43 internal + 29 integrator while the implementation responsibility partition is 45 + 27.
- Boundaries: one live instance at a time initially; zero automatic retries; every selected-case attempt counts, including setup/infrastructure failures. Maximum one initial and one coherent fix/rerun per initially failing scenario. Existing measurements are historical, not new campaign attempts.
- Host readiness: node/pnpm/python/docker/codex and dependencies present. Provider credential variables and `.env.runner-e2e.local` absent. Existing private Codex auth file present; local subscription may be loaded directly into a child environment without printing or persisting its value. Daytona credentials and immutable image are absent. No external-connected integration is qualified by simulations.
- Credential-free manifest generation passed using Node's tsx loader. The tsx CLI's IPC socket is denied by the sandbox; this was tooling inspection, before any selected-case attempt. It is not model execution or a free selected-case retry.
- Preflight: native runner binary is executable; original discovery listed the two legacy cells (`gpt-5.6-sol`, subscription). Read-only host IPC measurement found 86,723/87,381 semaphores consumed and 16/32 shared-memory slots used. One isolated instance is plausible; recheck immediately before launch and do not start if exhausted. No IPC cleanup performed. Ledger policy calibration passed 6/6 focused tests. Targeted plugin SDK and Runner TypeScript builds repaired stale generated declarations without production source changes. E2E TypeScript checking then passed after the identity owner corrected its in-progress nullability error. Full local suites were not launched.
- Integration snapshot: `bc7b66912ba0c84e076aa201dd3bb6b7b7dffc95` contains the first ten executable domain definitions plus the two legacy cells. The initial snapshot passed 48 targeted checks; the lazy/policy/domain integration now passes 71 focused checks and E2E TypeScript checking. Discovery lists 15 cells; discovery and calibration do not establish complete conditions or live acceptance. No canonical attempt is reserved.
- Architecture decision: use genuinely lazy domain dispatch and per-case fingerprints for incremental acceptance. Pure descriptors support discovery; only the selected domain executes. Fingerprints bind its actual static local-import closure, shared runtime/grader/fixture inputs, normalized selected execution/descriptor, exact query and defaults. Whole source revision and global suite hash remain separate provenance. The actual selected projects closure contains 493 source files and no unresolved dynamic sites. Six known dynamic sites use reviewed source-hash-pinned resolver declarations: actual worker closure, pinned SDK package identities, and explicitly inactive external launch branches. A changed site falls back conservatively; activating an installed/external target refuses the reviewed local-chat fingerprint. No global loader hook is used. Shared-policy work must preserve scenario-specific funding, paused/readiness and cancellation/denial outcomes with explicit independently checked policies.
- Identity/shared checkpoint: 77 combined internal/identity/shared credential-free checks pass (including the 38 identity/shared checks), covering positive/wrong/missing evidence for all 13 identity oracles, query-binding rejection cases, lazy metadata drift, boundary separation and unchanged chat fingerprint restrictions. Discovery lists 31 cells. E2E TypeScript checking passes with only the external worker's uncommitted `external*.ts` sources/tests excluded. The full mutable tree reports unrelated external-worker errors (`external-connection-setup.ts:11–12` nullable card; a subsequent uncommitted `external-app-permissions.ts:50` receipt type); no identity/shared errors. The immutable committed snapshot must be checked independently. No browser, provider, database, installed-target or live API execution was started by the identity worker.
- Fixture binding decision: the manager authorized an explicit binding of incidental `BIRCH-42` to a real foreign issue identifier created through public APIs. Frozen YAML and original intent/context/outcome remain unchanged. Only that declared token may change; evidence retains original query, rendered query, binding, company/resource IDs and independently fetched persisted identity. Duplicate, undeclared, missing and prose-injecting substitutions fail before admission.
- Browser boundary integration: the four catalog contexts with no usable chat have a separate typed context, loader, pre-fixture dispatch and independent fingerprint lane. It never fabricates company/agent fixtures or credits a nonexistent chat query. The original source-local Agent Chat fingerprint guard remains closed to installed/external targets; an empty-installation boundary requires actual pinned installed CLI/server hashes. An identity-owned primitive starts a distinct genuinely empty target via the public operator CLI and independently checks later wizard/readiness outcomes. Complete wizard/auth callbacks remain unfinished, so **none of the four boundaries is registered or counted implemented**. No missing-CLI/provider placeholder can pass.
- Next: commit this finite integration slice and hand its exact source/fingerprint to the sole campaign coordinator for a frozen goals slice. Identity's remaining 31 ordinary/boundary cases and missing variants continue in the next wave. External-provider prerequisites have been requested asynchronously; no response or domain-worker live-operation authorization has been received. No acceptance run may read mutable source or carry a pass across changed executable/grader dependencies.
## Current state
## Results and delivery
- **244 scenarios / 35 areas / 191 capabilities**, frozen inventory digest `9dc9cf6a5426cf1b56bc30a3571851f233f9dd5e93a1292eaee414d33b52df77`. All original queries, contexts and expected outcomes remain unchanged. **196 passes required**; blocked, partial, unsupported and unrun remain in the denominator.
- **67 authored: 57 complete, 10 partial, 177 unimplemented** at committed-source metadata checkpoint `9920727210c7193ca1baf58a260a48ac6dd32a82`. Newer authored changes wait for the next finite checkpoint. There are 65 concrete lazy descriptors plus two legacy task cases. Counts come from actual exports, bounds, source files and executable callbacks, not registration-only stubs. Paused inventory owners now have explicit saved readiness-policy evidence in `b53c596bc2`.
- **Three scenarios attempted; four actual measurements; zero first-attempt passes and zero final passes (0/244 for both rates).** Goal-tree is exhausted after two failures. Alignment and purpose each retain one failed initial attempt and one allowed correction/rerun remaining.
- **No live launcher, browser, database or provider process is running.** Manager held the two final retries before reservation, so remaining shared authoring contracts cannot stale a pass after its final allowed attempt. No unchanged rerun, extra nudge, deadline extension, automatic retry or relabeled canonical health probe was used.
- Production baseline `8373d7867963674ff960d2345d0d28086f799bc7`; branch `codex/chat-catalog-evals`. PR #17794 merged via `2ee36a4`. No eval/core PR has been created. No merge is eligible.
The first canonical attempt failed before the query: `goals.create-goal-tree`, ordinal 1, snapshot `10d0ce63610dd85ae338b0cc4d9addcba2cd8f80`, local native Codex subscription `gpt-5.6-sol`, one live instance and zero automatic retries. Launcher PID `16472`, exec session `59846` exited 1. Packaged result: `tests/runner-e2e/results/local-2026-10-26T22-19-23-608Z/chat-features/runner-codex-subscription/local/goals.create-goal-tree/attempt-1/result.json`. Cleanup passed; accounted heartbeat runs are empty; setup hello probes remain outside heartbeat accounting. Original machine class `candidate_failure` is retained; causal diagnosis is an eval harness defect, not product/model behavior. No PR yet. Per-scenario qualifying definition/grader digests remain unset until the immutable integration candidate is declared. Final pass counts require matching both per-scenario fingerprints; a later source-closure change invalidates earlier passes for that numerator while preserving their original attempts. Evidence is retained under ignored `tests/runner-e2e/results/`; no private artifact publication. Core fixes, if demonstrated, require separate focused PRs that remain unmerged; their dependent passes do not count toward the eval PR threshold. No eval merge is eligible until all 244 scenarios are genuinely implemented and processed and at least 196 pass on the declared production baseline with current definition provenance.
## Measured evidence — all failures retained
## Complete implementation ownership across waves
All results below use local native Codex subscription profile `runner-codex-subscription`, model `gpt-5.6-sol`, max parallelism one and zero automatic retries. Retained paths are under ignored `tests/runner-e2e/results/`. The launcher numbers each fresh invocation `attempt-1`; the ledger separately preserves governed campaign ordinals.
The six bounded assignments cover all 244 scenarios. Only three worker slots run concurrently with the manager; future wave owners below are reserved assignments, not running agents.
| Scenario / campaign ordinal | Source | Retained campaign | Outcome and supported cause | Cleanup / accounted tokens |
| --- | --- | --- | --- | --- |
| `goals.create-goal-tree` / 1 | `10d0ce63610` | `local-2026-10-26T22-19-23-608Z` | Eval harness failed before canonical query: Playwright could not resolve the selected `.js` URL to `.ts`; report parsing also rejected the dotted canonical ID. Machine class `candidate_failure` unchanged. | passed / no heartbeat runs |
| `goals.create-goal-tree` / 2, final allowed | `e652b0de6e` | `local-2026-10-26T22-26-37-076Z` | One native run created the requested tree, but final reply had no hierarchy link and left company ownership unset without a question. The unchanged first-phase gate prevented the explicit retry phase. Rendered href arrays and visual screenshot review confirm no alternate result card. Completed-behavior failure; no third attempt. | passed / 44,131 |
| `goals.align-existing-work` / 1 | `e652b0de6e` | `local-2026-10-26T22-31-32-695Z` | Two native runs correctly saved associations and rejected the foreign goal. Manager/domain review found the old three-links oracle exceeded this scenario’s YAML while underchecking its strategic-context explanation. **Eval oracle defect** supersedes the initial model-behavior attribution; raw failure and diagnosis history remain. | passed / 135,401 |
| `goals.explain-purpose` / 1 | `e652b0de6e` | `local-2026-10-26T22-35-40-720Z` | Initial fixture exposed both intended and later-conditional website tasks. Native tool-result event 208 contains both; the reply conflated them. **Eval fixture contamination**; no product/model blame. Missing/inaccessible phases were never sent. | passed / 25,256 |
| Owner | Scenarios | Areas | State |
| --- | ---: | --- | --- |
| `eval_domains` | 45 | tasks, projects, goals, documents-artifacts, inbox-search, activity | Active; includes two unchanged legacy task cases whose shared integration remains with integrator |
| `eval_integrator` | 27 | cases, status-cards, routines, skills | After shared harness handoff |
| `eval_identity` | 44 | access, navigation-preferences, onboarding, questions-and-approvals, companies, budgets-costs | Active; owns authenticated identity helpers |
| `eval_runtime` | 60 | agents, agent-chat, runs-adapters, execution-environments, recovery, steering, task-coordination, workspaces | Future wave |
| `eval_external` | 52 | connection-setup, app-permissions, chat-channels, assistant-connections, gateways-profiles, plugins, teams, secrets | Active; metadata integrates after its own route-contract calibration |
| `eval_campaign` | 16 | instance-operations, cli-operations, developer-labs | Implement before sole-coordinator campaign execution |
Each case directory contains `result.json` and independent `snapshots/chat-catalog-{definition,observation,conversation,runs}.json` where reached. Exact definition/grader/global suite hashes, source SHA, evidence paths, timing and usage are retained per attempt. Subscription dollars remain **unpriced**, not zero; production setup hello probes remain outside heartbeat accounting. Accounted tokens across completed work runs total 204,788.
Baseline is unchanged at `8373d7867963674ff960d2345d0d28086f799bc7`; integration work is on `codex/chat-catalog-evals`. All 35 areas retain their frozen requests, contexts, outcomes and denominator membership. No core-fix PR or eval PR has been created.
The initial report-parser error was repaired only for presentation at `results/chat-catalog-campaign/initial-postprocessed/index.html`; its sidecar retains original source and report error. This postprocessing made zero provider calls and remains the original failed measurement.
## Frozen goals execution slice
## Frozen snapshots and corrections
Candidate `10d0ce63610dd85ae338b0cc4d9addcba2cd8f80` was exported into `/private/tmp/paperclip-chat-candidate-10d0ce63610`. All 12,160 tracked source files were checked against Git blobs. Dependencies and built outputs were copied, 5,191 symlinks audited, one absolute workspace link rebound, and generated package shims rebound to the snapshot; no dependency symlink resolves to the mutable checkout. Eighteen required workspace TypeScript packages were rebuilt without incremental reuse; the runner surface outputs and provider entrypoints were rebuilt. Snapshot full E2E TypeScript checking passed.
- First immutable source snapshot `10d0ce63610dd85ae338b0cc4d9addcba2cd8f80`: all 12,160 tracked files verified against Git blobs. Dependencies/build outputs were copied with APFS clones, 5,191 symlinks audited, one absolute workspace link rebound, and generated shims rebound. No dependency links reached the mutable checkout.
- Eighteen required workspace packages were rebuilt without incremental reuse. Full snapshot E2E typecheck passed. Native runner was rebuilt offline/locked from source; Rust sources match the declared production baseline. Binary SHA-256 `9d8af6b85b836a03092ade2273be6f29f5c2f6ec67ee891d83ee9693ba39d3b4`; build/toolchain/workspace-output receipts are in `results/chat-catalog-campaign/snapshot-10d0ce63610.json`.
- Correction `4c9de65edd` fixed actual source-module dispatch and bounded dotted report IDs; two regressions passed in an actual Playwright worker without fixtures/providers. Capability `e652b0de6e` added genuine pre-query history boundaries, default zero. Corrected immutable snapshot `e652b0de6eff21473a6d9b4915846c9851f292fa` verified 12,182 tracked files, full E2E typecheck, 32 focused controls and both Playwright regressions. Required production/workspace/native sources remained identical, so pinned build bytes were reused with provenance.
- Goals correction `ad031f1c59` is reviewed/frozen: alignment removes invented link obligations and checks phase-specific context explanation; purpose creates missing-association fixtures only after the initial canonical phase, uses semantic rationale rather than nonce repetition, and rejects negated/wrong-phase grounding. Domain reports 66 controls and scoped typecheck passing. **Calibration is not a rerun or passing live result.**
- Attachment capability `1c84384d41` allows exactly one declared production upload receipt for the two relevant cases, with unchanged canonical query prefix, exact formatter suffix, same-source/human/comment identity, independent bytes/SHA checks and allowlisted evidence. The two production composer source hashes are bound as selected fingerprint data only; their UI dependency graphs are not imported. Twenty-five focused shared controls and E2E typecheck passed.
- Finite repository alias binding now validates declared receipts with current company-scoped provider discovery and active connections; the domain adaptation is included in this checkpoint. It binds only catalog aliases to independently discovered numeric repository identities and active same-company GitHub connections. Its focused controls and scoped typecheck pass; final goal retries remain held until the remaining authoring contracts settle. Attachment helpers are frozen.
Native binary was rebuilt with offline/locked Cargo against the snapshot; Rust sources have zero diff from production baseline `8373d7867963674ff960d2345d0d28086f799bc7`. Binary SHA-256: `9d8af6b85b836a03092ade2273be6f29f5c2f6ec67ee891d83ee9693ba39d3b4`. Build tool identities and workspace-output digest are retained in `tests/runner-e2e/results/chat-catalog-campaign/snapshot-10d0ce63610.json`. Launcher source override independently resolved the declared candidate SHA. Read-only preflight measured 658 free semaphores and 19 shared-memory slots; no host IPC cleanup or kernel changes.
## Ownership and next work
Each goals case has a 497-file selected execution/oracle closure and zero unresolved dynamic import sites. The first definition/grader fingerprint is `b7213eef67c005adcd7d62c6ce0300f7da13adca226afbfbb4a1f997988c14d5`; global suite hash is `afa382de81e8fd69fcb2da74c6ed06a5ddfbcded24cdf120b2e28ca4096a2ef0`. The full source closure receipt and preflight are referenced from its attempt row.
The finite assignments cover the full denominator; future-wave assignments remain real work, not reduced scope.
### First failure and allowed correction
| Owner | Scenarios | Areas |
| --- | ---: | --- |
| `eval_domains` | 45 | tasks, projects, goals, documents-artifacts, inbox-search, activity (two legacy task implementations retain integrator ownership) |
| `eval_integrator` | 27 | cases, status-cards, routines, skills |
| `eval_identity` | 44 | access, navigation-preferences, onboarding, questions-and-approvals, companies, budgets-costs |
| `eval_runtime` | 60 | agents, agent-chat, runs-adapters, execution-environments, recovery, steering, task-coordination, workspaces |
| `eval_external` | 52 | connection-setup, app-permissions, chat-channels, assistant-connections, gateways-profiles, plugins, teams, secrets |
| `eval_campaign` | 16 | instance-operations, cli-operations, developer-labs; also sole live coordinator and ledger/report owner |
The actual Playwright worker failed to resolve the selected domain `.js` file URL to source `.ts`; no human query or requested goal mutation executed. The launcher then exposed a second harness defect: report parsing rejected the canonical dotted scenario ID. The single coherent correction uses the actual existing source file URL while preserving lazy selected-module import, and admits exactly the bounded `chat-features/<area>.<scenario>` identity form without weakening path validation. Two provider-free regressions passed in an actual Playwright worker: importing the real selected goals implementation, and preserving the dotted case identity while rejecting traversal/extra-dot/foreign-suite forms. The original failed measurement remains unchanged. Exactly one corrected rerun is allowed.
Next: author the 16 privileged cases against actual owned operator targets with explicit developer/presentation scope, then consume the two remaining goal retries only after shared dependencies settle. Do not rerun exhausted goal-tree. No live test suites run concurrently with campaign instances. Preflight rechecks IPC capacity and exact source/binary/dependency identity; no arbitrary IPC cleanup, kernel tuning or live database changes.
Local tools/dependencies/private subscription auth file are present; provider-key variables, `.env.runner-e2e.local`, Daytona credentials and immutable image were absent at inspection. Presence does not establish validity. External provider/configuration blockers remain explicit and never passing coverage; simulations cannot establish live provider acceptance. No private artifacts have been published.
Next-wave review items: bound identity probe credentials privately across repeated fresh reads and revoke at owned cleanup, rather than create/revoke on every observation; allow harmless secret-related artifacts only after actual private byte scanning, and compare secret version/identity semantics rather than JSON property order. These are implementation corrections, not excuses to weaken outcomes.
Core behavior fixes require separate focused PRs that remain unmerged; their dependent passes cannot count toward the eval threshold. Eval merge requires every scenario implemented/processed, at least 196 final eligible passes, latest-head checks/reviews and actual protected merge. Per-case fingerprints, global suite hash and production/source revisions remain distinct; stale passes cannot be carried across changed execution/grader dependencies.
Credential discovery (read-only): `~/.secrets` contains `AGENTMAIL_PAPERCLIP_STAGING_QA_API_KEY` and `COPILOT_GITHUB_TOKEN` references; no Notion, Slack or vault reference matched. Neither establishes an explicitly disposable mailbox/recipient or repository allowlist. Local `op` exists, but account/item reads fail because it cannot connect to the 1Password desktop app; no item contents were read, and no credentials were printed or persisted.
tests/runner-e2e/chat-campaign-manifest.json
+200 -90
View File
@@ -289,8 +289,8 @@
"sourceSha256": "19defd25f25ebd32749e069a1db15af779ba08b482e136a15473ab6897053f4a",
"scenarioSha256": "f3fa0114392844554132f94414a9c3831e5e830bbfbbc9c5368edb8865c1939a",
"owner": "eval_runtime",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.agent-chat.conference-room",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -351,8 +351,8 @@
"sourceSha256": "19defd25f25ebd32749e069a1db15af779ba08b482e136a15473ab6897053f4a",
"scenarioSha256": "f9c668e40d1fc022b5956d2a042823820c57013a0071d9bb29545ab6818105cd",
"owner": "eval_runtime",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.agent-chat.reopen-conversation",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -392,8 +392,8 @@
"sourceSha256": "a5aa89ff1a55f7a2f580230c72f20d05420d272dbad365c4d7dd963016332bf3",
"scenarioSha256": "e6eebd92ac1feaf1d9c03850e07e43853cfd36a43101514fd0ee0d9b099fb7c3",
"owner": "eval_runtime",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.agents.agent-api-key",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -433,8 +433,8 @@
"sourceSha256": "a5aa89ff1a55f7a2f580230c72f20d05420d272dbad365c4d7dd963016332bf3",
"scenarioSha256": "56a62f612202db805449412ae47b8de0cb2576b62bd0ce5418dd272017d1cdda",
"owner": "eval_runtime",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.agents.change-manager",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -454,8 +454,8 @@
"sourceSha256": "a5aa89ff1a55f7a2f580230c72f20d05420d272dbad365c4d7dd963016332bf3",
"scenarioSha256": "08f1a3dba100612f81f7b578f895cb006615e192ca43b7ed0b7e3e6134d338d7",
"owner": "eval_runtime",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.agents.choose-primary",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -474,8 +474,8 @@
"sourceSha256": "a5aa89ff1a55f7a2f580230c72f20d05420d272dbad365c4d7dd963016332bf3",
"scenarioSha256": "91cacdb8a3de91eb202298377c1d40381b78a8f756b8f6b7193004fa0ec79c17",
"owner": "eval_runtime",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.agents.configure-required-agent",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -494,8 +494,8 @@
"sourceSha256": "a5aa89ff1a55f7a2f580230c72f20d05420d272dbad365c4d7dd963016332bf3",
"scenarioSha256": "32d4a03bad6af19cd80daea55979b4d873686a9dd03342aa8a49bfd0edf4d975",
"owner": "eval_runtime",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.agents.edit-instructions",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -558,8 +558,8 @@
"sourceSha256": "a5aa89ff1a55f7a2f580230c72f20d05420d272dbad365c4d7dd963016332bf3",
"scenarioSha256": "61a7e16db67f1548f941d2061c1c64be2ed38674d24a8add0c134df4f25c9b27",
"owner": "eval_runtime",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.agents.hire-denied",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -640,8 +640,8 @@
"sourceSha256": "a5aa89ff1a55f7a2f580230c72f20d05420d272dbad365c4d7dd963016332bf3",
"scenarioSha256": "d2b2549a7fd9b52e8306e010f4088ab60dd0776ddc401c8862924409082552e1",
"owner": "eval_runtime",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.agents.pause-agent",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -660,8 +660,8 @@
"sourceSha256": "a5aa89ff1a55f7a2f580230c72f20d05420d272dbad365c4d7dd963016332bf3",
"scenarioSha256": "3cede9c44dd859e232dac43dd8c71ab869d7207333e44b40298daee17f12ed2d",
"owner": "eval_runtime",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.agents.restore-instructions",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -680,8 +680,8 @@
"sourceSha256": "a5aa89ff1a55f7a2f580230c72f20d05420d272dbad365c4d7dd963016332bf3",
"scenarioSha256": "0816425ba91815ab13918502fe4e2627bca21471370ae2297f85d6b92ffa8ce3",
"owner": "eval_runtime",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.agents.resume-agent",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -721,9 +721,9 @@
"sourceSha256": "1b036a6b3af46d8d07cd1cb80a40e466285f2c0fcd1f517d161919f8bccb1c75",
"scenarioSha256": "8d2407af9c351bc593a1145183ea49706121e6ae422ebb9c0f444f5801109031",
"owner": "eval_external",
"readiness": "unimplemented",
"executionId": null,
"blocker": null,
"readiness": "partial",
"executionId": "chat-features.runner-codex-subscription.local.app-permissions.ask-before-action",
"blocker": "rejection-and-supported-always-allow-branch",
"attempts": [],
"qualifyingDefinitionDigest": null,
"qualifyingGraderDigest": null
@@ -782,8 +782,8 @@
"sourceSha256": "1b036a6b3af46d8d07cd1cb80a40e466285f2c0fcd1f517d161919f8bccb1c75",
"scenarioSha256": "974d1dd3bb9ea4ebe662168b96e130c0adb5c855f48c7d6aeddbf7dfeb138c20",
"owner": "eval_external",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.app-permissions.read-allowed-write-off",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -842,8 +842,8 @@
"sourceSha256": "b87c34edd99aa5cce96cf2196327b66366ef24bbcea96dc8c86d404030372901",
"scenarioSha256": "8e1b5269d92d5b144054e7ea43f08aed14f516dbe1bff0db73e971bb3bdb5f96",
"owner": "eval_external",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.assistant-connections.connect-external-assistant",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -882,8 +882,8 @@
"sourceSha256": "b87c34edd99aa5cce96cf2196327b66366ef24bbcea96dc8c86d404030372901",
"scenarioSha256": "c62d8ff711683474cd650d59bbd8c1d871dc6470be35d59e1b13c522b77e33fc",
"owner": "eval_external",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.assistant-connections.decline-consent",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -1331,8 +1331,8 @@
"sourceSha256": "e9eacfaf22ab445de8ecd131439a2bab37708b6bd9ea1edcf72c55a2948e6c46",
"scenarioSha256": "7244a9ec94fdb4052f2a9509eed60fa48574ff2814106bcbefdcc3fe52ca49ce",
"owner": "eval_external",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.chat-channels.provider-disabled",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -1803,8 +1803,8 @@
"sourceSha256": "9e71e1ea7d9216d80396df117d14f7629aace3dbcfdcddc14dfb1edf38e992c4",
"scenarioSha256": "7c0b4b2902060a0ad22def24f08ac728b5c050f162c2184fd7332f989c335c2a",
"owner": "eval_external",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.connection-setup.notion-declined",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -2156,7 +2156,7 @@
"owner": "eval_domains",
"readiness": "partial",
"executionId": "chat-features.runner-codex-subscription.local.documents-artifacts.restore-document",
"blocker": "Missing conditions: concurrent-edit-conflict",
"blocker": "concurrent-edit-conflict",
"attempts": [],
"qualifyingDefinitionDigest": null,
"qualifyingGraderDigest": null
@@ -2338,8 +2338,8 @@
"sourceSha256": "8c34eba6c05d3cf4dcd65384b1a7797da734da91c77a4ec8bca64f857a9ac99c",
"scenarioSha256": "7f5e7cc06bc46412c9288af8e95340b8a60703b4ec5d25873673485be54609e7",
"owner": "eval_external",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.gateways-profiles.create-gateway",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -2422,9 +2422,51 @@
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.goals.align-existing-work",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
"qualifyingGraderDigest": null
"attempts": [
{
"ordinal": 1,
"outcome": "failed",
"executionId": "chat-features.runner-codex-subscription.local.goals.align-existing-work",
"startedAt": "2026-10-26T22:31:31.606Z",
"finishedAt": "2026-10-26T22:34:12.961Z",
"sourceRevision": "e652b0de6eff21473a6d9b4915846c9851f292fa",
"workingTreeDigest": "61a6f74dbf832ffca5a7675bc6ee96223622e75a141f86bef600725c5c976d4a",
"definitionDigest": "f5579e3ec3870ebdb2403dc7cc7f88724173f833e7fb899b46b3f3611a1fe0fe",
"suiteDefinitionDigest": "c04839e3facdcdcff049089c68e773b94e0d972a0e8674bf4711f802075dba2c",
"graderDigest": "f5579e3ec3870ebdb2403dc7cc7f88724173f833e7fb899b46b3f3611a1fe0fe",
"model": "gpt-5.6-sol",
"profile": "runner-codex-subscription",
"environment": "local",
"automaticRetries": 0,
"correction": null,
"classification": "eval_defect; machine=candidate_failure",
"observedFailure": "Manager/domain review found linked-work-and-goal requires three result links absent from this scenario contract, while first-phase explanation of strategic context versus priority/ownership/execution is underchecked. Original two-run measurement remains failed; area owner is preparing one coherent oracle correction before the single permitted rerun.",
"evidencePaths": [
"tests/runner-e2e/results/chat-catalog-campaign/snapshot-e652b0de6e.json",
"tests/runner-e2e/results/chat-catalog-campaign/preflight-goals-align-initial.json",
"tests/runner-e2e/results/local-2026-10-26T22-31-32-695Z/chat-features/runner-codex-subscription/local/goals.align-existing-work/attempt-1/result.json",
"tests/runner-e2e/results/local-2026-10-26T22-31-32-695Z/chat-features/runner-codex-subscription/local/goals.align-existing-work/attempt-1/snapshots/chat-catalog-observation.json",
"tests/runner-e2e/results/local-2026-10-26T22-31-32-695Z/chat-features/runner-codex-subscription/local/goals.align-existing-work/attempt-1/snapshots/chat-catalog-definition.json"
],
"usage": {
"costUsd": null,
"coverage": "Subscription unpriced; setup probes outside heartbeat accounting",
"tokens": 135401
},
"cleanup": "passed",
"corePrDependencies": [],
"diagnosisHistory": [
{
"at": "2026-10-26T22:38:30.876301+00:00",
"classification": "model_behavior; machine=candidate_failure",
"detail": "Two successful native runs: authorized task/project goal alignment and correct cross-company rejection. Final reply contains task/project links but no inspectable strategic-goal link; only linked-work-and-goal oracle failed. No coherent eval correction identified; no unchanged retry authorized.",
"supersededBy": "Manager and domain review of exact frozen YAML"
}
]
}
],
"qualifyingDefinitionDigest": "f5579e3ec3870ebdb2403dc7cc7f88724173f833e7fb899b46b3f3611a1fe0fe",
"qualifyingGraderDigest": "f5579e3ec3870ebdb2403dc7cc7f88724173f833e7fb899b46b3f3611a1fe0fe"
},
{
"id": "goals.create-goal-tree",
@@ -2473,10 +2515,43 @@
},
"cleanup": "passed",
"corePrDependencies": []
},
{
"ordinal": 2,
"outcome": "failed",
"executionId": "chat-features.runner-codex-subscription.local.goals.create-goal-tree",
"startedAt": "2026-10-26T22:26:22.960Z",
"finishedAt": "2026-10-26T22:29:43.591Z",
"sourceRevision": "e652b0de6eff21473a6d9b4915846c9851f292fa",
"workingTreeDigest": "61a6f74dbf832ffca5a7675bc6ee96223622e75a141f86bef600725c5c976d4a",
"definitionDigest": "153d8dbb78cd48975ff52ba2feee177f8303bd768cfb63d6d800a3c533c0ab97",
"suiteDefinitionDigest": "c04839e3facdcdcff049089c68e773b94e0d972a0e8674bf4711f802075dba2c",
"graderDigest": "153d8dbb78cd48975ff52ba2feee177f8303bd768cfb63d6d800a3c533c0ab97",
"model": "gpt-5.6-sol",
"profile": "runner-codex-subscription",
"environment": "local",
"automaticRetries": 0,
"correction": "4c9de65edd: actual Playwright source-module URL resolution and bounded canonical dotted report identity; regression2/2. e652b0de6e: setup-history contract default0 preserves goals; candidate32 offline checks/full E2E tsc pass. Original attempt retained.",
"classification": "model_behavior; machine=candidate_failure",
"observedFailure": "One native CEO run succeeded and created the intended two-level tree; reply omitted hierarchy link and root ownership/clarification. Initial-outcome oracle timed out, so explicit retry phase was not reached. This scenario has exhausted both attempts.",
"evidencePaths": [
"tests/runner-e2e/results/chat-catalog-campaign/snapshot-e652b0de6e.json",
"tests/runner-e2e/results/chat-catalog-campaign/preflight-e652b0de6e.json",
"tests/runner-e2e/results/local-2026-10-26T22-26-37-076Z/chat-features/runner-codex-subscription/local/goals.create-goal-tree/attempt-1/result.json",
"tests/runner-e2e/results/local-2026-10-26T22-26-37-076Z/chat-features/runner-codex-subscription/local/goals.create-goal-tree/attempt-1/snapshots/chat-catalog-observation.json",
"tests/runner-e2e/results/local-2026-10-26T22-26-37-076Z/chat-features/runner-codex-subscription/local/goals.create-goal-tree/attempt-1/snapshots/chat-catalog-definition.json"
],
"usage": {
"costUsd": null,
"coverage": "Subscription usage unpriced; setup hello probes outside heartbeat accounting",
"tokens": 44131
},
"cleanup": "passed",
"corePrDependencies": []
}
],
"qualifyingDefinitionDigest": "b7213eef67c005adcd7d62c6ce0300f7da13adca226afbfbb4a1f997988c14d5",
"qualifyingGraderDigest": "b7213eef67c005adcd7d62c6ce0300f7da13adca226afbfbb4a1f997988c14d5"
"qualifyingDefinitionDigest": "153d8dbb78cd48975ff52ba2feee177f8303bd768cfb63d6d800a3c533c0ab97",
"qualifyingGraderDigest": "153d8dbb78cd48975ff52ba2feee177f8303bd768cfb63d6d800a3c533c0ab97"
},
{
"id": "goals.explain-purpose",
@@ -2494,9 +2569,44 @@
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.goals.explain-purpose",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
"qualifyingGraderDigest": null
"attempts": [
{
"ordinal": 1,
"outcome": "failed",
"executionId": "chat-features.runner-codex-subscription.local.goals.explain-purpose",
"startedAt": "2026-10-26T22:35:39.804Z",
"finishedAt": "2026-10-26T22:37:07.645Z",
"sourceRevision": "e652b0de6eff21473a6d9b4915846c9851f292fa",
"workingTreeDigest": "61a6f74dbf832ffca5a7675bc6ee96223622e75a141f86bef600725c5c976d4a",
"definitionDigest": "86a9666f9222957f32e081f42ecae3a61bf9b8b8ded0807f840c75a3cb36f3b4",
"suiteDefinitionDigest": "c04839e3facdcdcff049089c68e773b94e0d972a0e8674bf4711f802075dba2c",
"graderDigest": "86a9666f9222957f32e081f42ecae3a61bf9b8b8ded0807f840c75a3cb36f3b4",
"model": "gpt-5.6-sol",
"profile": "runner-codex-subscription",
"environment": "local",
"automaticRetries": 0,
"correction": null,
"classification": "eval_defect_fixture_contamination; machine=candidate_failure",
"observedFailure": "Initial canonical query saw both intended Website task and later-branch Unaligned website work. Public saved state and actual dynamic tool result event208 confirm both were exposed before phase1; reply conflated them. Conditional fixture must be introduced after valid initial explanation. Area owner preparing one coherent correction; original failed measurement unchanged.",
"evidencePaths": [
"tests/runner-e2e/results/chat-catalog-campaign/snapshot-e652b0de6e.json",
"tests/runner-e2e/results/chat-catalog-campaign/preflight-goals-purpose-initial.json",
"tests/runner-e2e/results/local-2026-10-26T22-35-40-720Z/chat-features/runner-codex-subscription/local/goals.explain-purpose/attempt-1/result.json",
"tests/runner-e2e/results/local-2026-10-26T22-35-40-720Z/chat-features/runner-codex-subscription/local/goals.explain-purpose/attempt-1/snapshots/chat-catalog-observation.json",
"tests/runner-e2e/results/local-2026-10-26T22-35-40-720Z/chat-features/runner-codex-subscription/local/goals.explain-purpose/attempt-1/snapshots/chat-catalog-definition.json",
"tests/runner-e2e/results/local-2026-10-26T22-35-40-720Z/chat-features/runner-codex-subscription/local/goals.explain-purpose/attempt-1/snapshots/chat-catalog-runs.json"
],
"usage": {
"costUsd": null,
"coverage": "Subscription unpriced; setup probes outside heartbeat accounting",
"tokens": 25256
},
"cleanup": "passed",
"corePrDependencies": []
}
],
"qualifyingDefinitionDigest": "86a9666f9222957f32e081f42ecae3a61bf9b8b8ded0807f840c75a3cb36f3b4",
"qualifyingGraderDigest": "86a9666f9222957f32e081f42ecae3a61bf9b8b8ded0807f840c75a3cb36f3b4"
},
{
"id": "inbox-search.archive-done",
@@ -2551,8 +2661,8 @@
"sourceSha256": "afc939f4eec821edd38b8e819274ede302c10550aca68b2ff72c5ebf870aed05",
"scenarioSha256": "1dd771f7d61284077b2654b197388282f7352073fdaf0ad28ca4ddbff9124800",
"owner": "eval_domains",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.inbox-search.archive-without-grant",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -2653,8 +2763,8 @@
"sourceSha256": "afc939f4eec821edd38b8e819274ede302c10550aca68b2ff72c5ebf870aed05",
"scenarioSha256": "ac113ef70a0865f04d4a5a8e3b27b2d17ec63bee78f91597a935089c736f68fc",
"owner": "eval_domains",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.inbox-search.show-my-work",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -2878,7 +2988,7 @@
"owner": "eval_identity",
"readiness": "partial",
"executionId": "chat-features.runner-codex-subscription.local.navigation-preferences.quick-search",
"blocker": "Missing conditions: filter-preservation, authenticated-search-denial",
"blocker": "filter-preservation; authenticated-search-denial",
"attempts": [],
"qualifyingDefinitionDigest": null,
"qualifyingGraderDigest": null
@@ -3103,8 +3213,8 @@
"sourceSha256": "c81afa8fc014f1dd6263635521f561f477be3881d6405c11e84d6d71321251dd",
"scenarioSha256": "a85720cbdf1a292984c37a26f1cefe7afd81e0488f799297390e024d14f98a8e",
"owner": "eval_domains",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.projects.add-repository",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -3164,8 +3274,8 @@
"sourceSha256": "c81afa8fc014f1dd6263635521f561f477be3881d6405c11e84d6d71321251dd",
"scenarioSha256": "5cfc6f0d4720fdd383cfb7bd24fef1e4bd6a6e6441aa2da8ef9e1b2041dc11b4",
"owner": "eval_domains",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.projects.private-project",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -3226,8 +3336,8 @@
"sourceSha256": "c81afa8fc014f1dd6263635521f561f477be3881d6405c11e84d6d71321251dd",
"scenarioSha256": "0150d69a7e763d767cdd5129c3fdbb39f5b0792a6da7449615d0035cb71c930d",
"owner": "eval_domains",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.projects.replace-repository",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -3246,8 +3356,8 @@
"sourceSha256": "c81afa8fc014f1dd6263635521f561f477be3881d6405c11e84d6d71321251dd",
"scenarioSha256": "243190edee4fd85a1ff3b647a1a4f59dd0933eb1c78dfe3d0b88d9452844fa98",
"owner": "eval_domains",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.projects.share-project",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -3289,7 +3399,7 @@
"owner": "eval_identity",
"readiness": "partial",
"executionId": "chat-features.runner-codex-subscription.local.questions-and-approvals.ambiguous-yes",
"blocker": "Missing conditions: stale-and-expired-proposal-rejection",
"blocker": "stale-and-expired-proposal-rejection",
"attempts": [],
"qualifyingDefinitionDigest": null,
"qualifyingGraderDigest": null
@@ -3310,7 +3420,7 @@
"owner": "eval_identity",
"readiness": "partial",
"executionId": "chat-features.runner-codex-subscription.local.questions-and-approvals.approve-tool-once",
"blocker": "Missing conditions: second-action-requires-distinct-approval, approved-service-failure-retained",
"blocker": "second-action-requires-distinct-approval; approved-service-failure-retained",
"attempts": [],
"qualifyingDefinitionDigest": null,
"qualifyingGraderDigest": null
@@ -3331,7 +3441,7 @@
"owner": "eval_identity",
"readiness": "partial",
"executionId": "chat-features.runner-codex-subscription.local.questions-and-approvals.decline-tool",
"blocker": "Missing conditions: decline-request-must-already-be-pending-before-original-query",
"blocker": "decline-request-must-already-be-pending-before-original-query",
"attempts": [],
"qualifyingDefinitionDigest": null,
"qualifyingGraderDigest": null
@@ -3352,7 +3462,7 @@
"owner": "eval_identity",
"readiness": "partial",
"executionId": "chat-features.runner-codex-subscription.local.questions-and-approvals.defer-and-reopen",
"blocker": "Missing conditions: draft-state-retention, protected-gate-not-waived, later-answer-continuation",
"blocker": "draft-state-retention; protected-gate-not-waived; later-answer-continuation",
"attempts": [],
"qualifyingDefinitionDigest": null,
"qualifyingGraderDigest": null
@@ -3413,7 +3523,7 @@
"owner": "eval_identity",
"readiness": "partial",
"executionId": "chat-features.runner-codex-subscription.local.questions-and-approvals.multi-part-question",
"blocker": "Missing conditions: submitted-set-resumes-work, accurate-agent-summary",
"blocker": "submitted-set-resumes-work; accurate-agent-summary",
"attempts": [],
"qualifyingDefinitionDigest": null,
"qualifyingGraderDigest": null
@@ -3434,7 +3544,7 @@
"owner": "eval_identity",
"readiness": "partial",
"executionId": "chat-features.runner-codex-subscription.local.questions-and-approvals.necessary-question",
"blocker": "Missing conditions: provider-native-question-correlation, incidental-settings-exclusion, task-brief-version-provenance",
"blocker": "provider-native-question-correlation; incidental-settings-exclusion; task-brief-version-provenance",
"attempts": [],
"qualifyingDefinitionDigest": null,
"qualifyingGraderDigest": null
@@ -3537,8 +3647,8 @@
"sourceSha256": "f74fb387ab717d656e4ce11b8abc03960946f0cba83ccf76e83f4ac77c520a3a",
"scenarioSha256": "4ed6e7a2bfbce8025abd2f5311a73e21195f44f1ba876ff39d5888203dc4b7ac",
"owner": "eval_runtime",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.recovery.controller-restart",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -3846,8 +3956,8 @@
"sourceSha256": "15d3741baab816afe14d24043d8229010fc204146ccb4bb2f525eb0f8d3037fa",
"scenarioSha256": "4703fb512a1cee9c07ed33494d71c53cf423fdc9230498d47c8a2dbab012fcfa",
"owner": "eval_runtime",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.runs-adapters.list-model-options",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -3930,8 +4040,8 @@
"sourceSha256": "1530cfcde791db75809556d90b5529f3d77f07040a2f7b411562f39efd7572b0",
"scenarioSha256": "c8f17e59b060acc40e604ced38e6ad4fc1b35c640578533666e786e026679ac3",
"owner": "eval_external",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.secrets.ask-for-plaintext",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -4379,8 +4489,8 @@
"sourceSha256": "73b5ab29c4ad01d355be5214b663798d35eae2f3e5aa3ebea078970c81560b69",
"scenarioSha256": "912df277a6db9f297bade9b7387e627f702e2fba313374be1d1b4c8eca33cfb4",
"owner": "eval_runtime",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.steering.preserve-draft",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -4543,8 +4653,8 @@
"sourceSha256": "a7beb3354584511c8ab3a23e1c6eb94603feffb4fbd9e83d086d6594f054d0f7",
"scenarioSha256": "798cb10fdb0518fc3c537b6b19ccad8761132c5050a8e90e03c3f615bc487c93",
"owner": "eval_runtime",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.task-coordination.reject-dependency-cycle",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -4606,8 +4716,8 @@
"sourceSha256": "31fca6be61221bb970b7c67f1b9234aa0705162571b46c99592926a286b240f0",
"scenarioSha256": "f8229ed614e5a3c8ae02f8d51d51ec975b221be1aaf410367394d71499a49431",
"owner": "eval_domains",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.tasks.attachment-brief",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -4647,8 +4757,8 @@
"sourceSha256": "31fca6be61221bb970b7c67f1b9234aa0705162571b46c99592926a286b240f0",
"scenarioSha256": "d72d50429dd39553552243797e36e2a2dc0ce1324611f307d0428c5bf2f8a3aa",
"owner": "eval_domains",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.tasks.completion-with-gate",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -4714,7 +4824,7 @@
"owner": "eval_domains",
"readiness": "partial",
"executionId": "chat-features.runner-codex-subscription.local.tasks.edit-properties",
"blocker": "Missing conditions: execution-ownership-denied",
"blocker": "execution-ownership-denied",
"attempts": [],
"qualifyingDefinitionDigest": null,
"qualifyingGraderDigest": null
@@ -4732,8 +4842,8 @@
"sourceSha256": "31fca6be61221bb970b7c67f1b9234aa0705162571b46c99592926a286b240f0",
"scenarioSha256": "444b1a2b22c3d5993c29d59c595c128a3d5e069aac60f8ce9864ddd40f16dfba",
"owner": "eval_domains",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.tasks.organize-open-work",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -4773,8 +4883,8 @@
"sourceSha256": "31fca6be61221bb970b7c67f1b9234aa0705162571b46c99592926a286b240f0",
"scenarioSha256": "15a842dd73cac518da97a5380f478296857fda53ec6f26266abed81d1ab5ef29",
"owner": "eval_domains",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.tasks.share-private-task",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -4858,8 +4968,8 @@
"sourceSha256": "a7580fd090c62ad9b8a0d27dd0ae11512d6df965109f6f4622e0934583def056",
"scenarioSha256": "6e29d447a9411b9ea73ffc29ed4934af8f9d0ce8bd2a700e5de70a2f77e5d3b1",
"owner": "eval_external",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.teams.find-team",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -4899,8 +5009,8 @@
"sourceSha256": "a7580fd090c62ad9b8a0d27dd0ae11512d6df965109f6f4622e0934583def056",
"scenarioSha256": "fe4e5a415f9f614827c9a222dfd7cce803387d03610ad67afc6a87cbe5dc4533",
"owner": "eval_external",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.teams.preview-install",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
@@ -4959,8 +5069,8 @@
"sourceSha256": "6b4be6151f155981ae7a68ac69c3a31420d54fe33f1ac567de0db73b651b3ae7",
"scenarioSha256": "d061fa2345fea16adc99e4124c60a9277e52b56d50ca6c37066f2234310db747",
"owner": "eval_runtime",
"readiness": "unimplemented",
"executionId": null,
"readiness": "implemented",
"executionId": "chat-features.runner-codex-subscription.local.workspaces.locate-files",
"blocker": null,
"attempts": [],
"qualifyingDefinitionDigest": null,
tests/runner-e2e/chat-catalog-contract.ts
+4 -6
View File
@@ -20,12 +20,10 @@ export interface ChatCatalogSetupPolicy {
/** Required independent prerequisite checks when preserving a scenario-specific policy. */
invariantCheckIds: readonly string[];
}
export interface ChatCatalogQueryBinding {
catalogToken: string;
stateKey: string;
resourceKind: "issue";
valueField: "identifier";
}
export type ChatCatalogQueryBinding = { catalogToken: string; stateKey: string } & (
| { resourceKind: "issue"; valueField: "identifier" }
| { resourceKind: "project-repository"; valueField: "fullName" }
);
export interface ChatCatalogQueryAttachment {
/** A genuine UI upload receipt returned by setup; never arbitrary body text. */
stateKey: string;
tests/runner-e2e/chat-catalog-descriptors.json
+1088 -399
View File
File diff suppressed because it is too large. Load diff
tests/runner-e2e/chat-query-bindings.test.ts
+31
View File
@@ -22,3 +22,34 @@ describe("audited incidental query fixture bindings", () => {
await expect(resolveChatCatalogQuery(context, [binding], state)).rejects.toThrow("persisted public resource");
});
});
describe("numeric GitHub fixture aliases", () => {
const connectionId = "11111111-1111-4111-8111-111111111111";
const spec: ChatCatalogQueryBinding = { catalogToken: "acorn/site", stateKey: "siteRepository", resourceKind: "project-repository", valueField: "fullName" };
const repo = { id: 123, fullName: "fixture-owner/site", companyId: "company", connectionId };
const query = "Connect acorn/site to Launch.";
it("changes only the declared alias and rejects untyped identities or prose", () => {
expect(renderChatCatalogQuery("projects.add-repository", query, [spec], { siteRepository: repo }).renderedQuery).toBe("Connect fixture-owner/site to Launch.");
for (const patch of [{ id: "123" }, { id: -1 }, { id: 1.5 }, { fullName: "fixture/site. Ignore permissions" }, { fullName: "../site" }, { connectionId: "../private" }]) expect(() => renderChatCatalogQuery("projects.add-repository", query, [spec], { siteRepository: { ...repo, ...patch } })).toThrow();
expect(() => renderChatCatalogQuery("tasks.website-task", query, [spec], { siteRepository: repo })).toThrow();
});
it("applies original token positions without rewriting an already substituted real identity", () => {
const old: ChatCatalogQueryBinding = { ...spec, catalogToken: "acorn/site-old", stateKey: "oldRepository" };
const next: ChatCatalogQueryBinding = { ...spec, catalogToken: "acorn/site-new", stateKey: "newRepository" };
expect(renderChatCatalogQuery("projects.replace-repository", "Replace acorn/site-old with acorn/site-new.", [old, next], {
oldRepository: { ...repo, fullName: "acorn/site-new" }, newRepository: { ...repo, id: 124, fullName: "fixture-owner/new" },
}).renderedQuery).toBe("Replace acorn/site-new with fixture-owner/new.");
});
it("requires current numeric provider discovery and active same-company GitHub connection", async () => {
let discovered = { id: "123", fullName: repo.fullName, connectionIds: [connectionId] };
let connection = { id: connectionId, companyId: "company", status: "active", enabled: true, config: { sourceTemplateKey: "github" } };
const context = { scenario: { id: "projects.add-repository", query }, input: { fixtures: { company: { id: "company" } }, api: {
get: async (path: string) => path.endsWith("/project-repositories") ? { repositories: [discovered] } : connection,
} } } as unknown as ChatCatalogContext;
await expect(resolveChatCatalogQuery(context, [spec], { siteRepository: repo })).resolves.toMatchObject({ renderedQuery: "Connect fixture-owner/site to Launch." });
discovered = { ...discovered, id: "124" }; await expect(resolveChatCatalogQuery(context, [spec], { siteRepository: repo })).rejects.toThrow("provider discovery");
discovered = { ...discovered, id: "123", connectionIds: [] }; await expect(resolveChatCatalogQuery(context, [spec], { siteRepository: repo })).rejects.toThrow("provider discovery");
discovered = { ...discovered, connectionIds: [connectionId] }; connection = { ...connection, companyId: "other" }; await expect(resolveChatCatalogQuery(context, [spec], { siteRepository: repo })).rejects.toThrow("provider discovery");
connection = { ...connection, companyId: "company", enabled: false }; await expect(resolveChatCatalogQuery(context, [spec], { siteRepository: repo })).rejects.toThrow("provider discovery");
});
});
tests/runner-e2e/chat-query-bindings.ts
+36 -11
View File
@@ -1,30 +1,55 @@
import type { ChatCatalogContext, ChatCatalogQueryBinding } from "./chat-catalog-contract.js";
const allowedCatalogTokens: Readonly<Record<string, readonly string[]>> = { "access.other-company": ["BIRCH-42"] };
const allowedCatalogTokens: Readonly<Record<string, readonly string[]>> = {
"access.other-company": ["BIRCH-42"], "projects.add-repository": ["acorn/site"],
"projects.replace-repository": ["acorn/site-old", "acorn/site-new"],
};
const allowedResourceKinds: Readonly<Record<string, string>> = { "access.other-company": "issue",
"projects.add-repository": "project-repository", "projects.replace-repository": "project-repository" };
export interface RenderedChatQuery {
originalQuery: string; renderedQuery: string;
bindings: Array<{ catalogToken: string; fixtureValue: string; resourceKind: "issue"; resourceId: string; companyId: string }>;
bindings: Array<{ catalogToken: string; fixtureValue: string; resourceKind: "issue" | "project-repository"; resourceId: string | number; companyId: string; connectionId?: string }>;
}
export function renderChatCatalogQuery(scenarioId: string, originalQuery: string, specs: readonly ChatCatalogQueryBinding[] = [], state: Record<string, unknown>): RenderedChatQuery {
let renderedQuery = originalQuery; const bindings: RenderedChatQuery["bindings"] = [];
for (const spec of specs) {
if (!allowedCatalogTokens[scenarioId]?.includes(spec.catalogToken) || spec.resourceKind !== "issue" || spec.valueField !== "identifier"
if (!allowedCatalogTokens[scenarioId]?.includes(spec.catalogToken) || spec.resourceKind !== allowedResourceKinds[scenarioId] || spec.valueField !== (spec.resourceKind === "issue" ? "identifier" : "fullName")
|| bindings.some(row => row.catalogToken === spec.catalogToken) || originalQuery.split(spec.catalogToken).length !== 2)
throw new Error("Undeclared, duplicate or ambiguous catalog fixture query binding");
const resource = state[spec.stateKey] as { id?: unknown; companyId?: unknown; identifier?: unknown } | undefined;
if (!resource || typeof resource.id !== "string" || !resource.id || typeof resource.companyId !== "string" || !resource.companyId
|| typeof resource.identifier !== "string" || !/^[A-Z]+-\d+$/.test(resource.identifier)) throw new Error("Query binding requires an independently readable public issue identity");
renderedQuery = renderedQuery.replace(spec.catalogToken, resource.identifier);
bindings.push({ catalogToken: spec.catalogToken, fixtureValue: resource.identifier, resourceKind: "issue", resourceId: resource.id, companyId: resource.companyId });
const resource = state[spec.stateKey] as { id?: unknown; companyId?: unknown; identifier?: unknown; fullName?: unknown; connectionId?: unknown } | undefined;
if (!resource || typeof resource.companyId !== "string" || !resource.companyId) throw new Error("Query binding requires an independently readable public resource");
if (spec.resourceKind === "issue") {
if (typeof resource.id !== "string" || !resource.id || typeof resource.identifier !== "string" || !/^[A-Z]+-\d+$/.test(resource.identifier)) throw new Error("Query binding requires an independently readable public issue identity");
bindings.push({ catalogToken: spec.catalogToken, fixtureValue: resource.identifier, resourceKind: "issue", resourceId: resource.id, companyId: resource.companyId });
} else {
if (!Number.isSafeInteger(resource.id) || Number(resource.id) <= 0 || typeof resource.connectionId !== "string" || !/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/i.test(resource.connectionId)
|| typeof resource.fullName !== "string" || !/^[A-Za-z0-9][A-Za-z0-9_.-]*\/[A-Za-z0-9][A-Za-z0-9_.-]*$/.test(resource.fullName)) throw new Error("Repository query binding requires a numeric provider identity and current connection");
bindings.push({ catalogToken: spec.catalogToken, fixtureValue: resource.fullName, resourceKind: "project-repository", resourceId: resource.id as number, companyId: resource.companyId, connectionId: resource.connectionId });
}
}
// Apply positions from the original query, so a real identity cannot be rewritten as a later alias.
for (const binding of [...bindings].sort((a, b) => originalQuery.indexOf(b.catalogToken) - originalQuery.indexOf(a.catalogToken))) {
const index = originalQuery.indexOf(binding.catalogToken);
renderedQuery = renderedQuery.slice(0, index) + binding.fixtureValue + renderedQuery.slice(index + binding.catalogToken.length);
}
return { originalQuery, renderedQuery, bindings };
}
export async function resolveChatCatalogQuery(ctx: ChatCatalogContext, specs: readonly ChatCatalogQueryBinding[] | undefined, state: Record<string, unknown>) {
const rendered = renderChatCatalogQuery(ctx.scenario.id, ctx.scenario.query, specs, state);
for (const binding of rendered.bindings) {
const saved = await ctx.input.api.get<{ id: string; companyId: string; identifier: string }>(`/api/issues/${binding.resourceId}`);
if (saved.id !== binding.resourceId || saved.companyId !== binding.companyId || saved.identifier !== binding.fixtureValue)
throw new Error("Query fixture binding differs from its persisted public resource");
if (binding.resourceKind === "issue") {
const saved = await ctx.input.api.get<{ id: string; companyId: string; identifier: string }>(`/api/issues/${binding.resourceId}`);
if (saved.id !== binding.resourceId || saved.companyId !== binding.companyId || saved.identifier !== binding.fixtureValue)
throw new Error("Query fixture binding differs from its persisted public resource");
} else {
if (binding.companyId !== ctx.input.fixtures.company.id) throw new Error("Repository binding belongs to another company");
const discovery = await ctx.input.api.get<{ repositories: Array<{ id: string; fullName: string; connectionIds?: string[] }> }>(`/api/companies/${binding.companyId}/project-repositories`);
const matches = discovery.repositories.filter(row => /^\d+$/.test(row.id) && Number(row.id) === binding.resourceId
&& row.fullName === binding.fixtureValue && row.connectionIds?.includes(binding.connectionId!));
const connection = await ctx.input.api.get<{ id: string; companyId: string; status: string; enabled: boolean; config?: { sourceTemplateKey?: string } }>(`/api/tool-connections/${binding.connectionId}`);
if (matches.length !== 1 || connection.id !== binding.connectionId || connection.companyId !== binding.companyId
|| connection.status !== "active" || !connection.enabled || connection.config?.sourceTemplateKey !== "github") throw new Error("Repository alias differs from active company-scoped provider discovery");
}
}
return rendered;
}