diff --git a/doc/plans/2026-10-26-chat-catalog-eval-progress.md b/doc/plans/2026-10-26-chat-catalog-eval-progress.md index 406e307dfcb..611366f2fd9 100644 --- a/doc/plans/2026-10-26-chat-catalog-eval-progress.md +++ b/doc/plans/2026-10-26-chat-catalog-eval-progress.md @@ -6,14 +6,14 @@ Updated: 2026-10-27. This is the single progress record for [the saved goal](202 - **Frozen inventory:244 scenarios /35areas /191capabilities;196 eligible passes required.** Inventory digest `9dc9cf6a5426cf1b56bc30a3571851f233f9dd5e93a1292eaee414d33b52df77`. Original queries, contexts and outcomes remain unchanged; blocked, partial and unrun cases stay in the denominator. - **All244 callbacks authored and authoring-resolved** at closed source `b65cce861b5667f2424bdfacbdfd610fa697ba60`:238 without declared gaps and6 concrete executable blocker probes,0 absent/unfinished (242 lazy definitions +2 legacy). Grok, named ChatGPT and private GitHub after-grant callbacks are integrated. This is authored coverage, not live acceptance. -- **72 scenarios processed /73 completed measurements:10 initial passes and63 failures;0 running at the closed checkpoint;0 final eligible passes.** Actual retained native usage:28,018,825 tokens; subscription dollars remain unpriced. The latest30-case batch finished2passed/28failed; all30 have verified cleanup and exact Docker removal. Closed receipt:`tests/runner-e2e/results/chat-catalog-campaign/closed-initial-checkpoint-72-73.json`. Two older cleanup uncertainties remain unresolved; original measurement bytes are preserved. +- **74 scenarios processed /75 completed measurements:11 initial passes and64 failures;0 running.** Actual retained native usage:28,615,045 tokens; subscription dollars remain unpriced. The concurrent qualification pair produced1pass/1failure with successful independent cleanup. Archive-company's selected identity still matches current source; its pass remains provisional until the final closed-source checkpoint. The other10 initial passes require their already-recorded selected-closure revalidation. Two older cleanup uncertainties remain unresolved; original measurement bytes are preserved. - **Single coherent final-B retry set remains held.** Goal-tree exhausted two measurements. All original outcomes, evidence and measured digests remain unchanged. Actual current selected identity review is retained in `closed-pass-identity-review-b65.json`; changed selected execution/grader closures require reviewed revalidation within the same max-two budget. Unrelated Git revisions and unchanged failed behavior do not justify reruns. -- **The30-case efbb batch completed; a two-case concurrent initial qualification is authorized** from immutable `c167a54162fadffe665b4d35a7a1308c6233b75d`:companies.archive-company and connection-setup.choose-provider, each1native/1source-chat human turn/300s; two one-case children, each maxParallel1/retries0. Exact source/fingerprints, qualified Docker/native,40 archived helper/shared controls plus parent prepare/negative/atomic controls passed. Prior launcher/writer exited and exact last container absence was independently verified. Sole-writer qualifying metadata is prepared before the parent lock; stop after the pair for review. No B rerun is included. +- **Two-case concurrency qualification completed; further admissions held for manager review.** Immutable `c167a54162fadffe665b4d35a7a1308c6233b75d`:companies.archive-company passed and connection-setup.choose-provider failed its actual reply check; each1native/1source human. Worker overlap25.479s; separate roots, ports53206/53205, containers and PostgreSQL system identifiers. Both containers and exact-root-labeled volumes are absent, both roots removed, and coordinator/child PIDs absent. A parent-only duplicate Playwright import prevented the passing completion from initially recording; isolated-process replay confirmed unchanged selected identity and finalized the existing result without rerun/reservation. Original parent error/stop preserved and owner lock retained. Receipt:`tests/runner-e2e/results/chat-catalog-campaign/two-lane-qualification-closed-c167.json`;5 offline replay controls passed. Heavy broad source verification is running separately before further timed initials. - **Infrastructure is qualified:**owned local Docker PostgreSQL18.6 at the pinned installed image, normal onboarding/migrations/API and final persisted-state scan passed with zero model/native/scenario work; exact resources removed. No further infrastructure health is needed for the unchanged lane. The proposed host SysV cleanup was never performed and is no longer needed. Source/build/native bytes remain bound to production baseline `8373d7867963674ff960d2345d0d28086f799bc7`; no core candidate is used as benchmark proof. -- **Separate core PRs remain OPEN/unmerged, auto-merge off.** Doctor [#17847](https://github.com/paperclipai/paperclip/pull/17847), head `a042fb4a28f2e1574972798ed80de2e616daf9fc`, and invite logging [#17848](https://github.com/paperclipai/paperclip/pull/17848), head `98a2432f88c33afacff2e014bd8c30630b26b339`, each have55 successful checks. Budget [#17857](https://github.com/paperclipai/paperclip/pull/17857), head `e5682d22fa8f7c0dd01ddb5e9cc42d941db6fe43`, and approval routing [#17858](https://github.com/paperclipai/paperclip/pull/17858), head `b60aaf19343389faf67580fb1ff7a561983bb4f5`, have CI/review pending at the latest manager checkpoint. None changes measured baseline8373. No eval PR or merge is eligible. +- **Separate core PRs remain OPEN/unmerged, auto-merge off.** Doctor [#17847](https://github.com/paperclipai/paperclip/pull/17847), head `a042fb4a28f2e1574972798ed80de2e616daf9fc`, and invite logging [#17848](https://github.com/paperclipai/paperclip/pull/17848), head `98a2432f88c33afacff2e014bd8c30630b26b339`, each have55 successful checks. Budget [#17857](https://github.com/paperclipai/paperclip/pull/17857), head `b6cd49d4b6926fae61a2c68a31ce7c659b69eaa6`, and approval routing [#17858](https://github.com/paperclipai/paperclip/pull/17858), head `b60aaf19343389faf67580fb1ff7a561983bb4f5`, have CI/review pending at the latest manager checkpoint. None changes measured baseline8373. No eval PR or merge is eligible. - **Genuine missing inputs:**AgentMail/GitHub read-only discovery was authorized and completed; designated self-mail/recipient and repository choices remain pending. Standard provider-key source references exist, but validity and disposable remote/operator targets remain unqualified. No provider send, remote setup or account discovery is implied by authoring coverage. -The efbb coordinator exited0 after30 retained measurements; no old launcher/writer remains. The authoritative ledger is the current manifest; immutable snapshot selection summaries are historical. The remaining initial-only draft covers172 IDs after this batch, including concrete missing-resource probes; its known two alias fixtures are now corrected in151488bf8c before future admission. +The efbb coordinator exited0 after30 retained measurements; no old launcher/writer remains. The authoritative ledger is the current manifest; immutable snapshot selection summaries are historical. The remaining initial-only draft originally covered172 IDs after efbb; the qualification pair consumed two, leaving170 unattempted IDs. It includes concrete missing-resource probes; its known two alias fixtures are corrected in151488bf8c before future admission. ## Current measured continuation and held corrections @@ -25,7 +25,7 @@ Necessary-question's retained observation contains no saved task, document or re ## Measured evidence — all failures retained -All results below use local native Codex subscription profile `runner-codex-subscription`, model `gpt-5.6-sol`, max parallelism one and zero automatic retries. Retained paths are under ignored `tests/runner-e2e/results/`. The launcher numbers each fresh invocation `attempt-1`; the ledger separately preserves governed campaign ordinals. +All results below use local native Codex subscription profile `runner-codex-subscription`, model `gpt-5.6-sol`, one case per launcher and zero automatic retries. The first72 scenarios were sequential; the separately recorded qualification pair used two concurrent isolated launchers. Retained paths are under ignored `tests/runner-e2e/results/`. The launcher numbers each fresh invocation `attempt-1`; the ledger separately preserves governed campaign ordinals. | Scenario / campaign ordinal | Source | Retained campaign | Outcome and supported cause | Cleanup / accounted tokens | | --- | --- | --- | --- | --- | diff --git a/tests/runner-e2e/chat-campaign-manifest.json b/tests/runner-e2e/chat-campaign-manifest.json index f8ac170d0e2..f440775a4a3 100644 --- a/tests/runner-e2e/chat-campaign-manifest.json +++ b/tests/runner-e2e/chat-campaign-manifest.json @@ -3075,7 +3075,61 @@ "readiness": "implemented", "executionId": "chat-features.runner-codex-subscription.local.companies.archive-company", "blocker": null, - "attempts": [], + "attempts": [ + { + "ordinal": 1, + "outcome": "passed", + "executionId": "chat-features.runner-codex-subscription.local.companies.archive-company", + "startedAt": "2026-10-27T10:29:19.521Z", + "finishedAt": "2026-10-27T10:31:23.533Z", + "sourceRevision": "c167a54162fadffe665b4d35a7a1308c6233b75d", + "workingTreeDigest": "6251a7763d81986af3d43a142a52c702da1bde2b57cee5aef2606e4418ccc698", + "definitionDigest": "e45461119cd11cbc7d9f3659a35bbd7a6c1ac72d5c5b2e4f187f97210ac8c7c7", + "suiteDefinitionDigest": "c0a8d311ea4dc7616bc8cef7a815457e5c013bd491d2ac6510f62500d3d0ac25", + "graderDigest": "e45461119cd11cbc7d9f3659a35bbd7a6c1ac72d5c5b2e4f187f97210ac8c7c7", + "model": "gpt-5.6-sol", + "profile": "runner-codex-subscription", + "environment": "local", + "automaticRetries": 0, + "correction": null, + "classification": "provisional_initial_pass_pending_final_closure", + "observedFailure": null, + "evidencePaths": [ + "tests/runner-e2e/results/chat-catalog-campaign/two-lane-initial-pair-c167a54162.json", + "tests/runner-e2e/results/two-lane-initial-c167a54162fa-b95423a10d79997385db/chat-features/runner-codex-subscription/local/companies.archive-company/attempt-1/result.json", + "tests/runner-e2e/results/chat-catalog-campaign/active-companies.archive-company.json", + "tests/runner-e2e/results/chat-catalog-campaign/initial-pass-identity-review-companies.archive-company.json", + "tests/runner-e2e/results/chat-catalog-campaign/operations/two-lane-exact-absence-c167.json", + "tests/runner-e2e/results/chat-catalog-campaign/operations/two-lane-archive-replayed-completion.json" + ], + "usage": { + "costUsd": null, + "tokens": 541825, + "coverage": "Actual heartbeat usage receipts, including cache; subscription cost unpriced unless complete billing; fixture costs excluded." + }, + "cleanup": "passed", + "corePrDependencies": [], + "databaseLane": { + "lane": "owned-docker-postgres", + "image": "postgres@sha256:74935e72241653ca55e0414067e6d8763aceb8a810eb51b452253ec3dcfc4336", + "imageId": "sha256:74935e72241653ca55e0414067e6d8763aceb8a810eb51b452253ec3dcfc4336", + "version": "18.6-1.pgdg13+2" + }, + "productIdentity": { + "kind": "source-build", + "productionSourceRevision": "8373d7867963674ff960d2345d0d28086f799bc7", + "snapshotSourceRevision": "c167a54162fadffe665b4d35a7a1308c6233b75d", + "nativeBinarySha256": "9d8af6b85b836a03092ade2273be6f29f5c2f6ec67ee891d83ee9693ba39d3b4", + "workspaceBuildDigest": "222e10a8a2a20d3015ec4470b952e2dd9b927dfe9390c4a5ed1e19d740c0e35b", + "receiptPaths": [ + "tests/runner-e2e/results/chat-catalog-campaign/snapshot-two-lane-c167a54162.json", + "tests/runner-e2e/results/chat-catalog-campaign/snapshot-10d0ce63610.json", + "tests/runner-e2e/results/chat-catalog-campaign/snapshot-0fdcb539a8.json" + ], + "coreCandidate": false + } + } + ], "qualifyingDefinitionDigest": "e45461119cd11cbc7d9f3659a35bbd7a6c1ac72d5c5b2e4f187f97210ac8c7c7", "qualifyingGraderDigest": "e45461119cd11cbc7d9f3659a35bbd7a6c1ac72d5c5b2e4f187f97210ac8c7c7", "productPolicy": { @@ -3405,7 +3459,58 @@ "readiness": "implemented", "executionId": "chat-features.runner-codex-subscription.local.connection-setup.choose-provider", "blocker": null, - "attempts": [], + "attempts": [ + { + "ordinal": 1, + "outcome": "failed", + "executionId": "chat-features.runner-codex-subscription.local.connection-setup.choose-provider", + "startedAt": "2026-10-27T10:29:19.551Z", + "finishedAt": "2026-10-27T10:30:11.290Z", + "sourceRevision": "c167a54162fadffe665b4d35a7a1308c6233b75d", + "workingTreeDigest": "c7fd9a3a9f33cdc907f3e8c61d5f40eeb107cc8f856e01552745cd438e4e27d7", + "definitionDigest": "2650b5969af7a28ce555ba5ec92dfb6cf3eb3c21dfce0b8ab99be9c41e2ff772", + "suiteDefinitionDigest": "c0a8d311ea4dc7616bc8cef7a815457e5c013bd491d2ac6510f62500d3d0ac25", + "graderDigest": "2650b5969af7a28ce555ba5ec92dfb6cf3eb3c21dfce0b8ab99be9c41e2ff772", + "model": "gpt-5.6-sol", + "profile": "runner-codex-subscription", + "environment": "local", + "automaticRetries": 0, + "correction": null, + "classification": "candidate_failure", + "observedFailure": "Actual reply does not explain the advertised Notion choices and unavailable prerequisite", + "evidencePaths": [ + "tests/runner-e2e/results/chat-catalog-campaign/two-lane-initial-pair-c167a54162.json", + "tests/runner-e2e/results/two-lane-initial-c167a54162fa-41b15d773be40166eeb1/chat-features/runner-codex-subscription/local/connection-setup.choose-provider/attempt-1/result.json", + "tests/runner-e2e/results/chat-catalog-campaign/active-connection-setup.choose-provider.json" + ], + "usage": { + "costUsd": null, + "tokens": 54395, + "coverage": "Actual heartbeat usage receipts, including cache; subscription cost unpriced unless complete billing; fixture costs excluded." + }, + "cleanup": "passed", + "corePrDependencies": [], + "databaseLane": { + "lane": "owned-docker-postgres", + "image": "postgres@sha256:74935e72241653ca55e0414067e6d8763aceb8a810eb51b452253ec3dcfc4336", + "imageId": "sha256:74935e72241653ca55e0414067e6d8763aceb8a810eb51b452253ec3dcfc4336", + "version": "18.6-1.pgdg13+2" + }, + "productIdentity": { + "kind": "source-build", + "productionSourceRevision": "8373d7867963674ff960d2345d0d28086f799bc7", + "snapshotSourceRevision": "c167a54162fadffe665b4d35a7a1308c6233b75d", + "nativeBinarySha256": "9d8af6b85b836a03092ade2273be6f29f5c2f6ec67ee891d83ee9693ba39d3b4", + "workspaceBuildDigest": "222e10a8a2a20d3015ec4470b952e2dd9b927dfe9390c4a5ed1e19d740c0e35b", + "receiptPaths": [ + "tests/runner-e2e/results/chat-catalog-campaign/snapshot-two-lane-c167a54162.json", + "tests/runner-e2e/results/chat-catalog-campaign/snapshot-10d0ce63610.json", + "tests/runner-e2e/results/chat-catalog-campaign/snapshot-0fdcb539a8.json" + ], + "coreCandidate": false + } + } + ], "qualifyingDefinitionDigest": "2650b5969af7a28ce555ba5ec92dfb6cf3eb3c21dfce0b8ab99be9c41e2ff772", "qualifyingGraderDigest": "2650b5969af7a28ce555ba5ec92dfb6cf3eb3c21dfce0b8ab99be9c41e2ff772", "productPolicy": {