diff --git a/doc/plans/2026-10-26-chat-catalog-eval-progress.md b/doc/plans/2026-10-26-chat-catalog-eval-progress.md index 05ed4050751..d3055183085 100644 --- a/doc/plans/2026-10-26-chat-catalog-eval-progress.md +++ b/doc/plans/2026-10-26-chat-catalog-eval-progress.md @@ -6,10 +6,10 @@ Updated: 2026-10-27. This is the single progress record for [the saved goal](202 - **Frozen inventory:244 scenarios /35areas /191capabilities;196 eligible passes required.** Inventory digest `9dc9cf6a5426cf1b56bc30a3571851f233f9dd5e93a1292eaee414d33b52df77`. Original queries, contexts and outcomes remain unchanged; blocked, partial and unrun cases stay in the denominator. - **All244 callbacks authored and authoring-resolved** in immutable remaining-initial source `68e8534e3127dec27ae979bbf9f33d8a26422e4b`:238 without declared gaps and6 concrete executable blocker probes,0 absent/unfinished (242 lazy definitions +2 legacy). Grok, named ChatGPT and private GitHub after-grant callbacks are integrated. This is authored coverage, not live acceptance. -- **Last closed checkpoint:90 scenarios processed /91 completed measurements,13 initial passes and78 failures,0 running.** Actual retained native usage:34,517,843 tokens; subscription dollars remain unpriced. The independent68e parent stopped after12 initials (1pass/11failures) on an actual exact-secret report match during device-login; HTML index and blob ZIP were omitted. The matched root is unknown in retained metadata; source contains generated device/user-code and exchanged-token registration plus recorded device-consent browser actions. This is not a cap-only omission. All12 results report cleanup passed/DBremoved; the final exact container is absent, private root gone, and pinned-engine exact-root label/process queries are empty. Original exact volume name was not retained outside the removed private receipt, so absence is reported by exact root labels rather than a reconstructed name. Final retained manifest:29files,0leaks,0missing; no fresh raw-root rescan claimed. Parent lock remains retained and admissions halted. Remaining132 selection (129source+3installed) is prepared from unchanged68e;22 invite-dependent IDs remain quarantined/unprocessed. Original outcomes and two older historical cleanup uncertainties remain unchanged. Closed receipt:`tests/runner-e2e/results/chat-catalog-campaign/operations/device-login-privacy-stop-68e.json`. Initial passes remain provisional pending actual final selected-identity checks; no unrelated Git SHA alone requires revalidation. +- **Current completed checkpoint:110 scenarios processed /111 completed measurements,13 initial passes and98 failures; gated-lab and interaction-fixture are active.** Actual retained native usage:41,972,223 tokens; subscription dollars remain unpriced. Earlier, the independent68e parent stopped at90processed/91measurements after12 initials (1pass/11failures) on an actual exact-secret report match during device-login; HTML index and blob ZIP were omitted. The matched root is unknown in retained metadata; source contains generated device/user-code and exchanged-token registration plus recorded device-consent browser actions. This is not a cap-only omission. All12 results report cleanup passed/DBremoved; the final exact container is absent, private root gone, and pinned-engine exact-root label/process queries are empty. Original exact volume name was not retained outside the removed private receipt, so absence is reported by exact root labels rather than a reconstructed name. Final retained manifest:29files,0leaks,0missing; no fresh raw-root rescan claimed. Original stopped-parent lock is preserved by hash. Manager authorized the remaining132 continuation (129source+3installed) from unchanged68e after closed reconciliation; actual first wave began11:49:48Z. The current sole writer/ledger records live state;22 invite-dependent IDs remain quarantined/unprocessed. Original outcomes and two older historical cleanup uncertainties remain unchanged. Closed receipt:`tests/runner-e2e/results/chat-catalog-campaign/operations/device-login-privacy-stop-68e.json`. Initial passes remain provisional pending actual final selected-identity checks; no unrelated Git SHA alone requires revalidation. Current continuation has20 completed failures, all cleanup passed/DBremoved; no privacy/infrastructure stop. Receipt:`tests/runner-e2e/results/chat-catalog-campaign/initial-checkpoint-110-111.json`. Processed setup attempts are not claimed as completed canonical chat journeys; those stage counts will be derived separately from retained dispatch/result evidence. - **Single coherent final-B retry set remains held.** Goal-tree exhausted two measurements. All original outcomes, evidence and measured digests remain unchanged. Actual current selected identity review is retained in `closed-pass-identity-review-b65.json`; changed selected execution/grader closures require reviewed revalidation within the same max-two budget. Unrelated Git revisions and unchanged failed behavior do not justify reruns. - **Two-case concurrency qualification completed and manager accepted continued waves of two.** Immutable `c167a54162fadffe665b4d35a7a1308c6233b75d`:companies.archive-company passed and connection-setup.choose-provider failed its actual reply check; each1native/1source human. Worker overlap25.479s; separate roots, ports53206/53205, containers and PostgreSQL system identifiers. Both containers and exact-root-labeled volumes are absent, both roots removed, and coordinator/child PIDs absent. A parent-only duplicate Playwright import prevented the passing completion from initially recording; isolated-process replay confirmed unchanged selected identity and finalized the existing result without rerun/reservation. Original parent error/stop preserved and owner lock retained. Receipt:`tests/runner-e2e/results/chat-catalog-campaign/two-lane-qualification-closed-c167.json`;5 offline replay controls passed. Full repository typecheck/build and final scoped E2E typecheck passed; broad test:run failed/incomplete on host embedded-PostgreSQL bootstrap, with all owned check processes absent. The qualified Docker lane is unchanged. -- **The independent144 continuation is now halted after12 actual initials.** Exact whitelist `independent-initial-selection-68e8534e31.json` preserved definitions/bounds and excluded22 unsafe enrollment IDs. First wave started11:17:56Z; device-login's real600s expiry was exercised without mocked time or deadline change. No consumed case is replayed. Prepared-only `remaining132-initial-selection-68e8534e31.json` excludes all consumed12 and retains129source/3installed identities byte-for-byte. Manager continuation remains required after this privacy stop. Original ownership/stop/lock history is preserved; no scanner weakening or B rerun. +- **The independent144 continuation preserved an actual privacy stop after12 initials; the remaining132 are now authorized/running.** Exact whitelist `independent-initial-selection-68e8534e31.json` preserved definitions/bounds and excluded22 unsafe enrollment IDs. First wave started11:17:56Z; device-login's real600s expiry was exercised without mocked time or deadline change. No consumed case is replayed. Prepared-only `remaining132-initial-selection-68e8534e31.json` excludes all consumed12 and retains129source/3installed identities byte-for-byte. Manager approved continuation after exact ownership/resource/evidence reconciliation; new parent uses distinct remaining132 state and never rewrites the old stop. Original ownership/stop/lock history is preserved; no scanner weakening or B rerun. - **Known invite-path dependency quarantine is source-qualified for22 of166 remaining IDs.** Exact selected setup call chains (including observerPerson wrappers) are recorded in `remaining166-invite-setup-dependencies-68e.json`;144 independent IDs remain, including3 distinct installed cases. This is a dependency inventory, not processed live setup proof. No safe equivalent public membership setup is already implemented/proven; no fake identities, database grants, scanner weakening or automatic continuation is introduced. - **Infrastructure is qualified:**owned local Docker PostgreSQL18.6 at the pinned installed image, normal onboarding/migrations/API and final persisted-state scan passed with zero model/native/scenario work; exact resources removed. No further infrastructure health is needed for the unchanged lane. The proposed host SysV cleanup was never performed and is no longer needed. Source/build/native bytes remain bound to production baseline `8373d7867963674ff960d2345d0d28086f799bc7`; no core candidate is used as benchmark proof. - **Separate core PRs remain OPEN/unmerged, auto-merge off.** Doctor [#17847](https://github.com/paperclipai/paperclip/pull/17847), head `a042fb4a28f2e1574972798ed80de2e616daf9fc`, and invite logging [#17848](https://github.com/paperclipai/paperclip/pull/17848), head `98a2432f88c33afacff2e014bd8c30630b26b339`, each have55 successful checks. Budget [#17857](https://github.com/paperclipai/paperclip/pull/17857), head `b6cd49d4b6926fae61a2c68a31ce7c659b69eaa6`, has55 successful/2skipped/1neutral checks and no outstanding checks (manager exact-head checkpoint). Approval routing [#17858](https://github.com/paperclipai/paperclip/pull/17858), head `b60aaf19343389faf67580fb1ff7a561983bb4f5`, has55 successful/2skipped/1neutral checks and no pending checks, Greptile5/5 with0unresolved threads. Connection choice [#17862](https://github.com/paperclipai/paperclip/pull/17862), latest repair head `a76a6120ec2b0767b32b62be4716f69214692688`, has fresh CI/review pending after13focused controls/server typecheck; prior failed checks remain historical evidence. None changes measured baseline8373. No eval PR or merge is eligible.