From 3605d08784ae9e2acc2ab654f79a00bec87e0215 Mon Sep 17 00:00:00 2001 From: Dotta Date: Tue, 27 Oct 2026 12:22:25 -0500 Subject: [PATCH] docs(evals): preserve remote timings and incomplete journey scope --- doc/plans/2026-10-26-chat-catalog-eval-goal.md | 2 +- doc/plans/2026-10-26-chat-catalog-eval-progress.md | 6 ++++-- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/doc/plans/2026-10-26-chat-catalog-eval-goal.md b/doc/plans/2026-10-26-chat-catalog-eval-goal.md index 047cd55a778..37a3ebf497b 100644 --- a/doc/plans/2026-10-26-chat-catalog-eval-goal.md +++ b/doc/plans/2026-10-26-chat-catalog-eval-goal.md @@ -4,7 +4,7 @@ Date: 2026-10-26 ## Authoritative user steering — 2026-10-27 -The existing Evals website and family renderer/history are the primary delivery surface: [Evals hub](https://pages.paperclip.ing/evals/) and [Product E2E history](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/). Retire the separate custom local HTML as the delivery approach; retain it only as historical evidence. The [published historical partial Codex campaign](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/historical-local-codex-chat-catalog-20261027-158measured-86unrun/) contains158 actually measured IDs,17 original initial passes,141 first-attempt failures and159 retained measurements;86 IDs remain unrun in that historical lane. These are original one-time local outcomes, not current-source/OpenCode qualification. Authored238 complete callbacks plus6 executable blocker definitions does not mean those behaviors have been verified. +The existing Evals website and family renderer/history are the primary delivery surface: [Evals hub](https://pages.paperclip.ing/evals/) and [Product E2E history](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/). Retire the separate custom local HTML as the delivery approach; retain it only as historical evidence. The [published historical partial Codex campaign](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/historical-local-codex-chat-catalog-20261027-158measured-86unrun/) contains158 actually measured IDs,17 original initial passes,141 first-attempt failures and159 retained measurements;86 IDs remain unrun in that historical lane. These are original one-time local outcomes, not current-source/OpenCode qualification. There are238 callback implementations without declared gaps and six incomplete or blocked journeys. All244 are registered, but registration and prerequisite probes do not establish complete intended E2E coverage or verified behavior. The [published OpenCode setup result](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-40530790741-1/) records actual framework attempt 40530790741: tasks.create-backlog initial 1 failed setup before the canonical query. 243 remote initials remain; no canonical chat or feature grading completed. The report was verified rendered in the browser using the existing Product history renderer. Automatic setup hello provider usage/cost is unknown, not zero. The same maximum-one coherent correction budget remains. Two earlier host-only qualification operations consumed no case ordinal. diff --git a/doc/plans/2026-10-26-chat-catalog-eval-progress.md b/doc/plans/2026-10-26-chat-catalog-eval-progress.md index e69e3001b77..3a35e773a42 100644 --- a/doc/plans/2026-10-26-chat-catalog-eval-progress.md +++ b/doc/plans/2026-10-26-chat-catalog-eval-progress.md @@ -4,6 +4,8 @@ Updated: 2026-10-27. This is the single progress record for [the saved goal](202 ## Current state +- **First remote feedback-loop timing:**run40530790741 spanned15:25:15–16:18:09UTC (52m54s). Retained workflow facts: authorization queue≈8m03s, lockfile queue6m12s, catalog queue3m44s, AWS build queue16m40s plus build3m38s, paid case queue10m19s plus worker2m34s. These stage durations are not additive causal attribution or model runtime. Actual host/Docker qualified; the first framework attempt then failed agent-readiness setup before the canonical query. + - **Existing Evals website is primary:**[published historical partial Codex campaign](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/historical-local-codex-chat-catalog-20261027-158measured-86unrun/) is live in the [canonical Product history](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/) and was opened/visually verified in the browser. It shows17/158 original initial passes,141 first-attempt failures and159 retained attempts, with86 historical unrun IDs listed in its published summary. It is not new-baseline confidence. The custom localhost page is retired as the delivery approach. Its exact owned preview server PID3634 was rechecked by UID/start/command/loopback listener, terminated and proven absent; historical HTML remains intact (`operations/interim-report-preview-retired.json`). No original result or ledger bytes changed; public projection excludes raw usage/error/transcripts/screenshots. Canonical normalized usage63,126,652tokens includes2,071,424cached tokens omitted from three goal rows in the old61,055,228-token ledger scope; both values are retained, and unpriced costs are not zero. - **New remote measurement:**[published OpenCode setup result](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-40530790741-1/), browser-verified through the existing Product history renderer. Run40530790741 at source0dd8 completed one actual tasks.create-backlog initial as an agent-readiness setup failure before canonical query;243 initials remain,0passes. Host/Docker/source/model/artifact identity is verified; cleanup passed. Heartbeat native runs0, but automatic hello provider calls/tokens/cost remain unknown. Finally grading snapshots do not establish returned feature grading. The maximum-one correction budget is unchanged; this is not reclassified as a host-only operation. - **New user host authorization15:21UTC:**future GitHub Actions execution may use the normal AWS fleet Spot or On-Demand fallback, with actual lifecycle retained and unknown hosts rejected. This supersedes the strict-Spot interpretation only for an explicitly new source/intent. Old617/941 intents and the two pre-evaluation refusals remain unchanged; those operations consumed no initials. The later actual framework attempt consumed one, leaving243. No local execution is permitted. @@ -13,12 +15,12 @@ Updated: 2026-10-27. This is the single progress record for [the saved goal](202 - **Historical detail:**the earlier custom report remains retained evidence, while the existing website above is the user-facing report. Its conservative local stage projection proved105 saved canonical queries,102 exact canonical-native admissions,14 returned feature grades and110 observation-grade snapshots; unproven stages remain unknown. Local reservation span15.539h, union of completed attempt wall intervals4.719h (not model time), summed walls5.879h. Those historical quantities do not describe the new remote lane. - **Frozen inventory:244 scenarios /35areas /191capabilities;196 eligible passes required.** Inventory digest `9dc9cf6a5426cf1b56bc30a3571851f233f9dd5e93a1292eaee414d33b52df77`. Original queries, contexts and outcomes remain unchanged; blocked, partial and unrun cases stay in the denominator. -- **All244 callbacks authored and authoring-resolved** in immutable remaining-initial source `68e8534e3127dec27ae979bbf9f33d8a26422e4b`:238 without declared gaps and6 concrete executable blocker probes,0 absent/unfinished (242 lazy definitions +2 legacy). Grok, named ChatGPT and private GitHub after-grant callbacks are integrated. This is authored coverage, not live acceptance. +- **All244 callbacks registered; six journeys remain incomplete or blocked** in immutable remaining-initial source `68e8534e3127dec27ae979bbf9f33d8a26422e4b`:238 implementations have no declared gaps; six incomplete or blocked journeys retain prerequisite probes (242 lazy definitions +2 legacy). No absent registry entry is implied, but six are not complete intended E2E journeys. Grok, named ChatGPT and private GitHub after-grant callbacks are integrated. This is registered implementation coverage, not complete intended-journey coverage or live acceptance. - **Closed historical local checkpoint:158 scenarios attempted/processed,159 completed measurements,17 original initial passes and142 failed measurements;86 scenarios unrun,0 running.** Historical initial rate is17/244 (6.97%) across the frozen inventory, or17/158 (10.76%) among attempted IDs. Final current-source qualification was not completed; no local pass carries into the remote lane. Actual retained usage is61,055,228 tokens; subscription dollars remain unpriced. The final12 initials completed with all cleanup passed/DBremoved; exact owner/12launchers,12full container IDs and roots are absent, with zero resources under their exact root labels on the unchanged pinned engine. Original volume names were not reconstructed. The user-stop/state/lock histories and all original result/attempt bytes are retained. Receipts:`operations/user-stop-local-admissions.json`, `operations/qualified-inputs-user-stop-exact-absence-68e.json`, `operations/local-campaign-user-stop-closed.json`, `local-campaign-closed-summary.json`. Two historical cleanup records remain unresolved; this is not a claim of newly observed live resources. No further local execution is authorized. - **Single coherent final-B retry set remains held.** Goal-tree exhausted two measurements. All original outcomes, evidence and measured digests remain unchanged. Actual current selected identity review is retained in `closed-pass-identity-review-b65.json`; changed selected execution/grader closures require reviewed revalidation within the same max-two budget. Unrelated Git revisions and unchanged failed behavior do not justify reruns. - **Two-case concurrency qualification completed and manager accepted continued waves of two.** Immutable `c167a54162fadffe665b4d35a7a1308c6233b75d`:companies.archive-company passed and connection-setup.choose-provider failed its actual reply check; each1native/1source human. Worker overlap25.479s; separate roots, ports53206/53205, containers and PostgreSQL system identifiers. Both containers and exact-root-labeled volumes are absent, both roots removed, and coordinator/child PIDs absent. A parent-only duplicate Playwright import prevented the passing completion from initially recording; isolated-process replay confirmed unchanged selected identity and finalized the existing result without rerun/reservation. Original parent error/stop preserved and owner lock retained. Receipt:`tests/runner-e2e/results/chat-catalog-campaign/two-lane-qualification-closed-c167.json`;5 offline replay controls passed. Full repository typecheck/build and final scoped E2E typecheck passed; broad test:run failed/incomplete on host embedded-PostgreSQL bootstrap, with all owned check processes absent. The qualified Docker lane is unchanged. - **The former local continuation is cancelled:**61 source initials,3 installed initials and22 guarded invitation prerequisites remain unrun (86 total). The prepared source/private-input selections remain provenance only and must not launch locally. Future remote execution uses the audited current coherent corrections and new OpenCode/OpenRouter model/host/database identity; it must not reuse obsolete68e merely to finish the old schedule. -- **Known invite-path dependency quarantine is source-qualified for22 of166 remaining IDs.** Exact selected setup call chains (including observerPerson wrappers) are recorded in `remaining166-invite-setup-dependencies-68e.json`;144 independent IDs remain, including3 distinct installed cases. This is a dependency inventory, not processed live setup proof. No safe equivalent public membership setup is already implemented/proven; no fake identities, database grants, scanner weakening or automatic continuation is introduced. Guarded immutable candidate `e515d60fe396ef1ecd45f3a274a46f310bee4d2b` (96b source plus exact metadata overlay) is prepared with238 complete/six concrete blocker definitions, scoped E2E typecheck passed, and22 selected fingerprints. No invitation-lane admission has occurred. Actual guard/probe or any earlier-publication failure must be recorded, with no token-path accept on the affected baseline and no invented zero-native measurement. +- **Known invite-path dependency quarantine is source-qualified for22 of166 remaining IDs.** Exact selected setup call chains (including observerPerson wrappers) are recorded in `remaining166-invite-setup-dependencies-68e.json`;144 independent IDs remain, including3 distinct installed cases. This is a dependency inventory, not processed live setup proof. No safe equivalent public membership setup is already implemented/proven; no fake identities, database grants, scanner weakening or automatic continuation is introduced. Guarded immutable candidate `e515d60fe396ef1ecd45f3a274a46f310bee4d2b` (96b source plus exact metadata overlay) is prepared with238 definitions without declared gaps and six incomplete or blocked journeys, scoped E2E typecheck passed, and22 selected fingerprints. No invitation-lane admission has occurred. Actual guard/probe or any earlier-publication failure must be recorded, with no token-path accept on the affected baseline and no invented zero-native measurement. - **New selected external authority:**the user authorized disposable mail only from the sole QA inbox to itself, and read/isolated clone of the existing designated QA private repository with no repository writes. The external owner independently qualified the actual inbox/account and exact repository/HTTPS clone, then removed its owned clone/helper. New private0700/0600 configuration/reference files are staged separately; actual credential values remain private and selected before capture. Five cases have qualified input fields (outbound-email, unknown-email-outcome, add/replace-repository, prepare-repository). Reply-to-email still lacks a distinct authorized reviewer/sender; private-repository-no-access still lacks the human App/browser authorization. The stable coordinator-only HMAC key commits the selected recipient expectation; credentials, private account/repository metadata and key bytes are not public evidence. Existing read-only discovery and its history remain unchanged. Host IPC cleanup is unnecessary and was not performed. - **Infrastructure is qualified:**owned local Docker PostgreSQL18.6 at the pinned installed image, normal onboarding/migrations/API and final persisted-state scan passed with zero model/native/scenario work; exact resources removed. No further infrastructure health is needed for the unchanged lane. The proposed host SysV cleanup was never performed and is no longer needed. Source/build/native bytes remain bound to production baseline `8373d7867963674ff960d2345d0d28086f799bc7`; no core candidate is used as benchmark proof. - **Separate core PRs remain OPEN/unmerged, auto-merge off.** Doctor [#17847](https://github.com/paperclipai/paperclip/pull/17847), head `a042fb4a28f2e1574972798ed80de2e616daf9fc`, and invite logging [#17848](https://github.com/paperclipai/paperclip/pull/17848), head `98a2432f88c33afacff2e014bd8c30630b26b339`, each have55 successful checks. Budget [#17857](https://github.com/paperclipai/paperclip/pull/17857), head `b6cd49d4b6926fae61a2c68a31ce7c659b69eaa6`, has55 successful/2skipped/1neutral checks and no outstanding checks (manager exact-head checkpoint). Approval routing [#17858](https://github.com/paperclipai/paperclip/pull/17858), head `b60aaf19343389faf67580fb1ff7a561983bb4f5`, has55 successful/2skipped/1neutral checks and no pending checks, Greptile5/5 with0unresolved threads. Connection choice [#17862](https://github.com/paperclipai/paperclip/pull/17862), latest repair head `a76a6120ec2b0767b32b62be4716f69214692688`, has48/48 required workflow checks green at exact head (workflow40497606929), Greptile5/5 and0unresolved threads;13focused controls/server typecheck passed and prior failed checks remain historical evidence. Performance metric [#17872](https://github.com/paperclipai/paperclip/pull/17872), head `3807ee526d2e761f6aa420452b390287542682bd`, has3 regression controls and UI typecheck/token-gates/build passed, with CI/review pending. None changes measured baseline8373. No eval PR or merge is eligible.