From b72ab4724798ab495a6974f2e9e480e4156ff833 Mon Sep 17 00:00:00 2001 From: Dotta Date: Mon, 12 Oct 2026 16:09:41 -0500 Subject: [PATCH] docs(evals): describe saved Claude onboarding defaults accurately Co-Authored-By: Paperclip --- doc/evals.md | 5 +++-- tests/runner-e2e/README.md | 20 ++++++++++---------- 2 files changed, 13 insertions(+), 12 deletions(-) diff --git a/doc/evals.md b/doc/evals.md index 97b17733496..09ec658a69d 100644 --- a/doc/evals.md +++ b/doc/evals.md @@ -399,8 +399,9 @@ Remaining native chat boundaries are in the explicit-only `agent-chat-qualification` suite: active task reassignment, user Retry after verified worker process loss, and multi-turn answers grounded in actual task records. See the [workflow and qualification limits](../tests/runner-e2e/README.md#remaining-native-agent-chat-qualification). -The 26 native `first-task` cells exercise onboarding before native selection -becomes the UI default. Live results and semantic answer reviews must accompany +The 26 native `first-task` cells inspect the production wizard's saved native +default on qualified targets, without an API runner switch. The 26 Legacy cells +select Legacy explicitly in Advanced. Live results and semantic answer reviews must accompany any qualification claim; catalog presence alone is not a pass. The explicit-only [native question/resume qualification](../tests/runner-e2e/README.md#task-continuation) diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index 6a41b49d07f..df40a460776 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -1086,13 +1086,12 @@ model. Configured and provider-observed model identities are reported separately There are thirteen cases on `legacy-codex`, `legacy-claude`, `runner-codex`, and `runner-acpx-claude`, local only (52 cells). `runner-codex` selects OpenAI in the production wizard, omits the runner override, and verifies the saved native -configuration without changing runners afterward. `legacy-codex` selects -Legacy runner in Advanced before authentication and creation. Both are labeled -`production-wizard`. Claude's creation default is unchanged: `legacy-claude` -retains its wizard configuration, while `runner-acpx-claude` explicitly changes -runtime through the public API before its first task and is labeled -`post-onboarding-runtime-switch`. Persona, managed instructions, skills, seeded -question, and task invocation are preserved. The native path removes the legacy +configuration without changing runners afterward. `runner-acpx-claude` does the +same for Claude on its qualified default targets and inspects the saved ACPX +Claude profile directly. Both Legacy profiles select Legacy runner in Advanced +before authentication and creation. All four paths are labeled +`production-wizard`; there is no post-onboarding runner switch. Persona, +managed instructions, skills, seeded question, and task invocation are preserved. The native path removes the legacy Paperclip operational skill because Runner supplies that contract through its protocol; other assigned skills, including `/first-task`, are retained. Each isolated onboarding company has a $5 task budget. Setup probes have a @@ -1454,9 +1453,10 @@ prompts, onboarding defaults, or provider permissions are changed. For native onboarding qualification, select `first-task` with profiles `runner-codex,runner-acpx-claude` (26 cells). Codex verifies the production -creation default with the experimental flag disabled. Claude still uses the -existing public-API runtime switch before provider work; that case does not -qualify a native Claude creation default. +creation default with the experimental flag disabled. Claude also inspects its +wizard's saved native default with the experimental flag disabled; select that +profile on a qualified default target. Retained results from the former API-switch +definition do not establish that the current creation-default journey passed. Current proof and remaining decisions are recorded in [the 21 September qualification report](QUALIFICATION-2026-09-21.md). In particular,