docs(evals): describe saved Claude onboarding defaults accurately

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip committed 2026-10-12 16:09:41 -05:00
1 parent d761bdefca
commit b72ab47247
2 files changed
+13 -12

No files matched your search

+3 -2
View File
@@ -399,8 +399,9 @@ Remaining native chat boundaries are in the explicit-only
`agent-chat-qualification` suite: active task reassignment, user Retry after
verified worker process loss, and multi-turn answers grounded in actual task
records. See the [workflow and qualification limits](../tests/runner-e2e/README.md#remaining-native-agent-chat-qualification).
The 26 native `first-task` cells exercise onboarding before native selection
becomes the UI default. Live results and semantic answer reviews must accompany
The 26 native `first-task` cells inspect the production wizard's saved native
default on qualified targets, without an API runner switch. The 26 Legacy cells
select Legacy explicitly in Advanced. Live results and semantic answer reviews must accompany
any qualification claim; catalog presence alone is not a pass.
The explicit-only [native question/resume qualification](../tests/runner-e2e/README.md#task-continuation)
+10 -10
View File
@@ -1086,13 +1086,12 @@ model. Configured and provider-observed model identities are reported separately
There are thirteen cases on `legacy-codex`, `legacy-claude`, `runner-codex`, and
`runner-acpx-claude`, local only (52 cells). `runner-codex` selects OpenAI in the
production wizard, omits the runner override, and verifies the saved native
configuration without changing runners afterward. `legacy-codex` selects
Legacy runner in Advanced before authentication and creation. Both are labeled
`production-wizard`. Claude's creation default is unchanged: `legacy-claude`
retains its wizard configuration, while `runner-acpx-claude` explicitly changes
runtime through the public API before its first task and is labeled
`post-onboarding-runtime-switch`. Persona, managed instructions, skills, seeded
question, and task invocation are preserved. The native path removes the legacy
configuration without changing runners afterward. `runner-acpx-claude` does the
same for Claude on its qualified default targets and inspects the saved ACPX
Claude profile directly. Both Legacy profiles select Legacy runner in Advanced
before authentication and creation. All four paths are labeled
`production-wizard`; there is no post-onboarding runner switch. Persona,
managed instructions, skills, seeded question, and task invocation are preserved. The native path removes the legacy
Paperclip operational skill because Runner supplies that contract through its
protocol; other assigned skills, including `/first-task`, are retained.
Each isolated onboarding company has a $5 task budget. Setup probes have a
@@ -1454,9 +1453,10 @@ prompts, onboarding defaults, or provider permissions are changed.
For native onboarding qualification, select `first-task` with profiles
`runner-codex,runner-acpx-claude` (26 cells). Codex verifies the production
creation default with the experimental flag disabled. Claude still uses the
existing public-API runtime switch before provider work; that case does not
qualify a native Claude creation default.
creation default with the experimental flag disabled. Claude also inspects its
wizard's saved native default with the experimental flag disabled; select that
profile on a qualified default target. Retained results from the former API-switch
definition do not establish that the current creation-default journey passed.
Current proof and remaining decisions are recorded in
[the 21 September qualification report](QUALIFICATION-2026-09-21.md). In particular,