feat: default new Codex agents to the qualified Paperclip Runner

Resolve creation intent across onboarding, hires, imports, and the UI. Preserve existing execution and qualify automatic defaults for shipped Linux x64 artifacts.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip committed 2026-10-09 10:12:15 -05:00
1 parent 835a022936
commit aa879604ae
91 files changed
+2718 -884

No files matched your search

+24 -22
View File
@@ -139,8 +139,9 @@ Paperclip task, run an agent, or replace a Product E2E result.
`--suite confirmation-replies` selects ten local native Claude/Codex cells:
conversational single-task approval, saved-plan approval, rejection, card-click
acceptance as a control, and ambiguous approval with two independent pending
proposals. The onboarding cells use the production wizard, runtime switch and
persona. The ambiguity fixture creates ordinary board cards through the public
proposals. The onboarding cells use the production wizard and persona. Codex
retains its automatic native selection; Claude uses an explicit runtime switch.
The ambiguity fixture creates ordinary board cards through the public
API, sends "Yes, go ahead" through the browser, requires both to stay pending
with a clarification reply and no execution, then approves only one and rejects
the other through separate browser messages.
@@ -184,8 +185,9 @@ pnpm test:e2e:runner -- --suite confirmation-replies
and native Claude, each with onboarding, idle handoff, busy handoff, two-task
handoff, and restart recovery. These exercise the production completion-delivery
path and agent-authored responses. There is no separate completion feature flag.
The onboarding cell reuses the real wizard and its existing pre-execution native
runtime switch, retaining the production persona.
The onboarding cell reuses the real wizard and retains the production persona.
Codex uses its automatic native default; Claude explicitly changes runtime
before execution.
The idle chat cell asks the agent to delegate one welcome note to a named worker and
report its result without another user message. A bounded local file read in
@@ -1082,18 +1084,19 @@ default is retained**, even when it differs from that profile's normal harness
model. Configured and provider-observed model identities are reported separately.
There are thirteen cases on `legacy-codex`, `legacy-claude`, `runner-codex`, and
`runner-acpx-claude`, local only (52 cells). Native profiles complete the same
production wizard using their legacy provider, then change only the agent's
runtime configuration via the public API before its first task. The wizard does
not currently offer native Runner. Persona, managed instructions, skills, seeded
question, and task invocation are preserved. Explicit model choices are retained;
an unset model resolves through the production runtime-switch defaults. The
production switch removes the legacy Paperclip operational skill because Runner
supplies its control-plane contract through its protocol; other assigned skills,
including `/first-task`, are retained. Native runtime permissions come from the
existing qualified profile. Evidence labels
this setup `post-onboarding-runtime-switch`; it does not claim a native wizard
path exists. Legacy setup is labeled `production-wizard`.
`runner-acpx-claude`, local only (52 cells). `runner-codex` selects OpenAI in the
production wizard, omits the runner override, and verifies the saved native
configuration without changing runners afterward. `legacy-codex` selects
Legacy runner in Advanced before authentication and creation. Both are labeled
`production-wizard`. Claude's creation default is unchanged: `legacy-claude`
retains its wizard configuration, while `runner-acpx-claude` explicitly changes
runtime through the public API before its first task and is labeled
`post-onboarding-runtime-switch`. Persona, managed instructions, skills, seeded
question, and task invocation are preserved. The native path removes the legacy
Paperclip operational skill because Runner supplies that contract through its
protocol; other assigned skills, including `/first-task`, are retained.
Each isolated onboarding company has a $5 task budget. Setup probes have a
separate campaign reservation because they do not create task cost events.
| First response / control | Complete journey |
| --- | --- |
@@ -1449,12 +1452,11 @@ released on failure and normal isolated-instance cleanup removes the workspace.
The usual provider billing and partial-attempt reporting apply. No production
prompts, onboarding defaults, or provider permissions are changed.
For pre-default native onboarding qualification, select all `first-task` cases
with profiles `runner-codex,runner-acpx-claude` (26 cells). The existing public-API
runtime switch occurs after the real wizard creates its first agent and before
any provider work. It preserves the wizard's model, persona, skills, and task.
This tests the native first-task process in advance of the UI/default rollout;
it does not certify a native option in the wizard, which is not offered yet.
For native onboarding qualification, select `first-task` with profiles
`runner-codex,runner-acpx-claude` (26 cells). Codex verifies the production
creation default with the experimental flag disabled. Claude still uses the
existing public-API runtime switch before provider work; that case does not
qualify a native Claude creation default.
Current proof and remaining decisions are recorded in
[the 21 September qualification report](QUALIFICATION-2026-09-21.md). In particular,
+4 -1
View File
@@ -99,6 +99,9 @@ function commonAgent(
capabilities:
"Completes deterministic standard, planning, and ask-mode runner acceptance tasks.",
adapterType,
// These are explicit legacy regression controls. Ordinary Codex creation
// defaults are exercised through the production onboarding wizard below.
...(adapterType === "codex_local" ? { runner: "legacy" as const } : {}),
adapterConfig,
defaultEnvironmentId: input.environmentId,
budgetMonthlyCents: 0,
@@ -1427,7 +1430,7 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
groups: ["onboarding"],
profiles: runnerProfiles.filter(profile => ["legacy-codex", "legacy-claude", "runner-codex", "runner-acpx-claude"].includes(profile.id)),
environments: [localEnvironment], tasks: firstTaskTasks, expectedMatrixSize: 52,
definitionMetadata: { version: 4, runGrading: "evidenced-nonexecution-and-refusal", credentialPersistenceCheck: false, questionChoiceMinimum: 2, nativeSetup: "post-onboarding-runtime-switch", productionInstructions: true, qualityGrading: "informational" },
definitionMetadata: { version: 5, runGrading: "evidenced-nonexecution-and-refusal", credentialPersistenceCheck: false, questionChoiceMinimum: 2, nativeSetup: "codex-production-default-claude-explicit-runtime-switch", productionInstructions: true, qualityGrading: "informational" },
},
{
id: "agent-chat", label: "Persistent Agent Chat",
+6 -4
View File
@@ -3,10 +3,10 @@ import type { RunnerApi } from "./api.js";
import type { LiveFixtureValues } from "./live-fixtures.js";
import type { CredentialName, MatrixExecution } from "./types.js";
/** Provision only credentials for the UI-created company. The production wizard
/** Provision credentials and a bounded budget for the UI-created company. The production wizard
* remains the sole creator/configurer of its first agent and onboarding task. */
export async function provisionFirstTaskFixtures(input: {
api: Pick<RunnerApi, "get" | "postSensitive">;
api: Pick<RunnerApi, "get" | "patch" | "postSensitive">;
execution: MatrixExecution;
nonce: string;
company: LiveFixtureValues["company"];
@@ -30,6 +30,7 @@ export async function provisionFirstTaskFixtures(input: {
const credential = execution.profile.credential;
const value = credentials[credential];
if (!value) throw new Error(`Missing credential ${credential}`);
await api.patch(`/api/companies/${company.id}`, { budgetMonthlyCents: 500 });
const environments = await api.get<Array<LiveFixtureValues["environment"]>>(
`/api/companies/${company.id}/environments?driver=local`,
);
@@ -59,8 +60,9 @@ export async function provisionFirstTaskFixtures(input: {
};
}
/** Reuse the qualified runtime configuration only, never the QA persona from
* buildAgent. Native onboarding is not offered by the production wizard yet. */
/** Reuse the qualified runtime configuration for explicit regression switches,
* never the QA persona from buildAgent. Codex default qualification does not
* call this helper; its wizard must persist the native selection directly. */
export function firstTaskNativeRuntimePatch(
execution: MatrixExecution,
fixtures: LiveFixtureValues,
+13 -2
View File
@@ -93,6 +93,11 @@ export async function setupFirstTaskFixtures(input: {
exact: true,
});
const savedKey = page.getByRole("combobox", { name: "Saved API key" });
if (execution.profile.id === "legacy-codex") {
await page.getByText("Advanced", { exact: true }).click();
await page.getByRole("button", { name: "Runner", exact: true }).click();
await page.getByRole("option", { name: "Legacy runner", exact: true }).click();
}
// Credential mode depends on the selected provider and its asynchronous key lookup.
await expect(savedKey.or(useKey).first()).toBeVisible();
if (await useKey.isVisible()) await useKey.click();
@@ -110,16 +115,22 @@ export async function setupFirstTaskFixtures(input: {
expect(agents).toHaveLength(1);
const wizardAdapter =
execution.profile.credential === "OPENAI_API_KEY"
? "codex_local"
? execution.profile.id === "runner-codex" ? "paperclip_runner" : "codex_local"
: "claude_local";
expect(agents[0].adapterType).toBe(wizardAdapter);
if (execution.profile.id === "runner-codex") {
expect(agents[0].adapterConfig?.provider).toBe("codex");
}
fixtures.onboardingRuntime = {
mode: "production-wizard",
originalAdapterType: wizardAdapter,
testedAdapterType: wizardAdapter,
originalModel: agents[0].adapterConfig?.model ?? null,
};
if (execution.profile.generation === "native") {
// Claude remains an explicit native regression path in this Codex-only
// rollout. Codex is never switched after onboarding: the saved default is
// the execution under test.
if (execution.profile.id === "runner-acpx-claude") {
const runtimePatch = firstTaskNativeRuntimePatch(
execution,
fixtures,
+6 -2
View File
@@ -695,9 +695,10 @@ Accept the card above and I write it. This task stays in review until then.`;
(e) => e.suite.id === "first-task" && e.profile.id === "legacy-codex",
)!;
const get = vi.fn().mockResolvedValue([{ id: "local", driver: "local" }]);
const patch = vi.fn().mockResolvedValue({});
const postSensitive = vi.fn().mockResolvedValue({ id: "encrypted-secret" });
const fixtures = await provisionFirstTaskFixtures({
api: { get, postSensitive },
api: { get, patch, postSensitive },
execution,
nonce: "test",
company: {
@@ -710,6 +711,9 @@ Accept the card above and I write it. This task stays in review until then.`;
expect(get).toHaveBeenCalledExactlyOnceWith(
"/api/companies/ui-created-company/environments?driver=local",
);
expect(patch).toHaveBeenCalledExactlyOnceWith(
"/api/companies/ui-created-company", { budgetMonthlyCents: 500 },
);
expect(postSensitive).toHaveBeenCalledExactlyOnceWith(
"/api/companies/ui-created-company/secrets",
expect.objectContaining({ key: "OPENAI_API_KEY", value: "fixture-key" }),
@@ -723,7 +727,7 @@ Accept the card above and I write it. This task stays in review until then.`;
expect(JSON.stringify(fixtures)).not.toContain("fixture-key");
await expect(
provisionFirstTaskFixtures({
api: { get, postSensitive },
api: { get, patch, postSensitive },
execution,
nonce: "test",
company: fixtures.company,
+4 -2
View File
@@ -870,10 +870,12 @@ for (const execution of executions) {
const receipt = verifyStockHarnessPreflight(process.env[STOCK_PREFLIGHT_ENV]);
await writeSanitizedJson(snapshotsDir, "stock-harness-preflight.json", receipt, secrets);
}
// Codex default journeys must work with the experimental gate disabled.
const nativeRolloutEnabled = execution.profile.provider !== "codex";
const experimental = await api.patch<{
enableNativeRunner: boolean;
}>("/api/instance/settings/experimental", {
enableNativeRunner: true,
enableNativeRunner: nativeRolloutEnabled,
...(["warm_three_turn", "everyday_workflow"].includes(execution.task.flow)
? { enableIsolatedWorkspaces: true }
: {}),
@@ -882,7 +884,7 @@ for (const execution of executions) {
? { enableRunnerPreviewIngress: true }
: {}),
});
expect(experimental.enableNativeRunner).toBe(true);
expect(experimental.enableNativeRunner).toBe(nativeRolloutEnabled);
if (execution.suite.id === "extended-harnesses" && execution.profile.qualificationCandidate === "pi" && execution.task.id === "file-edit-validate") {
piFileSeed = await seedPiFile(workspacePath, nonce);