mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 05:11:49 +01:00
feat: default new Codex agents to the qualified Paperclip Runner
Resolve creation intent across onboarding, hires, imports, and the UI. Preserve existing execution and qualify automatic defaults for shipped Linux x64 artifacts. Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
835a022936
commit
aa879604ae
91 files changed
+2718
-884
No files matched your search
+24
-22
@@ -139,8 +139,9 @@ Paperclip task, run an agent, or replace a Product E2E result.
|
||||
`--suite confirmation-replies` selects ten local native Claude/Codex cells:
|
||||
conversational single-task approval, saved-plan approval, rejection, card-click
|
||||
acceptance as a control, and ambiguous approval with two independent pending
|
||||
proposals. The onboarding cells use the production wizard, runtime switch and
|
||||
persona. The ambiguity fixture creates ordinary board cards through the public
|
||||
proposals. The onboarding cells use the production wizard and persona. Codex
|
||||
retains its automatic native selection; Claude uses an explicit runtime switch.
|
||||
The ambiguity fixture creates ordinary board cards through the public
|
||||
API, sends "Yes, go ahead" through the browser, requires both to stay pending
|
||||
with a clarification reply and no execution, then approves only one and rejects
|
||||
the other through separate browser messages.
|
||||
@@ -184,8 +185,9 @@ pnpm test:e2e:runner -- --suite confirmation-replies
|
||||
and native Claude, each with onboarding, idle handoff, busy handoff, two-task
|
||||
handoff, and restart recovery. These exercise the production completion-delivery
|
||||
path and agent-authored responses. There is no separate completion feature flag.
|
||||
The onboarding cell reuses the real wizard and its existing pre-execution native
|
||||
runtime switch, retaining the production persona.
|
||||
The onboarding cell reuses the real wizard and retains the production persona.
|
||||
Codex uses its automatic native default; Claude explicitly changes runtime
|
||||
before execution.
|
||||
|
||||
The idle chat cell asks the agent to delegate one welcome note to a named worker and
|
||||
report its result without another user message. A bounded local file read in
|
||||
@@ -1082,18 +1084,19 @@ default is retained**, even when it differs from that profile's normal harness
|
||||
model. Configured and provider-observed model identities are reported separately.
|
||||
|
||||
There are thirteen cases on `legacy-codex`, `legacy-claude`, `runner-codex`, and
|
||||
`runner-acpx-claude`, local only (52 cells). Native profiles complete the same
|
||||
production wizard using their legacy provider, then change only the agent's
|
||||
runtime configuration via the public API before its first task. The wizard does
|
||||
not currently offer native Runner. Persona, managed instructions, skills, seeded
|
||||
question, and task invocation are preserved. Explicit model choices are retained;
|
||||
an unset model resolves through the production runtime-switch defaults. The
|
||||
production switch removes the legacy Paperclip operational skill because Runner
|
||||
supplies its control-plane contract through its protocol; other assigned skills,
|
||||
including `/first-task`, are retained. Native runtime permissions come from the
|
||||
existing qualified profile. Evidence labels
|
||||
this setup `post-onboarding-runtime-switch`; it does not claim a native wizard
|
||||
path exists. Legacy setup is labeled `production-wizard`.
|
||||
`runner-acpx-claude`, local only (52 cells). `runner-codex` selects OpenAI in the
|
||||
production wizard, omits the runner override, and verifies the saved native
|
||||
configuration without changing runners afterward. `legacy-codex` selects
|
||||
Legacy runner in Advanced before authentication and creation. Both are labeled
|
||||
`production-wizard`. Claude's creation default is unchanged: `legacy-claude`
|
||||
retains its wizard configuration, while `runner-acpx-claude` explicitly changes
|
||||
runtime through the public API before its first task and is labeled
|
||||
`post-onboarding-runtime-switch`. Persona, managed instructions, skills, seeded
|
||||
question, and task invocation are preserved. The native path removes the legacy
|
||||
Paperclip operational skill because Runner supplies that contract through its
|
||||
protocol; other assigned skills, including `/first-task`, are retained.
|
||||
Each isolated onboarding company has a $5 task budget. Setup probes have a
|
||||
separate campaign reservation because they do not create task cost events.
|
||||
|
||||
| First response / control | Complete journey |
|
||||
| --- | --- |
|
||||
@@ -1449,12 +1452,11 @@ released on failure and normal isolated-instance cleanup removes the workspace.
|
||||
The usual provider billing and partial-attempt reporting apply. No production
|
||||
prompts, onboarding defaults, or provider permissions are changed.
|
||||
|
||||
For pre-default native onboarding qualification, select all `first-task` cases
|
||||
with profiles `runner-codex,runner-acpx-claude` (26 cells). The existing public-API
|
||||
runtime switch occurs after the real wizard creates its first agent and before
|
||||
any provider work. It preserves the wizard's model, persona, skills, and task.
|
||||
This tests the native first-task process in advance of the UI/default rollout;
|
||||
it does not certify a native option in the wizard, which is not offered yet.
|
||||
For native onboarding qualification, select `first-task` with profiles
|
||||
`runner-codex,runner-acpx-claude` (26 cells). Codex verifies the production
|
||||
creation default with the experimental flag disabled. Claude still uses the
|
||||
existing public-API runtime switch before provider work; that case does not
|
||||
qualify a native Claude creation default.
|
||||
|
||||
Current proof and remaining decisions are recorded in
|
||||
[the 21 September qualification report](QUALIFICATION-2026-09-21.md). In particular,
|
||||
|
||||
@@ -99,6 +99,9 @@ function commonAgent(
|
||||
capabilities:
|
||||
"Completes deterministic standard, planning, and ask-mode runner acceptance tasks.",
|
||||
adapterType,
|
||||
// These are explicit legacy regression controls. Ordinary Codex creation
|
||||
// defaults are exercised through the production onboarding wizard below.
|
||||
...(adapterType === "codex_local" ? { runner: "legacy" as const } : {}),
|
||||
adapterConfig,
|
||||
defaultEnvironmentId: input.environmentId,
|
||||
budgetMonthlyCents: 0,
|
||||
@@ -1427,7 +1430,7 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
|
||||
groups: ["onboarding"],
|
||||
profiles: runnerProfiles.filter(profile => ["legacy-codex", "legacy-claude", "runner-codex", "runner-acpx-claude"].includes(profile.id)),
|
||||
environments: [localEnvironment], tasks: firstTaskTasks, expectedMatrixSize: 52,
|
||||
definitionMetadata: { version: 4, runGrading: "evidenced-nonexecution-and-refusal", credentialPersistenceCheck: false, questionChoiceMinimum: 2, nativeSetup: "post-onboarding-runtime-switch", productionInstructions: true, qualityGrading: "informational" },
|
||||
definitionMetadata: { version: 5, runGrading: "evidenced-nonexecution-and-refusal", credentialPersistenceCheck: false, questionChoiceMinimum: 2, nativeSetup: "codex-production-default-claude-explicit-runtime-switch", productionInstructions: true, qualityGrading: "informational" },
|
||||
},
|
||||
{
|
||||
id: "agent-chat", label: "Persistent Agent Chat",
|
||||
|
||||
@@ -3,10 +3,10 @@ import type { RunnerApi } from "./api.js";
|
||||
import type { LiveFixtureValues } from "./live-fixtures.js";
|
||||
import type { CredentialName, MatrixExecution } from "./types.js";
|
||||
|
||||
/** Provision only credentials for the UI-created company. The production wizard
|
||||
/** Provision credentials and a bounded budget for the UI-created company. The production wizard
|
||||
* remains the sole creator/configurer of its first agent and onboarding task. */
|
||||
export async function provisionFirstTaskFixtures(input: {
|
||||
api: Pick<RunnerApi, "get" | "postSensitive">;
|
||||
api: Pick<RunnerApi, "get" | "patch" | "postSensitive">;
|
||||
execution: MatrixExecution;
|
||||
nonce: string;
|
||||
company: LiveFixtureValues["company"];
|
||||
@@ -30,6 +30,7 @@ export async function provisionFirstTaskFixtures(input: {
|
||||
const credential = execution.profile.credential;
|
||||
const value = credentials[credential];
|
||||
if (!value) throw new Error(`Missing credential ${credential}`);
|
||||
await api.patch(`/api/companies/${company.id}`, { budgetMonthlyCents: 500 });
|
||||
const environments = await api.get<Array<LiveFixtureValues["environment"]>>(
|
||||
`/api/companies/${company.id}/environments?driver=local`,
|
||||
);
|
||||
@@ -59,8 +60,9 @@ export async function provisionFirstTaskFixtures(input: {
|
||||
};
|
||||
}
|
||||
|
||||
/** Reuse the qualified runtime configuration only, never the QA persona from
|
||||
* buildAgent. Native onboarding is not offered by the production wizard yet. */
|
||||
/** Reuse the qualified runtime configuration for explicit regression switches,
|
||||
* never the QA persona from buildAgent. Codex default qualification does not
|
||||
* call this helper; its wizard must persist the native selection directly. */
|
||||
export function firstTaskNativeRuntimePatch(
|
||||
execution: MatrixExecution,
|
||||
fixtures: LiveFixtureValues,
|
||||
|
||||
@@ -93,6 +93,11 @@ export async function setupFirstTaskFixtures(input: {
|
||||
exact: true,
|
||||
});
|
||||
const savedKey = page.getByRole("combobox", { name: "Saved API key" });
|
||||
if (execution.profile.id === "legacy-codex") {
|
||||
await page.getByText("Advanced", { exact: true }).click();
|
||||
await page.getByRole("button", { name: "Runner", exact: true }).click();
|
||||
await page.getByRole("option", { name: "Legacy runner", exact: true }).click();
|
||||
}
|
||||
// Credential mode depends on the selected provider and its asynchronous key lookup.
|
||||
await expect(savedKey.or(useKey).first()).toBeVisible();
|
||||
if (await useKey.isVisible()) await useKey.click();
|
||||
@@ -110,16 +115,22 @@ export async function setupFirstTaskFixtures(input: {
|
||||
expect(agents).toHaveLength(1);
|
||||
const wizardAdapter =
|
||||
execution.profile.credential === "OPENAI_API_KEY"
|
||||
? "codex_local"
|
||||
? execution.profile.id === "runner-codex" ? "paperclip_runner" : "codex_local"
|
||||
: "claude_local";
|
||||
expect(agents[0].adapterType).toBe(wizardAdapter);
|
||||
if (execution.profile.id === "runner-codex") {
|
||||
expect(agents[0].adapterConfig?.provider).toBe("codex");
|
||||
}
|
||||
fixtures.onboardingRuntime = {
|
||||
mode: "production-wizard",
|
||||
originalAdapterType: wizardAdapter,
|
||||
testedAdapterType: wizardAdapter,
|
||||
originalModel: agents[0].adapterConfig?.model ?? null,
|
||||
};
|
||||
if (execution.profile.generation === "native") {
|
||||
// Claude remains an explicit native regression path in this Codex-only
|
||||
// rollout. Codex is never switched after onboarding: the saved default is
|
||||
// the execution under test.
|
||||
if (execution.profile.id === "runner-acpx-claude") {
|
||||
const runtimePatch = firstTaskNativeRuntimePatch(
|
||||
execution,
|
||||
fixtures,
|
||||
|
||||
@@ -695,9 +695,10 @@ Accept the card above and I write it. This task stays in review until then.`;
|
||||
(e) => e.suite.id === "first-task" && e.profile.id === "legacy-codex",
|
||||
)!;
|
||||
const get = vi.fn().mockResolvedValue([{ id: "local", driver: "local" }]);
|
||||
const patch = vi.fn().mockResolvedValue({});
|
||||
const postSensitive = vi.fn().mockResolvedValue({ id: "encrypted-secret" });
|
||||
const fixtures = await provisionFirstTaskFixtures({
|
||||
api: { get, postSensitive },
|
||||
api: { get, patch, postSensitive },
|
||||
execution,
|
||||
nonce: "test",
|
||||
company: {
|
||||
@@ -710,6 +711,9 @@ Accept the card above and I write it. This task stays in review until then.`;
|
||||
expect(get).toHaveBeenCalledExactlyOnceWith(
|
||||
"/api/companies/ui-created-company/environments?driver=local",
|
||||
);
|
||||
expect(patch).toHaveBeenCalledExactlyOnceWith(
|
||||
"/api/companies/ui-created-company", { budgetMonthlyCents: 500 },
|
||||
);
|
||||
expect(postSensitive).toHaveBeenCalledExactlyOnceWith(
|
||||
"/api/companies/ui-created-company/secrets",
|
||||
expect.objectContaining({ key: "OPENAI_API_KEY", value: "fixture-key" }),
|
||||
@@ -723,7 +727,7 @@ Accept the card above and I write it. This task stays in review until then.`;
|
||||
expect(JSON.stringify(fixtures)).not.toContain("fixture-key");
|
||||
await expect(
|
||||
provisionFirstTaskFixtures({
|
||||
api: { get, postSensitive },
|
||||
api: { get, patch, postSensitive },
|
||||
execution,
|
||||
nonce: "test",
|
||||
company: fixtures.company,
|
||||
|
||||
@@ -870,10 +870,12 @@ for (const execution of executions) {
|
||||
const receipt = verifyStockHarnessPreflight(process.env[STOCK_PREFLIGHT_ENV]);
|
||||
await writeSanitizedJson(snapshotsDir, "stock-harness-preflight.json", receipt, secrets);
|
||||
}
|
||||
// Codex default journeys must work with the experimental gate disabled.
|
||||
const nativeRolloutEnabled = execution.profile.provider !== "codex";
|
||||
const experimental = await api.patch<{
|
||||
enableNativeRunner: boolean;
|
||||
}>("/api/instance/settings/experimental", {
|
||||
enableNativeRunner: true,
|
||||
enableNativeRunner: nativeRolloutEnabled,
|
||||
...(["warm_three_turn", "everyday_workflow"].includes(execution.task.flow)
|
||||
? { enableIsolatedWorkspaces: true }
|
||||
: {}),
|
||||
@@ -882,7 +884,7 @@ for (const execution of executions) {
|
||||
? { enableRunnerPreviewIngress: true }
|
||||
: {}),
|
||||
});
|
||||
expect(experimental.enableNativeRunner).toBe(true);
|
||||
expect(experimental.enableNativeRunner).toBe(nativeRolloutEnabled);
|
||||
|
||||
if (execution.suite.id === "extended-harnesses" && execution.profile.qualificationCandidate === "pi" && execution.task.id === "file-edit-validate") {
|
||||
piFileSeed = await seedPiFile(workspacePath, nonce);
|
||||
|
||||
Reference in new issue
Block a user