mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-21 00:05:49 +02:00
fix: preserve Codex runner intent and private eval evidence
Retain native setup links and built-in settings, record explicit runner choices, and qualify native/legacy skill fixtures. Keep full grading artifacts private while projecting safe first-task summaries. Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
38199ea8e6
commit
1be8502f27
15 files changed
+454
-32
No files matched your search
@@ -170,3 +170,64 @@ cleanup must be verified before reuse. Old archived probe resources and missing
|
||||
compute billing remain unresolved. Do not substitute old artifacts or passing CI
|
||||
for these live gates. The next candidate must pass fresh CI and review before
|
||||
merge sign-off; actual private cloud composition remains post-merge work.
|
||||
|
||||
|
||||
### 2026-10-09 Codex-only candidate qualification update
|
||||
|
||||
Frozen execution candidate `38199ea8e6a276eecd35084ad84d691322aba9f4` is rebased
|
||||
onto master `835a022936ddc7a152b9feab0c0420d02249a6d1`. Its CI completed
|
||||
48 passing, four failing and four skipped checks. The two substantive failures
|
||||
are test fixtures: legacy skill expectations on Linux and a credential probe
|
||||
that depended on a developer daemon. Their narrow corrections pass locally:
|
||||
77 native/legacy/onboarding skill route cases and the selected-login refresh
|
||||
regression. The earlier Node 26 skill run ended without a result; it is not proof.
|
||||
Build, workspace typecheck, all browser E2E shards, policy checks, native
|
||||
compilation, and the clean public npm consumer pass. The consumer checks all
|
||||
18 packed packages contain no Codex payload and acquires host Codex from official
|
||||
npm. The merge and generated-lock CI stamps are recorded separately from source.
|
||||
|
||||
The existing Linux Product E2E campaign [37950165589](https://github.com/paperclipai/paperclip/actions/runs/37950165589)
|
||||
passed its only selected case on attempt one, without a retry. The ordinary
|
||||
onboarding picker saved `paperclip_runner` / `codex`. All 13 assertions passed,
|
||||
including an independently checked nonce-bearing document and follow-up with
|
||||
recorded native execution and the same workspace. Four runs succeeded with
|
||||
observed model `gpt-5.6-sol`; cleanup passed. Target source is exactly `38199ea8`.
|
||||
The workflow deliberately applies its verified resolved lock; the source receipt
|
||||
is dirty and must not be described as a pristine checkout. The report selected
|
||||
one of 52 catalog cases; it is not the entire catalog qualification.
|
||||
|
||||
The final run ledger contains four ready rate-card estimates totaling
|
||||
$1.265460800 (openai-standard-2026-09-30). Production pricing and accounting
|
||||
settled. The existing E2E numeric-only cost summary omits decimal-string
|
||||
costUsdExact and incorrectly reports unpriced/zero; this is a reporting follow-up,
|
||||
not a production pricing failure. The full $12 campaign hold remains reserved for
|
||||
setup and invoice uncertainty. Estimates are not provider invoices. No further
|
||||
paid campaign was dispatched.
|
||||
The existing public report includes private session identifiers in its result
|
||||
summary. It is withheld from broad linking while an allowlisted public projection
|
||||
is added at the existing result seam. Private artifacts and behavioral grading
|
||||
remain intact. This is an evidence-publication correction, not new eval machinery.
|
||||
|
||||
Fresh review scored 3/5, verified the previous three fixes, and found three UI
|
||||
edge cases: preserve original native setup-link intent on unqualified hosts;
|
||||
retain built-in settings on runner-only changes; and record an explicit choice
|
||||
when clicking the displayed automatic default. These fixes pass 171 focused UI
|
||||
checks, typecheck and token gates. A fresh static build passes; affected visual
|
||||
proof is in progress. Existing UI verification passed 28 desktop/eight mobile
|
||||
renders, six keyboard/footer cases and two touch/wheel journeys. Final CI/review
|
||||
must run after this combined follow-up is frozen.
|
||||
|
||||
The isolated preview remains [3125](http://127.0.0.1:3125/COD/dashboard).
|
||||
Its supervisor now uses the correct port and the backend restarted at
|
||||
`2026-10-09T15:28:08.305Z`. No user preview or the separate OAuth proxy was stopped.
|
||||
The first-agent name draft survived reload. The cached local daemon is not proof
|
||||
of the new native Rust seams; Linux hosted build/live execution supply that proof.
|
||||
Attended subscription login and current access to the dedicated managed staging
|
||||
stack remain unresolved. Canonical issue coordination is unavailable because the
|
||||
original test-drive service is offline; no unrelated company was used for updates.
|
||||
|
||||
Next action: finish public-evidence projection and affected UI proof, freeze one
|
||||
combined follow-up, require green exact-head CI and fresh 5/5 review, then complete
|
||||
the attended and managed staging journeys when access is available. The PR stays
|
||||
a draft and is neither merge-ready nor production-ready yet. Merge and production
|
||||
deployment remain outside authorization.
|
||||
@@ -3,6 +3,17 @@ import { readFile } from "node:fs/promises";
|
||||
import request from "supertest";
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
const runnerTarget = vi.hoisted(() => ({ driver: "local", platform: "linux", architecture: "x64" }));
|
||||
vi.mock("../services/agent-runner-selection.js", async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import("../services/agent-runner-selection.js")>();
|
||||
return {
|
||||
...actual,
|
||||
// Exercise the real selection/translation policy on an explicit target.
|
||||
resolveNewAgentRunnerForCompany: vi.fn(async (...[_db, _companyId, input]: Parameters<typeof actual.resolveNewAgentRunnerForCompany>) =>
|
||||
actual.resolveNewAgentRunner({ ...input, target: runnerTarget })),
|
||||
};
|
||||
});
|
||||
|
||||
const mockAgentService = vi.hoisted(() => ({
|
||||
getById: vi.fn(),
|
||||
update: vi.fn(),
|
||||
@@ -252,6 +263,8 @@ function makeAgent(adapterType: string) {
|
||||
|
||||
describe("agent skill routes", () => {
|
||||
beforeEach(async () => {
|
||||
runnerTarget.platform = "linux";
|
||||
runnerTarget.architecture = "x64";
|
||||
vi.resetModules();
|
||||
vi.doUnmock("../routes/agents.js");
|
||||
vi.doUnmock("../routes/authz.js");
|
||||
@@ -1244,31 +1257,45 @@ describe("agent skill routes", () => {
|
||||
.toContain("paperclipai/paperclip/first-task");
|
||||
});
|
||||
|
||||
it.each(["codex_local", "claude_local"].flatMap((adapterType) => [
|
||||
const codexRunnerCases = [
|
||||
{ adapterType: "codex_local", runner: "auto", platform: "linux", expectedType: "paperclip_runner" },
|
||||
{ adapterType: "codex_local", runner: "legacy", platform: "linux", expectedType: "codex_local" },
|
||||
{ adapterType: "codex_local", runner: "auto", platform: "darwin", expectedType: "codex_local" },
|
||||
];
|
||||
it.each([...codexRunnerCases, { adapterType: "claude_local", runner: "auto", platform: "linux", expectedType: "claude_local" }].flatMap((runnerCase) => [
|
||||
["agents", "paperclipai/paperclip/paperclip-create-agent"],
|
||||
["agent-hires", "paperclipai/paperclip/paperclip-create-agent"],
|
||||
["agents", "paperclip"],
|
||||
["agent-hires", "paperclip"],
|
||||
["agents", "paperclipai/paperclip/first-task"],
|
||||
["agent-hires", "paperclipai/paperclip/first-task"],
|
||||
].map(([route, skill]) => ({ adapterType, route, skill }))))("gives $adapterType onboarding agents core and first-task skills via $route, preserving $skill pins", async ({ adapterType, route, skill }) => {
|
||||
].map(([route, skill]) => ({ ...runnerCase, route, skill }))))("assigns onboarding skills via $route for $adapterType/$runner on $platform with $skill pins", async ({ adapterType, runner, platform, expectedType, route, skill }) => {
|
||||
runnerTarget.platform = platform;
|
||||
runnerTarget.architecture = platform === "darwin" ? "arm64" : "x64";
|
||||
mockInstanceSettingsService.getExperimental.mockResolvedValue({ enableBetaSkills: true });
|
||||
const versionId = "22222222-2222-4222-8222-222222222222";
|
||||
const res = await request(await createApp(createDb(route === "agent-hires")))
|
||||
.post(`/api/companies/company-1/${route}`)
|
||||
.send({
|
||||
name: "Chiff", role: "general", adapterType,
|
||||
name: "Chiff", role: "general", adapterType, ...(runner === "auto" ? {} : { runner }),
|
||||
onboardingFirstAgent: true,
|
||||
desiredSkills: [{ key: skill, versionId }],
|
||||
});
|
||||
expect(res.status, JSON.stringify(res.body)).toBe(201);
|
||||
const input = mockAgentService.create.mock.calls[0][1];
|
||||
expect(input.role).toBe("general");
|
||||
expect(input.adapterType).toBe(expectedType);
|
||||
const native = expectedType === "paperclip_runner";
|
||||
if (native) expect(input.adapterConfig.provider).toBe("codex");
|
||||
const canonicalKey = skill === "paperclip" ? "paperclipai/paperclip/paperclip" : skill;
|
||||
const expected = ["paperclip", "paperclip-board", "paperclip-converting-plans-to-tasks", "paperclip-create-agent", "para-memory-files", "first-task"]
|
||||
// Native PRP supplies the operational contract; role skills and their pins remain assigned.
|
||||
.filter((name) => !native || name !== "paperclip")
|
||||
.map((name) => ({ key: `paperclipai/paperclip/${name}`, versionId: `paperclipai/paperclip/${name}` === canonicalKey ? versionId : null }));
|
||||
expect(input.adapterConfig.paperclipSkillSync.desiredSkills).toEqual(expect.arrayContaining(expected));
|
||||
expect(input.adapterConfig.paperclipSkillSync.desiredSkills).toHaveLength(6);
|
||||
const desiredSkills = input.adapterConfig.paperclipSkillSync.desiredSkills.map((entry: string | { key: string; versionId: string | null }) =>
|
||||
typeof entry === "string" ? { key: entry, versionId: null } : entry);
|
||||
expect(desiredSkills).toEqual(expect.arrayContaining(expected));
|
||||
expect(desiredSkills).toHaveLength(expected.length);
|
||||
});
|
||||
|
||||
it.each(["agents", "agent-hires"])("leaves ordinary general agents' defaults unchanged via %s", async (route) => {
|
||||
@@ -1279,13 +1306,23 @@ describe("agent skill routes", () => {
|
||||
expect(mockAgentService.create.mock.calls[0][1].adapterConfig.paperclipSkillSync).toBeUndefined();
|
||||
});
|
||||
|
||||
it.each(["agents", "agent-hires"])("does not assign first-task to ordinary CEOs via %s", async (route) => {
|
||||
it.each(codexRunnerCases.flatMap((runnerCase) => ["agents", "agent-hires"].map((route) => ({ ...runnerCase, route }))))("assigns core skills without first-task to ordinary CEOs via $route using $runner on $platform", async ({ route, runner, platform, expectedType }) => {
|
||||
runnerTarget.platform = platform;
|
||||
runnerTarget.architecture = platform === "darwin" ? "arm64" : "x64";
|
||||
const res = await request(await createApp())
|
||||
.post(`/api/companies/company-1/${route}`)
|
||||
.send({ name: "CEO", role: "ceo", adapterType: "codex_local" });
|
||||
.send({ name: "CEO", role: "ceo", adapterType: "codex_local", ...(runner === "auto" ? {} : { runner }) });
|
||||
expect(res.status, JSON.stringify(res.body)).toBe(201);
|
||||
const desiredSkills = mockAgentService.create.mock.calls[0][1].adapterConfig.paperclipSkillSync.desiredSkills;
|
||||
expect(desiredSkills).toContain("paperclipai/paperclip/paperclip");
|
||||
const input = mockAgentService.create.mock.calls[0][1];
|
||||
expect(input.adapterType).toBe(expectedType);
|
||||
const native = expectedType === "paperclip_runner";
|
||||
if (native) expect(input.adapterConfig.provider).toBe("codex");
|
||||
const desiredSkills = input.adapterConfig.paperclipSkillSync.desiredSkills;
|
||||
const expected = ["paperclip", "paperclip-board", "paperclip-converting-plans-to-tasks", "paperclip-create-agent", "para-memory-files"]
|
||||
.filter((name) => !native || name !== "paperclip")
|
||||
.map((name) => `paperclipai/paperclip/${name}`);
|
||||
expect(desiredSkills).toEqual(expect.arrayContaining(expected));
|
||||
expect(desiredSkills).toHaveLength(expected.length);
|
||||
expect(desiredSkills).not.toContain("paperclipai/paperclip/first-task");
|
||||
});
|
||||
|
||||
|
||||
@@ -23,6 +23,7 @@ import * as codexAdapter from "@paperclipai/adapter-codex-local/server";
|
||||
import { WORKSPACE_RESTORE_LOCK_TIMEOUT_CODE } from "@paperclipai/adapter-utils/workspace-restore-merge";
|
||||
import * as executionTarget from "@paperclipai/adapter-utils/execution-target";
|
||||
import * as runnerProbe from "../vendor/paperclip-runner/index.js";
|
||||
import * as nativeCodexRunner from "../services/native-runtime/native-codex-runner.js";
|
||||
import { testNativeRunnerAuthentication } from "../services/native-runtime/setup-readiness.js";
|
||||
import { prepareManagedAiRuntime, withManagedAiProbe, assertManagedAiProjectAuth, isAiConnectionBusy } from "../services/ai-connection-runtime.js";
|
||||
import { execute as executeGemini, testEnvironment as testGeminiEnvironment } from "@paperclipai/adapter-gemini-local/server";
|
||||
@@ -180,7 +181,10 @@ describe("managed AI connections", () => {
|
||||
await service.save(companyId, owner, { provider: "openai", method: "subscription", ownership: "personal", name: owner, loginSessionId: "fixture", agentIds: [], allAgents: true }, auth("10"));
|
||||
const [row] = await service.quotaAccounts(companyId, owner);
|
||||
const [before] = await db.select({ adapterType: agents.adapterType, adapterConfig: agents.adapterConfig }).from(agents).where(eq(agents.id, agentId));
|
||||
const runnerBinary = path.join(home, "fixture-runnerd");
|
||||
const resolveBinary = vi.spyOn(nativeCodexRunner, "resolvePaperclipRunnerBinary").mockReturnValue(runnerBinary);
|
||||
const probe = vi.spyOn(runnerProbe, "probeNativeRunnerEnvironment").mockImplementation(async input => {
|
||||
expect(input.transportOptions?.runnerBinary).toBe(runnerBinary);
|
||||
expect(input.transportOptions?.sourceCodexHome).toContain("/provider");
|
||||
expect(await readFile(path.join(input.transportOptions!.sourceCodexHome!, "auth.json"), "utf8")).toBe(auth("10"));
|
||||
const providerHome = path.join(input.runtimeDirectory, "codex-home");
|
||||
@@ -203,7 +207,7 @@ describe("managed AI connections", () => {
|
||||
await expect(access(managedHome!)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
const [after] = await db.select({ adapterType: agents.adapterType, adapterConfig: agents.adapterConfig }).from(agents).where(eq(agents.id, agentId));
|
||||
expect(after).toEqual(before);
|
||||
} finally { probe.mockRestore(); }
|
||||
} finally { probe.mockRestore(); resolveBinary.mockRestore(); }
|
||||
});
|
||||
|
||||
it("rejects preparation when reconnect replaces the selected credential reference", async () => {
|
||||
|
||||
@@ -21,10 +21,11 @@ import {
|
||||
collectChatRunEvidence,
|
||||
} from "./chat-flow.js";
|
||||
import type { LiveFixtureValues } from "./live-fixtures.js";
|
||||
import type { CredentialName, MatrixExecution } from "./types.js";
|
||||
import type { CredentialName, MatrixExecution, RunnerE2EResult } from "./types.js";
|
||||
import { firstTaskScenario } from "./first-task-cases.js";
|
||||
import {
|
||||
activeRuns,
|
||||
digestText,
|
||||
snapshotInstruction,
|
||||
gradeFirstTask,
|
||||
firstTaskCompletionSettled,
|
||||
@@ -33,6 +34,117 @@ import {
|
||||
type Row,
|
||||
} from "./first-task-scoring.js";
|
||||
|
||||
/** Results enter the public history bundle. Full observations stay in the
|
||||
* access-controlled snapshots and are graded before this projection. */
|
||||
export function projectFirstTaskResult(result: RunnerE2EResult): RunnerE2EResult {
|
||||
const e = result.firstTask;
|
||||
if (!e) return result;
|
||||
const pick = (value: unknown, fields: readonly string[]): Row => {
|
||||
const row = value && typeof value === "object" ? value as Record<string, unknown> : {};
|
||||
return Object.fromEntries(fields.flatMap(key => {
|
||||
const item = row[key];
|
||||
return item === null || ["string", "number", "boolean"].includes(typeof item)
|
||||
? [[key, item]] : [];
|
||||
})) as Row;
|
||||
};
|
||||
const rows = (value: unknown): Row[] => Array.isArray(value)
|
||||
? value.filter(row => row && typeof row === "object") : [];
|
||||
const strings = (value: unknown): string[] => Array.isArray(value)
|
||||
? value.filter((item): item is string => typeof item === "string") : [];
|
||||
const usage = (value: unknown) => {
|
||||
const u = value as Row | undefined;
|
||||
const measurements = ["inputTokens", "outputTokens", "cachedInputTokens", "cacheWriteTokens", "cacheReadTokens", "cachedReadTokens", "input", "output", "promptTokens", "completionTokens", "cacheAdjustedCostUsd", "costUsd", "providerCostUsd", "costUsdExact"];
|
||||
return {
|
||||
...pick(u, [...measurements, "model", "provider", "biller", "billingType", "costStatus", "usageSource", "accountingReceiptReady"]),
|
||||
...Object.fromEntries(["runDelta", "total", "cumulative"].filter(key => u?.[key]).map(key => [key, pick(u?.[key], measurements)])),
|
||||
...(u?.cost ? { cost: pick(u.cost, ["currency", "amount", "total"]) } : {}),
|
||||
...(u?.pricingProvenance ? { pricingProvenance: pick(u.pricingProvenance, ["source", "version", "contextTier", "serviceTier", "inputCentsPerMillion", "outputCentsPerMillion", "cacheWriteCentsPerMillion", "cachedInputCentsPerMillion"]) } : {}),
|
||||
};
|
||||
};
|
||||
const questionSet = (value: unknown) => {
|
||||
const set = value as Row | undefined;
|
||||
return { ...pick(set, ["title", "description", "submitLabel"]), questions: rows(set?.questions).map(q => ({
|
||||
...pick(q, ["id", "prompt", "header", "helpText", "required", "answerMode", "selectionMode", "allowOther"]),
|
||||
options: rows(q.options).map(option => pick(option, ["id", "label", "description", "recommended", "freeText"])),
|
||||
...(q.customAnswer ? { customAnswer: pick(q.customAnswer, ["enabled", "label", "placeholder"]) } : {}),
|
||||
})) };
|
||||
};
|
||||
const interaction = (row: Row): Row => {
|
||||
const payload = row.payload as Row | undefined;
|
||||
const answers = row.result?.answers;
|
||||
const answerRows: Row[] = Array.isArray(answers) ? rows(answers) : answers && typeof answers === "object"
|
||||
? Object.entries(answers).map(([questionId, answer]) => ({ ...answer as Row, questionId })) : [];
|
||||
return {
|
||||
...pick(row, ["id", "issueId", "kind", "status", "createdAt", "resolvedAt", "resolvedByAgentId"]),
|
||||
...(row.resolvedByUserId ? { resolvedByUserId: "fixture-board" } : {}),
|
||||
payload: {
|
||||
...pick(payload, ["version", "title", "prompt", "detailsMarkdown", "acceptLabel", "rejectLabel", "rejectRequiresReason", "rejectReasonLabel", "submitLabel"]),
|
||||
...(payload?.questions ? questionSet(payload) : {}),
|
||||
...(payload?.questionSet ? { questionSet: questionSet(payload.questionSet) } : {}),
|
||||
...(payload?.target ? { target: pick(payload.target, ["type", "key", "label", "revisionNumber"]) } : {}),
|
||||
},
|
||||
...(row.result ? { result: {
|
||||
...pick(row.result, ["outcome", "reason", "commentId"]),
|
||||
answers: answerRows.map(answer => ({
|
||||
...pick(answer, ["id", "questionId", "text", "otherText", "customText"]),
|
||||
optionIds: strings(answer.optionIds ?? answer.selectedOptionIds),
|
||||
})),
|
||||
} } : {}),
|
||||
};
|
||||
};
|
||||
const privateInstructions = "Full instructions are retained in the access-controlled first-task snapshot.";
|
||||
const settings = e.runtimeSettings as Record<string, any> | undefined;
|
||||
const checkpointIds = new Set(e.checkpoints.map(c => c.id));
|
||||
const firstTask: FirstTaskEvidence = {
|
||||
caseId: e.caseId, nonce: e.nonce, onboardingIssueId: e.onboardingIssueId,
|
||||
agentId: e.agentId, initialTaskIds: [...e.initialTaskIds],
|
||||
...(e.source ? { source: { sha: e.source.sha, ref: e.source.ref, dirty: e.source.dirty } } : {}),
|
||||
configuredModel: e.configuredModel, observedModels: [...e.observedModels],
|
||||
instructions: e.instructions.map(i => ({
|
||||
path: /^(?:\/|[A-Za-z]:)|(?:^|\/)\.\.(?:\/|$)/.test(i.path) ? "private-instruction-path" : i.path,
|
||||
sha256: i.sha256, content: privateInstructions, contentSha256: digestText(privateInstructions), redacted: true,
|
||||
})),
|
||||
runtimeSettings: {
|
||||
...pick(settings, ["adapterType", "completionDeliveryProbe"]),
|
||||
adapterConfig: pick(settings?.adapterConfig, ["provider", "model", "modelReasoningEffort", "lifecycleMode", "codexPermissionMode"]),
|
||||
...(settings?.onboardingRuntime ? { onboardingRuntime: pick(settings.onboardingRuntime, ["mode", "originalAdapterType", "testedAdapterType", "originalModel"]) } : {}),
|
||||
},
|
||||
checkpoints: e.checkpoints.map(c => ({
|
||||
id: c.id, at: c.at, phase: c.phase, issueId: c.issueId,
|
||||
tasks: c.tasks.map(t => pick(t, ["id", "identifier", "title", "description", "status", "parentId", "assigneeAgentId", "createdAt", "completedAt"])),
|
||||
agents: c.agents.map(a => ({ ...pick(a, ["id", "name", "adapterType"]),
|
||||
adapterConfig: pick(a.adapterConfig, ["provider", "model", "modelReasoningEffort", "lifecycleMode", "codexPermissionMode"]) })),
|
||||
comments: c.comments.map(comment => ({ ...pick(comment, ["id", "issueId", "body", "authorAgentId", "createdAt"]),
|
||||
...(comment.authorUserId ? { authorUserId: "fixture-board" } : {}) })),
|
||||
interactions: c.interactions.map(interaction),
|
||||
documents: c.documents.map(d => pick(d, ["id", "issueId", "key", "title", "body", "format", "latestRevisionId", "latestRevisionNumber", "createdAt", "updatedAt"])),
|
||||
attachments: (c.attachments ?? []).map(a => pick(a, ["id", "issueId", "title", "originalFilename", "filename", "body", "contentVerified", "contentSha256", "byteSize", "createdAt"])),
|
||||
runs: c.runs.map(run => ({ ...pick(run, ["id", "issueId", "status", "startedAt", "finishedAt", "runtimeMode", "driverKind"]),
|
||||
...pick({ model: run.resultJson?.model ?? run.usageJson?.model }, ["model"]) })),
|
||||
})),
|
||||
checks: e.checks.map(check => ({
|
||||
id: check.id, passed: check.passed,
|
||||
...(check.notReached ? { notReached: "The journey did not reach this check; private evidence retains the reason." } : {}),
|
||||
evidence: check.evidence.filter(ref => checkpointIds.has(ref)),
|
||||
detail: `${check.id}: ${check.notReached ? "not reached" : check.passed ? "passed" : "failed"}. Full diagnostic evidence is retained in the access-controlled snapshot.`,
|
||||
})),
|
||||
};
|
||||
return {
|
||||
...result, firstTask,
|
||||
...(result.usage ? { usage: Array.isArray(result.usage.runs)
|
||||
? { runs: rows(result.usage.runs).map(run => ({ runId: run.runId, usage: run.usage ? usage(run.usage) : null })) }
|
||||
: usage(result.usage) } : {}),
|
||||
...(result.error ? { error: "First-task journey failed; inspect the recorded checks and access-controlled evidence for the original error." } : {}),
|
||||
matcherResults: (result.matcherResults ?? []).map(matcher => ({
|
||||
...matcher,
|
||||
detail: matcher.matcher.kind === "json_path"
|
||||
? firstTask.checks.find(check => matcher.matcher.kind === "json_path" && matcher.matcher.path === `firstTask.checks.${check.id}`)?.detail
|
||||
?? `${matcher.matcher.kind}: ${matcher.passed ? "passed" : "failed"}. Private evidence retains diagnostics.`
|
||||
: `${matcher.matcher.kind}: ${matcher.passed ? "passed" : "failed"}. Private evidence retains diagnostics.`,
|
||||
})),
|
||||
};
|
||||
}
|
||||
|
||||
/** The wizard creates the agent/task. Only credential provisioning uses Node fetch,
|
||||
* keeping plaintext keys out of Playwright's trace and recorded form values. */
|
||||
export async function setupFirstTaskFixtures(input: {
|
||||
|
||||
@@ -6,6 +6,8 @@ import { firstTaskRejectionReplyRecorded, isFirstTaskRejectionCancellation } fro
|
||||
import { createIssueThreadInteractionSchema } from "../../packages/shared/src/validators/issue.js";
|
||||
import { renderInteractionCard } from "./interaction-report.js";
|
||||
import { main as judgeCommand } from "./first-task-judge.js";
|
||||
import { projectFirstTaskResult } from "./first-task-flow.js";
|
||||
import { validateRetainedRunnerResult } from "./result-validation.js";
|
||||
import {
|
||||
firstTaskNativeRuntimePatch,
|
||||
provisionFirstTaskFixtures,
|
||||
@@ -31,6 +33,7 @@ import {
|
||||
QUALITY_DIMENSIONS,
|
||||
judgeFirstTask,
|
||||
pendingQuality,
|
||||
qualityInput,
|
||||
validateQualityScores,
|
||||
} from "./first-task-quality.js";
|
||||
import { renderFirstTaskDetails } from "./first-task-report.js";
|
||||
@@ -216,6 +219,81 @@ const scores = () =>
|
||||
evidence: ["response-1"],
|
||||
}));
|
||||
|
||||
describe("first-task public result projection", () => {
|
||||
it("keeps fixture-visible evidence while excluding private records from both normalized result fields", () => {
|
||||
const e = recording();
|
||||
const sentinel = "PRIVATE_SESSION_ACCOUNT_SENTINEL";
|
||||
e.instructions[0].content = sentinel;
|
||||
e.instructions[0].path = `/private/${sentinel}/AGENTS.md`;
|
||||
e.runtimeSettings = {
|
||||
adapterType: "paperclip_runner",
|
||||
adapterConfig: { provider: "codex", model: "gpt-5.6-sol", env: { OPENAI_API_KEY: sentinel }, codexHome: sentinel },
|
||||
runtimeConfig: { accountId: sentinel }, permissions: { privateTrace: sentinel },
|
||||
onboardingRuntime: { mode: "production-wizard", originalAdapterType: "paperclip_runner", testedAdapterType: "paperclip_runner", accountId: sentinel },
|
||||
};
|
||||
for (const checkpoint of e.checkpoints) {
|
||||
checkpoint.tasks.forEach(t => Object.assign(t, { executionState: { sessionId: sentinel }, responsibleUserId: sentinel }));
|
||||
checkpoint.agents.forEach(a => Object.assign(a, { accountId: sentinel, metadata: { auth: sentinel }, adapterType: "paperclip_runner",
|
||||
adapterConfig: { provider: "codex", model: "gpt-5.6-sol", env: { TOKEN: sentinel }, home: sentinel } }));
|
||||
checkpoint.comments.forEach(c => Object.assign(c, { hiddenReasoning: sentinel, metadata: { sessionId: sentinel },
|
||||
...(c.authorUserId ? { authorUserId: sentinel } : {}) }));
|
||||
checkpoint.interactions.forEach(i => Object.assign(i, { nativeQuestionId: sentinel, metadata: { sessionId: sentinel },
|
||||
payload: { ...i.payload, providerSessionId: sentinel, toolAction: { secret: sentinel } },
|
||||
result: { ...i.result, outcome: "answered", refreshToken: sentinel } }));
|
||||
checkpoint.documents.forEach(d => Object.assign(d, { privateStorageRef: sentinel }));
|
||||
checkpoint.runs.forEach(r => Object.assign(r, { nativeSessionId: sentinel, sessionIdBefore: sentinel, sessionIdAfter: sentinel,
|
||||
runnerProfileJson: { sessionCheckpoint: { providerSessionId: sentinel } }, contextSnapshot: { home: sentinel },
|
||||
resultJson: { model: "gpt-5.6-sol", privateAuth: sentinel }, stdoutExcerpt: sentinel, error: sentinel }));
|
||||
}
|
||||
e.checks = gradeFirstTask(e);
|
||||
e.checks.push({ id: "native-session-continuity", passed: false, evidence: ["finished-3"], detail: sentinel });
|
||||
const original = structuredClone(e);
|
||||
const raw = { ...result(e), error: sentinel, failureClass: "candidate_failure" as const,
|
||||
usage: { runs: [{ runId: "parent-run", usage: { inputTokens: 100, outputTokens: 20, cachedInputTokens: 30,
|
||||
costUsdExact: "0.499592800", costUsd: null, costStatus: "estimated", accountingReceiptReady: true,
|
||||
pricingProvenance: { source: "rate_card", version: "openai-standard-2026-09-30", accountId: sentinel },
|
||||
persistedSessionId: sentinel, providerRequestId: sentinel, accountingReceiptSourceId: sentinel } }] },
|
||||
matcherResults: e.checks.map(c => ({ matcher: { kind: "json_path" as const, path: `firstTask.checks.${c.id}`, expected: true }, passed: c.passed, detail: c.detail })) };
|
||||
raw.matcherResults.push({ matcher: { kind: "json_path", path: "cleanup.provider-retired", expected: true }, passed: false, detail: sentinel });
|
||||
const projected = projectFirstTaskResult(raw);
|
||||
const normalized = { results: [projected] };
|
||||
expect(JSON.stringify(normalized)).not.toContain(sentinel);
|
||||
expect(JSON.stringify(raw)).toContain(sentinel);
|
||||
expect(e).toEqual(original);
|
||||
expect(projected.firstTask!.checks.map(c => ({ id: c.id, passed: c.passed, notReached: Boolean(c.notReached) })))
|
||||
.toEqual(e.checks.map(c => ({ id: c.id, passed: c.passed, notReached: Boolean(c.notReached) })));
|
||||
expect(projected.matcherResults!.map(c => ({ matcher: c.matcher, passed: c.passed })))
|
||||
.toEqual(raw.matcherResults.map(c => ({ matcher: c.matcher, passed: c.passed })));
|
||||
expect(projected.firstTask!.runtimeSettings).toMatchObject({ adapterType: "paperclip_runner", adapterConfig: { provider: "codex", model: "gpt-5.6-sol" } });
|
||||
expect(projected.usage).toMatchObject({ runs: [{ runId: "parent-run", usage: { inputTokens: 100, outputTokens: 20,
|
||||
cachedInputTokens: 30, costUsdExact: "0.499592800", costStatus: "estimated", accountingReceiptReady: true,
|
||||
pricingProvenance: { source: "rate_card", version: "openai-standard-2026-09-30" } } }] });
|
||||
expect(summarizeExecutionBilling(projected)).toEqual(summarizeExecutionBilling(raw));
|
||||
expect(qualityInput(projected.firstTask!)).toContain(firstTaskScenario(e.caseId, e.nonce).acceptance);
|
||||
expect(qualityInput(projected.firstTask!)).toContain(firstTaskScenario(e.caseId, e.nonce).marker);
|
||||
expect(renderFirstTaskDetails(projected)).not.toContain(sentinel);
|
||||
expect(projected.firstTask!.checkpoints.at(-1)!.documents[0].body).toBe(e.checkpoints.at(-1)!.documents[0].body);
|
||||
validateRetainedRunnerResult(projected);
|
||||
});
|
||||
|
||||
it("preserves failures, unreached checks, cleanup controls and private artifact hashes", () => {
|
||||
const e = recording();
|
||||
e.checks = [{ id: "recorded-response", passed: false, evidence: ["response-1"], detail: "private diagnostic" },
|
||||
{ id: "durable-completion", passed: false, notReached: "private interrupted-session reason", evidence: ["finished-3"], detail: "private diagnostic" }];
|
||||
const raw = { ...result(e), error: "private transport/session error", failureClass: "candidate_failure" as const, cleanup: "failed" as const,
|
||||
matcherResults: [{ matcher: { kind: "json_path" as const, path: "cleanup.provider-retired", expected: true }, passed: false, detail: "private process details" }] };
|
||||
const projected = projectFirstTaskResult(raw);
|
||||
expect(projected).toMatchObject({ status: "failed", failureClass: "candidate_failure", cleanup: "failed" });
|
||||
expect(projected.firstTask!.checks[0].passed).toBe(false);
|
||||
expect(projected.firstTask!.checks[1]).toMatchObject({ passed: false, notReached: expect.any(String) });
|
||||
expect(projected.matcherResults![0]).toMatchObject({ matcher: raw.matcherResults[0].matcher, passed: false });
|
||||
expect(projected.firstTask!.instructions[0]).toMatchObject({ sha256: e.instructions[0].sha256, redacted: true });
|
||||
expect(raw.error).toBe("private transport/session error");
|
||||
expect(projected.error).not.toContain("private transport/session error");
|
||||
expect(projectFirstTaskResult(projected)).toEqual(projected);
|
||||
});
|
||||
});
|
||||
|
||||
describe("first-task attachment evidence", () => {
|
||||
it("reads persisted bytes, verifies their hash, and redacts the retained text", async () => {
|
||||
const body = "Welcome SECRET-VALUE";
|
||||
|
||||
@@ -34,7 +34,7 @@ import { captureStockHarness, gradeStockHarness, gradeStockHire } from "./stock-
|
||||
import { verifyStockHarnessPreflight, STOCK_PREFLIGHT_ENV } from "./stock-harness-admission.js";
|
||||
import { createTaskThroughUi, submitTaskReply } from "./user-actions.js";
|
||||
|
||||
import { runFirstTaskFlow, setupFirstTaskFixtures } from "./first-task-flow.js";
|
||||
import { projectFirstTaskResult, runFirstTaskFlow, setupFirstTaskFixtures } from "./first-task-flow.js";
|
||||
import { runChatFlow } from "./chat-flow.js";
|
||||
import { restartChatServer } from "./chat-restart.js";
|
||||
import { matchesRunCount, minimumRunCount } from "./run-count.js";
|
||||
@@ -3147,10 +3147,13 @@ for (const execution of executions) {
|
||||
...resultWithoutBilling,
|
||||
billing: summarizeExecutionBilling(resultWithoutBilling),
|
||||
};
|
||||
if (firstTaskEvidence) {
|
||||
await writeSanitizedJson(snapshotsDir, "first-task-result-private.json", result, secrets);
|
||||
}
|
||||
await mkdir(privateDir, { recursive: true });
|
||||
await writeFile(
|
||||
resultPath,
|
||||
`${JSON.stringify(sanitizeJson(result, secrets), null, 2)}\n`,
|
||||
`${JSON.stringify(sanitizeJson(projectFirstTaskResult(result), secrets), null, 2)}\n`,
|
||||
"utf8",
|
||||
);
|
||||
await testInfo.attach("runner-e2e-result", {
|
||||
|
||||
@@ -37,6 +37,10 @@ const EXTRA_SETTLE_MS: Record<string, number> = {
|
||||
// waiting. The affected element is masked (solid overlay in both baseline and
|
||||
// comparison) so the rest of the story still snapshot-verifies.
|
||||
const MASKED_SELECTORS: Record<string, string> = {
|
||||
// The unchanged dialog character animates through its SVG runtime even with
|
||||
// reduced motion. Keep the harness picker and footer fully visible.
|
||||
"agents-codex-runner-creation--experimental-native-harnesses":
|
||||
'[role="dialog"] [data-slot="agent-avatar"]',
|
||||
// DocumentAnnotationLayer's ::highlight range over "two selectors" ends 1-2
|
||||
// characters short on ~half of renders (anchor offsets race). Mask only the
|
||||
// paragraph that carries that highlight.
|
||||
|
||||
@@ -28,8 +28,8 @@ export function CodexRunnerSelect({
|
||||
<SelectPopover
|
||||
aria-label="Runner"
|
||||
disabled={disabled}
|
||||
value={value === "auto" ? defaultRunner ?? "auto" : value}
|
||||
displayValue={value === "auto" && !defaultRunner ? (pending ? "Checking runner availability…" : "Automatic selection unavailable") : undefined}
|
||||
value={value}
|
||||
displayValue={value === "auto" ? (defaultRunner ? `${defaultRunner === "paperclip" ? "Paperclip Runner" : "Legacy runner"} (default)` : pending ? "Checking runner availability…" : "Automatic selection unavailable") : undefined}
|
||||
onValueChange={next => onChange(next as AgentRunnerChoice)}
|
||||
options={[
|
||||
{ value: "paperclip", label: `Paperclip Runner${defaultRunner === "paperclip" ? " (default)" : ""}`, disabled: !supportedRunners.includes("paperclip") && value !== "paperclip" },
|
||||
|
||||
@@ -11,6 +11,9 @@ import type { BuiltInAgentState } from "@/api/builtInAgents";
|
||||
const provisionMock = vi.hoisted(() => vi.fn());
|
||||
const updateMock = vi.hoisted(() => vi.fn());
|
||||
const adapterModelsMock = vi.hoisted(() => vi.fn());
|
||||
const adapterDiscoveryMock = vi.hoisted(() => vi.fn());
|
||||
|
||||
vi.mock("@/api/adapters", () => ({ adaptersApi: { list: adapterDiscoveryMock } }));
|
||||
|
||||
vi.mock("@/api/builtInAgents", async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import("@/api/builtInAgents")>();
|
||||
@@ -72,6 +75,15 @@ function makeState(overrides: Partial<BuiltInAgentState> = {}): BuiltInAgentStat
|
||||
};
|
||||
}
|
||||
|
||||
function makeSavedAgent(adapterType: "codex_local" | "paperclip_runner", adapterConfig: Record<string, unknown>): NonNullable<BuiltInAgentState["agent"]> {
|
||||
return {
|
||||
id: "a1", companyId: "c1", name: "Briefs Agent", urlKey: "briefs-agent", role: "general", title: null, icon: null,
|
||||
status: "idle", reportsTo: null, capabilities: null, adapterType, adapterConfig, runtimeConfig: {}, defaultEnvironmentId: null,
|
||||
budgetMonthlyCents: 0, spentMonthlyCents: 0, pauseReason: null, pausedAt: null, permissions: { canCreateAgents: false },
|
||||
lastHeartbeatAt: null, metadata: null, createdAt: new Date(0), updatedAt: new Date(0),
|
||||
};
|
||||
}
|
||||
|
||||
async function flushReact() {
|
||||
for (let index = 0; index < 6; index += 1) {
|
||||
await Promise.resolve();
|
||||
@@ -117,6 +129,7 @@ describe("ConfigureBuiltInAgentModal (PAP-12978)", () => {
|
||||
provisionMock.mockReset();
|
||||
updateMock.mockReset();
|
||||
adapterModelsMock.mockReset().mockResolvedValue([]);
|
||||
adapterDiscoveryMock.mockReset().mockResolvedValue([{ type: "codex_local", supportedRunners: ["paperclip", "legacy"], defaultRunner: "legacy" }]);
|
||||
onOpenChange.mockReset();
|
||||
onConfigured.mockReset();
|
||||
});
|
||||
@@ -189,6 +202,40 @@ describe("ConfigureBuiltInAgentModal (PAP-12978)", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it.each(["codex_local", "paperclip_runner"] as const)("lets the server translate saved %s settings for a runner-only change", async adapterType => {
|
||||
const savedConfig = { model: "gpt-5", timeoutSec: 45, modelReasoningEffort: "high", env: { CODEX_HOME: { type: "secret_ref", secretId: "account-1" } }, ...(adapterType === "paperclip_runner" ? { provider: "codex", codexPermissionMode: "never", lifecycleMode: "per_turn" } : {}) };
|
||||
const state = makeState({ status: "ready", agentId: "a1", agent: makeSavedAgent(adapterType, savedConfig) });
|
||||
provisionMock.mockResolvedValue(state);
|
||||
await renderModal(state);
|
||||
const summary = document.querySelector("summary")!;
|
||||
flushSync(() => summary.dispatchEvent(new MouseEvent("click", { bubbles: true })));
|
||||
flushSync(() => document.querySelector('[aria-label="Runner"]')!.dispatchEvent(new MouseEvent("click", { bubbles: true })));
|
||||
await flushReact();
|
||||
const runner = adapterType === "paperclip_runner" ? "legacy" : "paperclip";
|
||||
flushSync(() => document.querySelector(`[role="option"][data-value="${runner}"]`)!.dispatchEvent(new MouseEvent("click", { bubbles: true })));
|
||||
await flushReact();
|
||||
flushSync(() => findButton("Configure")!.dispatchEvent(new MouseEvent("click", { bubbles: true })));
|
||||
await flushReact();
|
||||
expect(provisionMock).toHaveBeenCalledWith("c1", "briefs", { runner });
|
||||
expect(state.agent?.adapterConfig).toEqual(savedConfig);
|
||||
});
|
||||
|
||||
it("merges a built-in model edit into the saved native configuration", async () => {
|
||||
const savedConfig = { provider: "codex", model: "gpt-5", timeoutSec: 45, modelReasoningEffort: "high", env: { CODEX_HOME: { type: "secret_ref", secretId: "account-1" } } };
|
||||
const state = makeState({ status: "ready", agentId: "a1", agent: makeSavedAgent("paperclip_runner", savedConfig) });
|
||||
provisionMock.mockResolvedValue(state);
|
||||
await renderModal(state);
|
||||
const modelInput = document.querySelector('[data-testid="model-input"]') as HTMLInputElement;
|
||||
flushSync(() => {
|
||||
Object.getOwnPropertyDescriptor(HTMLInputElement.prototype, "value")!.set!.call(modelInput, "gpt-6-sol");
|
||||
modelInput.dispatchEvent(new Event("input", { bubbles: true }));
|
||||
});
|
||||
await flushReact();
|
||||
flushSync(() => findButton("Configure")!.dispatchEvent(new MouseEvent("click", { bubbles: true })));
|
||||
await flushReact();
|
||||
expect(provisionMock).toHaveBeenCalledWith("c1", "briefs", { adapterType: "paperclip_runner", adapterConfig: { ...savedConfig, model: "gpt-6-sol" } });
|
||||
});
|
||||
|
||||
it("shows a visible error and blocks provisioning for an unknown model", async () => {
|
||||
adapterModelsMock.mockResolvedValue([
|
||||
{ id: "claude-haiku-4-5", label: "Claude Haiku 4.5" },
|
||||
|
||||
@@ -134,10 +134,16 @@ export function ConfigureBuiltInAgentModal({
|
||||
mutationFn: async () => {
|
||||
const adapterConfig: Record<string, unknown> = {};
|
||||
if (model.trim()) adapterConfig.model = model.trim();
|
||||
const existing = state.agent;
|
||||
const sameHarness = existing && agentHarnessType(existing.adapterType, existing.adapterConfig) === adapterType;
|
||||
const savedModel = typeof existing?.adapterConfig.model === "string" ? existing.adapterConfig.model.trim() : "";
|
||||
// A runner-only change must let the server translate the full saved config.
|
||||
const adapterInput = sameHarness && normalizedModel === savedModel ? {}
|
||||
: sameHarness ? { adapterType: existing.adapterType, adapterConfig: { ...existing.adapterConfig, ...adapterConfig } }
|
||||
: { adapterType, adapterConfig };
|
||||
const result = await builtInAgentsApi.provision(companyId, definition.key, {
|
||||
adapterType: state.agent && agentHarnessType(state.agent.adapterType, state.agent.adapterConfig) === adapterType ? state.agent.adapterType : adapterType,
|
||||
...adapterInput,
|
||||
...(adapterType === "codex_local" && runner !== undefined ? { runner } : {}),
|
||||
adapterConfig,
|
||||
...(budgetMonthlyCents !== undefined ? { budgetMonthlyCents } : {}),
|
||||
});
|
||||
return result;
|
||||
|
||||
@@ -92,6 +92,7 @@ export function NewAgentSetup() {
|
||||
name={params.get("name") ?? ""}
|
||||
adapterType={params.get("adapterType") === "paperclip_runner" && (params.get("runnerProvider") ?? "codex") === "codex" ? "codex_local" : params.get("adapterType") ?? ""}
|
||||
initialRunner={params.get("runner") ?? (params.get("adapterType") === "paperclip_runner" && (params.get("runnerProvider") ?? "codex") === "codex" ? "paperclip" : "auto")}
|
||||
preserveNativeCodex={params.get("adapterType") === "paperclip_runner" && (params.get("runnerProvider") ?? "codex") === "codex"}
|
||||
runnerProvider={params.get("runnerProvider") ?? "codex"}
|
||||
createdAgentId={params.get("createdAgentId")}
|
||||
/>
|
||||
@@ -104,6 +105,7 @@ function Setup({
|
||||
adapterType,
|
||||
runnerProvider,
|
||||
initialRunner,
|
||||
preserveNativeCodex,
|
||||
createdAgentId,
|
||||
}: {
|
||||
companyId: string;
|
||||
@@ -111,6 +113,7 @@ function Setup({
|
||||
adapterType: string;
|
||||
runnerProvider: string;
|
||||
initialRunner: string;
|
||||
preserveNativeCodex: boolean;
|
||||
createdAgentId: string | null;
|
||||
}) {
|
||||
const navigate = useNavigate();
|
||||
@@ -118,6 +121,7 @@ function Setup({
|
||||
const { openNewIssue } = useDialogActions();
|
||||
const appearanceDraft = useAgentAppearanceDraft(`${companyId}:new-agent`);
|
||||
const [runner, setRunner] = useState<AgentRunnerChoice>(initialRunner === "legacy" || initialRunner === "paperclip" ? initialRunner : "auto");
|
||||
const requestAdapterType = preserveNativeCodex && runner === "paperclip" ? "paperclip_runner" : adapterType;
|
||||
const isRunner = adapterType === "paperclip_runner";
|
||||
const isDot = isRunner && runnerProvider === "openai_dot";
|
||||
const brandType = isDot ? "openai_dot" : isRunner
|
||||
@@ -335,7 +339,7 @@ function Setup({
|
||||
adapter.loaded &&
|
||||
!adapter.disabled &&
|
||||
!getAdapterDisplay(adapterType).comingSoon,
|
||||
);
|
||||
) && (requestAdapterType !== "paperclip_runner" || adapterType === "paperclip_runner" || adapterInventory.data?.some(adapter => adapter.type === "paperclip_runner" && adapter.loaded && !adapter.disabled));
|
||||
const ready = Boolean(
|
||||
available &&
|
||||
(adapterType !== "codex_local" || runner !== "auto" || Boolean(adapters.data)) &&
|
||||
@@ -380,6 +384,9 @@ function Setup({
|
||||
: {}),
|
||||
};
|
||||
const config = getUIAdapter(adapterType).buildAdapterConfig(values);
|
||||
if (preserveNativeCodex && runner === "paperclip") {
|
||||
Object.assign(config, { provider: "codex", codexPermissionMode: "never", lifecycleMode: "per_turn" });
|
||||
}
|
||||
if (isRunner)
|
||||
Object.assign(config, {
|
||||
provider: (runnerProvider === "claude" || runnerProvider === "grok") ? "acpx" : runnerProvider,
|
||||
@@ -464,7 +471,7 @@ function Setup({
|
||||
const config = await preparedConfig(nextConnection);
|
||||
const tested = await testAgentSetup({
|
||||
companyId,
|
||||
adapterType,
|
||||
adapterType: requestAdapterType,
|
||||
providerAdapter: brandType,
|
||||
...(adapterType === "codex_local" && runner !== "auto" ? { runner } : {}),
|
||||
adapterConfig: config,
|
||||
@@ -538,7 +545,7 @@ function Setup({
|
||||
appearance: appearanceDraft.appearance,
|
||||
role: existing.length ? "general" : "ceo",
|
||||
...(leader ? { reportsTo: leader.id } : {}),
|
||||
adapterType,
|
||||
adapterType: requestAdapterType,
|
||||
...(adapterType === "codex_local" && runner !== "auto" ? { runner } : {}),
|
||||
adapterConfig: config,
|
||||
defaultEnvironmentId:
|
||||
|
||||
@@ -200,6 +200,45 @@ afterEach(async () => {
|
||||
container.remove();
|
||||
});
|
||||
describe("New agent setup", () => {
|
||||
it("records an explicit legacy choice when it matches the Mac default, then preserves it on Linux", async () => {
|
||||
settings.get.mockResolvedValue({ defaultEnvironmentId: "local-1" });
|
||||
envApi.list.mockResolvedValue([
|
||||
{ id: "local-1", name: "Mac", driver: "local", status: "active", config: {} },
|
||||
{ id: "linux-ssh", name: "Linux", driver: "ssh", status: "active", config: {} },
|
||||
]);
|
||||
adapterApi.list.mockImplementation(async (context?: { environmentId?: string | null }) => state.adapters.map((adapter: any) =>
|
||||
adapter.type === "codex_local" ? { ...adapter, supportedRunners: context?.environmentId === "linux-ssh" ? ["paperclip", "legacy"] : ["legacy"], defaultRunner: context?.environmentId === "linux-ssh" ? "paperclip" : "legacy" } : adapter));
|
||||
await render("codex_local");
|
||||
await act(async () => (container.querySelector("summary") as HTMLElement).click());
|
||||
const runner = container.querySelector<HTMLButtonElement>('[aria-label="Runner"]')!;
|
||||
expect(runner.textContent).toBe("Legacy runner (default)");
|
||||
await act(async () => runner.click());
|
||||
await act(async () => document.querySelector<HTMLButtonElement>('[role="option"][data-value="legacy"]')!.click());
|
||||
await connect("OpenAI");
|
||||
expect(api.testEnvironment).toHaveBeenLastCalledWith("company-1", "codex_local", expect.objectContaining({ runner: "legacy" }));
|
||||
const environment = container.querySelector<HTMLSelectElement>('select[aria-label="Environment"]')!;
|
||||
await act(async () => { environment.value = "linux-ssh"; environment.dispatchEvent(new Event("change", { bubbles: true })); });
|
||||
await settle();
|
||||
await connect("OpenAI");
|
||||
await click("Finish setup");
|
||||
expect(api.testEnvironment).toHaveBeenLastCalledWith("company-1", "codex_local", expect.objectContaining({ environmentId: "linux-ssh", runner: "legacy" }));
|
||||
expect(api.hire.mock.calls[0][1]).toMatchObject({ adapterType: "codex_local", runner: "legacy" });
|
||||
});
|
||||
|
||||
it("preserves an old native Codex setup link on an unqualified target with the experiment off", async () => {
|
||||
settings.getExperimental.mockResolvedValue({ enableNativeRunner: false });
|
||||
adapterApi.list.mockImplementation(async () => state.adapters.map((adapter: any) => adapter.type === "codex_local" ? { ...adapter, supportedRunners: ["legacy"], defaultRunner: "legacy" } : adapter));
|
||||
api.testEnvironment.mockImplementation(async (_company, adapterType, payload) => {
|
||||
if (adapterType === "codex_local" && payload.runner === "paperclip") throw new Error("This target is not qualified for a new native default.");
|
||||
return { ...pass, adapterType, checks: [{ code: "paperclip_runner_codex_hello_probe_passed", level: "info", message: "Native runtime replied" }] };
|
||||
});
|
||||
await render("paperclip_runner", "codex");
|
||||
await connect("OpenAI");
|
||||
await click("Finish setup");
|
||||
expect(api.testEnvironment).toHaveBeenCalledWith("company-1", "paperclip_runner", expect.objectContaining({ runner: "paperclip", adapterConfig: expect.objectContaining({ provider: "codex" }) }));
|
||||
expect(api.hire.mock.calls[0][1]).toMatchObject({ adapterType: "paperclip_runner", runner: "paperclip", adapterConfig: { provider: "codex" } });
|
||||
});
|
||||
|
||||
it("discovers the runner for the selected target and rechecks after changing environments", async () => {
|
||||
settings.get.mockResolvedValue({ defaultEnvironmentId: "linux-ssh" });
|
||||
envApi.list.mockResolvedValue([
|
||||
@@ -801,8 +840,8 @@ describe("New agent setup", () => {
|
||||
await click("Finish setup");
|
||||
const config = api.hire.mock.calls[0][1].adapterConfig;
|
||||
if (runner === "codex") {
|
||||
expect(api.hire.mock.calls[0][1]).toMatchObject({ adapterType: "codex_local", runner: "paperclip" });
|
||||
expect(api.testEnvironment).toHaveBeenCalledWith("company-1", "codex_local", expect.objectContaining({ runner: "paperclip" }));
|
||||
expect(api.hire.mock.calls[0][1]).toMatchObject({ adapterType: "paperclip_runner", runner: "paperclip" });
|
||||
expect(api.testEnvironment).toHaveBeenCalledWith("company-1", "paperclip_runner", expect.objectContaining({ runner: "paperclip" }));
|
||||
} else expect(config.provider).toBe(runner === "claude" ? "acpx" : runner);
|
||||
if (runner === "claude") {
|
||||
expect(config.acpxAgent).toBe("claude");
|
||||
@@ -871,6 +910,7 @@ describe("New agent setup", () => {
|
||||
});
|
||||
it("blocks a disabled runner even when opened through a URL", async () => {
|
||||
state.adapters = [
|
||||
{ type: "codex_local", loaded: true, disabled: false },
|
||||
{ type: "paperclip_runner", loaded: true, disabled: true },
|
||||
];
|
||||
await render("paperclip_runner");
|
||||
|
||||
@@ -162,12 +162,9 @@ function installStorybookApiFixtures() {
|
||||
if (url.pathname === "/api/instance/settings/experimental") {
|
||||
return Response.json(experimentalSettings());
|
||||
}
|
||||
if (url.pathname === "/api/instance/settings/general") {
|
||||
return Response.json(instanceGeneralSettingsSchema.parse({}));
|
||||
}
|
||||
|
||||
if (url.pathname === "/api/health") {
|
||||
return Response.json({ status: "ok", deploymentMode: "authenticated", localAiLoginSupported: onboardingFixtureState.environments === "local" });
|
||||
return Response.json({ deploymentMode: "authenticated", localAiLoginSupported: onboardingFixtureState.environments === "local" });
|
||||
}
|
||||
|
||||
if (url.pathname === "/api/instance/settings") {
|
||||
@@ -509,8 +506,6 @@ function installStorybookApiFixtures() {
|
||||
{
|
||||
type: "codex_local",
|
||||
label: "Codex",
|
||||
supportedRunners: ["paperclip", "legacy"],
|
||||
defaultRunner: "paperclip",
|
||||
source: "builtin",
|
||||
modelsCount: 3,
|
||||
loaded: true,
|
||||
|
||||
@@ -90,7 +90,21 @@ export function AgentSettingsPreview({
|
||||
const queryClient = useQueryClient();
|
||||
useEffect(() => {
|
||||
const uninstall = fixtures.install();
|
||||
const previous = window.fetch;
|
||||
const codexFixture: typeof fetch = async (input, init) => {
|
||||
const url = new URL(typeof input === "string" ? input : input instanceof URL ? input.href : input.url, window.location.origin);
|
||||
if (url.pathname === "/api/adapters" || url.pathname === "/api/health") {
|
||||
const response = await previous(input, init);
|
||||
const data = await response.json();
|
||||
return Response.json(url.pathname === "/api/health" ? { ...data, status: "ok" }
|
||||
: data.map((adapter: { type: string }) => adapter.type === "codex_local" ? { ...adapter, supportedRunners: ["paperclip", "legacy"], defaultRunner: "paperclip" } : adapter));
|
||||
}
|
||||
return previous(input, init);
|
||||
};
|
||||
if (adapterType === "codex_local" || adapterType === "paperclip_runner") window.fetch = codexFixture;
|
||||
const fixtureQueryKeys = [
|
||||
queryKeys.health,
|
||||
queryKeys.adapters.all,
|
||||
queryKeys.agents.detail(ID),
|
||||
queryKeys.agents.detail(REF),
|
||||
queryKeys.agents.skills(ID),
|
||||
@@ -114,10 +128,11 @@ export function AgentSettingsPreview({
|
||||
navigate(`/PAP/agents/${REF}/${initialTab}`, { replace: true });
|
||||
setReady(true);
|
||||
return () => {
|
||||
if (window.fetch === codexFixture) window.fetch = previous;
|
||||
uninstall();
|
||||
clearFixtureQueries();
|
||||
};
|
||||
}, [fixtures, initialTab, queryClient, setSelectedCompanyId]);
|
||||
}, [adapterType, fixtures, initialTab, queryClient, setSelectedCompanyId]);
|
||||
if (!ready || selectedCompanyId !== COMPANY) return null;
|
||||
return (
|
||||
<Routes>
|
||||
|
||||
@@ -7,7 +7,7 @@ import { storybookHiredAgent } from "../../fixtures/paperclipData";
|
||||
import { resetOnboardingFixtureState, setOnboardingFixtureState } from "../../fixtures/onboardingEnvironment";
|
||||
|
||||
/** Real creation screen; existing Storybook API fixtures provide isolated accounts. */
|
||||
function installCodexCreationFixture({ setupFailure = false, nativeRunnerEnabled = false }: { setupFailure?: boolean; nativeRunnerEnabled?: boolean } = {}) {
|
||||
function installCodexCreationFixture({ setupFailure = false, nativeRunnerEnabled = false, unqualifiedTarget = false }: { setupFailure?: boolean; nativeRunnerEnabled?: boolean; unqualifiedTarget?: boolean } = {}) {
|
||||
resetOnboardingFixtureState();
|
||||
setOnboardingFixtureState({ environments: "local", authSignal: "present", localLoginStatus: "ready", savedManagedSubscription: "openai" });
|
||||
const previous = window.fetch;
|
||||
@@ -19,9 +19,15 @@ function installCodexCreationFixture({ setupFailure = false, nativeRunnerEnabled
|
||||
const response = await previous(input, init);
|
||||
return Response.json({ ...await response.json(), enableNativeRunner: nativeRunnerEnabled });
|
||||
}
|
||||
if (nativeRunnerEnabled && url.pathname === "/api/adapters") {
|
||||
if (url.pathname === "/api/instance/settings/general") return Response.json({ executionMode: "any" });
|
||||
if (url.pathname === "/api/health") {
|
||||
const response = await previous(input, init);
|
||||
return Response.json([...await response.json(), { type: "paperclip_runner", label: "Paperclip Runner", source: "builtin", loaded: true, disabled: false, modelsCount: 0 }]);
|
||||
return Response.json({ ...await response.json(), status: "ok" });
|
||||
}
|
||||
if (url.pathname === "/api/adapters") {
|
||||
const response = await previous(input, init);
|
||||
const adapters = (await response.json()).map((adapter: { type: string }) => adapter.type === "codex_local" ? { ...adapter, supportedRunners: unqualifiedTarget ? ["legacy"] : ["paperclip", "legacy"], defaultRunner: unqualifiedTarget ? "legacy" : "paperclip" } : adapter);
|
||||
return Response.json(nativeRunnerEnabled || unqualifiedTarget ? [...adapters, { type: "paperclip_runner", label: "Paperclip Runner", source: "builtin", loaded: true, disabled: false, modelsCount: 0 }] : adapters);
|
||||
}
|
||||
if (url.pathname.endsWith("/adapters/codex_local/models")) return Response.json(models);
|
||||
if (url.pathname.endsWith("/adapters/codex_local/test-environment")) {
|
||||
@@ -68,6 +74,13 @@ const connectCodex: NonNullable<Story["play"]> = async ({ canvasElement }) => {
|
||||
};
|
||||
export const AutomaticCodex: Story = { play: showRunner };
|
||||
export const ExplicitLegacy: Story = { parameters: { initialEntries: ["/PAP/agents/new?name=Nova&adapterType=codex_local&runner=legacy"] }, play: showRunner };
|
||||
export const OldNativeCodexLink: Story = {
|
||||
parameters: { initialEntries: ["/PAP/agents/new?name=Nova&adapterType=paperclip_runner&runnerProvider=codex"], codexCreationFixture: { unqualifiedTarget: true } },
|
||||
play: async context => {
|
||||
await showRunner(context);
|
||||
await expect(await within(context.canvasElement).findByRole("button", { name: "Runner" })).toHaveTextContent("Paperclip Runner");
|
||||
},
|
||||
};
|
||||
export const AutomaticLight: Story = { ...AutomaticCodex, globals: { theme: "light" } };
|
||||
export const LegacyMobile: Story = { ...ExplicitLegacy, globals: { viewport: { value: "mobile1", isRotated: false } } };
|
||||
export const SetupFailure: Story = { parameters: { codexCreationFixture: { setupFailure: true } }, play: async context => {
|
||||
|
||||
Reference in new issue
Block a user