mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-04 19:28:46 +01:00
Reconcile Claude platform packages with merged Codex defaults
Co-Authored-By: Paperclip <noreply@paperclip.ing> * codex/claude-runner-default: (85 commits) feat: default new Codex agents to Paperclip Runner on Linux and macOS (#15683) docs(skills): add slow-query diagnosis and fix skill (#16303) feat(sentry): add opt-in run failure reporting with saved run references (#16300) refactor(server): isolate provider message admission and runtime policies (#16289) refactor(server): isolate provider webhook lifecycle decisions (#16283) fix(workspaces): keep cleanup Git scans fresh and dedupe page reads (#16302) fix(sentry): keep provider-rejected Grok wakes out of run failure reports (#16297) fix(runner): report Runner disconnect age when a Dot bridge exits (#16296) refactor(server): isolate chat conversation and control policies (#16276) perf(recovery): use indexed execution identity lookups (#16291) perf(runs): bound history and reduce repeated context reads (#16286) fix(native-runtime): report which close barrier failed (#16298) feat(skills): select and validate tracked GitHub branches (#16288) fix(observability): sample native runs independently of triggers (#16285) fix(github): retry and recover review publication (#16284) perf(workspaces): wake terminal cleanup from committed changes (#16282) fix(mcp): deduplicate discovered tool descriptors (#16281) perf: remove history scans from task and chat queries (#16278) docs(skills): add maintenance bot setup workflow (#16279) fix(observability): time execution configuration steps (#16236) ... # Conflicts: # scripts/verify-grok-npm-install.mjs
This commit is contained in:
commit
b030683ca2
1176 files changed
+129461
-261897
No files matched your search
@@ -124,6 +124,19 @@ attribution. Failed setup before fixture completion retains unavailable/unknown
|
||||
cost coverage and its holds. A connection-only probe or name-only snapshot cannot
|
||||
qualify native defaults.
|
||||
|
||||
The manual managed first-task cell is the exception to fixture prequalification:
|
||||
`confirmation-replies.runner-codex.daytona.task-card-accept` leaves the saved
|
||||
environment unqualified until the ordinary onboarding Connect action. It
|
||||
observes exactly one platform response and one selected-runtime response, fresh
|
||||
scoped native discovery and the saved automatic choice. Both setup leases must
|
||||
have authoritative response IDs, match this company/environment, and reach
|
||||
released/success before setup is accepted. Exact IDs join the existing task-run
|
||||
lease collector and runtime estimate, including failed setup. New uncorrelated
|
||||
or response-less leases remain unknown and unattributed; incomplete coverage
|
||||
removes the estimate and retains its budget holds. No retry or extra fixture
|
||||
probe closes that gap. The existing environment deletion, expiry backstops and
|
||||
instance cleanup still apply on every outcome.
|
||||
|
||||
Daytona creates sandbox environments through the public API. The core fixture
|
||||
keeps `reuseLease:false` and `runnerLifecycleMode:"per_turn"`. The dedicated
|
||||
warm-continuity fixture uses `reuseLease:true` and
|
||||
|
||||
@@ -2152,6 +2152,29 @@ the changed chief-of-staff persona and skill selections; it needs no fixture
|
||||
change for that default selection. Hiring from that wizard-created chief of
|
||||
staff remains a separate follow-up qualification.
|
||||
|
||||
The explicit-only cell `confirmation-replies.runner-codex.daytona.task-card-accept`
|
||||
extends that wizard journey to one fresh managed company. Select this exact ID;
|
||||
the automatic roster and all local prompts remain unchanged. Its fixture seeds
|
||||
encrypted credentials and an immutable OCI image environment, but does not
|
||||
prequalify the image or create an agent. Connect owns the existing saved
|
||||
environment probe, fresh scoped adapter discovery, and selected runtime setup.
|
||||
The case requires one confirmed platform probe and one native Codex setup lease,
|
||||
then verifies the automatically saved native configuration after reload.
|
||||
|
||||
The completed child must expose a downloaded `garden-welcome.txt` attachment
|
||||
whose byte size, SHA-256, content and scenario marker are independently checked.
|
||||
A saved document alone cannot pass. Existing task-card grading and the same-task
|
||||
follow-up continuity checks remain required. This cell has one attempt, zero
|
||||
automatic retries and a shared fifteen-minute deadline with cleanup time
|
||||
reserved. The script authors three turns; four task runs were observed in the
|
||||
historical local journey. The existing twelve-run guard and $5 company task cap
|
||||
remain, with native setup, one completion judge request, sandbox time and cleanup
|
||||
requiring separate reservation. Registration and provider-free controls do not
|
||||
establish live qualification or funding within the remaining paid reserve.
|
||||
|
||||
This OCI-image Core case does not prove Cloud signup, warm/cold provisioning or
|
||||
name-only snapshot qualification. Those remain separate acceptance gates.
|
||||
|
||||
### Hiring completion accounting evidence
|
||||
|
||||
The v3 hiring grader uses turn-accounting v2 in both executable guards. It requires complete per-run public event streams, exact native tool-use/result pairing and canonical execution IDs for completion actions. Only successful known GET issue/document/comment operations, verified reads/discovery, and attributed native chat finish are admitted. Writes, failed mutation attempts, incomplete streams and unknown actions cannot pass. Separate ACPX host request IDs and provider execution IDs are not joined by name/order/count; missing mapping is uncomparable action coverage, not a measured task failure. The original source-read and exact template checks remain unchanged.
|
||||
|
||||
@@ -126,6 +126,16 @@ The Paperclip server process also receives none; the browser posts each value
|
||||
once to the encrypted company secret API and agents/environments retain only
|
||||
secret references.
|
||||
|
||||
The explicit managed first-task cell observes the wizard's two owned setup
|
||||
responses only to correlate their exact lease IDs with same-company environment
|
||||
records. Raw probe/setup responses, account references, opaque provider lease
|
||||
identities and setup screenshots remain private. Public first-task evidence
|
||||
continues through the existing projection. Missing or foreign lease identities
|
||||
cannot acquire attribution or a zero-cost estimate; incomplete setup coverage
|
||||
stays unavailable even if later task-run billing succeeds. Platform qualification
|
||||
uses the existing permission-gated environment probe, without granting new
|
||||
permissions or accepting fixture-authored qualification metadata.
|
||||
|
||||
The context-integrity catalog lists six pending Kimi/Grok cells for discovery:
|
||||
two cases each across `legacy-kimi-cli`, `legacy-kimi-acp`, and `legacy-grok`.
|
||||
The launcher and direct Playwright runner reject these profiles before loading
|
||||
|
||||
@@ -32,6 +32,23 @@ import {
|
||||
} from "./selectors.js";
|
||||
|
||||
describe("runner E2E catalog", () => {
|
||||
it("keeps the single managed onboarding variant explicit-only without changing local onboarding coverage", () => {
|
||||
const id = "confirmation-replies.runner-codex.daytona.task-card-accept";
|
||||
const suite = runnerSuites.find(candidate => candidate.id === "confirmation-replies")!;
|
||||
const remote = runnerMatrix.filter(entry => entry.suite.id === suite.id && entry.environment.id === "daytona");
|
||||
expect(remote.map(entry => entry.id)).toEqual([id]);
|
||||
expect(suite.manualOnly).toBe(true);
|
||||
expect(remote[0]!.task.flow).toBe("first_task");
|
||||
expect(remote[0]!.task.expectedRunCount).toBe(4);
|
||||
expect(remote[0]!.task.automaticRetryPolicy).toBe("single_attempt");
|
||||
expect(remote[0]!.requiredCredentials).toEqual(["OPENAI_API_KEY", "DAYTONA_API_KEY"]);
|
||||
expect(runnerMatrix.filter(entry => entry.suite.id === "first-task")).toHaveLength(52);
|
||||
expect(runnerMatrix.filter(entry => entry.suite.id === suite.id && entry.environment.id === "local")).toHaveLength(12);
|
||||
expect(selectRunnerExecutions(parseRunnerSelectors(["--id", id])).map(entry => entry.id)).toEqual([id]);
|
||||
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(entry => entry.id === id)).toBe(false);
|
||||
expect(selectRunnerExecutions(parseRunnerSelectors(["--profile", "runner-codex", "--environment", "daytona"])).some(entry => entry.id === id)).toBe(false);
|
||||
});
|
||||
|
||||
it("keeps private repository skill delivery explicit and bounded in both environments", () => {
|
||||
const cases = runnerMatrix.filter(entry => entry.suite.id === "repository-skills");
|
||||
expect(cases).toHaveLength(2);
|
||||
@@ -159,10 +176,10 @@ describe("runner E2E catalog", () => {
|
||||
expect(localIntegrityTasks).toHaveLength(2);
|
||||
expect(openRouterBreadthTasks).toHaveLength(3);
|
||||
expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
|
||||
3, 4, 8, 10, 4, 4, 6, 63, 12, 6, 2, 30, 3, 16, 16, 2, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1, 116, 2,
|
||||
3, 4, 8, 10, 4, 4, 6, 63, 12, 6, 2, 30, 3, 16, 16, 2, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 13, 10, 48, 16, 10, 2, 1, 1, 116, 2,
|
||||
]);
|
||||
expect(validateRunnerCatalog()).toHaveLength(729);
|
||||
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(729);
|
||||
expect(validateRunnerCatalog()).toHaveLength(730);
|
||||
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(730);
|
||||
expect(
|
||||
runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
|
||||
).toHaveLength(48);
|
||||
|
||||
@@ -101,8 +101,6 @@ function commonAgent(
|
||||
capabilities:
|
||||
"Completes deterministic standard, planning, and ask-mode runner acceptance tasks.",
|
||||
adapterType,
|
||||
// These are explicit legacy regression controls. Ordinary Codex creation
|
||||
// defaults are exercised through the production onboarding wizard below.
|
||||
...(adapterType === "codex_local" ? { runner: "legacy" as const } : {}),
|
||||
adapterConfig,
|
||||
defaultEnvironmentId: input.environmentId,
|
||||
@@ -1498,10 +1496,14 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
|
||||
groups: ["chat", "native"],
|
||||
profiles: runnerProfiles.filter(profile => ["runner-codex", "runner-acpx-claude"].includes(profile.id))
|
||||
.map(profile => productionStoryProfile(defaultPermissionProfile(profile))),
|
||||
environments: [localEnvironment],
|
||||
environments: [localEnvironment, runnerEnvironments.find(environment => environment.id === "daytona")!],
|
||||
excludedExecutionIds: ["runner-codex", "runner-acpx-claude"].flatMap(profile =>
|
||||
[...firstTaskTasks.filter(task => ["task-reply-accept", "interview-plan-accept", "reject-no-execution", "task-card-accept"].includes(task.id)), ...chatConfirmationTasks]
|
||||
.map(task => `confirmation-replies.${profile}.daytona.${task.id}`)
|
||||
.filter(id => id !== "confirmation-replies.runner-codex.daytona.task-card-accept")),
|
||||
tasks: [...firstTaskTasks.filter(task => ["task-reply-accept", "interview-plan-accept", "reject-no-execution", "task-card-accept"].includes(task.id)), ...chatConfirmationTasks],
|
||||
expectedMatrixSize: 12,
|
||||
definitionMetadata: { version: 6, instructions: "production", grading: "card-message-provenance-before-child-creation", unansweredQuestionComposer: "history-card-only-including-fresh-dismissal", completionObservation: "120-seconds", scheduling: "explicit-only" },
|
||||
expectedMatrixSize: 13,
|
||||
definitionMetadata: { version: 7, managedOnboarding: "ui-owned-qualification-and-physical-file", managedExpectedTaskRuns: 4, managedAutomaticRetries: 0, instructions: "production", grading: "card-message-provenance-before-child-creation", unansweredQuestionComposer: "history-card-only-including-fresh-dismissal", completionObservation: "120-seconds", scheduling: "explicit-only" },
|
||||
},
|
||||
{
|
||||
id: "completion-updates", label: "Delegated Completion Updates", manualOnly: true,
|
||||
@@ -1679,8 +1681,10 @@ export function buildRunnerMatrix(
|
||||
suiteDefinitionHash: suiteDefinitionHash(suite),
|
||||
profile,
|
||||
environment,
|
||||
task: suite.id === "extended-harnesses" && profile.qualificationCandidate === "pi" && task.id === "file-edit-validate"
|
||||
? { ...task, buildPrompt: piFilePrompt } : task,
|
||||
task: suite.id === "confirmation-replies" && profile.id === "runner-codex" && environment.id === "daytona" && task.id === "task-card-accept"
|
||||
? { ...task, expectedRunCount: 4, automaticRetryPolicy: "single_attempt" as const }
|
||||
: suite.id === "extended-harnesses" && profile.qualificationCandidate === "pi" && task.id === "file-edit-validate"
|
||||
? { ...task, buildPrompt: piFilePrompt } : task,
|
||||
groups: [
|
||||
...new Set([
|
||||
...suite.groups,
|
||||
|
||||
@@ -1,17 +1,23 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { runsCompletionUpdateProbe, COMPLETION_QUALITY_CONFIG, completionQualityControls, completionQualityStatus, completionQualityRequest, judgeCompletionQuality, reserveCompletionQuality, validateCompletionQuality } from "./completion-quality.js";
|
||||
import { runnerMatrix } from "./catalog.js";
|
||||
import { MANAGED_FIRST_TASK_CELL } from "./first-task-fixtures.js";
|
||||
const observation = { sourceId: "chat", marker: "REF", worker: { id: "task", title: "Welcome", status: "done", completedAt: "2026-09-01T00:00:00Z" }, documents: [{ id: "doc", issueId: "task", key: "welcome", body: "Welcome to the garden. Meet at 10:30." }], comments: [{ id: "reply", issueId: "chat", authorAgentId: "agent", createdAt: "2026-09-01T00:01:00Z", body: "The note is ready at /issues/task", createdByRunId: "run" }], runs: [] };
|
||||
const criteria = Object.keys(COMPLETION_QUALITY_CONFIG.rubric).map(id => ({ id, passed: true, rationale: "Supported by the saved note and reply", evidenceIds: ["reply", "doc"] }));
|
||||
const reports = [{ replyId: "reply", rationale: "Reports the completed task", completedTaskIdsReferenced: ["task"], resultAccessTaskIds: ["task"], correctsReplyIds: [] as string[] }];
|
||||
describe("completion semantic qualification", () => {
|
||||
it.each(["runner-codex", "runner-acpx-claude"])("requires a completion judge only for accepted work on %s", profile => {
|
||||
const cells = runnerMatrix.filter(e => e.suite.id === "confirmation-replies" && e.profile.id === profile);
|
||||
const cells = runnerMatrix.filter(e => e.suite.id === "confirmation-replies" && e.profile.id === profile && e.environment.id === "local");
|
||||
expect(cells).toHaveLength(6);
|
||||
expect(cells.filter(runsCompletionUpdateProbe).map(e => e.task.id).sort()).toEqual([
|
||||
"interview-plan-accept", "task-card-accept", "task-reply-accept",
|
||||
]);
|
||||
});
|
||||
it("requires the existing completion judge for accepted managed onboarding work", () => {
|
||||
const managed = runnerMatrix.find(e => e.id === MANAGED_FIRST_TASK_CELL);
|
||||
expect(managed).toBeDefined();
|
||||
expect(runsCompletionUpdateProbe(managed!)).toBe(true);
|
||||
});
|
||||
it("retains the completion suite's gate and excludes ordinary chat journeys", () => {
|
||||
const completion = runnerMatrix.filter(e => e.suite.id === "completion-updates");
|
||||
expect(completion).toHaveLength(10);
|
||||
|
||||
@@ -3,7 +3,7 @@ import { describe, expect, it, vi } from "vitest";
|
||||
import { gradeConfirmationReply, assertAmbiguousReplyUnresolved, ambiguousConfirmationFixtures } from "./confirmation-replies.js";
|
||||
import { firstTaskScenario } from "./first-task-cases.js";
|
||||
import type { FirstTaskEvidence } from "./first-task-scoring.js";
|
||||
import { provisionFirstTaskFixtures } from "./first-task-fixtures.js";
|
||||
import { MANAGED_FIRST_TASK_CELL, provisionFirstTaskFixtures } from "./first-task-fixtures.js";
|
||||
import { runnerMatrix } from "./catalog.js";
|
||||
import { parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js";
|
||||
|
||||
@@ -111,7 +111,7 @@ describe("confirmation-reply independent oracle", () => {
|
||||
payload: { questions: [{ prompt, options: [{ label: "Welcome note" }, { label: "Poster" }] }] },
|
||||
}] })).toThrow();
|
||||
});
|
||||
it.each(runnerMatrix.filter(e => e.suite.id === "confirmation-replies" && e.task.flow === "first_task"))("provisions the real onboarding fixture contract for $id", async execution => {
|
||||
it.each(runnerMatrix.filter(e => e.suite.id === "confirmation-replies" && e.task.flow === "first_task" && e.environment.id === "local"))("provisions the real local onboarding fixture contract for $id", async execution => {
|
||||
const get = vi.fn().mockResolvedValue([{ id: "local", driver: "local" }]);
|
||||
const patch = vi.fn().mockResolvedValue({ id: "company" });
|
||||
const postSensitive = vi.fn().mockResolvedValue({ id: "secret" });
|
||||
@@ -137,12 +137,14 @@ describe("confirmation-reply independent oracle", () => {
|
||||
originalIds: ["a", "b"], tasks: [], reply: "", agentId: "agent", answerId: "reply" })).toThrow();
|
||||
});
|
||||
|
||||
it("selects exactly twelve explicit-only cases using production native profiles", () => {
|
||||
it("selects twelve local and one managed explicit-only cases using production native profiles", () => {
|
||||
const cells = runnerMatrix.filter(e => e.suite.id === "confirmation-replies");
|
||||
expect(cells).toHaveLength(12);
|
||||
expect(cells).toHaveLength(13);
|
||||
expect(cells.filter(c => c.environment.id === "local")).toHaveLength(12);
|
||||
expect(cells.filter(c => c.environment.id === "daytona").map(c => c.id)).toEqual([MANAGED_FIRST_TASK_CELL]);
|
||||
expect(new Set(cells.map(c => c.profile.id))).toEqual(new Set(["runner-codex", "runner-acpx-claude"]));
|
||||
expect(new Set(cells.map(c => c.task.id)).size).toBe(6);
|
||||
expect(selectRunnerExecutions(parseRunnerSelectors(["--suite", "confirmation-replies"]))).toHaveLength(12);
|
||||
expect(selectRunnerExecutions(parseRunnerSelectors(["--suite", "confirmation-replies"]))).toHaveLength(13);
|
||||
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(c => c.suite.id === "confirmation-replies")).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,21 +1,70 @@
|
||||
import { normalizePaperclipOperationalSkillPreference } from "../../packages/adapter-utils/src/server-utils.js";
|
||||
import { isDeepStrictEqual } from "node:util";
|
||||
import type { RunnerApi } from "./api.js";
|
||||
import type { LiveFixtureValues } from "./live-fixtures.js";
|
||||
import { deleteDaytonaEnvironment, installFixtureDaytonaProvider, type LiveFixtureValues } from "./live-fixtures.js";
|
||||
import { isImmutableDaytonaImage } from "./catalog.js";
|
||||
import type { CredentialName, MatrixExecution } from "./types.js";
|
||||
|
||||
/** Provision credentials and a bounded budget for the UI-created company. The production wizard
|
||||
* remains the sole creator/configurer of its first agent and onboarding task. */
|
||||
export const MANAGED_FIRST_TASK_CELL = "confirmation-replies.runner-codex.daytona.task-card-accept";
|
||||
export function isManagedFirstTask(execution: Pick<MatrixExecution, "id">): boolean {
|
||||
return execution.id === MANAGED_FIRST_TASK_CELL;
|
||||
}
|
||||
|
||||
interface FirstTaskRuntimeBound {
|
||||
timeoutSec: 180;
|
||||
maxConcurrentRuns: 1;
|
||||
before: Record<string, unknown>;
|
||||
after: Record<string, unknown>;
|
||||
}
|
||||
|
||||
export type FirstTaskFixtureValues = LiveFixtureValues & {
|
||||
onboardingRuntimeBound?: FirstTaskRuntimeBound;
|
||||
};
|
||||
|
||||
export async function startFirstTaskWithRuntimeBound(input: {
|
||||
api: Pick<RunnerApi, "get" | "patch">;
|
||||
execution: MatrixExecution;
|
||||
agent: Record<string, any>;
|
||||
observe: (bound: FirstTaskRuntimeBound) => void;
|
||||
start: () => Promise<void>;
|
||||
}): Promise<void> {
|
||||
if (input.execution.profile.id === "runner-codex") {
|
||||
const snapshot = (agent: Record<string, any>) => structuredClone(Object.fromEntries([
|
||||
"id", "companyId", "adapterType", "adapterConfig", "runtimeConfig", "permissions", "defaultEnvironmentId", "budgetMonthlyCents", "metadata",
|
||||
].map(key => [key, agent?.[key]]))) as Record<string, any>;
|
||||
const before = snapshot(input.agent);
|
||||
if (!before.id || !before.companyId || before.adapterType !== "paperclip_runner" || before.adapterConfig?.provider !== "codex") {
|
||||
throw new Error("First-task runtime bound requires the wizard's original native Codex selection");
|
||||
}
|
||||
const agentPath = `/api/agents/${encodeURIComponent(before.id)}`;
|
||||
if (!isDeepStrictEqual(snapshot(await input.api.get(agentPath)), before)) {
|
||||
throw new Error("The automatic wizard configuration changed before the first-task runtime bound");
|
||||
}
|
||||
const runtimeConfig = { ...before.runtimeConfig, heartbeat: { ...before.runtimeConfig?.heartbeat, maxConcurrentRuns: 1 } };
|
||||
await input.api.patch(agentPath, { adapterConfig: { timeoutSec: 180 }, runtimeConfig });
|
||||
const after = snapshot(await input.api.get(agentPath));
|
||||
const expected = { ...before, adapterConfig: { ...before.adapterConfig, timeoutSec: 180 }, runtimeConfig };
|
||||
if (!isDeepStrictEqual(after, expected)) {
|
||||
throw new Error("The first-task runtime bound was not confirmed with the original execution and account settings");
|
||||
}
|
||||
input.observe({ timeoutSec: 180, maxConcurrentRuns: 1, before, after });
|
||||
}
|
||||
await input.start();
|
||||
}
|
||||
|
||||
export async function provisionFirstTaskFixtures(input: {
|
||||
api: Pick<RunnerApi, "get" | "patch" | "postSensitive">;
|
||||
api: Pick<RunnerApi, "get" | "patch" | "postSensitive"> & Partial<Pick<RunnerApi, "post" | "delete">>;
|
||||
execution: MatrixExecution;
|
||||
nonce: string;
|
||||
daytonaImage?: string;
|
||||
company: LiveFixtureValues["company"];
|
||||
credentials: Partial<Record<CredentialName, string>>;
|
||||
}): Promise<LiveFixtureValues> {
|
||||
}): Promise<FirstTaskFixtureValues> {
|
||||
const { api, execution, nonce, company, credentials } = input;
|
||||
const managed = isManagedFirstTask(execution);
|
||||
if (
|
||||
!["first-task", "completion-updates", "confirmation-replies"].includes(execution.suite.id) || execution.task.flow !== "first_task" ||
|
||||
execution.environment.id !== "local" ||
|
||||
(execution.environment.id !== "local" && !managed) ||
|
||||
![
|
||||
"legacy-codex",
|
||||
"legacy-claude",
|
||||
@@ -27,42 +76,89 @@ export async function provisionFirstTaskFixtures(input: {
|
||||
"First-task fixtures require a supported local onboarding profile",
|
||||
);
|
||||
}
|
||||
const credential = execution.profile.credential;
|
||||
const value = credentials[credential];
|
||||
if (!value) throw new Error(`Missing credential ${credential}`);
|
||||
if (managed && (!isImmutableDaytonaImage(input.daytonaImage) || !api.post || !api.delete)) {
|
||||
throw new Error("Managed onboarding requires the immutable image and normal environment create/cleanup APIs");
|
||||
}
|
||||
const required = managed ? [execution.profile.credential, "DAYTONA_API_KEY" as const] : [execution.profile.credential];
|
||||
for (const name of required) if (!credentials[name]) throw new Error(`Missing credential ${name}`);
|
||||
const previousSettings = managed ? await api.get<{
|
||||
defaultEnvironmentId: string | null;
|
||||
experimental: { enableManagedSandboxOnly: boolean };
|
||||
}>("/api/instance/settings") : null;
|
||||
if (managed && (!previousSettings ||
|
||||
(previousSettings.defaultEnvironmentId !== null && typeof previousSettings.defaultEnvironmentId !== "string") ||
|
||||
typeof previousSettings.experimental?.enableManagedSandboxOnly !== "boolean")) {
|
||||
throw new Error("Managed onboarding fixture prior instance settings could not be read");
|
||||
}
|
||||
await api.patch(`/api/companies/${company.id}`, { budgetMonthlyCents: 500 });
|
||||
const environments = await api.get<Array<LiveFixtureValues["environment"]>>(
|
||||
`/api/companies/${company.id}/environments?driver=local`,
|
||||
);
|
||||
const environment = environments.find((e) => e.driver === "local");
|
||||
if (!environment)
|
||||
throw new Error("Onboarding fixture local environment missing");
|
||||
const secret = await api.postSensitive<{ id: string }>(
|
||||
`/api/companies/${company.id}/secrets`,
|
||||
{
|
||||
name: `First task ${credential}`,
|
||||
key: credential,
|
||||
value,
|
||||
},
|
||||
);
|
||||
let localEnvironment: LiveFixtureValues["environment"] | undefined;
|
||||
if (!managed) {
|
||||
const environments = await api.get<Array<LiveFixtureValues["environment"]>>(`/api/companies/${company.id}/environments?driver=local`);
|
||||
localEnvironment = environments.find(candidate => candidate.driver === "local");
|
||||
if (!localEnvironment) throw new Error("Onboarding fixture local environment missing");
|
||||
}
|
||||
const secretRefs: LiveFixtureValues["secretRefs"] = {};
|
||||
for (const name of required) {
|
||||
const secret = await api.postSensitive<{ id: string }>(`/api/companies/${company.id}/secrets`, {
|
||||
name: `First task ${name}`, key: name, value: credentials[name],
|
||||
});
|
||||
secretRefs[name] = { type: "secret_ref", secretId: secret.id, version: "latest" };
|
||||
}
|
||||
let environment: LiveFixtureValues["environment"];
|
||||
let teardown = async () => {};
|
||||
if (managed) {
|
||||
const remoteApi = api as Pick<RunnerApi, "post" | "delete">;
|
||||
await installFixtureDaytonaProvider(remoteApi, execution);
|
||||
environment = await remoteApi.post<LiveFixtureValues["environment"]>(`/api/companies/${company.id}/environments`,
|
||||
execution.environment.buildEnvironment({ secretRefs, daytonaImage: input.daytonaImage, executionId: nonce }));
|
||||
let restoreDefault = false;
|
||||
let restoreManagedOnly = false;
|
||||
let environmentDeleted = false;
|
||||
teardown = async () => {
|
||||
const failures: unknown[] = [];
|
||||
if (restoreManagedOnly) {
|
||||
try {
|
||||
await api.patch("/api/instance/settings/experimental", { enableManagedSandboxOnly: previousSettings!.experimental.enableManagedSandboxOnly });
|
||||
restoreManagedOnly = false;
|
||||
} catch (error) { failures.push(error); }
|
||||
}
|
||||
if (restoreDefault) {
|
||||
try {
|
||||
await api.patch("/api/instance/settings", { defaultEnvironmentId: previousSettings!.defaultEnvironmentId });
|
||||
restoreDefault = false;
|
||||
} catch (error) { failures.push(error); }
|
||||
}
|
||||
if (!restoreDefault && !environmentDeleted) {
|
||||
try {
|
||||
await deleteDaytonaEnvironment(remoteApi, environment.id);
|
||||
environmentDeleted = true;
|
||||
} catch (error) { failures.push(error); }
|
||||
}
|
||||
if (failures.length) throw new AggregateError(failures, restoreDefault
|
||||
? "Managed onboarding fixture cleanup failed; instance default restoration is unconfirmed and environment deletion was not attempted"
|
||||
: "Managed onboarding fixture cleanup failed");
|
||||
};
|
||||
try {
|
||||
restoreDefault = true;
|
||||
await api.patch("/api/instance/settings", { defaultEnvironmentId: environment.id });
|
||||
restoreManagedOnly = true;
|
||||
await api.patch("/api/instance/settings/experimental", { enableManagedSandboxOnly: true });
|
||||
} catch (error) {
|
||||
try { await teardown(); } catch (cleanup) { throw new AggregateError([error, cleanup], "Managed onboarding fixture setup and cleanup failed"); }
|
||||
throw error;
|
||||
}
|
||||
} else environment = localEnvironment!;
|
||||
|
||||
return {
|
||||
company,
|
||||
environment,
|
||||
secretRefs: {
|
||||
[credential]: {
|
||||
type: "secret_ref",
|
||||
secretId: secret.id,
|
||||
version: "latest",
|
||||
},
|
||||
},
|
||||
secretRefs,
|
||||
agent: { id: "", name: `Garden lead ${nonce}`, companyId: company.id },
|
||||
teardown: async () => {}, // Existing launcher removes the complete isolated instance.
|
||||
...(managed ? { setupLeases: [], setupQualification: { recorded: false, adapterType: null, platformResponses: 0, adapterResponses: 0 } } : {}),
|
||||
teardown,
|
||||
};
|
||||
}
|
||||
|
||||
/** Reuse the qualified runtime configuration for explicit regression switches,
|
||||
* never the QA persona from buildAgent. Codex default qualification does not
|
||||
* call this helper; its wizard must persist the native selection directly. */
|
||||
export function firstTaskNativeRuntimePatch(
|
||||
execution: MatrixExecution,
|
||||
fixtures: LiveFixtureValues,
|
||||
|
||||
@@ -9,9 +9,12 @@ import { waitForFirstTaskReply } from "./first-task-replies.js";
|
||||
import { observeCompletionUpdate } from "./completion-update-flow.js";
|
||||
import {
|
||||
provisionFirstTaskFixtures,
|
||||
isManagedFirstTask,
|
||||
startFirstTaskWithRuntimeBound,
|
||||
type FirstTaskFixtureValues,
|
||||
} from "./first-task-fixtures.js";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { expect, type Page } from "@playwright/test";
|
||||
import { expect, type Page, type Request, type Response } from "@playwright/test";
|
||||
import { readFile } from "node:fs/promises";
|
||||
import { pollUntil, type RunnerApi } from "./api.js";
|
||||
import {
|
||||
@@ -33,8 +36,6 @@ import {
|
||||
type Row,
|
||||
} from "./first-task-scoring.js";
|
||||
|
||||
/** Results enter the public history bundle. Full observations stay in the
|
||||
* access-controlled snapshots and are graded before this projection. */
|
||||
export function projectFirstTaskResult(result: RunnerE2EResult): RunnerE2EResult {
|
||||
const e = result.firstTask;
|
||||
if (!e) return result;
|
||||
@@ -144,16 +145,138 @@ export function projectFirstTaskResult(result: RunnerE2EResult): RunnerE2EResult
|
||||
};
|
||||
}
|
||||
|
||||
/** The wizard creates the agent/task. Only credential provisioning uses Node fetch,
|
||||
* keeping plaintext keys out of Playwright's trace and recorded form values. */
|
||||
export function managedFirstTaskFileCheck(e: FirstTaskEvidence) {
|
||||
const last = e.checkpoints.at(-1);
|
||||
const children = last?.tasks.filter(task => task.parentId === e.onboardingIssueId && task.status === "done") ?? [];
|
||||
const marker = firstTaskScenario(e.caseId, e.nonce).marker;
|
||||
const passed = children.length === 1 && (last?.attachments ?? []).some(attachment =>
|
||||
attachment.issueId === children[0]!.id && attachment.originalFilename === "garden-welcome.txt"
|
||||
&& /^text\/(?:plain|markdown)(?:;|$)/i.test(String(attachment.contentType))
|
||||
&& attachment.contentVerified === true && typeof attachment.body === "string"
|
||||
&& attachment.contentSha256 === digestText(attachment.body) && attachment.body.includes(marker));
|
||||
return { id: "managed-onboarding-physical-artifact", passed, evidence: last ? [last.id] : [],
|
||||
detail: "The completed child has a downloaded, size/hash-verified garden-welcome.txt attachment with the requested marker." };
|
||||
}
|
||||
|
||||
export async function observeManagedFirstTaskSetup(page: Page, api: RunnerApi, fixtures: LiveFixtureValues, deadlineAt: number) {
|
||||
const before = new Set((await api.get<Row[]>(`/api/environments/${fixtures.environment.id}/leases`)).map(lease => lease.id));
|
||||
const startedAt = Date.now();
|
||||
const qualification = fixtures.setupQualification!;
|
||||
const operations: Promise<void>[] = [];
|
||||
let rejectFailure!: (error: unknown) => void;
|
||||
const failure = new Promise<never>((_resolve, reject) => { rejectFailure = reject; });
|
||||
void failure.catch(() => undefined);
|
||||
const requestStarts = new WeakMap<Request, number>();
|
||||
const pendingSetupRequests = new Set<Request>();
|
||||
let platformEndedAt = Infinity;
|
||||
let qualifiedDefault = false;
|
||||
let automaticRequest = false;
|
||||
function phase(request: Request): "platform" | "adapter" | "discovery" | null {
|
||||
const url = new URL(request.url());
|
||||
if (request.method() === "POST" && url.pathname === `/api/environments/${fixtures.environment.id}/probe`
|
||||
&& url.searchParams.get("companyId") === fixtures.company.id) return "platform";
|
||||
if (request.method() === "POST" && url.pathname === `/api/companies/${fixtures.company.id}/adapters/codex_local/test-environment`) return "adapter";
|
||||
if (request.method() === "GET" && url.pathname === "/api/adapters" && url.searchParams.get("companyId") === fixtures.company.id
|
||||
&& url.searchParams.get("environmentId") === fixtures.environment.id) return "discovery";
|
||||
return null;
|
||||
}
|
||||
const onRequest = (request: Request) => {
|
||||
const kind = phase(request);
|
||||
if (kind) requestStarts.set(request, Date.now());
|
||||
if (kind === "platform" || kind === "adapter") pendingSetupRequests.add(request);
|
||||
};
|
||||
const onRequestFailed = (request: Request) => {
|
||||
if (pendingSetupRequests.has(request)) fixtures.setupLeaseAccountingIncomplete = true;
|
||||
};
|
||||
const onResponse = (response: Response) => {
|
||||
const request = response.request();
|
||||
const kind = phase(request);
|
||||
if (!kind) return;
|
||||
const responseAt = Date.now();
|
||||
if (kind === "platform") { qualification.platformResponses++; platformEndedAt = responseAt; }
|
||||
if (kind === "adapter") qualification.adapterResponses++;
|
||||
if (qualification.platformResponses > 1 || qualification.adapterResponses > 1) rejectFailure(new Error("The UI repeated a managed setup request"));
|
||||
const operation = (async () => {
|
||||
const body = await response.json() as Row | Row[];
|
||||
if (kind === "discovery") {
|
||||
if (Array.isArray(body) && responseAt >= platformEndedAt) {
|
||||
const codex = body.find(adapter => adapter.type === "codex_local");
|
||||
qualifiedDefault = codex?.defaultRunner === "paperclip" && codex?.supportedRunners?.includes("paperclip") === true;
|
||||
if (!qualifiedDefault) rejectFailure(new Error("The selected image is not qualified for native Codex onboarding"));
|
||||
}
|
||||
return;
|
||||
}
|
||||
const data = body as Row;
|
||||
if (kind === "platform") {
|
||||
qualification.recorded = response.ok() && data.ok === true && data.details?.platformQualificationRecorded === true;
|
||||
if (!qualification.recorded) rejectFailure(new Error("The UI environment qualification did not pass"));
|
||||
}
|
||||
else {
|
||||
qualification.adapterType = response.ok() && data.status === "pass" ? data.adapterType : null;
|
||||
if (qualification.adapterType !== "paperclip_runner") rejectFailure(new Error("The UI native runtime setup did not pass"));
|
||||
automaticRequest = request.postDataJSON()?.runner === undefined && (requestStarts.get(request) ?? 0) >= platformEndedAt;
|
||||
}
|
||||
const leaseId = kind === "platform" ? data.details?.leaseId
|
||||
: /(?:^|;\s*)paperclipLeaseId=([a-zA-Z0-9-]+)(?:;|$)/.exec(String(data.checks?.find((check: Row) => check.code === "sandbox_test_identity")?.detail ?? ""))?.[1];
|
||||
if (typeof leaseId !== "string" || !leaseId) {
|
||||
fixtures.setupLeaseAccountingIncomplete = true;
|
||||
return;
|
||||
}
|
||||
const lease = await api.get<Row>(`/api/environment-leases/${encodeURIComponent(leaseId)}`, { timeout: 10_000 });
|
||||
if (before.has(leaseId) || lease.companyId !== fixtures.company.id || lease.environmentId !== fixtures.environment.id
|
||||
|| lease.provider !== "daytona" || lease.heartbeatRunId || lease.issueId) throw new Error("Setup response references a foreign or non-setup lease");
|
||||
pendingSetupRequests.delete(request);
|
||||
if (!fixtures.setupLeases!.some(entry => entry.leaseId === leaseId)) fixtures.setupLeases!.push({ leaseId, phase: kind,
|
||||
durationMs: responseAt - (requestStarts.get(request) ?? startedAt) });
|
||||
})();
|
||||
void operation.catch(error => { fixtures.setupLeaseAccountingIncomplete = true; rejectFailure(error); });
|
||||
operations.push(operation);
|
||||
};
|
||||
page.on("request", onRequest);
|
||||
page.on("response", onResponse);
|
||||
page.on("requestfailed", onRequestFailed);
|
||||
const stop = async (requireSuccess: boolean) => {
|
||||
page.off("request", onRequest);
|
||||
page.off("response", onResponse);
|
||||
page.off("requestfailed", onRequestFailed);
|
||||
const settled = await Promise.allSettled(operations);
|
||||
if (pendingSetupRequests.size > 0) fixtures.setupLeaseAccountingIncomplete = true;
|
||||
const failures = settled.flatMap(result => result.status === "rejected" ? [result.reason] : []);
|
||||
let leases: Row[];
|
||||
try { leases = await api.get<Row[]>(`/api/environments/${fixtures.environment.id}/leases`, { timeout: 10_000 }); }
|
||||
catch (error) { fixtures.setupLeaseAccountingIncomplete = true; throw error; }
|
||||
for (const lease of leases) {
|
||||
if (before.has(lease.id) || lease.companyId !== fixtures.company.id || lease.environmentId !== fixtures.environment.id
|
||||
|| lease.provider !== "daytona" || lease.heartbeatRunId || lease.issueId) continue;
|
||||
if (!fixtures.setupLeases!.some(entry => entry.leaseId === lease.id)) fixtures.setupLeaseAccountingIncomplete = true;
|
||||
}
|
||||
if (failures.length) throw new AggregateError(failures, "Managed onboarding setup evidence could not be verified");
|
||||
if (!requireSuccess) return;
|
||||
if (!qualification.recorded || qualification.platformResponses !== 1 || qualification.adapterResponses !== 1
|
||||
|| qualification.adapterType !== "paperclip_runner" || !automaticRequest || !qualifiedDefault
|
||||
|| fixtures.setupLeaseAccountingIncomplete || fixtures.setupLeases!.length !== 2 || !["platform", "adapter"].every(kind => fixtures.setupLeases!.some(lease => lease.phase === kind))) {
|
||||
throw new Error("The UI did not prove one qualified automatic native setup with both owned leases");
|
||||
}
|
||||
await pollUntil({ label: "managed onboarding setup lease cleanup", deadlineAt: Math.min(deadlineAt, Date.now() + 120_000),
|
||||
load: () => api.get<Row[]>(`/api/environments/${fixtures.environment.id}/leases`),
|
||||
accept: rows => fixtures.setupLeases!.every(entry => rows.some(row => row.id === entry.leaseId
|
||||
&& row.companyId === fixtures.company.id && row.environmentId === fixtures.environment.id
|
||||
&& row.status === "released" && row.cleanupStatus === "success")),
|
||||
});
|
||||
};
|
||||
return Object.assign(stop, { failure });
|
||||
}
|
||||
|
||||
export async function setupFirstTaskFixtures(input: {
|
||||
page: Page;
|
||||
api: RunnerApi;
|
||||
execution: MatrixExecution;
|
||||
nonce: string;
|
||||
credentials: Partial<Record<CredentialName, string>>;
|
||||
daytonaImage?: string;
|
||||
deadlineAt?: number;
|
||||
observe: (fixtures: LiveFixtureValues) => void;
|
||||
}): Promise<LiveFixtureValues> {
|
||||
}): Promise<FirstTaskFixtureValues> {
|
||||
const { page, api, execution, nonce } = input;
|
||||
await api.patch("/api/instance/settings/experimental", {
|
||||
enableClassicTaskInterface: false,
|
||||
@@ -178,6 +301,7 @@ export async function setupFirstTaskFixtures(input: {
|
||||
execution,
|
||||
nonce,
|
||||
credentials: input.credentials,
|
||||
daytonaImage: input.daytonaImage,
|
||||
company: {
|
||||
id: company.id,
|
||||
name: company.name,
|
||||
@@ -215,13 +339,20 @@ export async function setupFirstTaskFixtures(input: {
|
||||
await page
|
||||
.getByRole("combobox", { name: "Saved API key" })
|
||||
.selectOption(`company:${secret.secretId}`);
|
||||
await page
|
||||
.getByRole("button", { name: /^(Connect|Next)$/, exact: true })
|
||||
.last()
|
||||
.click();
|
||||
await expect(
|
||||
page.getByRole("button", { name: "Get started", exact: true }),
|
||||
).toBeVisible({ timeout: 120_000 });
|
||||
const managed = isManagedFirstTask(execution);
|
||||
if (managed && (!input.deadlineAt || input.deadlineAt - Date.now() <= 120_000)) throw new Error("Managed onboarding has no bounded setup deadline");
|
||||
const stopSetup = managed ? await observeManagedFirstTaskSetup(page, api, fixtures, input.deadlineAt! - 60_000) : null;
|
||||
let setupError: unknown;
|
||||
try {
|
||||
await page.getByRole("button", { name: /^(Connect|Next)$/, exact: true }).last().click();
|
||||
const visible = expect(page.getByRole("button", { name: "Get started", exact: true })).toBeVisible({
|
||||
timeout: managed ? input.deadlineAt! - Date.now() - 120_000 : 120_000,
|
||||
});
|
||||
if (stopSetup) await Promise.race([visible, stopSetup.failure]); else await visible;
|
||||
} catch (error) { setupError = error; }
|
||||
try { await stopSetup?.(!setupError); }
|
||||
catch (error) { setupError = setupError ? new AggregateError([setupError, error], "Managed onboarding and setup evidence failed") : error; }
|
||||
if (setupError) throw setupError;
|
||||
const agents = await api.get<Row[]>(`/api/companies/${company.id}/agents`);
|
||||
expect(agents).toHaveLength(1);
|
||||
const wizardAdapter =
|
||||
@@ -249,16 +380,27 @@ export async function setupFirstTaskFixtures(input: {
|
||||
name: agents[0].name,
|
||||
};
|
||||
input.observe(fixtures);
|
||||
await page.getByRole("button", { name: "Get started", exact: true }).click();
|
||||
if (managed) {
|
||||
await page.reload({ waitUntil: "domcontentloaded" });
|
||||
const persisted = await api.get<Row>(`/api/agents/${fixtures.agent.id}`);
|
||||
expect(persisted.adapterType).toBe("paperclip_runner");
|
||||
expect(persisted.adapterConfig?.provider).toBe("codex");
|
||||
}
|
||||
await startFirstTaskWithRuntimeBound({
|
||||
api, execution, agent: agents[0],
|
||||
observe: bound => { fixtures.onboardingRuntimeBound = bound; input.observe(fixtures); },
|
||||
start: () => page.getByRole("button", { name: "Get started", exact: true }).click(),
|
||||
});
|
||||
return fixtures;
|
||||
}
|
||||
|
||||
export async function runFirstTaskFlow(input: {
|
||||
page: Page;
|
||||
api: RunnerApi;
|
||||
fixtures: LiveFixtureValues;
|
||||
fixtures: FirstTaskFixtureValues;
|
||||
execution: MatrixExecution;
|
||||
nonce: string;
|
||||
deadlineAt?: number;
|
||||
observe: (issue: any, runs: any[], evidence: FirstTaskEvidence) => void;
|
||||
secrets: readonly string[];
|
||||
createOrdinary: (title: string, prompt: string) => Promise<Row>;
|
||||
@@ -267,8 +409,14 @@ export async function runFirstTaskFlow(input: {
|
||||
}) {
|
||||
const { page, api, fixtures, execution, nonce } = input;
|
||||
const scenario = firstTaskScenario(execution.task.id, nonce);
|
||||
const deadlineAt =
|
||||
Date.now() + execution.task.attemptTimeoutMs.local - 120_000;
|
||||
const managed = isManagedFirstTask(execution);
|
||||
if (managed) {
|
||||
const attachment = "Attach the finished note as a UTF-8 text file named garden-welcome.txt.";
|
||||
scenario.prompt += ` ${attachment}`;
|
||||
scenario.facts += ` ${attachment}`;
|
||||
}
|
||||
const deadlineAt = managed ? (input.deadlineAt ?? Date.now()) - 120_000
|
||||
: Date.now() + execution.task.attemptTimeoutMs.local - 120_000;
|
||||
const tasksPath = `/api/companies/${fixtures.company.id}/issues?limit=100`;
|
||||
const allRuns = async () => {
|
||||
const runs = await api.get<Row[]>(
|
||||
@@ -353,6 +501,7 @@ export async function runFirstTaskFlow(input: {
|
||||
};
|
||||
e.checkpoints.push(checkpoint);
|
||||
e.checks = gradeFirstTask(e);
|
||||
if (managed && e.checkpoints.at(-1)?.phase === "finished") e.checks.push(managedFirstTaskFileCheck(e));
|
||||
if (execution.suite.id === "confirmation-replies" && scenario.id !== "task-card-accept") e.checks.push(...gradeConfirmationReply(e));
|
||||
input.observe(issue, runs, e);
|
||||
await input.evidence("first-task.json", e);
|
||||
@@ -443,6 +592,7 @@ export async function runFirstTaskFlow(input: {
|
||||
e.runtimeSettings = {
|
||||
completionDeliveryProbe: runsCompletionUpdateProbe(execution),
|
||||
onboardingRuntime: fixtures.onboardingRuntime,
|
||||
...(fixtures.onboardingRuntimeBound ? { onboardingRuntimeBound: fixtures.onboardingRuntimeBound } : {}),
|
||||
adapterType: agent.adapterType,
|
||||
adapterConfig: agent.adapterConfig,
|
||||
runtimeConfig: agent.runtimeConfig,
|
||||
@@ -714,6 +864,7 @@ export async function runFirstTaskFlow(input: {
|
||||
await snapshot("finished");
|
||||
}
|
||||
e.checks = gradeFirstTask(e);
|
||||
if (managed && e.checkpoints.at(-1)?.phase === "finished") e.checks.push(managedFirstTaskFileCheck(e));
|
||||
if (execution.suite.id === "confirmation-replies" && scenario.id !== "task-card-accept") e.checks.push(...gradeConfirmationReply(e));
|
||||
await input.evidence("first-task.json", e);
|
||||
await input.capture(
|
||||
|
||||
@@ -6,11 +6,15 @@ import { firstTaskRejectionReplyRecorded, isFirstTaskRejectionCancellation } fro
|
||||
import { createIssueThreadInteractionSchema } from "../../packages/shared/src/validators/issue.js";
|
||||
import { renderInteractionCard } from "./interaction-report.js";
|
||||
import { main as judgeCommand } from "./first-task-judge.js";
|
||||
import { projectFirstTaskResult } from "./first-task-flow.js";
|
||||
import { projectFirstTaskResult, managedFirstTaskFileCheck, observeManagedFirstTaskSetup } from "./first-task-flow.js";
|
||||
import { validateRetainedRunnerResult } from "./result-validation.js";
|
||||
import { buildRuntimeUsage } from "./billing.js";
|
||||
import { fixtureRuntimeCostCoverage, selectFixtureRuntimeLeases } from "./live-fixtures.js";
|
||||
import {
|
||||
firstTaskNativeRuntimePatch,
|
||||
provisionFirstTaskFixtures,
|
||||
startFirstTaskWithRuntimeBound,
|
||||
MANAGED_FIRST_TASK_CELL,
|
||||
} from "./first-task-fixtures.js";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { mkdtemp, mkdir, writeFile, readFile, rm } from "node:fs/promises";
|
||||
@@ -230,6 +234,7 @@ describe("first-task public result projection", () => {
|
||||
adapterConfig: { provider: "codex", model: "gpt-5.6-sol", env: { OPENAI_API_KEY: sentinel }, codexHome: sentinel },
|
||||
runtimeConfig: { accountId: sentinel }, permissions: { privateTrace: sentinel },
|
||||
onboardingRuntime: { mode: "production-wizard", originalAdapterType: "paperclip_runner", testedAdapterType: "paperclip_runner", accountId: sentinel },
|
||||
onboardingRuntimeBound: { timeoutSec: 180, maxConcurrentRuns: 1, before: { runtimeConfig: { accountId: sentinel } }, after: { runtimeConfig: { accountId: sentinel } } },
|
||||
};
|
||||
for (const checkpoint of e.checkpoints) {
|
||||
checkpoint.tasks.forEach(t => Object.assign(t, { executionState: { sessionId: sentinel }, responsibleUserId: sentinel }));
|
||||
@@ -815,6 +820,110 @@ Accept the card above and I write it. This task stays in review until then.`;
|
||||
expect(postSensitive).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
describe("bounded automatic Codex onboarding admission", () => {
|
||||
function harness(environment = "local", profile = "runner-codex") {
|
||||
const execution = runnerMatrix.find(entry => entry.task.flow === "first_task" && entry.profile.id === profile && entry.environment.id === environment)!;
|
||||
const original = {
|
||||
id: "agent", companyId: "company", adapterType: "paperclip_runner",
|
||||
adapterConfig: { provider: "codex", timeoutSec: 0, model: "gpt-5.6-sol", modelReasoningEffort: "high", codexPermissionMode: "readOnly",
|
||||
env: { OPENAI_API_KEY: { type: "secret_ref", secretId: "encrypted-key", version: "latest" } }, paperclipSkillSync: { desiredSkills: ["paperclipai/paperclip/first-task"] } },
|
||||
runtimeConfig: { aiConnection: { provider: "openai", method: "api_key", mode: "responsible_user" },
|
||||
heartbeat: { maxConcurrentRuns: 20, enabled: false, intervalSec: 0, wakeOnAssignment: true, customHeartbeatSetting: "retained" }, customSetting: "retained" },
|
||||
permissions: { canCreateAgents: true }, defaultEnvironmentId: environment, budgetMonthlyCents: 500, metadata: { ownerUserId: "qa-user" },
|
||||
};
|
||||
let stored = structuredClone(original);
|
||||
const calls: string[] = [];
|
||||
const api = {
|
||||
get: vi.fn(async (_path: string) => { calls.push("GET"); return structuredClone(stored); }),
|
||||
patch: vi.fn(async (_path: string, data: any) => {
|
||||
calls.push("PATCH");
|
||||
stored.adapterConfig = { ...stored.adapterConfig, ...data.adapterConfig };
|
||||
if (data.runtimeConfig) stored.runtimeConfig = structuredClone(data.runtimeConfig);
|
||||
return structuredClone(stored);
|
||||
}),
|
||||
};
|
||||
const observe = vi.fn(() => { calls.push("OBSERVE"); });
|
||||
const start = vi.fn(async () => { calls.push("START"); });
|
||||
const admit = () => startFirstTaskWithRuntimeBound({ api: api as any, execution, agent: original, observe, start });
|
||||
return { api, original, calls, observe, start, admit, stored: () => stored };
|
||||
}
|
||||
|
||||
it.each(["local", "daytona"])("bounds %s without choosing a runner, provider, model, or account", async environment => {
|
||||
const h = harness(environment);
|
||||
const original = structuredClone(h.original);
|
||||
const expected = { ...original, adapterConfig: { ...original.adapterConfig, timeoutSec: 180 },
|
||||
runtimeConfig: { ...original.runtimeConfig, heartbeat: { ...original.runtimeConfig.heartbeat, maxConcurrentRuns: 1 } } };
|
||||
await h.admit();
|
||||
expect(h.stored()).toEqual(expected);
|
||||
expect(h.api.patch).toHaveBeenCalledExactlyOnceWith("/api/agents/agent", { adapterConfig: { timeoutSec: 180 }, runtimeConfig: expected.runtimeConfig });
|
||||
expect(h.calls).toEqual(["GET", "PATCH", "GET", "OBSERVE", "START"]);
|
||||
expect(h.original).toEqual(original);
|
||||
expect(h.observe).toHaveBeenCalledExactlyOnceWith({ timeoutSec: 180, maxConcurrentRuns: 1, before: original, after: expected });
|
||||
});
|
||||
|
||||
it.each(["read-failure", "patch-failure", "verification-failure", "missing-timeout", "missing-concurrency"])("does not admit task execution after %s", async mode => {
|
||||
const h = harness();
|
||||
if (mode === "read-failure") h.api.get.mockRejectedValue(new Error("Agent read failed"));
|
||||
if (mode === "patch-failure") h.api.patch.mockRejectedValue(new Error("Timeout patch failed"));
|
||||
if (mode === "verification-failure") h.api.get.mockResolvedValueOnce(structuredClone(h.original)).mockRejectedValueOnce(new Error("Verification read failed"));
|
||||
if (mode === "missing-timeout") h.api.patch.mockImplementation(async (_path, data) => {
|
||||
h.stored().runtimeConfig = structuredClone(data.runtimeConfig);
|
||||
return structuredClone(h.stored());
|
||||
});
|
||||
if (mode === "missing-concurrency") h.api.patch.mockImplementation(async (_path, data) => {
|
||||
h.stored().adapterConfig = { ...h.stored().adapterConfig, ...data.adapterConfig };
|
||||
return structuredClone(h.stored());
|
||||
});
|
||||
await expect(h.admit()).rejects.toThrow();
|
||||
expect(h.start).not.toHaveBeenCalled();
|
||||
expect(h.observe).not.toHaveBeenCalled();
|
||||
if (mode === "read-failure") expect(h.api.patch).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each(["legacy", "wrong-provider", "foreign-agent", "changed-account"])("rejects %s before changing the automatic wizard choice", async mode => {
|
||||
const h = harness();
|
||||
if (mode === "legacy") h.original.adapterType = "codex_local";
|
||||
if (mode === "wrong-provider") h.original.adapterConfig.provider = "claude";
|
||||
if (mode === "foreign-agent") h.api.get.mockResolvedValue({ ...h.original, companyId: "another-company" });
|
||||
if (mode === "changed-account") h.api.get.mockResolvedValue({ ...h.original, runtimeConfig: { ...h.original.runtimeConfig,
|
||||
aiConnection: { ...h.original.runtimeConfig.aiConnection, method: "subscription" } } });
|
||||
await expect(h.admit()).rejects.toThrow();
|
||||
expect(h.api.patch).not.toHaveBeenCalled();
|
||||
expect(h.start).not.toHaveBeenCalled();
|
||||
expect(h.observe).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each(["runner", "provider", "model", "account", "permissions", "environment", "instructions", "heartbeat-policy", "other-runtime", "missing-heartbeat"])("refuses a persisted %s change instead of treating the bound PATCH as proof", async field => {
|
||||
const h = harness();
|
||||
const altered = structuredClone(h.original);
|
||||
altered.adapterConfig.timeoutSec = 180;
|
||||
altered.runtimeConfig.heartbeat.maxConcurrentRuns = 1;
|
||||
if (field === "runner") altered.adapterType = "codex_local";
|
||||
if (field === "provider") altered.adapterConfig.provider = "claude";
|
||||
if (field === "model") altered.adapterConfig.model = "another-model";
|
||||
if (field === "account") altered.runtimeConfig.aiConnection.mode = "fixed";
|
||||
if (field === "permissions") altered.permissions.canCreateAgents = false;
|
||||
if (field === "environment") altered.defaultEnvironmentId = "another-environment";
|
||||
if (field === "instructions") altered.adapterConfig.paperclipSkillSync.desiredSkills = [];
|
||||
if (field === "heartbeat-policy") altered.runtimeConfig.heartbeat.wakeOnAssignment = false;
|
||||
if (field === "other-runtime") altered.runtimeConfig.customSetting = "lost";
|
||||
if (field === "missing-heartbeat") delete (altered.runtimeConfig as Partial<typeof altered.runtimeConfig>).heartbeat;
|
||||
h.api.get.mockResolvedValueOnce(structuredClone(h.original)).mockResolvedValueOnce(altered);
|
||||
await expect(h.admit()).rejects.toThrow("runtime bound was not confirmed");
|
||||
expect(h.start).not.toHaveBeenCalled();
|
||||
expect(h.observe).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each(["legacy-codex", "legacy-claude", "runner-acpx-claude"])("leaves %s admission unchanged", async profile => {
|
||||
const h = harness("local", profile);
|
||||
await h.admit();
|
||||
expect(h.api.get).not.toHaveBeenCalled();
|
||||
expect(h.api.patch).not.toHaveBeenCalled();
|
||||
expect(h.observe).not.toHaveBeenCalled();
|
||||
expect(h.start).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
});
|
||||
|
||||
it.each(["runner-codex", "runner-acpx-claude"])(
|
||||
"switches only the runtime for %s, preserving onboarding assets and the default model",
|
||||
(id) => {
|
||||
@@ -1442,3 +1551,342 @@ describe("recorded refusal cancellation", () => {
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("managed onboarding through the production wizard", () => {
|
||||
const image = `registry.example.test/runtime@sha256:${"a".repeat(64)}`;
|
||||
const execution = () => runnerMatrix.find(entry => entry.id === MANAGED_FIRST_TASK_CELL)!;
|
||||
|
||||
function managedFixtureApi(previousDefault: string | null = "previous-default", previousManagedOnly = false) {
|
||||
const state = { defaultEnvironmentId: previousDefault, experimental: { enableManagedSandboxOnly: previousManagedOnly, enableAgentChat: true } };
|
||||
const calls: string[] = [];
|
||||
const api = {
|
||||
get: vi.fn(async () => structuredClone(state)),
|
||||
patch: vi.fn(async (path: string, data: any) => {
|
||||
calls.push(`PATCH ${path}`);
|
||||
if (path === "/api/instance/settings") state.defaultEnvironmentId = data.defaultEnvironmentId;
|
||||
if (path === "/api/instance/settings/experimental") Object.assign(state.experimental, data);
|
||||
return structuredClone(state);
|
||||
}),
|
||||
postSensitive: vi.fn(async (_path: string, data: any) => ({ id: `encrypted-${data.key}` })),
|
||||
post: vi.fn(async (path: string, data: any) => {
|
||||
calls.push(`POST ${path}`);
|
||||
return path === "/api/plugins/install" ? { id: "plugin", status: "ready" } : { id: "environment", driver: "sandbox", config: data.config };
|
||||
}),
|
||||
delete: vi.fn(async (path: string) => { calls.push(`DELETE ${path}`); }),
|
||||
};
|
||||
const provision = () => provisionFirstTaskFixtures({ api: api as any, execution: execution(), nonce: "private", company: { id: "company", name: "Fresh" },
|
||||
daytonaImage: image, credentials: { OPENAI_API_KEY: "synthetic-key", DAYTONA_API_KEY: "synthetic-platform" } });
|
||||
return { api, state, calls, provision };
|
||||
}
|
||||
|
||||
it.each([[null, false], ["previous-default", false], ["previous-default", true]] as const)(
|
||||
"restores prior default %s and managed-only %s before deleting the managed fixture", async (previousDefault, previousManagedOnly) => {
|
||||
const h = managedFixtureApi(previousDefault, previousManagedOnly);
|
||||
const fixtures = await h.provision();
|
||||
expect(h.state).toMatchObject({ defaultEnvironmentId: "environment", experimental: { enableManagedSandboxOnly: true } });
|
||||
let deletedState: typeof h.state | undefined;
|
||||
h.api.delete.mockImplementation(async () => { deletedState = structuredClone(h.state); });
|
||||
await fixtures.teardown();
|
||||
expect(deletedState).toMatchObject({ defaultEnvironmentId: previousDefault, experimental: { enableManagedSandboxOnly: previousManagedOnly } });
|
||||
expect(h.api.get).toHaveBeenCalledExactlyOnceWith("/api/instance/settings");
|
||||
expect(h.api.patch).toHaveBeenLastCalledWith("/api/instance/settings", { defaultEnvironmentId: previousDefault });
|
||||
expect(h.state.experimental.enableAgentChat).toBe(true);
|
||||
expect(h.api.delete).toHaveBeenCalledExactlyOnceWith("/api/environments/environment?destroyReusableSandboxLeases=true", { allowNotFound: true });
|
||||
},
|
||||
);
|
||||
|
||||
it.each(["default", "managed-only"])("restores a %s setup write whose response is lost after committing", async phase => {
|
||||
const h = managedFixtureApi();
|
||||
const patch = h.api.patch.getMockImplementation()!;
|
||||
let failed = false;
|
||||
h.api.patch.mockImplementation(async (path, data) => {
|
||||
const result = await patch(path, data);
|
||||
if (!failed && (phase === "default" ? data.defaultEnvironmentId === "environment" : data.enableManagedSandboxOnly === true)) {
|
||||
failed = true;
|
||||
throw new Error("Setup response lost after commit");
|
||||
}
|
||||
return result;
|
||||
});
|
||||
await expect(h.provision()).rejects.toThrow("Setup response lost after commit");
|
||||
expect(h.state).toEqual({ defaultEnvironmentId: "previous-default", experimental: { enableManagedSandboxOnly: false, enableAgentChat: true } });
|
||||
expect(h.api.delete).toHaveBeenCalledTimes(1);
|
||||
if (phase === "default") expect(h.api.patch.mock.calls.some(([path]) => path === "/api/instance/settings/experimental")).toBe(false);
|
||||
});
|
||||
|
||||
it("retains setup and restoration failures instead of deleting an unconfirmed instance default", async () => {
|
||||
const h = managedFixtureApi();
|
||||
const patch = h.api.patch.getMockImplementation()!;
|
||||
h.api.patch.mockImplementation(async (path, data) => {
|
||||
if (path === "/api/instance/settings/experimental" && data.enableManagedSandboxOnly === true) throw new Error("Managed mode setup failed");
|
||||
if (path === "/api/instance/settings" && data.defaultEnvironmentId === "previous-default") throw new Error("Default restoration failed");
|
||||
return patch(path, data);
|
||||
});
|
||||
const error = await h.provision().catch(error => error);
|
||||
expect(error).toBeInstanceOf(AggregateError);
|
||||
expect(error.errors[0].message).toBe("Managed mode setup failed");
|
||||
expect(error.errors[1].errors[0].message).toBe("Default restoration failed");
|
||||
expect(error.errors[1].message).toContain("environment deletion was not attempted");
|
||||
expect(h.api.delete).not.toHaveBeenCalled();
|
||||
expect(h.state.defaultEnvironmentId).toBe("environment");
|
||||
});
|
||||
|
||||
it("keeps failed default restoration outstanding and only deletes after a later confirmed restoration", async () => {
|
||||
const h = managedFixtureApi();
|
||||
const fixtures = await h.provision();
|
||||
const patch = h.api.patch.getMockImplementation()!;
|
||||
h.api.patch.mockImplementation(async (path, data) => {
|
||||
if (path === "/api/instance/settings") throw new Error("Default restoration failed");
|
||||
return patch(path, data);
|
||||
});
|
||||
await expect(fixtures.teardown()).rejects.toThrow("environment deletion was not attempted");
|
||||
expect(h.state).toMatchObject({ defaultEnvironmentId: "environment", experimental: { enableManagedSandboxOnly: false } });
|
||||
expect(h.api.delete).not.toHaveBeenCalled();
|
||||
h.api.patch.mockImplementation(patch);
|
||||
await fixtures.teardown();
|
||||
await fixtures.teardown();
|
||||
expect(h.state.defaultEnvironmentId).toBe("previous-default");
|
||||
expect(h.api.delete).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("reports failed managed-mode restoration while still removing an environment that is no longer the default", async () => {
|
||||
const h = managedFixtureApi();
|
||||
const fixtures = await h.provision();
|
||||
h.state.experimental.enableAgentChat = false;
|
||||
const patch = h.api.patch.getMockImplementation()!;
|
||||
h.api.patch.mockImplementation(async (path, data) => {
|
||||
if (path === "/api/instance/settings/experimental") throw new Error("Managed mode restoration failed");
|
||||
return patch(path, data);
|
||||
});
|
||||
const error = await fixtures.teardown().catch(error => error);
|
||||
expect(error).toBeInstanceOf(AggregateError);
|
||||
expect(error.errors.map((error: Error) => error.message)).toEqual(["Managed mode restoration failed"]);
|
||||
expect(h.state.defaultEnvironmentId).toBe("previous-default");
|
||||
expect(h.api.delete).toHaveBeenCalledTimes(1);
|
||||
h.api.patch.mockImplementation(patch);
|
||||
await fixtures.teardown();
|
||||
expect(h.state.experimental).toEqual({ enableManagedSandboxOnly: false, enableAgentChat: false });
|
||||
expect(h.api.delete).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("keeps failed environment deletion red after successful settings restoration", async () => {
|
||||
const h = managedFixtureApi();
|
||||
const fixtures = await h.provision();
|
||||
h.api.delete.mockRejectedValue(new Error("Provider cleanup unconfirmed"));
|
||||
vi.useFakeTimers();
|
||||
try {
|
||||
const rejected = expect(fixtures.teardown()).rejects.toThrow("Managed onboarding fixture cleanup failed");
|
||||
await vi.runAllTimersAsync();
|
||||
await rejected;
|
||||
expect(h.state).toMatchObject({ defaultEnvironmentId: "previous-default", experimental: { enableManagedSandboxOnly: false } });
|
||||
h.api.delete.mockResolvedValue();
|
||||
await fixtures.teardown();
|
||||
const deletes = h.api.delete.mock.calls.length;
|
||||
await fixtures.teardown();
|
||||
expect(h.api.delete).toHaveBeenCalledTimes(deletes);
|
||||
} finally { vi.useRealTimers(); }
|
||||
});
|
||||
|
||||
it.each(["read-failure", "missing-default", "missing-mode"])("fails before fixture mutations when prior settings have %s", async mode => {
|
||||
const h = managedFixtureApi();
|
||||
if (mode === "read-failure") h.api.get.mockRejectedValue(new Error("Settings read failed"));
|
||||
else h.api.get.mockResolvedValue((mode === "missing-default" ? { experimental: { enableManagedSandboxOnly: false } } : { defaultEnvironmentId: null, experimental: {} }) as any);
|
||||
await expect(h.provision()).rejects.toThrow(mode === "read-failure" ? "Settings read failed" : "prior instance settings could not be read");
|
||||
expect(h.api.patch).not.toHaveBeenCalled();
|
||||
expect(h.api.postSensitive).not.toHaveBeenCalled();
|
||||
expect(h.api.post).not.toHaveBeenCalled();
|
||||
expect(h.api.delete).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("prepares only encrypted credentials and an immutable managed default, with no fixture probe or agent", async () => {
|
||||
const calls: string[] = [];
|
||||
const api = {
|
||||
get: vi.fn(async () => ({ defaultEnvironmentId: null, experimental: { enableManagedSandboxOnly: false } })),
|
||||
patch: vi.fn(async (path: string) => { calls.push(`PATCH ${path}`); return {}; }),
|
||||
postSensitive: vi.fn(async (_path: string, data: any) => ({ id: `encrypted-${data.key}` })),
|
||||
post: vi.fn(async (path: string, data: any) => {
|
||||
calls.push(`POST ${path}`);
|
||||
if (path === "/api/plugins/install") return { id: "plugin", status: "ready" };
|
||||
expect(path).toBe("/api/companies/company/environments");
|
||||
expect(data.config).toMatchObject({ image, apiKey: { type: "secret_ref", secretId: "encrypted-DAYTONA_API_KEY", version: "latest" } });
|
||||
expect(data).not.toHaveProperty("metadata");
|
||||
return { id: "environment", driver: "sandbox", config: data.config };
|
||||
}),
|
||||
delete: vi.fn(async () => {}),
|
||||
};
|
||||
const fixtures = await provisionFirstTaskFixtures({ api: api as any, execution: execution(), nonce: "private", company: { id: "company", name: "Fresh" },
|
||||
daytonaImage: image, credentials: { OPENAI_API_KEY: "synthetic-key", DAYTONA_API_KEY: "synthetic-platform" } });
|
||||
expect(calls).toEqual(["PATCH /api/companies/company", "POST /api/plugins/install", "POST /api/companies/company/environments", "PATCH /api/instance/settings", "PATCH /api/instance/settings/experimental"]);
|
||||
expect(api.patch).toHaveBeenCalledWith("/api/instance/settings", { defaultEnvironmentId: "environment" });
|
||||
expect(api.patch).toHaveBeenCalledWith("/api/instance/settings/experimental", { enableManagedSandboxOnly: true });
|
||||
expect(fixtures.agent.id).toBe("");
|
||||
expect(fixtures.setupLeases).toEqual([]);
|
||||
expect(JSON.stringify(fixtures)).not.toContain("synthetic");
|
||||
await fixtures.teardown();
|
||||
expect(api.delete).toHaveBeenCalledExactlyOnceWith("/api/environments/environment?destroyReusableSandboxLeases=true", { allowNotFound: true });
|
||||
});
|
||||
|
||||
it.each(["missing-key", "mutable-image", "other-cell"])("rejects %s before admitting a managed browser fixture", async mode => {
|
||||
const api = { get: vi.fn(), patch: vi.fn(), postSensitive: vi.fn(), post: vi.fn(), delete: vi.fn() };
|
||||
await expect(provisionFirstTaskFixtures({ api, execution: mode === "other-cell" ? { ...execution(), id: "other" } : execution(), nonce: "private", company: { id: "company", name: "Fresh" },
|
||||
daytonaImage: mode === "mutable-image" ? "ubuntu:24.04" : image,
|
||||
credentials: mode === "missing-key" ? { OPENAI_API_KEY: "synthetic" } : { OPENAI_API_KEY: "synthetic", DAYTONA_API_KEY: "synthetic" } })).rejects.toThrow();
|
||||
expect(api.post).not.toHaveBeenCalled();
|
||||
expect(api.postSensitive).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
function fileEvidence() {
|
||||
const e = recording("task-card-accept");
|
||||
const last = e.checkpoints.at(-1)!;
|
||||
last.tasks = [{ id: "child", parentId: e.onboardingIssueId, status: "done" }];
|
||||
const body = `Welcome beginners. Join our free Saturday meetup ${firstTaskScenario(e.caseId, e.nonce).marker}.`;
|
||||
last.attachments = [{ id: "file", issueId: "child", originalFilename: "garden-welcome.txt", contentType: "text/plain", body,
|
||||
contentVerified: true, contentSha256: digestText(body) }];
|
||||
return e;
|
||||
}
|
||||
it("requires the downloaded verified physical artifact", () => {
|
||||
expect(managedFirstTaskFileCheck(fileEvidence()).passed).toBe(true);
|
||||
});
|
||||
it.each(["document-only", "unverified", "wrong-hash", "wrong-child", "wrong-name", "wrong-marker"])("rejects %s as physical-file proof", async mode => {
|
||||
const e = fileEvidence();
|
||||
const last = e.checkpoints.at(-1)!;
|
||||
const file = last.attachments![0]!;
|
||||
if (mode === "document-only") { last.documents = [{ ...file, key: "welcome" }]; last.attachments = []; }
|
||||
if (mode === "unverified") file.contentVerified = false;
|
||||
if (mode === "wrong-hash") file.contentSha256 = "0".repeat(64);
|
||||
if (mode === "wrong-child") file.issueId = "foreign";
|
||||
if (mode === "wrong-name") file.originalFilename = "plan.txt";
|
||||
if (mode === "wrong-marker") { file.body = "A different welcome note"; file.contentSha256 = digestText(file.body); }
|
||||
expect(managedFirstTaskFileCheck(e).passed).toBe(false);
|
||||
});
|
||||
|
||||
function setupHarness(mode = "success") {
|
||||
const listeners = new Map<string, Set<(event: any) => void>>();
|
||||
const page = { on: (name: string, callback: (event: any) => void) => { const set = listeners.get(name) ?? new Set(); set.add(callback); listeners.set(name, set); },
|
||||
off: (name: string, callback: (event: any) => void) => listeners.get(name)?.delete(callback) };
|
||||
const fixtures = { company: { id: "company", name: "Fresh" }, environment: { id: "environment", driver: "sandbox" },
|
||||
agent: { id: "", companyId: "company", name: "Ada" }, secretRefs: {}, setupLeases: [],
|
||||
setupQualification: { recorded: false, adapterType: null, platformResponses: 0, adapterResponses: 0 }, teardown: async () => {} } as import("./live-fixtures.js").LiveFixtureValues;
|
||||
const rows = ["platform", "adapter"].map(id => ({ id, companyId: "company", environmentId: "environment", provider: "daytona", heartbeatRunId: null, issueId: null,
|
||||
status: mode === "cleanup" ? "failed" : "released", cleanupStatus: "success" }));
|
||||
if (mode === "foreign") rows[0]!.companyId = "another-company";
|
||||
let begun = false;
|
||||
const api = { get: vi.fn(async (path: string) => path.includes("environment-leases/") ? rows.find(row => path.endsWith(`/${row.id}`))
|
||||
: begun ? [...rows, { ...rows[0], id: "unrelated", issueId: "other-task" }] : []) } as unknown as RunnerApi;
|
||||
const start = (kind: "platform" | "discovery" | "adapter", runner?: string) => {
|
||||
begun = true;
|
||||
const request = { url: () => `http://fixture.test${kind === "platform" ? "/api/environments/environment/probe?companyId=company"
|
||||
: kind === "discovery" ? "/api/adapters?companyId=company&environmentId=environment" : "/api/companies/company/adapters/codex_local/test-environment"}`,
|
||||
method: () => kind === "discovery" ? "GET" : "POST", postDataJSON: () => runner ? { runner } : {} };
|
||||
for (const listener of listeners.get("request") ?? []) listener(request);
|
||||
return request;
|
||||
};
|
||||
const respond = (request: ReturnType<typeof start>, body: any) => {
|
||||
const response = { request: () => request, json: async () => body, ok: () => true };
|
||||
for (const listener of listeners.get("response") ?? []) listener(response);
|
||||
};
|
||||
const emit = (kind: "platform" | "discovery" | "adapter", body: any, runner?: string) => respond(start(kind, runner), body);
|
||||
const fail = (request: ReturnType<typeof start>) => {
|
||||
for (const listener of listeners.get("requestfailed") ?? []) listener(request);
|
||||
};
|
||||
return { fixtures, page: page as unknown as import("@playwright/test").Page, api, emit, start, respond, fail, listeners, rows };
|
||||
}
|
||||
|
||||
it.each(["missing-response", "request-failed"])("retains unknown adapter setup cost after %s without an attributable row", async mode => {
|
||||
const h = setupHarness();
|
||||
h.rows.splice(1);
|
||||
const stop = await observeManagedFirstTaskSetup(h.page, h.api, h.fixtures, Date.now() + 1000);
|
||||
h.emit("platform", { ok: true, details: { platformQualificationRecorded: true, leaseId: "platform" } });
|
||||
const request = h.start("adapter");
|
||||
if (mode === "request-failed") h.fail(request);
|
||||
await stop(false);
|
||||
expect(h.fixtures.setupLeases!.map(row => row.leaseId)).toEqual(["platform"]);
|
||||
expect(h.fixtures.setupLeaseAccountingIncomplete).toBe(true);
|
||||
expect(h.fixtures.setupQualification!.adapterResponses).toBe(0);
|
||||
const leases = selectFixtureRuntimeLeases({ leases: h.rows.map(row => ({ ...row,
|
||||
acquiredAt: "2026-10-13T00:00:00Z", releasedAt: "2026-10-13T00:00:02Z", metadata: { cpu: 4, memory: 4, disk: 10 } })),
|
||||
runIds: [], setupLeaseIds: h.fixtures.setupLeases!.map(row => row.leaseId) });
|
||||
const usage = fixtureRuntimeCostCoverage(buildRuntimeUsage({ environmentId: "daytona", runs: [], leases }), h.fixtures);
|
||||
expect(usage.leaseCount).toBe(1);
|
||||
expect(usage.costStatus).toBe("unavailable");
|
||||
expect(usage).not.toHaveProperty("estimatedListCostUsd");
|
||||
expect([...h.listeners.values()].every(listeners => listeners.size === 0)).toBe(true);
|
||||
});
|
||||
|
||||
it("retains unknown failed platform setup without a response or visible lease row", async () => {
|
||||
const h = setupHarness();
|
||||
h.rows.splice(0);
|
||||
const stop = await observeManagedFirstTaskSetup(h.page, h.api, h.fixtures, Date.now() + 1000);
|
||||
h.fail(h.start("platform"));
|
||||
await stop(false);
|
||||
expect(h.fixtures.setupLeases).toEqual([]);
|
||||
expect(h.fixtures.setupLeaseAccountingIncomplete).toBe(true);
|
||||
expect(h.fixtures.setupQualification!.recorded).toBe(false);
|
||||
});
|
||||
|
||||
it("does not clear failed-request accounting uncertainty when later responses correlate", async () => {
|
||||
const h = setupHarness();
|
||||
const stop = await observeManagedFirstTaskSetup(h.page, h.api, h.fixtures, Date.now() + 1000);
|
||||
h.emit("platform", { ok: true, details: { platformQualificationRecorded: true, leaseId: "platform" } });
|
||||
const request = h.start("adapter");
|
||||
h.fail(request);
|
||||
h.respond(request, { adapterType: "paperclip_runner", status: "pass", checks: [{ code: "sandbox_test_identity", detail: "paperclipLeaseId=adapter" }] });
|
||||
await stop(false);
|
||||
expect(h.fixtures.setupLeases!.map(row => row.leaseId)).toEqual(["platform", "adapter"]);
|
||||
expect(h.fixtures.setupLeaseAccountingIncomplete).toBe(true);
|
||||
});
|
||||
|
||||
it.each(["success", "missing-discovery", "explicit-runner", "duplicate", "foreign", "cleanup"])("verifies exact UI setup responses and owned cleanup for %s", async mode => {
|
||||
const h = setupHarness(mode);
|
||||
const stop = await observeManagedFirstTaskSetup(h.page, h.api, h.fixtures, mode === "cleanup" ? Date.now() - 1 : Date.now() + 1000);
|
||||
const platform = { ok: true, details: { platformQualificationRecorded: true, leaseId: "platform" } };
|
||||
h.emit("platform", platform);
|
||||
if (mode === "duplicate") h.emit("platform", platform);
|
||||
if (mode !== "missing-discovery") h.emit("discovery", [{ type: "codex_local", defaultRunner: "paperclip", supportedRunners: ["paperclip", "legacy"] }]);
|
||||
h.emit("adapter", { adapterType: "paperclip_runner", status: "pass", checks: [{ code: "sandbox_test_identity", detail: "paperclipLeaseId=adapter; providerLeaseId=private-provider" }] }, mode === "explicit-runner" ? "paperclip" : undefined);
|
||||
if (mode === "success") { await stop(true); expect(h.fixtures.setupLeases!.map(row => row.leaseId)).toEqual(["platform", "adapter"]); }
|
||||
else await expect(stop(true)).rejects.toThrow();
|
||||
expect(h.fixtures.setupLeases!.some(row => row.leaseId === "unrelated")).toBe(false);
|
||||
expect([...h.listeners.values()].every(listeners => listeners.size === 0)).toBe(true);
|
||||
});
|
||||
|
||||
it("reports a rejected platform setup immediately without waiting for the long browser deadline", async () => {
|
||||
const h = setupHarness();
|
||||
const stop = await observeManagedFirstTaskSetup(h.page, h.api, h.fixtures, Date.now() + 1000);
|
||||
const rejected = expect(stop.failure).rejects.toThrow("UI environment qualification did not pass");
|
||||
h.emit("platform", { ok: false, details: { leaseId: "platform", platformQualificationRecorded: false } });
|
||||
await rejected;
|
||||
await stop(false);
|
||||
expect(h.fixtures.setupLeases!.map(row => row.leaseId)).toEqual(["platform"]);
|
||||
});
|
||||
|
||||
it("keeps an uncorrelated lease out of setup billing and retains unavailable cost coverage", async () => {
|
||||
const h = setupHarness();
|
||||
const stop = await observeManagedFirstTaskSetup(h.page, h.api, h.fixtures, Date.now() + 1000);
|
||||
h.emit("platform", { ok: false, details: { leaseId: "platform", platformQualificationRecorded: false } });
|
||||
await stop(false);
|
||||
expect(h.fixtures.setupLeases!.map(row => row.leaseId)).toEqual(["platform"]);
|
||||
expect(h.fixtures.setupLeaseAccountingIncomplete).toBe(true);
|
||||
expect(h.fixtures.setupQualification!.recorded).toBe(false);
|
||||
});
|
||||
|
||||
it("retains exact failed setup leases for accounting without granting qualification", async () => {
|
||||
const h = setupHarness();
|
||||
const stop = await observeManagedFirstTaskSetup(h.page, h.api, h.fixtures, Date.now() + 1000);
|
||||
h.emit("platform", { ok: true, details: { leaseId: "platform", platformQualificationRecorded: true } });
|
||||
h.emit("adapter", { adapterType: "paperclip_runner", status: "fail", checks: [{ code: "sandbox_test_identity", detail: "paperclipLeaseId=adapter; providerLeaseId=private-provider" }] });
|
||||
await stop(false);
|
||||
expect(h.fixtures.setupLeases!.map(row => row.leaseId)).toEqual(["platform", "adapter"]);
|
||||
expect(h.fixtures.setupQualification!.adapterType).toBeNull();
|
||||
});
|
||||
|
||||
it("keeps a missing response lease ID unknown even without visible lease history", async () => {
|
||||
const h = setupHarness();
|
||||
h.rows.splice(0);
|
||||
const stop = await observeManagedFirstTaskSetup(h.page, h.api, h.fixtures, Date.now() + 1000);
|
||||
h.emit("platform", { ok: false, details: { platformQualificationRecorded: false } });
|
||||
await stop(false);
|
||||
expect(h.fixtures.setupLeases).toEqual([]);
|
||||
expect(h.fixtures.setupLeaseAccountingIncomplete).toBe(true);
|
||||
expect(h.fixtures.setupQualification!.recorded).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -2,7 +2,7 @@ import { describe, expect, it } from "vitest";
|
||||
import type { RunnerApi } from "./api.js";
|
||||
import { buildRuntimeUsage } from "./billing.js";
|
||||
import { runnerMatrix } from "./catalog.js";
|
||||
import { setupLiveFixtures, selectFixtureRuntimeLeases } from "./live-fixtures.js";
|
||||
import { setupLiveFixtures, selectFixtureRuntimeLeases, fixtureRuntimeCostCoverage } from "./live-fixtures.js";
|
||||
|
||||
describe("live runner fixtures", () => {
|
||||
it.each(["runner-codex", "runner-acpx-claude", "runner-opencode"].flatMap(profile =>
|
||||
@@ -398,3 +398,31 @@ describe("live runner fixtures", () => {
|
||||
await fixtures.teardown();
|
||||
});
|
||||
});
|
||||
|
||||
describe("owned browser setup lease accounting", () => {
|
||||
it("includes both exact setup IDs and task runs while excluding unrelated leases", () => {
|
||||
const leases = [{ id: "platform", provider: "daytona" }, { id: "adapter", provider: "daytona" },
|
||||
{ id: "task", provider: "daytona", heartbeatRunId: "run" }, { id: "unrelated", provider: "daytona" }];
|
||||
expect(selectFixtureRuntimeLeases({ leases, runIds: ["run"], setupLeaseIds: ["platform", "adapter"] }).map(lease => lease.id)).toEqual(["platform", "adapter", "task"]);
|
||||
expect(selectFixtureRuntimeLeases({ leases, runIds: [], setupLeaseIds: ["platform"] }).map(lease => lease.id)).toEqual(["platform"]);
|
||||
expect(selectFixtureRuntimeLeases({ leases, runIds: [], setupLeaseIds: [] })).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("managed setup runtime cost coverage", () => {
|
||||
it("prices both exact setup windows with task usage and keeps missing attribution unknown", () => {
|
||||
const leases = ["platform", "adapter", "task", "unrelated"].map((id, index) => ({ id, provider: "daytona", heartbeatRunId: id === "task" ? "run" : null,
|
||||
acquiredAt: "2026-10-13T00:00:00Z", releasedAt: `2026-10-13T00:00:0${index + 1}Z`, metadata: { cpu: 4, memory: 4, disk: 10 } }));
|
||||
const selected = selectFixtureRuntimeLeases({ leases, runIds: ["run"], setupLeaseIds: ["platform", "adapter"] });
|
||||
const usage = buildRuntimeUsage({ environmentId: "daytona", runs: [], leases: selected });
|
||||
expect(usage.leaseCount).toBe(3);
|
||||
expect(usage.leaseDurationMs).toBe(6000);
|
||||
expect(usage.estimatedListCostUsd).toBeGreaterThan(0);
|
||||
expect(fixtureRuntimeCostCoverage(usage, {})).toEqual(usage);
|
||||
const incomplete = fixtureRuntimeCostCoverage(usage, { setupLeaseAccountingIncomplete: true });
|
||||
expect(incomplete.costStatus).toBe("unavailable");
|
||||
expect(incomplete.costSource).toBe("provider_cost_unavailable");
|
||||
expect(incomplete).not.toHaveProperty("estimatedListCostUsd");
|
||||
expect(incomplete.leaseCount).toBe(3);
|
||||
});
|
||||
});
|
||||
@@ -13,6 +13,7 @@ import type {
|
||||
MatrixExecution,
|
||||
SecretReference,
|
||||
SecretReferenceMap,
|
||||
RunnerE2ERuntimeUsage,
|
||||
} from "./types.js";
|
||||
|
||||
interface CompanyRecord {
|
||||
@@ -65,6 +66,9 @@ export interface LiveFixtureValues {
|
||||
project?: ProjectRecord;
|
||||
aiConnection?: ManagedAccountFixture;
|
||||
setupProbe?: { leaseId: string; durationMs: number; defaultRunner?: string };
|
||||
setupLeases?: Array<{ leaseId: string; phase: "platform" | "adapter"; durationMs: number }>;
|
||||
setupLeaseAccountingIncomplete?: boolean;
|
||||
setupQualification?: { recorded: boolean; adapterType: string | null; platformResponses: number; adapterResponses: number };
|
||||
|
||||
onboardingRuntime?: {
|
||||
mode: "production-wizard" | "post-onboarding-runtime-switch";
|
||||
@@ -83,17 +87,24 @@ function value<T>(resolved: ReadonlyMap<string, unknown>, id: string): T {
|
||||
|
||||
export function selectFixtureRuntimeLeases<T extends {
|
||||
id: string; provider?: string | null; heartbeatRunId?: string | null; issueId?: string | null;
|
||||
}>(input: { leases: T[]; runIds: string[]; issueId?: string; setupProbeLeaseId?: string }): T[] {
|
||||
}>(input: { leases: T[]; runIds: string[]; issueId?: string; setupProbeLeaseId?: string; setupLeaseIds?: readonly string[] }): T[] {
|
||||
const selectedRunIds = new Set(input.runIds);
|
||||
const setupIds = new Set([...(input.setupLeaseIds ?? []), ...(input.setupProbeLeaseId ? [input.setupProbeLeaseId] : [])]);
|
||||
const relevant = input.leases.filter(lease => lease.provider === "daytona" && (
|
||||
lease.id === input.setupProbeLeaseId || selectedRunIds.size === 0
|
||||
setupIds.has(lease.id) || (input.setupLeaseIds === undefined && selectedRunIds.size === 0)
|
||||
|| (lease.heartbeatRunId && selectedRunIds.has(lease.heartbeatRunId))
|
||||
|| (input.issueId && lease.issueId === input.issueId)
|
||||
));
|
||||
return relevant.length > 0 ? relevant : input.leases;
|
||||
return input.setupLeaseIds !== undefined || relevant.length > 0 ? relevant : input.leases;
|
||||
}
|
||||
|
||||
async function deleteDaytonaEnvironment(api: RunnerApi, environmentId: string) {
|
||||
export function fixtureRuntimeCostCoverage(usage: RunnerE2ERuntimeUsage, fixtures: Pick<LiveFixtureValues, "setupLeaseAccountingIncomplete">): RunnerE2ERuntimeUsage {
|
||||
if (!fixtures.setupLeaseAccountingIncomplete) return usage;
|
||||
const { estimatedListCostUsd: _estimate, ...known } = usage;
|
||||
return { ...known, costStatus: "unavailable", costSource: "provider_cost_unavailable" };
|
||||
}
|
||||
|
||||
export async function deleteDaytonaEnvironment(api: Pick<RunnerApi, "delete">, environmentId: string) {
|
||||
const deadlineAt = Date.now() + 120_000;
|
||||
let lastError: unknown;
|
||||
while (Date.now() < deadlineAt) {
|
||||
@@ -115,6 +126,14 @@ async function deleteDaytonaEnvironment(api: RunnerApi, environmentId: string) {
|
||||
);
|
||||
}
|
||||
|
||||
export async function installFixtureDaytonaProvider(api: Pick<RunnerApi, "post">, execution: MatrixExecution) {
|
||||
const installed = await verifyInstalledDaytonaPlugin(process.env, [execution]);
|
||||
return api.post<PluginRecord>("/api/plugins/install", {
|
||||
packageName: installed?.packageRoot ?? path.resolve(import.meta.dirname, "../../packages/plugins/sandbox-providers/daytona"),
|
||||
isLocalPath: true,
|
||||
}, { timeout: 120_000 });
|
||||
}
|
||||
|
||||
export async function setupLiveFixtures(input: {
|
||||
api: RunnerApi;
|
||||
execution: MatrixExecution;
|
||||
@@ -131,14 +150,7 @@ export async function setupLiveFixtures(input: {
|
||||
registry.register<PluginRecord>({
|
||||
id: "sandbox-provider",
|
||||
async setup() {
|
||||
const installed = await verifyInstalledDaytonaPlugin(process.env, [execution]);
|
||||
return api.post<PluginRecord>("/api/plugins/install", {
|
||||
packageName: installed?.packageRoot ?? path.resolve(
|
||||
import.meta.dirname,
|
||||
"../../packages/plugins/sandbox-providers/daytona",
|
||||
),
|
||||
isLocalPath: true,
|
||||
}, { timeout: 120_000 });
|
||||
return installFixtureDaytonaProvider(api, execution);
|
||||
},
|
||||
async teardown() {
|
||||
// The plugin is installed only in the isolated instance/database. The
|
||||
|
||||
@@ -180,7 +180,7 @@ describe("Pi controls catalog admission", () => {
|
||||
expect(cells).toHaveLength(4); expect(cells.every(c => c.profile.qualificationCandidate === "pi" && c.task.expectedRunCount === 1)).toBe(true);
|
||||
expect(cells.map(c => `${c.environment.id}/${c.task.id}`).sort()).toEqual(["daytona/pending-permission-stop", "daytona/same-turn-steering", "local/pending-permission-stop", "local/same-turn-steering"]);
|
||||
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(c => c.suite.id === "pi-controls")).toBe(false);
|
||||
expect(validateRunnerCatalog()).toHaveLength(729);
|
||||
expect(validateRunnerCatalog()).toHaveLength(730);
|
||||
for (const cell of cells) expect(buildRunnerE2EProcessEnvironment({}, [cell]).PAPERCLIP_RUNNER_ACPX_QUALIFICATION).toBeUndefined();
|
||||
expect(() => assertRemoteNativeEvidencePrerequisites(cells, {})).toThrow();
|
||||
});
|
||||
|
||||
@@ -52,7 +52,7 @@ import { runnerExecutionById } from "./catalog.js";
|
||||
import { classifyFailure } from "./failure-classifier.js";
|
||||
import { runnerE2EServerControlPaths } from "./harness-env.js";
|
||||
import { setupConnectionReview } from "./connection-reviews.js";
|
||||
import { setupLiveFixtures, selectFixtureRuntimeLeases, type LiveFixtureValues } from "./live-fixtures.js";
|
||||
import { setupLiveFixtures, selectFixtureRuntimeLeases, fixtureRuntimeCostCoverage, type LiveFixtureValues } from "./live-fixtures.js";
|
||||
import { evaluateMatcher, persistedFinalRunMessage, type MatcherResult } from "./matchers.js";
|
||||
import { readRegisteredArtifacts } from "./registered-artifact.js";
|
||||
import {
|
||||
@@ -701,7 +701,7 @@ for (const execution of executions) {
|
||||
);
|
||||
runtimeLeases = selectFixtureRuntimeLeases({ leases: listed,
|
||||
runIds: selectedRuns.map(run => run.id), issueId: issue?.id,
|
||||
setupProbeLeaseId: fixtures.setupProbe?.leaseId });
|
||||
setupProbeLeaseId: fixtures.setupProbe?.leaseId, setupLeaseIds: fixtures.setupLeases?.map(lease => lease.leaseId) });
|
||||
};
|
||||
|
||||
const cancelActiveRunsForCleanup = async () => {
|
||||
@@ -868,7 +868,6 @@ for (const execution of executions) {
|
||||
const receipt = verifyStockHarnessPreflight(process.env[STOCK_PREFLIGHT_ENV]);
|
||||
await writeSanitizedJson(snapshotsDir, "stock-harness-preflight.json", receipt, secrets);
|
||||
}
|
||||
// Codex default journeys must work with the experimental gate disabled.
|
||||
const nativeRolloutEnabled = execution.profile.provider !== "codex";
|
||||
const experimental = await api.patch<{
|
||||
enableNativeRunner: boolean;
|
||||
@@ -893,7 +892,7 @@ for (const execution of executions) {
|
||||
await setupGitStreamingWorkspace(workspacePath);
|
||||
}
|
||||
fixtures = execution.task.flow === "first_task"
|
||||
? await setupFirstTaskFixtures({ page, api, execution, nonce, credentials, observe: value => { fixtures = value; } })
|
||||
? await setupFirstTaskFixtures({ page, api, execution, nonce, credentials, daytonaImage: process.env.PAPERCLIP_E2E_DAYTONA_IMAGE, deadlineAt: startedAtMs + deadlineMs, observe: value => { fixtures = value; } })
|
||||
: await setupLiveFixtures({
|
||||
api,
|
||||
execution,
|
||||
@@ -1161,7 +1160,7 @@ for (const execution of executions) {
|
||||
if (matcherResults.length === 0) matcherResults = [{ matcher: { kind: "issue_status", expected: "in_review" }, passed: true, detail: "Chat workflow and durable handoff/session assertions passed" }];
|
||||
} else if (execution.task.flow === "first_task") {
|
||||
const firstTask = await runFirstTaskFlow({
|
||||
page, api, fixtures, execution, nonce, secrets,
|
||||
page, api, fixtures, execution, nonce, deadlineAt: startedAtMs + deadlineMs, secrets,
|
||||
observe: (currentIssue, runs, evidence) => {
|
||||
issue = currentIssue; selectedRuns = runs; firstTaskEvidence = evidence;
|
||||
matcherResults = evidence.checks.map(check => ({ matcher: { kind: "json_path" as const, path: `firstTask.checks.${check.id}`, expected: true }, passed: check.passed, detail: check.detail }));
|
||||
@@ -3104,12 +3103,12 @@ for (const execution of executions) {
|
||||
}
|
||||
|
||||
const finishedAtMs = Date.now();
|
||||
const runtimeUsage = buildRuntimeUsage({
|
||||
const runtimeUsage = fixtureRuntimeCostCoverage(buildRuntimeUsage({
|
||||
environmentId: execution.environment.id,
|
||||
runs: selectedRuns,
|
||||
leases: runtimeLeases,
|
||||
fallbackFinishedAt: new Date(finishedAtMs),
|
||||
});
|
||||
}), fixtures ?? {});
|
||||
const resultWithoutBilling: RunnerE2EResult = {
|
||||
schema: "paperclip.runner-e2e.result/v2",
|
||||
executionId: execution.id,
|
||||
|
||||
Reference in new issue
Block a user