Merge pinned production for the post-custody chat pilot

Preserve frozen 244-scenario YAML intent and 191-capability coverage. Production baseline: 39e811c9c5.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip committed 2026-10-27 21:03:19 -05:00
commit 0195fcb016
398 files changed
+90564 -66016

No files matched your search

+72 -2
View File
@@ -223,7 +223,8 @@ stable key-sorted JSON hashing. The model configuration contains `provider`,
`mode` (the manifest route), `price`, `maximumInputTokens`,
`maximumOutputTokens`, and `reservation` (the nine-decimal value from
`evalRequestReservation`). An operating configuration contains allowance
`reference`, `runner`, `verifiedAt`, `expiresAt`, and a `component` object with
`reference`, `runner`, `verifiedAt`, `expiresAt`, optional `accountingReadiness`
(only when present), and a `component` object with
its `component`, `maxUsd`, and `reference`. Changing any of those values
invalidates the approval hash. Evidence metadata itself is excluded from this
configuration hash.
@@ -439,8 +440,11 @@ attempt. Installing the ACP package alone does not satisfy this setup path.
Bounded cases keep the gateway in the setup phase until the created agent's
asynchronous lifecycle verification reaches ready. This wait uses the remaining
attempt deadline, allows at most 45 seconds, and stops on an observed setup error
attempt deadline, allows at most 45 seconds locally and the bounded attempt window
for Daytona provisioning, and stops on an observed setup error
or hold. The private `budgeted-setup.json` snapshot retains state transitions.
Budgeted Daytona fixtures give the saved-account hello probe five minutes inside
that attempt window; its former 90-second default could expire during cold startup.
Only then does the case begin agent-phase accounting and create its task.
Older evidence that advanced after fixture creation may label hello probes as
agent cost. Its campaign totals still include those requests; retain that phase
@@ -897,3 +901,69 @@ an operations manifest. They use the same metered connection, prices, campaign
journal, setup boundary, and durable attempt allowance as core compatibility.
Collection and the independent saved-evidence oracle require no additional
model call. The synthetic items need review as test data, not product incidents.
### Qualify model spend while operating reconciliation is pending
Set `accountingReadiness: "model_requests"` on the reviewed allowance to use
complete model receipts as the scheduled readiness gate. This is an explicit
operator policy. The report retains the full-accounting status, operating
reservations and model-readiness status separately. It does not release any
historical hold or certify a full-lifecycle vendor cost bound. Missing model
receipts, conflicts, active requests, excess commitments, and the existing
product, cleanup, feedback and trace checks still fail qualification.
### Explicit bounded Daytona gateway qualification
A manual manifest can set `remoteGateway: "cloudflare"` for the native Codex
or Claude marker/feedback Daytona cells. Install `cloudflared` on the controller
PATH, or set `PAPERCLIP_EVAL_CLOUDFLARED` to its executable path. Empty overrides
use the PATH default. The parent starts its existing model
gateway and native OTLP relay, then exposes only their authenticated endpoints
through one owned HTTPS quick tunnel. It verifies a fresh health nonce before
fixture setup, waits up to 120 seconds for registration and 120 seconds for
public readiness, and does not retry tunnel creation or provider work.
Only attempt-scoped tokens leave the controller. Provider and Braintrust keys
stay in the parent; the fixture/server retains the existing Daytona provisioning
credential separately. The model gateway still enforces its campaign allowance,
request count, concurrency, deadline and receipt journal. The bridge streams
responses and preserves terminal-usage draining after a client disconnect.
The tunnel receives no ambient provider credentials or Cloudflare config. Its
exposure closes at the campaign deadline or four hours after startup, whichever
comes first, and on attempt cleanup. Owned gateway,
relay and tunnel cleanup failures remain failures. A configured remote route
is not live acceptance; retain the actual Daytona result, cleanup, feedback,
native traces and complete model receipts.
The Daytona image runs Linux/x64. ARM or macOS controllers use the image's
verified daemon by default; only an explicit cross-compiled remote override
replaces it.
Daily and weekly selections remain local. This explicit remote option adds no
scheduled coverage or public report publication.
The worker generates the attempt's trace parent. The relay validates the project
in both the header and payload, but does not pin that worker-generated parent.
Treat the attempt OTLP token as authority to submit trace attribution within
this eval project. Keep it limited to the trusted harness/provider runtime;
never expose it to test prompts or unrelated clients. The project check is not
an attempt-isolation guarantee. Indexing verification is a separate retained
or scheduled check; the manual launcher alone does not perform it.
### Hosted report dependencies
The trusted report job applies a reviewed `report-lock.patch` before its frozen
install. `report-lock.json` pins the input lock, patch and resulting lock hashes.
This updates existing patch identities and the already declared yaml dependency
without registry resolution. The worker target and its lock artifact never
control report dependencies. An unknown committed lock fails before install;
update and review this correction when dependency inputs change. The original
failed run remains failed when retained evidence gets a new presentation.
The trusted authorization job checks this correction before catalog/build and
paid work. Unknown dependency locks fail before model admission. Manual history
publication applies the same correction before its frozen install. Qualification
image builds apply it only when target and workflow revisions match; a different
target keeps its own lock and does not require these reporting helper files.
+2 -1
View File
@@ -118,7 +118,8 @@ ownership manifests. An expired lease permits cleanup recovery, not reuse.
Managed self-hosted instances are removed after every attempt. Existing Cloud
stacks retain the QA account and declared baseline while removing verified
run-owned fixtures. Successful disposable stacks are deleted. Failed disposable
stacks are stopped and retained for 24 hours, then swept. Interrupted cleanup
stacks are stopped and retained for up to 24 hours. The deployment provider
declares earlier cleanup eligibility to leave time for its sweeper. Interrupted cleanup
remains visible and retains exclusive stack ownership. QA identity and inbox
never belong to disposable ownership.
+39 -3
View File
@@ -261,6 +261,16 @@ status, known adapters, and allowlisted outcome/check/server codes. The case
report renders these fields explicitly; older results without the receipt remain
valid. This diagnostic does not change the journey grade or qualify a live run.
When the canonical first-task journey fails `onboarding_work_before_acceptance`,
`productJourney.onboardingPreAcceptance` retains the failed checkpoint and every
observed guard violation: unexpected task set, existing attachment, non-planning
document, or first task already Done. The closed version-1 receipt contains only
counts, whether the first task matches the canonical parent, and closed first-task
and canonical-parent statuses. Reports validate and display these observations;
they do not infer a product cause. Only the failing boundary is retained, without
task/document IDs, document keys, text, or bodies. Historical results may omit the
receipt. The original error, failed grade, and dependent blocked outcomes remain.
## Live provider connection journeys
See [PROVIDER-CONNECTIONS.md](PROVIDER-CONNECTIONS.md) for the explicit-only
@@ -1262,9 +1272,17 @@ intentionally uncommitted workspace patch while keeping every target job on one
identical dependency resolution. The shared-build job compiles the selected
campaign's TypeScript outputs and native binaries once, then each paid cell
verifies and extracts the immutable bundle. Remote native cells similarly reuse
one verified provider pack. Report
sanitization and AWS history publication do not consume the target lockfile;
they explicitly check out and install from the trusted workflow commit. The
one verified provider pack. Report sanitization and AWS history publication
explicitly check out the trusted workflow commit. The report applies the
reviewed, hash-pinned `report-lock.patch` to the committed lock before its
frozen install. It does not resolve registry versions at run time or accept a
worker target lock. The patch changes existing package patch identities and
links the already declared yaml dependency; existing package tarball integrity entries
stay unchanged. The correction adds exactly yaml@2.9.1 with pinned integrity. The
trusted authorization job rejects unknown locks before paid work. Manual history
publication uses the same reviewed correction. Qualification image builds apply
it only to a target equal to the workflow revision; other targets keep their own
lock. Unknown lock contents fail before installation. The
workflow definition, runner-group permission, and protected-environment
deployment still come from the default branch. Do not select the target branch
in GitHub's **Use workflow from** control.
@@ -3051,3 +3069,21 @@ codes/categories. Public pause fences automatic setup retries; failed pause
requests the existing owned supervisor's shutdown before further API observations.
Multiple probe receipts fail health. Setup-probe usage/cost is unavailable, never
zero. This diagnostic cannot qualify a catalog scenario or rewrite prior results.
### Model-spend readiness and operating reconciliation
A reviewed operating allowance can opt into `accountingReadiness: "model_requests"`.
This policy requires at least one model request and complete final model-cost
receipts before scheduled qualification succeeds. Missing or conflicting model
receipts, active requests, failed product/cleanup/trace/feedback checks, and a
campaign commitment above its limit still fail qualification. The gateway still
reserves every request before forwarding it. Attempts, deadlines and concurrency
remain bounded.
The report shows the model subtotal and model-readiness status separately from
full accounting status. Unknown operating charges remain unknown. Operating
reservations stay in the same campaign admission allowance until reconciled;
they are not reported as charges or converted into reviewed lifecycle bounds.
This mode therefore does not certify a full-lifecycle operating-cost ceiling.
Omitting the policy preserves the stricter full-accounting gate. Historical
journals and their holds are not rewritten by changing future campaign policy.
+10
View File
@@ -28,6 +28,16 @@ Unknown codes become `unrecognized`; raw bodies, messages, URLs, headers, and
browser errors are excluded. Reports validate the receipt before rendering its
explicit fields, and historical results may omit it.
The optional `onboardingPreAcceptance` receipt captures only a failing first-task
acceptance guard. Its closed version-1 fields are a known checkpoint, up to four
distinct guard-violation codes, nonnegative integer task/attachment/document
counts, a canonical-parent match boolean, and closed first-task/canonical-parent
statuses (`missing` and `unknown` remain explicit). It excludes all identifiers,
task text, document keys and bodies, filenames, URLs, credentials, and raw API
responses. Validation rejects extra or inconsistent fields before report use;
older results may omit the receipt. It preserves the original failure grade and
does not attribute a product cause or authorize acceptance.
## Attended connection authentication
See [PROVIDER-CONNECTIONS.md](PROVIDER-CONNECTIONS.md) for the explicit-only
+28 -1
View File
@@ -4,7 +4,7 @@ import { promisify } from "node:util";
import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { buildBraintrustExperiment, importBraintrustExperiment, renderBraintrustExperiment, type BraintrustExperimentReceipt } from "./braintrust-experiment.js";
import { buildBraintrustExperiment, experimentEvidenceHash, importBraintrustExperiment, renderBraintrustExperiment, type BraintrustExperimentReceipt } from "./braintrust-experiment.js";
import { regenerateRunnerDashboard } from "./dashboard-regenerate.js";
import { buildRunnerCampaign } from "./history.js";
import { runnerMatrix } from "./catalog.js";
@@ -326,3 +326,30 @@ it.each(["public", "wrong-project", "wrong-hash", "partial-insert", "receiver-er
expect(JSON.stringify(receipts)).not.toMatch(/private-key|private-upstream/);
if (["partial-insert", "receiver-error"].includes(kind)) expect(receipts[0]).toMatchObject({ status: "created", acceptedRows: 0 });
});
it("preserves retained experiment identity when adding journal-derived token presentation", async () => {
const { openEvalCostLedger } = await import("./cost-ledger.js");
const { readEvalOperationsReport } = await import("./operations-report.js");
const { campaign, originals } = fixture();
const root = await mkdtemp(path.join(tmpdir(), "experiment-token-presentation-")), file = path.join(root, "ledger.jsonl");
const ledger = await openEvalCostLedger(file, { campaignId: campaign.campaignId, maxUsd: "1", deadlineAt: "2099-01-01T00:00:00Z", maxRequests: 2, maxConcurrent: 1 });
try {
await ledger.reserve("setup", { executionId: originals[0]!.executionId, attemptId: "first", phase: "setup" }, "0.2");
await ledger.observe({ observationId: "usage", requestId: "setup", provider: "fixture", source: "gateway", currency: "USD", route: "direct", classification: "reported", costUsd: "0.05", final: true,
tokens: { input: 1, cachedRead: 2, cachedWrite: 3, output: 4 } });
await ledger.finish("setup", "cancelled");
const enriched = await readEvalOperationsReport(file);
if (enriched.status === "unavailable") throw new Error("Expected token report");
campaign.operations = structuredClone(enriched);
delete campaign.operations.modelAccounting!.tokenUsage;
const legacyHash = experimentEvidenceHash(campaign);
const receipt = await importBraintrustExperiment({ projectId, apiKey: "key" }, campaign, originals, async () => {}, receiver(campaign, originals));
campaign.braintrustExperiment = receipt;
campaign.operations = enriched;
expect(experimentEvidenceHash(campaign)).toBe(legacyHash);
expect(renderBraintrustExperiment(campaign)).toContain("2/2 rows accepted");
campaign.operations.journalHash = "f".repeat(64);
expect(() => renderBraintrustExperiment(campaign)).toThrow("Invalid retained");
} finally { await ledger.close(); await rm(root, { recursive: true, force: true }); }
});
+3 -1
View File
@@ -114,13 +114,15 @@ export interface BraintrustExperimentReceipt {
}
export function experimentEvidenceHash(campaign: RunnerE2ECampaign) {
const operations = campaign.operations ? structuredClone(campaign.operations) : null;
if (operations && operations.status !== "unavailable" && operations.modelAccounting) delete operations.modelAccounting.tokenUsage;
return digest({ campaignId: campaign.campaignId, expected: [...campaign.expected].sort(),
selected: campaign.results.map(result => ({ executionId: result.executionId, accountingIdentity: result.accountingIdentity,
attempt: result.attempt, startedAt: result.attempt === 0 ? null : result.startedAt, sourceRevision: result.source?.sha ?? null,
suiteDefinitionHash: result.suiteDefinitionHash ?? null, suiteId: result.suiteId, profileId: result.profileId,
caseId: result.caseId, provider: result.provider, model: result.model, environmentId: result.environmentId,
runtimeMode: result.runtimeMode })).sort((a, b) => digest(a).localeCompare(digest(b))),
accounting: campaign.accounting, operations: campaign.operations ?? null, feedbackReview: campaign.feedbackReview ?? null });
accounting: campaign.accounting, operations, feedbackReview: campaign.feedbackReview ?? null });
}
function safeBraintrustUrl(value: unknown): value is string {
+10
View File
@@ -211,3 +211,13 @@ it("retains finalization export failures without changing the recorded product r
}, transport)).toMatchObject({ status: "failed", errorCode: "invalid_finalization_context" });
expect(transport).not.toHaveBeenCalled();
});
it("binds remote OTLP to the explicitly assigned eval gateway origin", () => {
const origin = "https://synthetic-eval.trycloudflare.com";
const env = { PAPERCLIP_E2E_BRAINTRUST_PROJECT_ID: projectId, BRAINTRUST_API_KEY: "synthetic-token",
PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN: origin, PAPERCLIP_E2E_OTLP_RELAY_URL: `${origin}/otel` };
expect(evalBraintrustConfig(env)).toMatchObject({ remoteOrigin: origin, endpointBaseUrl: `${origin}/otel` });
expect(() => evalBraintrustConfig({ ...env, PAPERCLIP_E2E_OTLP_RELAY_URL: "https://different-eval.trycloudflare.com/otel" })).toThrow();
expect(buildPaperclipServerEnvironment(env).PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN).toBeUndefined();
});
+5 -2
View File
@@ -1,13 +1,15 @@
import { randomBytes } from "node:crypto";
import { createRequire } from "node:module";
import { isEvalRemoteOrigin } from "./eval-remote-gateway.js";
import type { RunnerApi } from "./api.js";
import type { FailureClass, RunnerE2EResult } from "./types.js";
export const EVAL_OBSERVABILITY_SECRET_NAMES = ["BRAINTRUST_API_KEY"] as const;
const uuid = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
export interface EvalBraintrustConfig { projectId: string; apiKey: string; endpointBaseUrl?: string }
export interface EvalBraintrustConfig { projectId: string; apiKey: string; endpointBaseUrl?: string; remoteOrigin?: string }
function evalOtlpEndpoint(config: EvalBraintrustConfig) {
if (config.endpointBaseUrl !== undefined && !/^http:\/\/127\.0\.0\.1:[1-9][0-9]{0,4}\/otel$/.test(config.endpointBaseUrl)) {
if (config.remoteOrigin !== undefined && !isEvalRemoteOrigin(config.remoteOrigin)) throw new Error("Invalid eval remote OTLP origin");
if (config.endpointBaseUrl !== undefined && !(config.remoteOrigin && config.endpointBaseUrl === `${config.remoteOrigin}/otel`) && !/^http:\/\/127\.0\.0\.1:[1-9][0-9]{0,4}\/otel$/.test(config.endpointBaseUrl)) {
throw new Error("Eval OTLP relay must use the parent-owned loopback endpoint");
}
if (config.endpointBaseUrl && Number(new URL(config.endpointBaseUrl).port) > 65535) throw new Error("Invalid eval OTLP relay port");
@@ -20,6 +22,7 @@ export function evalBraintrustConfig(env: NodeJS.ProcessEnv): EvalBraintrustConf
throw new Error("Eval Braintrust export requires an explicit project UUID and API key");
}
const config = { projectId, apiKey: env.BRAINTRUST_API_KEY.trim(),
...(env.PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN ? { remoteOrigin: env.PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN } : {}),
...(env.PAPERCLIP_E2E_OTLP_RELAY_URL ? { endpointBaseUrl: env.PAPERCLIP_E2E_OTLP_RELAY_URL } : {}) };
evalOtlpEndpoint(config);
return config;
+26 -4
View File
@@ -21,7 +21,7 @@ it("keeps asynchronous hello probes in setup until the same agent is ready", asy
const get = vi.fn().mockResolvedValueOnce({ ...ready, status: "paused", lifecycleState: "verifying", adapterConfig: { private: true } }).mockResolvedValue(ready);
let saved: EvalSetupObservation[] = [];
try {
await finishBudgetedEvalSetup({ executionId: identity.executionId, agentId: "agent", companyId: "company",
await finishBudgetedEvalSetup({ executionId: identity.executionId, environmentId: "local", agentId: "agent", companyId: "company",
deadlineAt: Date.now() + 5_000, api: { get } as unknown as RunnerApi,
observe: async observations => {
expect((await readEvalAttemptPhase(f.file, identity)).phase).toBe("setup");
@@ -43,7 +43,7 @@ it.each([
] as const)("preserves setup attribution and the failed state when readiness is rejected: %s", async (state, reason) => {
const f = await fixture(), observe = vi.fn(async () => {}), get = vi.fn().mockResolvedValue(state);
try {
await expect(finishBudgetedEvalSetup({ executionId: identity.executionId, agentId: "agent", companyId: "company",
await expect(finishBudgetedEvalSetup({ executionId: identity.executionId, environmentId: "local", agentId: "agent", companyId: "company",
deadlineAt: Date.now() + 5_000, api: { get } as unknown as RunnerApi, observe }, f.env)).rejects.toThrow(reason);
expect(get).toHaveBeenCalledOnce();
expect(observe).toHaveBeenCalledOnce();
@@ -54,7 +54,7 @@ it.each([
it("does not start a readiness request after the attempt deadline", async () => {
const f = await fixture(), get = vi.fn();
try {
await expect(finishBudgetedEvalSetup({ executionId: identity.executionId, agentId: "agent", companyId: "company",
await expect(finishBudgetedEvalSetup({ executionId: identity.executionId, environmentId: "local", agentId: "agent", companyId: "company",
deadlineAt: Date.now() - 1, api: { get } as unknown as RunnerApi, observe: async () => {} }, f.env)).rejects.toThrow("Timed out");
expect(get).not.toHaveBeenCalled();
expect((await readEvalAttemptPhase(f.file, identity)).phase).toBe("setup");
@@ -66,10 +66,32 @@ it("does not advance when saving readiness crosses the deadline", async () => {
const deadlineAt = Date.now() + 5_000;
const now = vi.spyOn(Date, "now");
try {
await expect(finishBudgetedEvalSetup({ executionId: identity.executionId, agentId: "agent", companyId: "company",
await expect(finishBudgetedEvalSetup({ executionId: identity.executionId, environmentId: "local", agentId: "agent", companyId: "company",
deadlineAt, api: { get } as unknown as RunnerApi,
observe: async () => { now.mockReturnValue(deadlineAt); } }, f.env)).rejects.toThrow("Timed out saving");
expect(get).toHaveBeenCalledOnce();
expect((await readEvalAttemptPhase(f.file, identity)).phase).toBe("setup");
} finally { now.mockRestore(); await rm(f.directory, { recursive: true, force: true }); }
});
it.each(["local", "daytona"] as const)("uses the bounded attempt window for remote setup while keeping the local limit: %s", async environmentId => {
const f = await fixture();
let time = Date.now();
const now = vi.spyOn(Date, "now").mockImplementation(() => time);
const get = vi.fn().mockResolvedValueOnce({ ...ready, status: "paused", lifecycleState: "verifying" }).mockResolvedValue(ready);
let observations = 0;
try {
const result = finishBudgetedEvalSetup({ executionId: identity.executionId, environmentId, agentId: "agent", companyId: "company",
deadlineAt: time + 15 * 60_000, api: { get } as unknown as RunnerApi,
observe: async () => { if (++observations === 1) time += 46_000; } }, f.env);
if (environmentId === "daytona") {
await result;
expect(get).toHaveBeenCalledTimes(2);
expect((await readEvalAttemptPhase(f.file, identity)).phase).toBe("agent");
} else {
await expect(result).rejects.toThrow("Timed out");
expect(get).toHaveBeenCalledOnce();
expect((await readEvalAttemptPhase(f.file, identity)).phase).toBe("setup");
}
} finally { now.mockRestore(); await rm(f.directory, { recursive: true, force: true }); }
});
+3 -1
View File
@@ -1,3 +1,4 @@
import type { RunnerEnvironmentId } from "./types.js";
import type { Agent } from "../../packages/shared/src/types/agent.js";
import { pollUntil, type RunnerApi } from "./api.js";
import { beginEvalAgentPhase } from "./attempt-phase.js";
@@ -7,6 +8,7 @@ export type EvalSetupObservation = SetupState & { observedAt: string };
export async function finishBudgetedEvalSetup(input: {
executionId: string;
environmentId: RunnerEnvironmentId;
agentId: string;
companyId: string;
deadlineAt: number;
@@ -14,7 +16,7 @@ export async function finishBudgetedEvalSetup(input: {
observe: (observations: EvalSetupObservation[]) => Promise<void>;
}, env: NodeJS.ProcessEnv = process.env) {
if (env.PAPERCLIP_E2E_GATEWAY_BASE_URL) {
const deadlineAt = Math.min(input.deadlineAt, Date.now() + 45_000);
const deadlineAt = input.environmentId === "daytona" ? input.deadlineAt : Math.min(input.deadlineAt, Date.now() + 45_000);
const observations: EvalSetupObservation[] = [];
let previous: string | undefined;
const scoped = (agent: SetupState) => agent.id === input.agentId && agent.companyId === input.companyId;
+16
View File
@@ -1,6 +1,7 @@
import type { RunnerE2EResult } from "./types.js";
import { isBlockedConnectionResult } from "./connection-evidence.js";
import { validateOnboardingConnectEvidence } from "./product-journeys/onboarding-connect.js";
import { validateOnboardingPreAcceptanceEvidence } from "./product-journeys/onboarding-pre-acceptance.js";
const html = (value: unknown) =>
String(value ?? "").replace(
/[&<>"']/g,
@@ -31,6 +32,20 @@ function renderOnboardingConnect(result: RunnerE2EResult): string {
${response.truncatedChecks ? "<p>Only the first 32 validation checks were retained.</p>" : ""}`;
}).join("")}</details>`;
}
function renderOnboardingPreAcceptance(result: RunnerE2EResult): string {
const receipt = result.productJourney?.onboardingPreAcceptance;
if (receipt === undefined) return "";
try { validateOnboardingPreAcceptanceEvidence(receipt); }
catch { return "<p>Onboarding pre-acceptance evidence unavailable: invalid receipt.</p>"; }
return `<details class="case-context"><summary>Onboarding pre-acceptance evidence</summary>
<p>Checkpoint: <code>${html(receipt.checkpoint)}</code></p>
<p>Observed guard violations:</p><ul>${receipt.violations.map(code => `<li><code>${html(code)}</code></li>`).join("")}</ul>
<dl>${[
["Tasks", receipt.taskCount], ["Attachments", receipt.attachmentCount], ["Documents", receipt.documentCount],
["Non-planning documents", receipt.nonPlanningDocumentCount], ["First task matches canonical parent", receipt.firstTaskMatchesCanonicalParent],
["First task status", receipt.firstTaskStatus], ["Canonical parent status", receipt.canonicalParentStatus],
].map(([label, value]) => `<div><dt>${html(label)}</dt><dd>${html(value)}</dd></div>`).join("")}</dl></details>`;
}
/** An interrupted first-task journey is not evidence of failed downstream behavior. */
export function isIncompleteFirstTaskResult(
result: RunnerE2EResult,
@@ -93,6 +108,7 @@ export function renderCaseOutcome(
${result.productJourney?.operations ? `<p>Eval operations: ${html(result.productJourney.operations.campaignId)} · ${result.productJourney.operations.requestCount} requests · model cost $${html(result.productJourney.operations.modelCostUsd ?? "unknown")} (setup $${html(result.productJourney.operations.setupCostUsd ?? "unknown")}, counted once). Feedback: ${html(result.productJourney.operations.feedback.collection)} · ${result.productJourney.operations.feedback.complaints} complaints · ${result.productJourney.operations.feedback.suggestions} suggestions.</p>` : ""}
${result.productJourney?.operations?.traceIds.length ? `<details><summary>Trace correlation IDs</summary><ul>${result.productJourney.operations.traceIds.map(id => `<li><code>${html(id)}</code></li>`).join("")}</ul></details>` : ""}
${renderOnboardingConnect(result)}
${renderOnboardingPreAcceptance(result)}
${result.providerConnection ? `<p>Target: ${html(result.providerConnection.target.origin)} · ${html(result.providerConnection.target.commit ?? "revision unavailable")} · ${result.providerConnection.assisted ? "Assisted login" : "Unassisted"} · ${html(result.providerConnection.authFreshness)} browser · ${html(result.providerConnection.entry)} / ${html(result.providerConnection.method)} · stopped at ${html(result.providerConnection.phase)}</p>` : ""}
${failures.length ? `<p>Failed checks:</p><ul>${failures.map((c) => `<li><strong>${html(c.id)}</strong> — ${html(c.detail)}</li>`).join("")}</ul>` : checks.length ? `<p>No behavioral matcher failed.${notReached.length ? " The journey is incomplete; unexercised checks are not passes." : failed ? " The overall failure came from a separate run, cleanup, or evidence check." : ""}</p>` : ""}
${notReached.length ? `<p>Not reached:</p><ul>${notReached.map((c) => `<li><strong>${html(c.id)}</strong> — ${html(c.notReached)}</li>`).join("")}</ul>` : ""}
@@ -66,6 +66,8 @@ export const DAYTONA_IMAGE_INPUT_PATHS = [
"packages/paperclip-runner/test/fixtures/copilot-profile-v57-identity.json",
"packages/paperclip-runner/test/fixtures/copilot-profile-v58-identity.json",
"packages/paperclip-runner/test/fixtures/copilot-profile-v61-identity.json",
"packages/paperclip-runner/test/fixtures/copilot-profile-v65-identity.json",
"packages/paperclip-runner/test/fixtures/copilot-profile-v66-identity.json",
"packages/paperclip-runner/scripts/materialize-cursor-distribution.mjs",
"packages/paperclip-runner/scripts/cursor-runtime-patch.mjs",
"packages/paperclip-runner/scripts/build-verified-provider-entrypoints.mjs",
@@ -3,6 +3,7 @@ import { tmpdir } from "node:os";
import path from "node:path";
import { afterEach, expect, it, vi } from "vitest";
import { openEvalCostLedger, readEvalCostLedger } from "./cost-ledger.js";
import { startEvalRemoteBridge } from "./eval-remote-gateway.js";
import { forwardEvalRequest, startEvalGateway, type EvalGatewayReceipt, type EvalGatewayRoute } from "./eval-gateway.js";
const cleanups: Array<() => Promise<void>> = [];
@@ -181,3 +182,19 @@ it.each(["persist", "observe", "finish"] as const)("records a disconnect during
expect(f.ledger.snapshot()).toMatchObject({ knownUsd: "0.000070000", heldUsd: "0.000000000", unknownRequests: 0 });
expect(transport).toHaveBeenCalledOnce();
});
it("retains terminal usage when a remote bridge client disconnects during shutdown", async () => {
const f = await fixture(), started = deferred<void>(), release = deferred<void>();
const transport = vi.fn<typeof fetch>(async () => { started.resolve(); await release.promise; return finalResponse("openai"); });
const gateway = await startEvalGateway({ ...f, transport }); cleanups.push(() => gateway.close());
const bridge = await startEvalRemoteBridge({ gateway, deadlineAt: f.deadlineAt }); cleanups.push(() => bridge.close());
const client = new AbortController();
const result = fetch(bridge.baseUrl + "/responses", { method: "POST", signal: client.signal,
headers: { authorization: `Bearer ${gateway.token}` }, body: JSON.stringify({ model: "fixture", input: "hello" }) }).catch(error => error);
await started.promise; client.abort(); await bridge.close();
const closing = gateway.close(); release.resolve(); await closing;
expect(await result).not.toBeInstanceOf(Response);
expect(transport).toHaveBeenCalledOnce();
expect(f.receipts.at(-1)).toMatchObject({ clientDisconnected: true, final: true, usageComplete: true, costUsd: "0.000070000" });
expect(f.ledger.snapshot()).toMatchObject({ knownUsd: "0.000070000", heldUsd: "0.000000000", unknownRequests: 0, inflight: 0 });
});
@@ -0,0 +1,101 @@
import { mkdtemp, readFile, writeFile, rm, readdir } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { createServer } from "node:http";
import { once } from "node:events";
import { expect, it, vi } from "vitest";
import { isEvalRemoteOrigin, startEvalRemoteBridge, startEvalRemoteGateway } from "./eval-remote-gateway.js";
it("forwards only authenticated model/OTLP routes and closes owned exposure", async () => {
const calls: Array<{ url: string; auth?: string; body: string }> = [];
const upstream = createServer(async (request, response) => {
let body = ""; for await (const chunk of request) body += chunk;
calls.push({ url: request.url!, auth: request.headers.authorization, body });
response.writeHead(200, { "content-type": "text/event-stream" }); response.end("data: synthetic\n\n");
});
upstream.listen(0, "127.0.0.1"); await once(upstream, "listening");
const address = upstream.address(); if (!address || typeof address === "string") throw new Error("Fixture did not bind");
const url = `http://127.0.0.1:${address.port}`, modelToken = "a".repeat(64), otlpToken = "b".repeat(64);
const bridge = await startEvalRemoteBridge({ gateway: { baseUrl: url, token: modelToken },
observability: { baseUrl: url + "/otel", token: otlpToken }, deadlineAt: Date.now() + 60_000 });
try {
expect(await (await fetch(bridge.baseUrl + "/health")).json()).toEqual({ nonce: bridge.nonce });
for (const path of ["/v1/responses", "/v1/messages?beta=true", "/otel/v1/traces", "/otel/v1/logs"]) {
expect((await fetch(bridge.baseUrl + path, { method: "POST" })).status).toBe(401);
const token = path.startsWith("/otel") ? otlpToken : modelToken;
const response = await fetch(bridge.baseUrl + path, { method: "POST", headers: { authorization: `Bearer ${token}`, "content-type": "application/json" }, body: "{}" });
expect(response.status).toBe(200); expect(await response.text()).toBe("data: synthetic\n\n");
}
expect(calls).toHaveLength(4);
expect(calls.map(call => call.url)).toEqual(["/v1/responses", "/v1/messages?beta=true", "/otel/v1/traces", "/otel/v1/logs"]);
expect(calls.map(call => call.auth)).toEqual([`Bearer ${modelToken}`, `Bearer ${modelToken}`, `Bearer ${otlpToken}`, `Bearer ${otlpToken}`]);
expect((await fetch(bridge.baseUrl + "/admin", { method: "POST", headers: { authorization: `Bearer ${modelToken}` } })).status).toBe(404);
expect((await fetch(bridge.baseUrl + "/otel/v1/traces", { method: "POST", headers: { authorization: `Bearer ${modelToken}` } })).status).toBe(401);
await bridge.close(); await expect(fetch(bridge.baseUrl + "/health")).rejects.toThrow();
} finally { await bridge.close(); upstream.closeAllConnections(); await new Promise<void>(resolve => upstream.close(() => resolve())); }
});
it("admits only HTTPS quick-tunnel origins and bounded loopback destinations", async () => {
expect(isEvalRemoteOrigin("https://synthetic-fixture.trycloudflare.com")).toBe(true);
for (const value of ["https://foreign.invalid", "http://synthetic.trycloudflare.com", "https://synthetic.trycloudflare.com/path", "https://synthetic.trycloudflare.com?token=private"]) expect(isEvalRemoteOrigin(value)).toBe(false);
await expect(startEvalRemoteBridge({ gateway: { baseUrl: "https://foreign.invalid", token: "a".repeat(64) }, deadlineAt: Date.now() + 60_000 })).rejects.toThrow("policy");
});
it("fails before remote admission when the tunnel executable is unavailable", async () => {
await expect(startEvalRemoteGateway({ gateway: { baseUrl: "http://127.0.0.1:1", token: "a".repeat(64) },
deadlineAt: Date.now() + 5000 }, { PAPERCLIP_EVAL_CLOUDFLARED: "/missing/synthetic-cloudflared", PATH: "" }))
.rejects.toThrow("Remote eval tunnel executable unavailable");
});
it("closes remote exposure at its original campaign deadline", async () => {
const bridge = await startEvalRemoteBridge({ gateway: { baseUrl: "http://127.0.0.1:1", token: "a".repeat(64) }, deadlineAt: Date.now() + 50 });
try {
await new Promise(resolve => setTimeout(resolve, 100));
await expect(fetch(bridge.baseUrl + "/health")).rejects.toThrow();
} finally { await bridge.close(); }
});
it.each(["explicit", "empty-override"])("waits for registration and the correct public nonce, then terminates the owned tunnel: %s", async mode => {
const root = await mkdtemp(path.join(tmpdir(), "eval-fake-tunnel-"));
const executable = path.join(root, "cloudflared"), state = path.join(root, "state.json"), stopped = path.join(root, "stopped");
const origin = "https://synthetic-owned.trycloudflare.com";
const script = `#!${process.execPath}
const fs = require("node:fs");
fs.writeFileSync(${JSON.stringify(state)}, JSON.stringify({baseUrl:process.argv.at(-1),pid:process.pid,env:process.env}));
process.stderr.write("${origin}\\n");
setTimeout(() => process.stderr.write("Registered tunnel connection\\n"), 20);
process.on("SIGTERM", () => { fs.writeFileSync(${JSON.stringify(stopped)}, "stopped"); process.exit(0); });
setInterval(() => {}, 1000);
`;
await writeFile(executable, script, { mode: 0o700 });
const realFetch = globalThis.fetch; let healthChecks = 0;
const fetchSpy = vi.spyOn(globalThis, "fetch").mockImplementation(async url => {
expect(String(url)).toBe(origin + "/health");
if (++healthChecks === 1) return new Response(JSON.stringify({ nonce: "foreign" }), { status: 200 });
const { baseUrl } = JSON.parse(await readFile(state, "utf8"));
return realFetch(baseUrl + "/health");
});
let remote: Awaited<ReturnType<typeof startEvalRemoteGateway>> | undefined;
try {
remote = await startEvalRemoteGateway({ gateway: { baseUrl: "http://127.0.0.1:1", token: "a".repeat(64) },
deadlineAt: Date.now() + 10_000 }, { PAPERCLIP_EVAL_CLOUDFLARED: mode === "explicit" ? executable : "", PATH: root, OPENAI_API_KEY: "must-not-leave" });
expect(remote.baseUrl).toBe(origin); expect(healthChecks).toBe(2);
const { baseUrl, pid, env } = JSON.parse(await readFile(state, "utf8"));
expect(env.OPENAI_API_KEY).toBeUndefined(); expect(env.HOME).not.toBe(process.env.HOME);
await remote.close();
expect(await readFile(stopped, "utf8")).toBe("stopped");
expect(() => process.kill(pid, 0)).toThrow();
await expect(realFetch(baseUrl + "/health")).rejects.toThrow();
await expect(readFile(path.join(env.HOME, "anything"))).rejects.toThrow();
} finally { fetchSpy.mockRestore(); await remote?.close(); await rm(root, { recursive: true, force: true }); }
}, 15_000);
it("cleans the private tunnel home after synchronous spawn rejection", async () => {
const before = new Set((await readdir(tmpdir())).filter(name => name.startsWith("eval-cloudflare-")));
await expect(startEvalRemoteGateway({ gateway: { baseUrl: "http://127.0.0.1:1", token: "a".repeat(64) },
deadlineAt: Date.now() + 60_000 }, { PAPERCLIP_EVAL_CLOUDFLARED: "invalid\0executable", PATH: "" })).rejects.toThrow();
expect((await readdir(tmpdir())).filter(name => name.startsWith("eval-cloudflare-") && !before.has(name))).toEqual([]);
});
+119
View File
@@ -0,0 +1,119 @@
import { spawn } from "node:child_process";
import { randomBytes, timingSafeEqual } from "node:crypto";
import { once } from "node:events";
import { mkdtemp, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { createServer, request as httpRequest } from "node:http";
export const isEvalRemoteOrigin = (value: string) => /^https:\/\/[a-z0-9-]+\.trycloudflare\.com$/.test(value);
export async function startEvalRemoteBridge(input: {
gateway: { baseUrl: string; token: string };
observability?: { baseUrl: string; token: string };
deadlineAt: number;
}) {
if (!Number.isFinite(input.deadlineAt) || input.deadlineAt <= Date.now() || input.deadlineAt > Date.now() + 4 * 3_600_000
|| !/^http:\/\/127\.0\.0\.1:[1-9][0-9]{0,4}$/.test(input.gateway.baseUrl)
|| input.observability && !/^http:\/\/127\.0\.0\.1:[1-9][0-9]{0,4}\/otel$/.test(input.observability.baseUrl)
|| !/^[a-f0-9]{64}$/.test(input.gateway.token)
|| input.observability && !/^[a-f0-9]{64}$/.test(input.observability.token)) throw new Error("Invalid remote eval bridge policy");
try { new URL(input.gateway.baseUrl); if (input.observability) new URL(input.observability.baseUrl); }
catch { throw new Error("Invalid remote eval bridge policy"); }
const nonce = randomBytes(32).toString("hex");
let closed = false;
const server = createServer((incoming, outgoing) => {
if (closed || Date.now() >= input.deadlineAt) { outgoing.writeHead(503).end(); return; }
if (incoming.method === "GET" && incoming.url === "/health") {
outgoing.writeHead(200, { "content-type": "application/json", "cache-control": "no-store" }).end(JSON.stringify({ nonce })); return;
}
const model = ["/responses", "/v1/responses", "/messages", "/messages?beta=true", "/v1/messages", "/v1/messages?beta=true"].includes(incoming.url ?? "");
const otlp = ["/otel/v1/traces", "/otel/v1/logs"].includes(incoming.url ?? "");
const destination = model ? input.gateway : otlp ? input.observability : undefined;
if (incoming.method !== "POST" || !destination) { outgoing.writeHead(404).end(); return; }
const auth = String(incoming.headers.authorization ?? "").replace(/^Bearer /, "") || String(incoming.headers["x-api-key"] ?? "");
const presented = Buffer.from(auth), expected = Buffer.from(destination.token);
if (presented.length !== expected.length || !timingSafeEqual(presented, expected)) { outgoing.writeHead(401).end(); return; }
const suffix = otlp ? incoming.url!.slice("/otel".length) : incoming.url!;
const headers: Record<string, string> = { authorization: `Bearer ${destination.token}` };
for (const name of ["content-type", "content-encoding", "anthropic-beta", "anthropic-version", "x-bt-parent"]) {
const value = incoming.headers[name]; if (typeof value === "string") headers[name] = value;
}
const upstream = httpRequest(destination.baseUrl + suffix, { method: "POST", headers }, response => {
if (outgoing.destroyed) { response.destroy(); return; }
outgoing.writeHead(response.statusCode ?? 502, response.headers); response.pipe(outgoing);
response.on("error", () => outgoing.destroy());
});
upstream.on("error", () => { if (!outgoing.headersSent) outgoing.writeHead(502).end(); else outgoing.destroy(); });
incoming.on("aborted", () => upstream.destroy());
incoming.on("error", () => upstream.destroy());
outgoing.on("error", () => upstream.destroy());
outgoing.on("close", () => { if (!outgoing.writableEnded) upstream.destroy(); });
incoming.pipe(upstream);
});
server.requestTimeout = 15_000; server.headersTimeout = 10_000;
server.listen(0, "127.0.0.1"); await once(server, "listening");
const address = server.address();
if (!address || typeof address === "string") throw new Error("Remote eval bridge did not bind loopback");
let timer: ReturnType<typeof setTimeout>;
let closing: Promise<void> | undefined;
const close = () => closing ??= (async () => {
closed = true; clearTimeout(timer);
server.closeAllConnections(); await new Promise<void>(resolve => server.close(() => resolve()));
})();
timer = setTimeout(() => { void close().catch(() => {}); }, input.deadlineAt - Date.now()); timer.unref();
return { baseUrl: `http://127.0.0.1:${address.port}`, nonce, close };
}
export async function startEvalRemoteGateway(input: Parameters<typeof startEvalRemoteBridge>[0], env: NodeJS.ProcessEnv) {
const bridge = await startEvalRemoteBridge(input);
let home: string;
try { home = await mkdtemp(path.join(tmpdir(), "eval-cloudflare-")); }
catch (error) { await bridge.close(); throw error; }
let tunnel: ReturnType<typeof spawn> | undefined;
let closing: Promise<void> | undefined, deadlineTimer: ReturnType<typeof setTimeout> | undefined;
const close = () => closing ??= (async () => {
clearTimeout(deadlineTimer);
const errors: unknown[] = [];
try { await bridge.close(); } catch (error) { errors.push(error); }
try {
if (tunnel?.pid && tunnel.exitCode === null && tunnel.signalCode === null) {
const exited = once(tunnel, "exit").catch(() => {});
tunnel.kill("SIGTERM"); const timer = setTimeout(() => tunnel?.kill("SIGKILL"), 5000);
try { await exited; } finally { clearTimeout(timer); }
}
} catch (error) { errors.push(error); }
try { await rm(home, { recursive: true, force: true }); } catch (error) { errors.push(error); }
if (errors.length) throw new AggregateError(errors, "Remote eval exposure cleanup failed");
})();
try {
tunnel = spawn(env.PAPERCLIP_EVAL_CLOUDFLARED?.trim() || "cloudflared",
["tunnel", "--no-autoupdate", "--protocol", "http2", "--url", bridge.baseUrl],
{ stdio: ["ignore", "pipe", "pipe"], env: { PATH: env.PATH, HOME: home, TMPDIR: home } });
deadlineTimer = setTimeout(() => { void close().catch(() => {}); }, input.deadlineAt - Date.now()); deadlineTimer.unref();
const child = tunnel;
const origin = await new Promise<string>((resolve, reject) => {
let output = "";
const timer = setTimeout(() => finish(new Error("Remote eval tunnel startup timed out")), Math.min(120_000, input.deadlineAt - Date.now()));
const finish = (error?: Error, value?: string) => { clearTimeout(timer); if (error) reject(error); else resolve(value!); };
child.once("error", () => finish(new Error("Remote eval tunnel executable unavailable")));
child.once("exit", () => finish(new Error("Remote eval tunnel exited")));
const read = (chunk: Buffer) => {
output = (output + chunk.toString()).slice(-32768);
const value = output.match(/https:\/\/[a-z0-9-]+\.trycloudflare\.com/)?.[0];
if (value && output.includes("Registered tunnel connection")) finish(undefined, value);
};
child.stdout!.on("data", read); child.stderr!.on("data", read);
});
const readyBy = Math.min(Date.now() + 120_000, input.deadlineAt);
while (Date.now() < readyBy) {
try {
const response = await fetch(origin + "/health", { redirect: "error", signal: AbortSignal.timeout(Math.min(5000, readyBy - Date.now())) });
if (response.ok && (await response.json() as { nonce?: string }).nonce === bridge.nonce) return { baseUrl: origin, close };
await response.body?.cancel();
} catch {}
await new Promise(resolve => setTimeout(resolve, 1000));
}
throw new Error("Remote eval tunnel not publicly reachable before admission");
} catch (error) { await close(); throw error; }
}
+9 -4
View File
@@ -5,6 +5,7 @@ import { installedDaytonaPluginKeys } from "./installed-daytona-plugin.js";
import { QUALIFIED_ACPX_PROFILES } from "../../packages/paperclip-runner/src/drivers/acpx/qualified-profiles.js";
import { assertInstalledCliSelection, installedCliKeys, usesInstalledCli } from "./installed-cli.js";
import path from "node:path";
import { isEvalRemoteOrigin } from "./eval-remote-gateway.js";
import { chatNeedsApiTools, isManagedHiringCase } from "./chat-cases.js";
import { CREDENTIAL_NAMES } from "./types.js";
import type { MatrixExecution } from "./types.js";
@@ -17,6 +18,7 @@ const AMBIENT_PAPERCLIP_CREDENTIAL_KEYS = [
"PAPERCLIP_E2E_GATEWAY_TOKEN",
"PAPERCLIP_E2E_GATEWAY_PHASE_PATH",
"PAPERCLIP_E2E_OTLP_RELAY_URL",
"PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN",
"PAPERCLIP_SETUP_TOKEN",
"PAPERCLIP_SECRETS_MASTER_KEY",
"PAPERCLIP_SECRETS_MASTER_KEY_FILE",
@@ -51,6 +53,8 @@ export function applyEvalGatewayEnvironment(env: NodeJS.ProcessEnv, credential:
|| PROVIDER_SECRET_KEY.test(name) && /KEY|TOKEN|SECRET|PASSWORD/.test(name)) delete env[name];
}
delete env.PAPERCLIP_E2E_OTLP_RELAY_URL;
delete env.PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN;
if (isEvalRemoteOrigin(gateway.baseUrl)) env.PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN = gateway.baseUrl;
env[credential] = gateway.token;
env.PAPERCLIP_E2E_GATEWAY_BASE_URL = gateway.baseUrl;
env.PAPERCLIP_E2E_GATEWAY_TOKEN = gateway.token;
@@ -113,13 +117,13 @@ export function resolvePaperclipRemoteRunnerBinaryForHarness(
runnerBinary: string | undefined,
configuredPath = process.env.PAPERCLIP_RUNNER_REMOTE_BINARY_PATH,
platform: NodeJS.Platform = process.platform,
architecture: NodeJS.Architecture = process.arch,
): string | undefined {
if (configuredPath?.trim()) return configuredPath;
if (!runnerBinary) return undefined;
// Daytona runs Linux. A default debug binary built by a macOS developer is
// Mach-O and cannot be staged into that sandbox. Leave the remote override
// unset so the pinned Daytona image's verified runnerd is discovered instead.
if (platform !== "linux") return undefined;
// The pinned Daytona runtime is Linux/x64. Other controller binaries cannot
// run there; discover its verified runnerd unless an explicit cross-build is set.
if (platform !== "linux" || architecture !== "x64") return undefined;
return executions.some(
(execution) =>
execution.profile.generation === "native" &&
@@ -147,6 +151,7 @@ export function buildRunnerE2EProcessEnvironment(
else scrubChatPrivateEnvironment(result, []);
delete result.PAPERCLIP_E2E_GATEWAY_PHASE_PATH;
delete result.PAPERCLIP_E2E_OTLP_RELAY_URL;
delete result.PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN;
// Announcements are unrelated to the scenarios and obscure screenshot evidence.
result.PAPERCLIP_ANNOUNCEMENTS_ENABLED = "false";
delete result.OPENCODE_ALLOW_ALL_MODELS;
+3
View File
@@ -631,6 +631,9 @@ async function runAttempt(input: {
};
if (gateway) {
applyEvalGatewayEnvironment(childEnv, execution.profile.credential, gateway);
// The fixture/server controller needs Daytona provisioning access. The
// remote provider still receives only the scoped model and OTLP tokens.
if (execution.environment.id === "daytona") childEnv.DAYTONA_API_KEY = process.env.DAYTONA_API_KEY;
}
// The database URLs are stripped here and again at the Playwright web-server
// boundary so a developer's shell can never redirect this paid test.
+5 -2
View File
@@ -1,4 +1,5 @@
import { CHAT_SETUP_HEALTH_SUITE } from "./chat-setup-health.js";
import { isEvalRemoteOrigin } from "./eval-remote-gateway.js";
import type { AiConnectionBinding } from "../../packages/shared/src/ai-connections.js";
import { verifyInstalledDaytonaPlugin } from "./installed-daytona-plugin.js";
import { NATIVE_COMPLETION_BUDGET_CENTS } from "./native-completion-defaults.js";
@@ -283,8 +284,9 @@ export async function setupLiveFixtures(input: {
const gatewayUrl = process.env.PAPERCLIP_E2E_GATEWAY_BASE_URL;
const gatewayToken = process.env.PAPERCLIP_E2E_GATEWAY_TOKEN;
const budgetedGateway = Boolean(gatewayUrl || gatewayToken);
if (budgetedGateway && (!gatewayUrl || !gatewayToken || !/^http:\/\/127\.0\.0\.1:[0-9]+$/.test(gatewayUrl)
|| execution.environment.id !== "local" || !["core-compatibility", "feedback-persistence"].includes(execution.suite.id))) throw new Error("Unqualified fixture gateway route");
if (budgetedGateway && (!gatewayUrl || !gatewayToken || !(execution.environment.id === "local" ? /^http:\/\/127\.0\.0\.1:[0-9]+$/.test(gatewayUrl)
: isEvalRemoteOrigin(gatewayUrl) && gatewayUrl === process.env.PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN)
|| !["local", "daytona"].includes(execution.environment.id) || !["core-compatibility", "feedback-persistence"].includes(execution.suite.id))) throw new Error("Unqualified fixture gateway route");
if (managedHiring || savedCopilot || budgetedGateway) {
registry.register<ManagedAccountFixture>({
id: "ai-connection",
@@ -386,6 +388,7 @@ export async function setupLiveFixtures(input: {
if (managedHiring || savedCopilot || budgetedGateway) {
const account = value<ManagedAccountFixture>(resolved, "ai-connection");
const config = agent.adapterConfig as Record<string, unknown>;
if (budgetedGateway && execution.environment.id === "daytona") config.helloProbeTimeoutSec = 300;
delete config.env;
agent.runtimeConfig = {
...(agent.runtimeConfig as Record<string, unknown>),
+52 -2
View File
@@ -3,8 +3,10 @@ import { tmpdir } from "node:os";
import path from "node:path";
import { expect, it } from "vitest";
import { EvalSpendState, openEvalCostLedger, readEvalCostLedger, type EvalSpendLimits } from "./cost-ledger.js";
import { EVAL_OVERHEAD_COMPONENTS, scheduledOperatingOverhead, validateOperatingOverhead, type EvalOperatingOverhead } from "./operating-overhead.js";
import { readEvalOperationsReport, renderEvalOperationsReport } from "./operations-report.js";
import { EVAL_OVERHEAD_COMPONENTS, operatingReservationBound, scheduledOperatingOverhead, validateOperatingOverhead, type EvalOperatingOverhead } from "./operating-overhead.js";
import { evalAccountingStatus, evalReadinessStatus, readEvalOperationsReport, renderEvalOperationsReport } from "./operations-report.js";
import { costBoundConfigurationSha256 } from "./cost-bound-evidence.js";
const now = Date.parse("2026-10-21T00:00:00Z");
const allowance = (): EvalOperatingOverhead => ({ version: 1, reference: "fixture-only", runner: "aws-fleet",
@@ -117,3 +119,51 @@ it("stops new admission after allowance expiry while preserving late invoice rec
expect(() => state.validate({ type: "observe", observation: { observationId: "late-invoice", requestId: "overhead:ci_compute",
source: "invoice", provider: "aws", currency: "USD", route: "infrastructure", classification: "reconciled", costUsd: "0.5", final: true } }, expiredAt)).not.toThrow();
});
it("separates model readiness from unresolved overhead without releasing holds or masking model gaps", async () => {
const directory = await mkdtemp(path.join(tmpdir(), "eval-model-readiness-"));
const file = path.join(directory, "ledger.jsonl");
const overhead = { ...currentAllowance(), accountingReadiness: "model_requests" as const };
const ledger = await openEvalCostLedger(file, { campaignId: "model-readiness", maxUsd: "20",
deadlineAt: "2099-01-01T00:00:00Z", maxRequests: 10, maxConcurrent: 1, maxAttemptsPerExecution: 1, overhead });
try {
expect(evalReadinessStatus(await readEvalOperationsReport(file))).toBe("blocked");
await ledger.admitAttempt(modelIdentity.executionId, modelIdentity.attemptId);
await ledger.reserve("model", modelIdentity, "2");
expect(evalReadinessStatus(await readEvalOperationsReport(file))).toBe("blocked");
await ledger.finish("model", "cancelled");
expect(evalReadinessStatus(await readEvalOperationsReport(file))).toBe("blocked");
await ledger.observe({ observationId: "model-final", requestId: "model", source: "provider", provider: "anthropic",
currency: "USD", route: "direct", classification: "reported", costUsd: "0.5", final: true });
const report = await readEvalOperationsReport(file);
expect(evalAccountingStatus(report)).toBe("blocked");
expect(evalReadinessStatus(report)).toBe("receipts_complete");
expect(evalReadinessStatus({ ...report, accountingReadiness: undefined } as never)).toBe("blocked");
expect(report).toMatchObject({ accountingReadiness: "model_requests", heldUsd: "4.000000000",
committedUsd: "4.500000000", modelAccounting: { requests: 1, knownUsd: "0.500000000", unresolvedRequests: 0 } });
expect(renderEvalOperationsReport(report)).toContain("Operating costs remain separately unreconciled");
await ledger.observe({ observationId: "overrun", requestId: "overhead:ci_compute", source: "invoice", provider: "aws",
currency: "USD", route: "infrastructure", classification: "reconciled", costUsd: "21", final: true });
expect(evalReadinessStatus(await readEvalOperationsReport(file))).toBe("blocked");
} finally { await ledger.close(); await rm(directory, { recursive: true, force: true }); }
});
it("rejects unknown readiness policies instead of silently loosening accounting", () => {
expect(() => validateOperatingOverhead({ ...allowance(), accountingReadiness: "ignore_costs" } as never, "20"))
.toThrow("Invalid accounting readiness policy");
});
it("binds optional readiness policy to reviewed operating evidence without changing legacy hash recipes", () => {
for (const accountingReadiness of [undefined, "model_requests"] as const) {
const value = { ...allowance(), ...(accountingReadiness ? { accountingReadiness } : {}) };
const component = value.components[0]!;
const calculation = { reference: value.reference, runner: value.runner, verifiedAt: value.verifiedAt,
expiresAt: value.expiresAt, component: { ...component }, ...(accountingReadiness ? { accountingReadiness } : {}) };
component.provisionalBound = { version: 1, scope: "full_lifecycle", evidenceSha256: "a".repeat(64),
configurationSha256: costBoundConfigurationSha256(calculation) };
expect(operatingReservationBound(value, component)?.calculationSha256).toBe(component.provisionalBound.configurationSha256);
const changed = { ...value, accountingReadiness: accountingReadiness ? undefined : "model_requests" as const };
expect(() => operatingReservationBound(changed, component)).toThrow("Reviewed cost bound configuration changed");
}
});
+4 -1
View File
@@ -4,6 +4,7 @@ import { usdToUnits } from "../../packages/shared/src/money.js";
export const EVAL_OVERHEAD_COMPONENTS = ["ci_compute", "artifact_storage", "observability", "shared_subscription"] as const;
export interface EvalOperatingOverhead {
version: 1;
accountingReadiness?: "model_requests";
reference: string;
runner: "github-hosted" | "aws-fleet";
verifiedAt: string;
@@ -20,13 +21,15 @@ const reference = (value: unknown) => typeof value === "string" && /^[A-Za-z0-9]
export function operatingReservationBound(allowance: EvalOperatingOverhead, component: EvalOperatingOverhead["components"][number]) {
if (component.provisionalBound === undefined) return undefined;
return reservationBound("operating_allowance", component.provisionalBound, {
...(allowance.accountingReadiness ? { accountingReadiness: allowance.accountingReadiness } : {}),
reference: allowance.reference, runner: allowance.runner, verifiedAt: allowance.verifiedAt, expiresAt: allowance.expiresAt,
component: { component: component.component, maxUsd: component.maxUsd, reference: component.reference },
});
}
export function validateOperatingOverhead(value: EvalOperatingOverhead, totalUsd: string): bigint {
object(value, ["version", "reference", "runner", "verifiedAt", "expiresAt", "components"]);
object(value, ["version", "accountingReadiness", "reference", "runner", "verifiedAt", "expiresAt", "components"]);
if (value.accountingReadiness !== undefined && value.accountingReadiness !== "model_requests") throw new Error("Invalid accounting readiness policy");
if (value.version !== 1 || !reference(value.reference) || !["github-hosted", "aws-fleet"].includes(value.runner)
|| typeof value.verifiedAt !== "string" || typeof value.expiresAt !== "string"
|| !Number.isFinite(Date.parse(value.verifiedAt)) || !Number.isFinite(Date.parse(value.expiresAt))
+69 -1
View File
@@ -2,7 +2,7 @@ import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { expect, it } from "vitest";
import { openEvalCostLedger } from "./cost-ledger.js";
import { openEvalCostLedger, type EvalCostObservation } from "./cost-ledger.js";
import { readEvalOperationsReport, renderEvalOperationsReport, validateEvalOperationsReport } from "./operations-report.js";
import { buildRunnerCampaign } from "./history.js";
import { regenerateRunnerDashboard } from "./dashboard-regenerate.js";
@@ -91,3 +91,71 @@ it("keeps an in-flight final receipt reserved until the forwarding request finis
expect(report).toMatchObject({ inflight: 1, knownUsd: "0.050000000", heldUsd: "0.150000000", committedUsd: "0.200000000", unresolvedRequests: 0 });
} finally { await ledger.close(); await rm(root, { recursive: true, force: true }); }
});
it("reports final setup and canceled-request tokens once while exposing missing and conflicting usage", async () => {
const root = await mkdtemp(path.join(tmpdir(), "eval-token-report-")), file = path.join(root, "ledger.jsonl");
const ledger = await openEvalCostLedger(file, { campaignId: "tokens", maxUsd: "10", deadlineAt: "2099-01-01T00:00:00Z", maxRequests: 10, maxConcurrent: 1 });
const observation = { provider: "fixture", currency: "USD" as const, route: "direct" as const, classification: "reported" as const, costUsd: "0.05", final: true };
try {
for (const [requestId, phase] of [["setup", "setup"], ["canceled", "agent"], ["missing", "agent"], ["conflict", "agent"], ["full-conflict", "agent"], ["reasoning-conflict", "agent"], ["overhead", "infrastructure"]] as const) {
await ledger.reserve(requestId, { executionId: "cell", attemptId: "first", phase }, "1");
if (requestId !== "missing") {
const tokens = { input: 10, cachedRead: 20, cachedWrite: 30, output: 5, ...(requestId === "reasoning-conflict" ? {} : { reasoning: 2 }) };
await ledger.observe({ ...observation, observationId: requestId + "-partial", requestId, source: "gateway", tokens: { ...tokens, input: 1 }, final: false });
await ledger.observe({ ...observation, observationId: requestId + "-gateway", requestId, source: "gateway", tokens });
await ledger.observe({ ...observation, observationId: requestId + "-provider", requestId, source: "provider", tokens });
if (requestId === "setup") await ledger.observe({ ...observation, observationId: "native-telemetry", requestId, source: "native_otlp", tokens: { ...tokens, input: 999 } });
if (requestId === "conflict") await ledger.observe({ ...observation, observationId: "conflicting-usage", requestId, source: "provider", tokens: { input: 11 } });
if (requestId === "reasoning-conflict") await ledger.observe({ ...observation, observationId: "partial-reasoning", requestId, source: "provider", tokens: { reasoning: 9 } });
if (requestId === "full-conflict") await ledger.observe({ ...observation, observationId: "full-conflicting-usage", requestId, source: "provider", tokens: { ...tokens, input: 11 } });
}
await ledger.finish(requestId, requestId === "canceled" ? "cancelled" : "succeeded");
}
const report = await readEvalOperationsReport(file);
if (report.status === "unavailable") throw new Error("Expected readable token report");
expect(report.modelAccounting?.tokenUsage).toEqual({ requests: 2, conflictingRequests: 3,
totals: { input: 20, cachedRead: 40, cachedWrite: 60, output: 10, reasoning: 4 },
setup: { requests: 1, totals: { input: 10, cachedRead: 20, cachedWrite: 30, output: 5, reasoning: 2 } } });
const rendered = renderEvalOperationsReport(report);
expect(rendered).toContain("2/6 model requests have unambiguous final usage; 3 conflicting; 1 missing");
expect(rendered).toContain("Reported reasoning: 4, included in output");
expect(rendered).toContain("Setup token usage (1 requests, included above)");
const emptySetup = structuredClone(report);
emptySetup.modelAccounting!.tokenUsage!.setup.requests = 0;
expect(() => validateEvalOperationsReport(emptySetup)).toThrow("Invalid retained");
const emptyOverall = structuredClone(report);
emptyOverall.modelAccounting!.tokenUsage!.requests = 0;
emptyOverall.modelAccounting!.tokenUsage!.setup = { requests: 0, totals: { input: 0, cachedRead: 0, cachedWrite: 0, output: 0, reasoning: 0 } };
expect(() => validateEvalOperationsReport(emptyOverall)).toThrow("Invalid retained");
const changed = structuredClone(report);
changed.modelAccounting!.tokenUsage!.setup.totals.input = 21;
expect(() => validateEvalOperationsReport(changed)).toThrow("Invalid retained");
changed.modelAccounting!.tokenUsage!.setup.totals.input = Number.MAX_SAFE_INTEGER + 1;
expect(() => validateEvalOperationsReport(changed)).toThrow("Invalid retained");
} finally { await ledger.close(); await rm(root, { recursive: true, force: true }); }
});
it("preserves agreed final usage across every partial-field subset and observation order", async () => {
const root = await mkdtemp(path.join(tmpdir(), "eval-token-subsets-")), file = path.join(root, "ledger.jsonl");
const ledger = await openEvalCostLedger(file, { campaignId: "subsets", maxUsd: "10", deadlineAt: "2099-01-01T00:00:00Z", maxRequests: 64, maxConcurrent: 1 });
const values = { input: 10, cachedRead: 20, cachedWrite: 30, output: 5, reasoning: 2 };
const keys = ["input", "cachedRead", "cachedWrite", "output", "reasoning"] as const;
try {
for (let mask = 0; mask < 32; mask++) for (const reverse of [false, true]) {
const requestId = `request-${mask}-${reverse}`, partial: EvalCostObservation["tokens"] = {};
keys.forEach((key, index) => { if (mask & 1 << index) partial[key] = values[key]; });
await ledger.reserve(requestId, { executionId: "cell", attemptId: "attempt", phase: "agent" }, "0.2");
const sources = [{ source: "gateway" as const, tokens: { input: 10, cachedRead: 20, cachedWrite: 30, output: 5 } }, { source: "provider" as const, tokens: partial }];
if (reverse) sources.reverse();
for (const observation of sources) await ledger.observe({ ...observation, observationId: requestId + observation.source, requestId,
provider: "fixture", currency: "USD", route: "direct", classification: "reported", costUsd: "0.05", final: true });
await ledger.finish(requestId, "cancelled");
}
const report = await readEvalOperationsReport(file);
if (report.status === "unavailable") throw new Error("Expected usable accounting for agreeing partial receipts");
expect(report.modelAccounting?.tokenUsage).toMatchObject({ requests: 64, conflictingRequests: 0,
totals: { input: 640, cachedRead: 1280, cachedWrite: 1920, output: 320, reasoning: 64 } });
expect(report.knownUsd).toBe("3.200000000");
} finally { await ledger.close(); await rm(root, { recursive: true, force: true }); }
});
+75 -1
View File
@@ -3,10 +3,17 @@ import { usdToUnits } from "../../packages/shared/src/money.js";
import { html, isPublicReportTimestamp } from "./report-format.js";
const money = (n: bigint) => `${n / 1_000_000_000n}.${String(n % 1_000_000_000n).padStart(9, "0")}`;
interface ModelTokenTotals { input: number; cachedRead: number; cachedWrite: number; output: number; reasoning: number }
interface ModelTokenUsage { requests: number; conflictingRequests: number; totals: ModelTokenTotals; setup: { requests: number; totals: ModelTokenTotals } }
const tokenFields = ["input", "cachedRead", "cachedWrite", "output", "reasoning"] as const;
const emptyTokenTotals = (): ModelTokenTotals => ({ input: 0, cachedRead: 0, cachedWrite: 0, output: 0, reasoning: 0 });
interface EvalOperationsAvailableReport {
version: 1;
scope: "shared_gateway_ledger";
status?: never;
accountingReadiness?: "model_requests";
modelAccounting?: { requests: number; unresolvedRequests: number; conflictingRequests: number; inflight: number; knownUsd: string; heldUsd: string; tokenUsage?: ModelTokenUsage };
ledgerCampaignId: string;
journalHash: string;
records: number;
@@ -39,7 +46,7 @@ export function validateEvalOperationsReport(report: EvalOperationsReport): void
}
const dollars = (value: unknown): bigint => typeof value === "string" && /^(?:0|[1-9][0-9]*)\.[0-9]{9}$/.test(value) ? usdToUnits(value) : fail();
const count = (value: unknown) => { if (!Number.isSafeInteger(value) || Number(value) < 0) fail(); };
fields(report, ["version", "scope", "ledgerCampaignId", "journalHash", "records", "deadlineAt", "limitUsd", "knownUsd", "heldUsd", "committedUsd", "requests", "unresolvedRequests", "boundedUnresolvedRequests", "conflictingRequests", "inflight", "breakdown", "attempts"]);
fields(report, ["version", "scope", "accountingReadiness", "modelAccounting", "ledgerCampaignId", "journalHash", "records", "deadlineAt", "limitUsd", "knownUsd", "heldUsd", "committedUsd", "requests", "unresolvedRequests", "boundedUnresolvedRequests", "conflictingRequests", "inflight", "breakdown", "attempts"]);
if (report.version !== 1 || report.scope !== "shared_gateway_ledger" || !validEvalLedgerId(report.ledgerCampaignId)
|| !/^[a-f0-9]{64}$/.test(report.journalHash) || !isPublicReportTimestamp(report.deadlineAt) || !Array.isArray(report.attempts)) fail();
[report.records, report.requests, report.unresolvedRequests, report.conflictingRequests, report.inflight].forEach(count);
@@ -48,6 +55,29 @@ export function validateEvalOperationsReport(report: EvalOperationsReport): void
count(report.boundedUnresolvedRequests);
if (report.boundedUnresolvedRequests > report.unresolvedRequests - report.conflictingRequests) fail();
}
if (report.accountingReadiness !== undefined && (report.accountingReadiness !== "model_requests" || !report.modelAccounting)) fail();
if (report.modelAccounting !== undefined) {
const model = report.modelAccounting;
fields(model, ["requests", "unresolvedRequests", "conflictingRequests", "inflight", "knownUsd", "heldUsd", "tokenUsage"]);
if (model.tokenUsage !== undefined) {
const usage = model.tokenUsage;
fields(usage, ["requests", "conflictingRequests", "totals", "setup"]);
fields(usage.setup, ["requests", "totals"]);
[usage.requests, usage.conflictingRequests, usage.setup.requests].forEach(count);
if (usage.requests + usage.conflictingRequests > model.requests || usage.setup.requests > usage.requests) fail();
for (const [requests, totals] of [[usage.requests, usage.totals], [usage.setup.requests, usage.setup.totals]] as const) {
fields(totals, [...tokenFields]);
tokenFields.forEach(key => count(totals[key]));
if (totals.reasoning > totals.output || requests === 0 && tokenFields.some(key => totals[key] !== 0)) fail();
}
if (tokenFields.some(key => usage.setup.totals[key] > usage.totals[key])) fail();
}
[model.requests, model.unresolvedRequests, model.conflictingRequests, model.inflight].forEach(count);
if (model.requests > report.requests || model.unresolvedRequests > model.requests || model.unresolvedRequests > report.unresolvedRequests
|| model.conflictingRequests > model.unresolvedRequests || model.conflictingRequests > report.conflictingRequests
|| model.inflight > model.requests || model.inflight > report.inflight
|| dollars(model.knownUsd) > dollars(report.knownUsd) || dollars(model.heldUsd) > dollars(report.heldUsd)) fail();
}
dollars(report.limitUsd);
const known = dollars(report.knownUsd), held = dollars(report.heldUsd);
if (known + held !== dollars(report.committedUsd)) fail();
@@ -86,7 +116,35 @@ async function readAvailableEvalOperationsReport(file: string): Promise<EvalOper
group.heldUsd = money(usdToUnits(group.heldUsd) + held);
groups.set(key, group);
}
const modelRequests = snapshot.requests.filter(request => request.phase !== "infrastructure");
const modelKnown = modelRequests.reduce((sum, request) => sum + usdToUnits(request.costUsd ?? "0"), 0n);
const modelHeld = modelRequests.reduce((sum, request) => {
const cost = usdToUnits(request.costUsd ?? "0"), reserved = usdToUnits(request.reservedUsd);
return sum + ((!request.finished || !request.receiptComplete) && reserved > cost ? reserved - cost : 0n);
}, 0n);
const tokenUsage: ModelTokenUsage = { requests: 0, conflictingRequests: 0, totals: emptyTokenTotals(), setup: { requests: 0, totals: emptyTokenTotals() } };
for (const request of loaded.state.requests.values()) {
if (request.identity.phase === "infrastructure") continue;
const candidates = request.observations.filter(observation => observation.final && observation.tokens
&& ["gateway", "provider"].includes(observation.source));
const values = tokenFields.map(key => new Set(candidates.flatMap(observation => observation.tokens![key] === undefined ? [] : [observation.tokens![key]!] )));
const complete = candidates.find(observation => ["input", "cachedRead", "cachedWrite", "output"].every(key => observation.tokens![key as keyof ModelTokenTotals] !== undefined));
if (values.some(value => value.size > 1) || complete && candidates.some(observation => (observation.tokens!.reasoning ?? 0) > complete.tokens!.output!)) { tokenUsage.conflictingRequests++; continue; }
if (!complete) continue;
const tokens = Object.fromEntries(tokenFields.map(key => [key, complete.tokens![key] ?? 0])) as unknown as ModelTokenTotals;
tokens.reasoning = candidates.find(observation => observation.tokens!.reasoning !== undefined)?.tokens?.reasoning ?? 0;
tokenUsage.requests++;
if (request.identity.phase === "setup") tokenUsage.setup.requests++;
for (const key of tokenFields) {
tokenUsage.totals[key] += tokens[key];
if (request.identity.phase === "setup") tokenUsage.setup.totals[key] += tokens[key];
}
}
const report: EvalOperationsAvailableReport = { version: 1, scope: "shared_gateway_ledger", ledgerCampaignId: snapshot.campaignId,
...(loaded.state.limits.overhead?.accountingReadiness ? { accountingReadiness: loaded.state.limits.overhead.accountingReadiness } : {}),
modelAccounting: { requests: modelRequests.length, unresolvedRequests: modelRequests.filter(request => !request.receiptComplete).length,
conflictingRequests: modelRequests.filter(request => request.conflict).length, inflight: modelRequests.filter(request => !request.finished).length,
knownUsd: money(modelKnown), heldUsd: money(modelHeld), tokenUsage },
journalHash: loaded.previous, records: loaded.sequence, deadlineAt: new Date(loaded.state.limits.deadlineAt).toISOString(),
limitUsd: snapshot.limitUsd, knownUsd: snapshot.knownUsd, heldUsd: snapshot.heldUsd, committedUsd: snapshot.committedUsd,
requests: snapshot.requests.length, unresolvedRequests: snapshot.unknownRequests, boundedUnresolvedRequests: snapshot.boundedUnresolvedRequests, conflictingRequests: snapshot.conflictingRequests,
@@ -103,12 +161,28 @@ export function evalAccountingStatus(report: EvalOperationsReport): "unavailable
return report.unresolvedRequests ? "provisional" : "receipts_complete";
}
export function evalReadinessStatus(report: EvalOperationsReport): ReturnType<typeof evalAccountingStatus> {
const full = evalAccountingStatus(report);
if (report.status === "unavailable" || report.accountingReadiness !== "model_requests") return full;
const model = report.modelAccounting!;
if (!model.requests || model.unresolvedRequests || model.inflight || report.conflictingRequests || report.inflight
|| usdToUnits(report.committedUsd) > usdToUnits(report.limitUsd)) return "blocked";
return "receipts_complete";
}
export function operationsReportText(report: EvalOperationsReport): string[] {
if (report.status === "unavailable") return [
"Request ledger accounting unavailable: the journal could not be read or validated. Request amounts, counts, and held reservations are unknown.",
"Retained test results are preserved. Recover and validate the original journal before resuming paid work; do not reset the allowance or treat missing amounts as zero.",
];
return [
...(report.modelAccounting ? [`Model requests: ${report.modelAccounting.requests}; $${report.modelAccounting.knownUsd} known token cost; $${report.modelAccounting.heldUsd} additional held; ${report.modelAccounting.unresolvedRequests} unresolved.`] : []),
...(report.modelAccounting?.tokenUsage ? (() => {
const model = report.modelAccounting!, usage = model.tokenUsage!, total = usage.totals, setup = usage.setup.totals;
return [`Ledger token usage: ${usage.requests}/${model.requests} model requests have unambiguous final usage; ${usage.conflictingRequests} conflicting; ${model.requests - usage.requests - usage.conflictingRequests} missing. Totals: ${total.input} uncached input / ${total.cachedRead} cache-read input / ${total.cachedWrite} cache-write input / ${total.output} output. Reported reasoning: ${total.reasoning}, included in output.`,
`Setup token usage (${usage.setup.requests} requests, included above): ${setup.input} uncached input / ${setup.cachedRead} cache-read input / ${setup.cachedWrite} cache-write input / ${setup.output} output. Completed, failed and canceled requests are included once; token coverage is separate from invoice reconciliation.`];
})() : []),
...(report.accountingReadiness === "model_requests" ? [`Readiness scope: model requests; status: ${evalReadinessStatus(report)}. Operating costs remain separately unreconciled unless final receipts establish otherwise. Their reservations remain inside campaign admission; this does not establish a full-lifecycle operating-cost bound.`] : []),
`Accounting status: ${evalAccountingStatus(report)}. ${report.boundedUnresolvedRequests ?? 0} pending charges have reviewed bounds; ${report.unresolvedRequests - (report.boundedUnresolvedRequests ?? 0)} do not. Provisional completion preserves holds and requires later reconciliation; it does not establish invoice completeness.`,
`Shared request ledger ${report.ledgerCampaignId}: $${report.knownUsd} known subtotal; $${report.heldUsd} additional held reservations; $${report.committedUsd} committed against $${report.limitUsd}.`,
`Charges: $${report.breakdown.reconciled} reconciled; $${report.breakdown.reported} reported; $${report.breakdown.estimated} estimated; $${report.breakdown.allocated} allocated.`,
+55 -5
View File
@@ -3,6 +3,7 @@ import { access, mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promise
import { createHash } from "node:crypto";
import path from "node:path";
import { tmpdir } from "node:os";
import * as remoteGateway from "./eval-remote-gateway.js";
import { runnerMatrix } from "./catalog.js";
import { validateEvalOperationsManifest, openEvalOperations, type EvalOperationsManifest } from "./operations.js";
import { setupLiveFixtures } from "./live-fixtures.js";
@@ -17,7 +18,7 @@ function manifest(): EvalOperationsManifest {
models: [{ provider: "openai", maximumInputTokens: 1000, maximumOutputTokens: 100, timeoutMs: 1000,
price: { provider: "openai", model: cell.profile.model, version: "fixture-only", sourceUrl: "https://provider.invalid/fixture", inputUsdPerMillion: "2", cachedReadUsdPerMillion: "1", outputUsdPerMillion: "10", cacheWrite30mUsdPerMillion: "2", cacheWrite12hUsdPerMillion: "2" } }] };
}
afterEach(() => { vi.unstubAllEnvs(); vi.unstubAllGlobals(); });
afterEach(() => { vi.restoreAllMocks(); vi.unstubAllEnvs(); vi.unstubAllGlobals(); });
it("admits only the exact bounded local selection and explicitly priced models", () => {
expect(validateEvalOperationsManifest(manifest(), [cell], 1)).toMatchObject({ route: "direct" });
expect(() => validateEvalOperationsManifest({ ...manifest(), executionIds: [] }, [cell], 1)).toThrow("selection");
@@ -163,16 +164,21 @@ it.each([
cell.id,
"feedback-persistence.runner-codex.local.complaint-and-suggestion",
"feedback-persistence.runner-acpx-claude.local.complaint-and-suggestion",
"core-compatibility.runner-acpx-claude.daytona.message-marker",
])("routes %s through a shared connection without ambient provider bindings", async executionId => {
const selected = runnerMatrix.find(execution => execution.id === executionId)!;
const anthropic = selected.profile.credential === "ANTHROPIC_API_KEY";
const admitted = manifest(); admitted.executionIds = [selected.id];
const remote = selected.environment.id === "daytona";
const origin = remote ? "https://fixture.trycloudflare.com" : "http://127.0.0.1:43123";
const admitted = manifest();
if (remote) admitted.remoteGateway = "cloudflare"; admitted.executionIds = [selected.id];
admitted.models[0]!.provider = anthropic ? "anthropic" : "openai";
admitted.models[0]!.price.provider = admitted.models[0]!.provider;
admitted.models[0]!.price.model = selected.profile.model;
if (anthropic) Object.assign(admitted.models[0]!.price, { cacheWrite5mUsdPerMillion: "2", cacheWrite1hUsdPerMillion: "2" });
expect(validateEvalOperationsManifest(admitted, [selected], 0)).toBe(admitted);
vi.stubEnv("PAPERCLIP_E2E_GATEWAY_BASE_URL", "http://127.0.0.1:43123");
vi.stubEnv("PAPERCLIP_E2E_GATEWAY_BASE_URL", origin);
if (remote) vi.stubEnv("PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN", origin);
vi.stubEnv("PAPERCLIP_E2E_GATEWAY_TOKEN", "ephemeral-fixture-token");
const id = "11111111-1111-4111-8111-111111111111", grantId = "22222222-2222-4222-8222-222222222222";
let connection: unknown, agent: Record<string, any> | undefined, company: Record<string, unknown> | undefined;
@@ -180,14 +186,58 @@ it.each([
async get() { return [{ id, driver: "local" }]; },
async postSensitive(url: string, body: unknown) { if (url.endsWith("/ai-connections")) { connection = body; return { connectionId: id, grantId }; } return { id }; },
async post(url: string, body: Record<string, unknown>) { if (url.endsWith("/agents")) agent = body; if (url === "/api/companies") company = body; return { id, name: "fixture" }; },
async delete() {},
} as unknown as RunnerApi;
const fixtures = await setupLiveFixtures({ api, execution: selected, executionNonce: "fixture", workspacePath: "/tmp/fixture", credentials: { [selected.profile.credential]: "ephemeral-fixture-token" } });
const fixtures = await setupLiveFixtures({ api, execution: selected, executionNonce: "fixture", workspacePath: "/tmp/fixture", credentials: { [selected.profile.credential]: "ephemeral-fixture-token", ...(remote ? { DAYTONA_API_KEY: "fixture-daytona" } : {}) },
...(remote ? { daytonaImage: `ghcr.io/paperclip/image@sha256:${"a".repeat(64)}` } : {}) });
await fixtures.teardown();
expect(createAiConnectionSchema.parse(connection)).toMatchObject({ ownership: "shared", apiKey: "ephemeral-fixture-token", routing: { kind: "gateway", protocol: anthropic ? "messages" : "responses", baseUrl: `http://127.0.0.1:43123${anthropic ? "" : "/v1"}`, auth: "bearer" } });
expect(createAiConnectionSchema.parse(connection)).toMatchObject({ ownership: "shared", apiKey: "ephemeral-fixture-token", routing: { kind: "gateway", protocol: anthropic ? "messages" : "responses", baseUrl: `${origin}${anthropic ? "" : "/v1"}`, auth: "bearer" } });
expect(aiConnectionBindingSchema.parse(agent!.runtimeConfig.aiConnection)).toEqual({ provider: anthropic ? "anthropic" : "openai", method: "api_key", mode: "shared", connectionId: id, grantId });
expect(agent!.adapterConfig.env).toBeUndefined();
expect(agent!.adapterConfig.helloProbeTimeoutSec).toBe(remote ? 300 : undefined);
if (selected.suite.id === "feedback-persistence") {
expect(company!.budgetMonthlyCents).toBe(1_000);
expect(agent!.budgetMonthlyCents).toBe(1_000);
}
});
it("requires explicit remote gateway admission and infrastructure credentials before opening a journal", async () => {
const remote = runnerMatrix.find(e => e.suite.id === cell.suite.id && e.profile.id === cell.profile.id && e.environment.id === "daytona")!;
const value = { ...manifest(), executionIds: [remote.id], remoteGateway: "cloudflare" as const };
expect(validateEvalOperationsManifest(value, [remote], 0)).toBe(value);
const root = await mkdtemp(path.join(tmpdir(), "eval-remote-preflight-"));
try {
value.ledgerPath = path.join(root, "ledger.jsonl");
const file = path.join(root, "manifest.json"); await writeFile(file, JSON.stringify(value));
await expect(openEvalOperations(file, [remote], 0, { OPENAI_API_KEY: "synthetic" })).rejects.toThrow("credential");
await expect(access(value.ledgerPath)).rejects.toMatchObject({ code: "ENOENT" });
} finally { await rm(root, { recursive: true, force: true }); }
});
it("gives remote workers scoped gateway/relay URLs and closes both local listeners and the tunnel", async () => {
const remote = runnerMatrix.find(e => e.suite.id === cell.suite.id && e.profile.id === cell.profile.id && e.environment.id === "daytona")!;
const root = await mkdtemp(path.join(tmpdir(), "eval-remote-operation-"));
const close = vi.fn(async () => {}), origin = "https://synthetic-owned.trycloudflare.com";
const start = vi.spyOn(remoteGateway, "startEvalRemoteGateway").mockResolvedValue({ baseUrl: origin, close });
const value = { ...manifest(), executionIds: [remote.id], remoteGateway: "cloudflare" as const, ledgerPath: path.join(root, "ledger.jsonl") };
value.limits.deadlineAt = new Date(Date.now() + 24 * 3_600_000).toISOString();
const file = path.join(root, "manifest.json"), env = { OPENAI_API_KEY: "provider-private", DAYTONA_API_KEY: "infra-private",
BRAINTRUST_API_KEY: "trace-private", PAPERCLIP_E2E_BRAINTRUST_PROJECT_ID: "11111111-1111-4111-8111-111111111111" };
let operations: Awaited<ReturnType<typeof openEvalOperations>> | undefined;
try {
await writeFile(file, JSON.stringify(value)); operations = await openEvalOperations(file, [remote], 0, env);
const endpoint = await operations.start(remote, "remote-one", 1);
const worker: NodeJS.ProcessEnv = { ...env }; applyEvalGatewayEnvironment(worker, remote.profile.credential, endpoint);
expect(worker).toMatchObject({ PAPERCLIP_E2E_GATEWAY_BASE_URL: origin, PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN: origin,
PAPERCLIP_E2E_OTLP_RELAY_URL: `${origin}/otel`, OPENAI_API_KEY: endpoint.token, BRAINTRUST_API_KEY: endpoint.observability!.token });
expect(JSON.stringify(worker)).not.toMatch(/provider-private|infra-private|trace-private/);
const local = start.mock.calls[0]![0];
expect(local.gateway.baseUrl).toMatch(/^http:\/\/127\.0\.0\.1:/);
expect(local.deadlineAt).toBeLessThanOrEqual(Date.now() + 4 * 3_600_000);
expect(local.deadlineAt).toBeLessThan(Date.parse(value.limits.deadlineAt));
await operations.close(); operations = undefined; expect(close).toHaveBeenCalledOnce();
await expect(fetch(local.gateway.baseUrl)).rejects.toThrow();
await expect(fetch(local.observability!.baseUrl)).rejects.toThrow();
} finally { await operations?.close(); await rm(root, { recursive: true, force: true }); }
});
+18 -7
View File
@@ -4,6 +4,7 @@ import { createHash } from "node:crypto";
import { createEvalAttemptPhase, readEvalAttemptPhase } from "./attempt-phase.js";
import { EvalSpendState, openEvalCostLedger, type EvalSpendLimits } from "./cost-ledger.js";
import { evalRequestReservation, startEvalGateway, type EvalGatewayRoute, type EvalGatewayReceipt } from "./eval-gateway.js";
import { startEvalRemoteGateway } from "./eval-remote-gateway.js";
import type { MatrixExecution } from "./types.js";
import { evalBraintrustConfig } from "./braintrust.js";
import { startEvalOtlpRelay } from "./eval-otlp-relay.js";
@@ -17,13 +18,15 @@ export interface EvalOperationsManifest {
maxAttemptsPerExecution: number;
route: "direct" | "braintrust";
projectId?: string;
remoteGateway?: "cloudflare";
models: Array<Omit<EvalGatewayRoute, "apiKey" | "projectId" | "endpointName" | "mode">>;
}
const fields = (value: object, names: string[]) => {
if (!value || typeof value !== "object" || Array.isArray(value) || Object.keys(value).some(name => !names.includes(name))) throw new Error("Invalid eval operations manifest fields");
};
export function validateEvalOperationsManifest(value: EvalOperationsManifest, executions: readonly MatrixExecution[], maxRetries: number) {
fields(value, ["version", "limits", "ledgerPath", "executionIds", "maxAttemptsPerExecution", "route", "projectId", "models"]);
fields(value, ["version", "limits", "ledgerPath", "executionIds", "maxAttemptsPerExecution", "route", "projectId", "remoteGateway", "models"]);
if (value.remoteGateway !== undefined && value.remoteGateway !== "cloudflare") throw new Error("Invalid remote eval gateway policy");
const initial = new EvalSpendState(value.limits).snapshot();
const initialModelAllowance = usdToUnits(value.limits.maxUsd) - usdToUnits(initial.committedUsd);
if (value.version !== 1 || !["direct", "braintrust"].includes(value.route) || !path.isAbsolute(value.ledgerPath)
@@ -46,9 +49,11 @@ export function validateEvalOperationsManifest(value: EvalOperationsManifest, ex
models.set(key, usdToUnits(evalRequestReservation({ ...model, mode: value.route, apiKey: "validation" })));
}
for (const execution of executions) {
if (execution.environment.id !== "local" || !["core-compatibility", "feedback-persistence"].includes(execution.suite.id)
if (!(execution.environment.id === "local" || execution.environment.id === "daytona" && value.remoteGateway === "cloudflare"
&& ["runner-codex", "runner-acpx-claude"].includes(execution.profile.id)) || !["core-compatibility", "feedback-persistence"].includes(execution.suite.id)
|| !["runner-codex", "runner-acpx-claude", "legacy-codex", "legacy-claude"].includes(execution.profile.id)
|| execution.requiredCredentials.length !== 1) throw new Error(`Unqualified budgeted eval route: ${execution.id}`);
|| execution.requiredCredentials.length !== (execution.environment.id === "daytona" ? 2 : 1)
|| execution.requiredCredentials.filter(name => name !== "DAYTONA_API_KEY").length !== 1) throw new Error(`Unqualified budgeted eval route: ${execution.id}`);
const provider = execution.profile.credential === "ANTHROPIC_API_KEY" ? "anthropic" : "openai";
if (!models.has(`${provider}:${execution.profile.model}`)) throw new Error(`Missing exact model pricing: ${execution.id}`);
if (models.get(`${provider}:${execution.profile.model}`)! > initialModelAllowance) {
@@ -65,7 +70,8 @@ export async function openEvalOperations(file: string, executions: readonly Matr
|| Date.parse(manifest.limits.deadlineAt) >= Date.parse(manifest.limits.overhead.expiresAt))) {
throw new Error("Operating overhead allowance does not cover the campaign window");
}
const required = manifest.route === "braintrust" ? ["BRAINTRUST_API_KEY"] : executions.map(execution => execution.profile.credential);
const required = [...(manifest.route === "braintrust" ? ["BRAINTRUST_API_KEY"] : executions.map(execution => execution.profile.credential)),
...(executions.some(execution => execution.environment.id === "daytona") ? ["DAYTONA_API_KEY"] : [])];
if (required.some(name => !env[name]?.trim())) throw new Error("Missing eval gateway upstream credential");
const observability = evalBraintrustConfig({ ...env, PAPERCLIP_E2E_OTLP_RELAY_URL: undefined });
if (observability && manifest.route === "braintrust" && observability.projectId !== manifest.projectId) throw new Error("Eval gateway and OTLP projects differ");
@@ -99,14 +105,19 @@ export async function openEvalOperations(file: string, executions: readonly Matr
persist: (receipt: EvalGatewayReceipt) => atomicJson(path.join(receiptDirectory, `${receipt.requestId}.json`), receipt),
});
let relay: Awaited<ReturnType<typeof startEvalOtlpRelay>> | undefined;
let remote: Awaited<ReturnType<typeof startEvalRemoteGateway>> | undefined;
try {
if (observability) relay = await startEvalOtlpRelay({ projectId: observability.projectId, apiKey: observability.apiKey,
campaignId: manifest.limits.campaignId, attemptId, deadlineAt: Math.min(Date.parse(manifest.limits.deadlineAt), Date.now() + 4 * 3_600_000),
receiptPath: path.join(`${manifest.ledgerPath}.otlp`, `${createHash("sha256").update(attemptId).digest("hex")}.jsonl`) });
} catch (error) { await gateway.close(); throw error; }
if (execution.environment.id === "daytona") remote = await startEvalRemoteGateway({ gateway, observability: relay,
deadlineAt: Math.min(Date.parse(manifest.limits.deadlineAt), Date.now() + 4 * 3_600_000) }, env);
} catch (error) { await Promise.allSettled([gateway.close(), relay?.close()]); throw error; }
let closing: Promise<void> | undefined;
const endpoint = { ...gateway, phasePath, observability: relay, close: () => closing ??= (async () => {
const results = await Promise.allSettled([gateway.close(), relay?.close()]);
const endpoint = { ...gateway, baseUrl: remote?.baseUrl ?? gateway.baseUrl, phasePath,
observability: relay ? { ...relay, baseUrl: remote ? `${remote.baseUrl}/otel` : relay.baseUrl } : undefined, close: () => closing ??= (async () => {
const remoteResults = await Promise.allSettled([remote?.close()]);
const results = [...remoteResults, ...await Promise.allSettled([gateway.close(), relay?.close()])];
active.delete(endpoint);
if (results.some(result => result.status === "rejected")) throw new Error("Eval gateway or OTLP relay cleanup failed");
})() };
@@ -188,8 +188,8 @@ describe("Pi controls catalog admission", () => {
// Extended v3 states the strict file oracle's task-wide Bash limit.
const pi = runnerMatrix.find(c => c.profile.qualificationCandidate === "pi")!.profile;
expect(pi.modelQualification?.qualificationId).toBe("pi:0.0.33:1.0.0:openrouter");
expect(runnerSuites.find(s => s.id === "pi-native")!.definitionMetadata).toMatchObject({ version: 23, remoteProcExit: "separately-confirmed-absence-after-read-failure", taskCreation: "explicit-title-and-creation-response-id", agentMemoryParent: "public-managed-file-seed-before-admission", incompleteTerminalCleanup: "retirement-retained-with-failed-watch", profileVersion: 29, agentMemoryContent: "utf8-nonce-plus-final-lf", agentMemoryPrompt: "single-json-write-and-content-bound-native-read-both-runs", agentMemoryReadAuthority: "local-withheld-or-exact-remote-agent-run-file", taskPromptTransport: "fenced-markdown-paste-and-multiline-literal-escapes", nativeFinish: "current-contract-objective-evidence-refs", providerFaultExecutable: "stable-preinstalled-runner-link-and-snapshot-node-inode-with-held-bootstrap-fd-3-or-7" });
expect(runnerSuites.find(s => s.id === "pi-controls")!.definitionMetadata).toMatchObject({ version: 10, remoteProcExit: "separately-confirmed-absence-after-read-failure", taskCreation: "explicit-title-and-creation-response-id", profileVersion: 29,
expect(runnerSuites.find(s => s.id === "pi-native")!.definitionMetadata).toMatchObject({ version: 23, remoteProcExit: "separately-confirmed-absence-after-read-failure", taskCreation: "explicit-title-and-creation-response-id", agentMemoryParent: "public-managed-file-seed-before-admission", incompleteTerminalCleanup: "retirement-retained-with-failed-watch", profileVersion: 30, agentMemoryContent: "utf8-nonce-plus-final-lf", agentMemoryPrompt: "single-json-write-and-content-bound-native-read-both-runs", agentMemoryReadAuthority: "local-withheld-or-exact-remote-agent-run-file", taskPromptTransport: "fenced-markdown-paste-and-multiline-literal-escapes", nativeFinish: "current-contract-objective-evidence-refs", providerFaultExecutable: "stable-preinstalled-runner-link-and-snapshot-node-inode-with-held-bootstrap-fd-3-or-7" });
expect(runnerSuites.find(s => s.id === "pi-controls")!.definitionMetadata).toMatchObject({ version: 10, remoteProcExit: "separately-confirmed-absence-after-read-failure", taskCreation: "explicit-title-and-creation-response-id", profileVersion: 30,
remoteProcessIdentity: "observer-pid-startTicks-bootId",
controlPlaneSettlement: "required-scoped-result-and-terminal-after-runner" });
expect(runnerSuites.find(s => s.id === "extended-harnesses")!.definitionMetadata).toMatchObject({ version: 6, piFileArtifactTitle: "exact-filename", piFileCommandTransport: "fenced-bash-markdown-paste" });
@@ -199,7 +199,7 @@ describe("Pi controls catalog admission", () => {
}
const hashes = Object.fromEntries(runnerSuites.filter(s => ["pi-native", "native-active-stop", "extended-harnesses", "rich-acp-warm-continuity"].includes(s.id)).map(s => [s.id, suiteDefinitionHash(s)]));
expect(hashes).toEqual({
"pi-native": "d7d691088af76d460e353a3982f7732e6a65b2e5a23a66d2ef0d534f9d8dae48",
"pi-native": "04d49c6e098c24091d2939108a97722d1b8353e39192b87e98ee33377f183001",
"native-active-stop": "4654a2169bff96cf8377fdd62441f5cabea5c613f447e518830f6b1efd82c051",
"rich-acp-warm-continuity": "7e8a0012f2f67fba03cd94403b096a07cfc95cdb7aa568b4c3fd18b852d735f3",
"extended-harnesses": "d5bcf5de0ade3a54e4e1560e76feb5d8a787039d56d2379cd501626d85ed7418",
@@ -0,0 +1 @@
export function prepareReportLock(root?: string): Promise<'already-reviewed' | 'reviewed-correction'>;
+31
View File
@@ -0,0 +1,31 @@
import { createHash } from 'node:crypto';
import { execFileSync } from 'node:child_process';
import { readFile, writeFile, mkdtemp, rm } from 'node:fs/promises';
import path from 'node:path';
import { tmpdir } from 'node:os';
import { pathToFileURL } from 'node:url';
const sha256 = value => createHash('sha256').update(value).digest('hex');
export async function prepareReportLock(root = path.resolve(import.meta.dirname, '../..')) {
const policy = JSON.parse(await readFile(path.join(root, 'tests/runner-e2e/report-lock.json'), 'utf8'));
const lockPath = path.join(root, 'pnpm-lock.yaml');
const bytes = await readFile(lockPath), current = sha256(bytes);
if (current === policy.outputSha256) return 'already-reviewed';
if (current !== policy.inputSha256) throw new Error('Reporting lock changed; review the dependency correction before install');
const patch = await readFile(path.join(root, 'tests/runner-e2e/report-lock.patch'));
if (sha256(patch) !== policy.patchSha256) throw new Error('Reviewed reporting lock patch changed');
const result = execFileSync('git', ['apply', '--numstat', '-'], { cwd: root, input: patch, encoding: 'utf8' });
if (!/^\d+\s+\d+\s+pnpm-lock\.yaml\n$/.test(result)) throw new Error('Reporting correction must change only the lockfile');
const temporary = await mkdtemp(path.join(tmpdir(), 'eval-reviewed-lock-'));
try {
await writeFile(path.join(temporary, 'pnpm-lock.yaml'), bytes);
execFileSync('git', ['apply', '-'], { cwd: temporary, input: patch });
const corrected = await readFile(path.join(temporary, 'pnpm-lock.yaml'));
if (sha256(corrected) !== policy.outputSha256) throw new Error('Reviewed reporting lock result changed');
await writeFile(lockPath, corrected);
return 'reviewed-correction';
} finally { await rm(temporary, { recursive: true, force: true }); }
}
if (process.argv[1] && import.meta.url === pathToFileURL(path.resolve(process.argv[1])).href) {
console.log(await prepareReportLock());
}
@@ -0,0 +1,40 @@
import { mkdtemp, mkdir, readFile, writeFile, copyFile, rm } from 'node:fs/promises';
import { createHash } from 'node:crypto';
import { execFileSync } from 'node:child_process';
import { tmpdir } from 'node:os';
import path from 'node:path';
import { expect, it } from 'vitest';
import { parse } from 'yaml';
import { prepareReportLock } from './prepare-report-lock.mjs';
it.each(['valid', 'unknown-lock', 'changed-patch', 'wrong-output'])(
'installs only the reviewed dependency correction: %s', async variant => {
const root = await mkdtemp(path.join(tmpdir(), 'eval-report-lock-test-'));
const here = import.meta.dirname, relative = path.join(root, 'tests/runner-e2e');
try {
await mkdir(relative, { recursive: true });
for (const file of ['report-lock.json', 'report-lock.patch']) await copyFile(path.join(here, file), path.join(relative, file));
const input = execFileSync('git', ['show', 'HEAD:pnpm-lock.yaml'], { cwd: path.resolve(here, '../..') });
const policy = JSON.parse(await readFile(path.join(relative, 'report-lock.json'), 'utf8'));
expect(createHash('sha256').update(input).digest('hex')).toBe(policy.inputSha256);
const lock = path.join(root, 'pnpm-lock.yaml');
await writeFile(lock, variant === 'unknown-lock' ? 'unknown\n' : input);
if (variant === 'changed-patch') await writeFile(path.join(relative, 'report-lock.patch'), 'changed');
if (variant === 'wrong-output') { policy.outputSha256 = '0'.repeat(64); await writeFile(path.join(relative, 'report-lock.json'), JSON.stringify(policy)); }
if (variant === 'valid') {
expect(await prepareReportLock(root)).toBe('reviewed-correction');
expect(createHash('sha256').update(await readFile(lock)).digest('hex')).toBe(policy.outputSha256);
expect(await prepareReportLock(root)).toBe('already-reviewed');
const before = parse(input.toString()).packages;
const after = parse(await readFile(lock, 'utf8')).packages;
expect(Object.keys(after).filter(key => !Object.hasOwn(before, key))).toEqual(['yaml@2.9.1']);
for (const [key, value] of Object.entries(before)) expect(after[key]).toEqual(value);
expect(after['yaml@2.9.1'].resolution.integrity).toBe('sha512-3NxN8+78OdzbT7C/WjGsyfPAtJaN3FNDsWxv7Y7mcDsT/oOmgW8BpyQQFFBnvZE3j9Y2Sdz1ULFLezL7Eb2yFw==');
} else {
await expect(prepareReportLock(root)).rejects.toThrow();
expect(await readFile(lock)).toEqual(variant === 'unknown-lock' ? Buffer.from('unknown\n') : input);
}
} finally { await rm(root, { recursive: true, force: true }); }
},
);
@@ -5,6 +5,10 @@ import os from "node:os";
import { pathToFileURL } from "node:url";
import { createConnectionProof } from "../connection-evidence.js";
import { retainOnboardingArtifact, validateOnboardingArtifact } from "./artifact-evidence.js";
import { validateOnboardingPreAcceptanceEvidence, type OnboardingPreAcceptanceEvidence } from "./onboarding-pre-acceptance.js";
const preAcceptanceEvidence: OnboardingPreAcceptanceEvidence = { version: 1, checkpoint: "awaiting-proposal", violations: ["attachment_present"],
taskCount: 1, attachmentCount: 1, documentCount: 0, nonPlanningDocumentCount: 0,
firstTaskMatchesCanonicalParent: true, firstTaskStatus: "in_review", canonicalParentStatus: "in_review" };
const persistenceFault = vi.hoisted(() => ({ kind: null as "write" | "rename" | null, active: false, latestReport: null as any }));
vi.mock("node:fs/promises", async importOriginal => {
const actual = await importOriginal<typeof import("node:fs/promises")>();
@@ -33,7 +37,7 @@ import { regenerateRunnerDashboard } from "../dashboard-regenerate.js";
const roots: string[]=[];
afterEach(async()=>{vi.useRealTimers(); vi.restoreAllMocks(); persistenceFault.kind=null; persistenceFault.active=false; persistenceFault.latestReport=null; process.exitCode=0; await Promise.all(roots.splice(0).map(root=>rm(root,{recursive:true,force:true})));});
type Hook = "collect" | "release" | "stop" | "finalizeEvidence" | "checkpointEvidence";
async function run(options:{authFailure?:boolean;onboardingFailure?:boolean;missingArtifact?:boolean;drift?:boolean;cleanupFailure?:boolean;collectionFailure?:boolean;mini?:boolean;missingReceipt?:boolean;zeroRequests?:boolean;missingCheck?:boolean;finalizationFailure?:boolean;persistenceFailure?:"write"|"rename";persistenceAfter?:"collection"|"release";stallHook?:Hook;onHook?:(hook:Hook,signal?:AbortSignal)=>void}={}) {
async function run(options:{authFailure?:boolean;onboardingFailure?:boolean;preAcceptanceFailure?:boolean;missingArtifact?:boolean;drift?:boolean;cleanupFailure?:boolean;collectionFailure?:boolean;mini?:boolean;missingReceipt?:boolean;zeroRequests?:boolean;missingCheck?:boolean;finalizationFailure?:boolean;persistenceFailure?:"write"|"rename";persistenceAfter?:"collection"|"release";stallHook?:Hook;onHook?:(hook:Hook,signal?:AbortSignal)=>void}={}) {
const root=await mkdtemp(path.join(os.tmpdir(),"journey-campaign-test-")); roots.push(root);
const module=path.join(root,"provider.mjs");
const proof = createConnectionProof();
@@ -44,7 +48,7 @@ let onboarded=false;
const context={identity,company:{id:'company-1',issuePrefix:'QA'},api:{post:async()=>({id:'issue-fixture-1',identifier:'QA-1'})},startingState:'onboarding',capabilities:['company','board-user','task-write'],page:{async screenshot(){} }};
export const provider={version:1, async acquire(input){return {
async authenticate(){${options.authFailure ? "throw {outcome:'failed',code:'login_code_expired'};" : ""}},
async onboard(evidence){${options.onboardingFailure ? "throw {outcome:'failed',code:'onboarding_failed'};" : `${options.missingArtifact ? "" : `evidence.artifact=${JSON.stringify(artifact)};`}evidence.checks.push(...['fresh-workspace','ui-connection-and-native-agent','first-task-complete','downloadable-artifact-verified'${options.missingCheck ? "" : ",'native-trace-verified'"}].map(id=>({id,passed:true})));onboarded=true;return context;`}},
async onboard(evidence){${options.preAcceptanceFailure ? `evidence.onboardingPreAcceptance=${JSON.stringify(preAcceptanceEvidence)};throw {outcome:'failed',code:'onboarding_work_before_acceptance'};` : options.onboardingFailure ? "throw {outcome:'failed',code:'onboarding_failed'};" : `${options.missingArtifact ? "" : `evidence.artifact=${JSON.stringify(artifact)};`}evidence.checks.push(...['fresh-workspace','ui-connection-and-native-agent','first-task-complete','downloadable-artifact-verified'${options.missingCheck ? "" : ",'native-trace-verified'"}].map(id=>({id,passed:true})));onboarded=true;return context;`}},
async prepare(){throw Error('onboarding receipt was not reused')},
async identity(){return {...identity,commit:'${(options.drift?"b":"a").repeat(40)}'}},
async collect(){${options.collectionFailure ? "throw Error('private collection error');" : options.missingReceipt ? "return [];" : `return onboarded?[{journeyId:'cloud-onboarding',runIds:['run-1'],usage:{runs:[{runId:'run-1',usage:{costUsd:0.01}}]},operations:{campaignId:input.campaignId,attemptId:input.campaignId+':cloud-onboarding:1',modelCostUsd:0.01,setupCostUsd:0.005,requestCount:${options.zeroRequests ? 0 : 2},feedback:{collection:'complete',complaints:0,suggestions:0},traceIds:['a'.repeat(32)]}}]:[];`}}, async own(){}, async release(){${options.cleanupFailure ? "throw Error('private raw error');" : "return 'released';"}}, async stop(){}
@@ -109,6 +113,17 @@ describe("product journey campaign boundary outcomes",()=>{
expect(campaign.results.map((r:any)=>r.productJourney.outcome)).toEqual(["failed","blocked"]);
expect(campaign.results[0].productJourney.phases.find((p:any)=>p.phase==="journey").outcome).toBe("failed");
});
it("preserves failed pre-acceptance evidence in normalized results and blocks dependent reuse", async () => {
const campaign = await run({ preAcceptanceFailure: true });
const [onboarding, dependent] = campaign.results;
expect(campaign.results.map((row:any) => row.productJourney.outcome)).toEqual(["failed", "blocked"]);
expect(onboarding).toMatchObject({ status: "failed", failureClass: "candidate_failure", error: "onboarding_work_before_acceptance" });
expect(onboarding.productJourney.onboardingPreAcceptance).toEqual(preAcceptanceEvidence);
expect(() => validateOnboardingPreAcceptanceEvidence(onboarding.productJourney.onboardingPreAcceptance)).not.toThrow();
expect(onboarding.productJourney.phases).toContainEqual({ phase: "journey", outcome: "failed", code: "onboarding_work_before_acceptance" });
expect(dependent.error).toBe("prerequisite_journey_failed");
expect(dependent.productJourney.onboardingPreAcceptance).toBeUndefined();
});
it("reuses a verified onboarding context for the same shared story",async()=>{
const campaign=await run();
expect(campaign.results.map((r:any)=>r.status)).toEqual(["passed","passed"]);
+1 -1
View File
@@ -9,7 +9,7 @@ export function preparationPreview(config: TargetConfig) {
target: { id: config.id, mode: config.mode, executionEnvironment: config.executionEnvironment },
journeys: config.journeys.map(id => productJourneys.find(journey => journey.id === id)!),
configuration: config.baseline,
lifetime: config.mode === "existing-cloud" ? "persistent; clean run-owned resources only" : config.mode === "new-cloud" ? "disposable; delete on pass, retain failures for 24 hours" : "managed instance; remove after every attempt",
lifetime: config.mode === "existing-cloud" ? "persistent; clean run-owned resources only" : config.mode === "new-cloud" ? "disposable; delete on pass, retain failures up to 24 hours" : "managed instance; remove after every attempt",
limits: config.limits, authentication: config.mode === "managed-self-hosted" ? "local-trusted" : "normal email-code sign-in",
modelProfileId: config.modelProfileId ?? null,
maxDisposableStacks: config.mode === "new-cloud" ? 1 : 0,
@@ -2,6 +2,8 @@ import { describe, expect, it, vi } from "vitest";
import { cloudOnboarding, type CloudOnboardingInput } from "./cloud-onboarding.js";
import { onboardingFirstTask } from "./onboarding-first-task.js";
import { retainOnboardingArtifact } from "./artifact-evidence.js";
import { JourneyError } from "./types.js";
import type { OnboardingPreAcceptanceEvidence } from "./onboarding-pre-acceptance.js";
vi.mock("@playwright/test", () => ({ expect: (locator: { isVisible(): Promise<boolean> }) => ({
toBeVisible: async () => { expect(await locator.isVisible()).toBe(true); },
@@ -118,6 +120,20 @@ function fixture(entry: Entry, options: { intentFailure?: boolean; qualification
}
describe("Cloud onboarding visible controls", () => {
it("retains pre-acceptance evidence while preserving the failed error and withholding completion markers", async () => {
const f = fixture("portfolio");
const receipt: OnboardingPreAcceptanceEvidence = { version: 1, checkpoint: "awaiting-proposal", violations: ["attachment_present"],
taskCount: 1, attachmentCount: 1, documentCount: 0, nonPlanningDocumentCount: 0,
firstTaskMatchesCanonicalParent: true, firstTaskStatus: "in_review", canonicalParentStatus: "in_review" };
vi.mocked(onboardingFirstTask).mockImplementationOnce(async input => {
input.onPreAcceptanceFailure?.(receipt);
throw new JourneyError("failed", "onboarding_work_before_acceptance");
});
await expect(cloudOnboarding(f.input)).rejects.toMatchObject({ outcome: "failed", code: "onboarding_work_before_acceptance" });
expect(JSON.parse(JSON.stringify(f.input.evidence)).onboardingPreAcceptance).toEqual(receipt);
expect(f.input.evidence.checks.map(check => check.id)).toEqual(["fresh-workspace", "ui-connection-and-native-agent"]);
expect(f.events).not.toContain("native:verified");
});
it.each(["first-run", "portfolio", "create-form"] as const)("creates from %s with intent before dispatch and qualification before paid Connect", async entry => {
const f = fixture(entry);
const result = await cloudOnboarding(f.input);
@@ -83,7 +83,8 @@ export async function cloudOnboarding(input: CloudOnboardingInput): Promise<Jour
await input.phase("agent");
const completed = await onboardingFirstTask({ context, agent: { id: agent.id }, connectionId: connection.id,
proof: createConnectionProof(), campaignId, maxRuns: input.maxRuns, deadlineAt: input.deadlineAt,
own: input.own, observeRun: input.observeRun, onArtifact: artifact => { evidence.artifact = artifact; } });
own: input.own, observeRun: input.observeRun, onArtifact: artifact => { evidence.artifact = artifact; },
onPreAcceptanceFailure: receipt => { evidence.onboardingPreAcceptance = receipt; } });
await input.verifyExecution(context, [completed.artifact.originatingRunId]);
evidence.checks.push({ id: "first-task-complete", passed: true },
{ id: "downloadable-artifact-verified", passed: true }, { id: "native-trace-verified", passed: true });
+1 -1
View File
@@ -99,7 +99,7 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri
try { sha = execFileSync("git", ["rev-parse", "HEAD"], { cwd: repositoryRoot, encoding: "utf8", stdio: ["ignore", "pipe", "ignore"] }).trim(); } catch { /* Packaged harnesses may not include Git metadata. */ }
const source = resolveRunnerE2ESource({ sha, ref: null, workflowRunUrl: null });
const definitionHash = createHash("sha256").update(JSON.stringify(productJourneys));
for (const name of ["catalog.ts", "task-properties.ts", "cloud-onboarding.ts", "onboarding-connect.ts", "onboarding-first-task.ts", "artifact-evidence.ts", "fixtures.ts", "launch.ts", "types.ts", "collection.ts", "local-target.ts"]) definitionHash.update(await readFile(path.join(import.meta.dirname, name)));
for (const name of ["catalog.ts", "task-properties.ts", "cloud-onboarding.ts", "onboarding-connect.ts", "onboarding-first-task.ts", "onboarding-pre-acceptance.ts", "artifact-evidence.ts", "fixtures.ts", "launch.ts", "types.ts", "collection.ts", "local-target.ts"]) definitionHash.update(await readFile(path.join(import.meta.dirname, name)));
const suiteDefinitionHash = definitionHash.digest("hex");
const expected = config.journeys.map(id => `product-journeys.${config.id}.${id === "cloud-onboarding" ? config.modelProfileId : "browser"}.${config.executionEnvironment}.${id}`);
let reportPersisted = false;
@@ -4,6 +4,7 @@ import { createConnectionProof } from "../connection-evidence.js";
import { connectionDelay } from "../connection-target.js";
import { onboardingFirstTask, type OnboardingFirstTaskInput } from "./onboarding-first-task.js";
import { validateOnboardingArtifact } from "./artifact-evidence.js";
import { validateOnboardingPreAcceptanceEvidence } from "./onboarding-pre-acceptance.js";
vi.mock("../connection-target.js", () => ({ connectionDelay: vi.fn(async () => {}) }));
type Row = { id: string; [key: string]: any };
@@ -92,6 +93,90 @@ function fixture(options: Options = {}) {
afterEach(() => { vi.restoreAllMocks(); vi.mocked(connectionDelay).mockReset().mockResolvedValue(undefined); });
describe("Cloud canonical onboarding first task", () => {
it.each([
["unexpected task set", "unexpected_task_set", (state: Parameters<NonNullable<Options["mutate"]>>[1]) => { state.tasks.push({ id: "private-early-task", title: "private task text", status: "todo" }); }, { taskCount: 2 }],
["attachment", "attachment_present", (state: Parameters<NonNullable<Options["mutate"]>>[1]) => { state.attachments.push({ id: "private-attachment", issueId: "parent", originalFilename: "private-file.txt" }); }, { attachmentCount: 1 }],
["non-planning document", "non_planning_document_present", (state: Parameters<NonNullable<Options["mutate"]>>[1]) => { state.documents.push({ id: "private-document", issueId: "parent", key: "private-result-key", body: "private document body" }); }, { documentCount: 1, nonPlanningDocumentCount: 1 }],
["first task done", "first_task_done", (state: Parameters<NonNullable<Options["mutate"]>>[1]) => { state.tasks[0]!.status = "done"; }, { firstTaskStatus: "done", canonicalParentStatus: "done" }],
] as const)("retains the observed %s violation at the failing acceptance boundary", async (_label, violation, mutate, observed) => {
const f = fixture({ mutate(phase, state) { if (phase === "proposal") mutate(state); } });
const capture = vi.fn();
f.input.onPreAcceptanceFailure = capture;
await expect(onboardingFirstTask(f.input)).rejects.toMatchObject({ outcome: "failed", code: "onboarding_work_before_acceptance" });
expect(capture).toHaveBeenCalledExactlyOnceWith({ version: 1, checkpoint: "awaiting-proposal", violations: [violation],
taskCount: 1, attachmentCount: 0, documentCount: 0, nonPlanningDocumentCount: 0,
firstTaskMatchesCanonicalParent: true, firstTaskStatus: "in_review", canonicalParentStatus: "in_review", ...observed });
expect(() => validateOnboardingPreAcceptanceEvidence(JSON.parse(JSON.stringify(capture.mock.calls[0]![0])))).not.toThrow();
const retained = JSON.stringify(capture.mock.calls[0]![0]);
for (const text of ["private-", "parent", "proposal-run", "qa.example.test", "connection", "campaign-1"]) expect(retained).not.toContain(text);
expect(f.events).not.toContain("click:Approve");
});
it("retains every simultaneous guard violation from the same failed snapshot", async () => {
const f = fixture({ plan: true, mutate(phase, state) {
if (phase !== "proposal") return;
state.tasks[0]!.status = "done";
state.tasks.push({ id: "private-early-task" });
state.attachments.push({ id: "private-attachment", issueId: "parent" });
state.documents.push({ id: "private-document", issueId: "parent", key: "private-result", body: "private body" });
} });
const capture = vi.fn();
f.input.onPreAcceptanceFailure = capture;
await expect(onboardingFirstTask(f.input)).rejects.toMatchObject({ code: "onboarding_work_before_acceptance" });
expect(capture).toHaveBeenCalledExactlyOnceWith({ version: 1, checkpoint: "awaiting-proposal",
violations: ["unexpected_task_set", "attachment_present", "non_planning_document_present", "first_task_done"],
taskCount: 2, attachmentCount: 1, documentCount: 2, nonPlanningDocumentCount: 1,
firstTaskMatchesCanonicalParent: true, firstTaskStatus: "done", canonicalParentStatus: "done" });
expect(f.events).not.toContain("click:Confirm selection");
});
it.each(["missing", "replaced", "reordered", "unknown-status"] as const)("records closed task observations when the snapshot is %s", async mode => {
const f = fixture({ mutate(phase, state) {
if (phase !== "proposal") return;
if (mode === "missing") state.tasks = [];
if (mode === "replaced") state.tasks = [{ id: "private-replacement", status: "done" }];
if (mode === "reordered") state.tasks.unshift({ id: "private-first", status: "done" });
if (mode === "unknown-status") { state.tasks[0]!.status = "private future status"; state.attachments.push({ id: "private-attachment", issueId: "parent" }); }
} });
const capture = vi.fn();
f.input.onPreAcceptanceFailure = capture;
await expect(onboardingFirstTask(f.input)).rejects.toMatchObject({ code: "onboarding_work_before_acceptance" });
expect(capture).toHaveBeenCalledOnce();
expect(capture.mock.calls[0]![0]).toMatchObject({
violations: mode === "unknown-status" ? ["attachment_present"] : mode === "missing" ? ["unexpected_task_set"] : ["unexpected_task_set", "first_task_done"],
taskCount: mode === "missing" ? 0 : mode === "reordered" ? 2 : 1,
firstTaskMatchesCanonicalParent: mode === "unknown-status",
firstTaskStatus: mode === "missing" ? "missing" : mode === "unknown-status" ? "unknown" : "done",
canonicalParentStatus: mode === "reordered" ? "in_review" : mode === "unknown-status" ? "unknown" : "missing",
});
expect(JSON.stringify(capture.mock.calls[0]![0])).not.toContain("private");
expect(f.events).not.toContain("click:Approve");
});
it("retains an opening-boundary failure before the opening answer dispatches work", async () => {
const f = fixture({ mutate(phase, state) { if (phase === "opening") state.tasks[0]!.status = "done"; } });
const capture = vi.fn();
f.input.onPreAcceptanceFailure = capture;
await expect(onboardingFirstTask(f.input)).rejects.toMatchObject({ code: "onboarding_work_before_acceptance" });
expect(capture).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({ checkpoint: "before-opening-answer", violations: ["first_task_done"] }));
expect(f.events).not.toContain("click:Continue");
});
it("does not claim the first-slot Done guard when only a later canonical parent is Done", async () => {
const f = fixture({ mutate(phase, state) {
if (phase !== "proposal") return;
state.tasks[0]!.status = "done";
state.tasks.unshift({ id: "private-first", status: "todo" });
} });
const capture = vi.fn();
f.input.onPreAcceptanceFailure = capture;
await expect(onboardingFirstTask(f.input)).rejects.toMatchObject({ code: "onboarding_work_before_acceptance" });
expect(capture).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({ violations: ["unexpected_task_set"],
firstTaskMatchesCanonicalParent: false, firstTaskStatus: "todo", canonicalParentStatus: "done" }));
});
it.each([false, true])("does not retain failure snapshots during clean canonical success, plan mode %s", async plan => {
const f = fixture({ plan });
const capture = vi.fn();
f.input.onPreAcceptanceFailure = capture;
await expect(onboardingFirstTask(f.input)).resolves.toMatchObject({ onboardingIssueId: "parent", childIssueId: "child" });
expect(capture).not.toHaveBeenCalled();
});
it.each(["reload", "download-link"] as const)("retains the independently verified download when later %s fails", async readbackFailure => {
const f = fixture({ readbackFailure });
const capture = vi.fn();
@@ -6,6 +6,7 @@ import { activeRuns, firstTaskCompletionSettled, type Row } from "../first-task-
import { isBlockedUnstartedWake, isTerminalUnstartedWake } from "../non-execution-wake.js";
import { JourneyError, type JourneyContext, type OnboardingArtifactEvidence, type TargetSession } from "./types.js";
import { retainOnboardingArtifact } from "./artifact-evidence.js";
import { preAcceptanceTaskStatus, preAcceptanceViolations, type OnboardingPreAcceptanceEvidence } from "./onboarding-pre-acceptance.js";
export interface OnboardingFirstTaskInput {
context: JourneyContext;
@@ -18,6 +19,7 @@ export interface OnboardingFirstTaskInput {
own: TargetSession["own"];
observeRun(run: Row): Promise<void>;
onArtifact?(artifact: OnboardingArtifactEvidence): void;
onPreAcceptanceFailure?(evidence: OnboardingPreAcceptanceEvidence): void;
}
export interface OnboardingFirstTaskResult {
runIds: string[];
@@ -91,9 +93,19 @@ export async function onboardingFirstTask(input: OnboardingFirstTaskInput): Prom
fail("onboarding_first_task_failed");
}
};
const assertBeforeAcceptance = (snapshot: Snapshot, parentId: string) => {
if (snapshot.tasks.length !== 1 || snapshot.tasks[0]!.id !== parentId || snapshot.attachments.length ||
snapshot.documents.some(document => !isPlanningDocument(document)) || snapshot.tasks[0]!.status === "done") fail("onboarding_work_before_acceptance");
const assertBeforeAcceptance = (snapshot: Snapshot, parentId: string, checkpoint: OnboardingPreAcceptanceEvidence["checkpoint"]) => {
const observed = {
taskCount: snapshot.tasks.length, attachmentCount: snapshot.attachments.length,
documentCount: snapshot.documents.length, nonPlanningDocumentCount: snapshot.documents.filter(document => !isPlanningDocument(document)).length,
firstTaskMatchesCanonicalParent: snapshot.tasks[0]?.id === parentId,
firstTaskStatus: preAcceptanceTaskStatus(snapshot.tasks[0]),
canonicalParentStatus: preAcceptanceTaskStatus(snapshot.tasks.find(task => task.id === parentId)),
};
const violations = preAcceptanceViolations(observed);
if (violations.length) {
input.onPreAcceptanceFailure?.({ version: 1, checkpoint, violations, ...observed });
fail("onboarding_work_before_acceptance");
}
assertRuns(snapshot, [parentId]);
};
@@ -105,7 +117,7 @@ export async function onboardingFirstTask(input: OnboardingFirstTaskInput): Prom
await input.own({ kind: "issue", name: parent.title, id: parent.id });
snapshot = await load(parent.id);
if (seen.size) fail("onboarding_opening_started_model_work");
assertBeforeAcceptance(snapshot, parent.id);
assertBeforeAcceptance(snapshot, parent.id, "before-opening-answer");
const opening = snapshot.interactions.filter(row => row.kind === "ask_user_questions" && row.status === "pending");
const questions = opening[0]?.payload?.questionSet?.questions ?? opening[0]?.payload?.questions ?? [];
const question = questions.find((row: Row) => row.id === "first-task-opening");
@@ -129,7 +141,7 @@ export async function onboardingFirstTask(input: OnboardingFirstTaskInput): Prom
let proposal: Row | undefined;
while (!proposal) {
snapshot = await load(parent.id);
assertBeforeAcceptance(snapshot, parent.id);
assertBeforeAcceptance(snapshot, parent.id, "awaiting-proposal");
const pending = snapshot.interactions.filter(row => row.status === "pending");
const proposals = pending.filter(isProposal);
if (proposals.length > 1 || pending.some(row => row.kind === "ask_user_questions" && row.id !== opening[0]!.id)) fail("onboarding_proposal_scope_unclear");
@@ -0,0 +1,105 @@
import { describe, expect, it } from "vitest";
import { validateRetainedRunnerResult } from "../result-validation.js";
import { renderCaseOutcome } from "../case-outcome.js";
import type { RunnerE2EResult } from "../types.js";
import type { OnboardingPreAcceptanceEvidence } from "./onboarding-pre-acceptance.js";
function result(): RunnerE2EResult {
return { schema: "paperclip.runner-e2e.result/v2", suiteId: "product-journeys", suiteDefinitionHash: "test",
executionId: "product-journeys.qa.runner-codex.local.cloud-onboarding", profileId: "runner-codex", environmentId: "local", caseId: "cloud-onboarding",
provider: "codex", model: "synthetic-model", runtimeMode: "native", status: "failed", error: "onboarding_work_before_acceptance", cleanup: "passed", attempt: 1,
startedAt: "2026-01-01T00:00:00Z", finishedAt: "2026-01-01T00:00:01Z", durationMs: 1000,
productJourney: { outcome: "failed", startingState: "onboarding", phases: [{ phase: "journey", outcome: "failed", code: "onboarding_work_before_acceptance" }],
checks: [], privacy: "private", resourceDisposition: "retained" } };
}
function preAcceptanceReceipt(): OnboardingPreAcceptanceEvidence {
return { version: 1, checkpoint: "awaiting-proposal", violations: ["attachment_present"],
taskCount: 1, attachmentCount: 1, documentCount: 0, nonPlanningDocumentCount: 0,
firstTaskMatchesCanonicalParent: true, firstTaskStatus: "in_review", canonicalParentStatus: "in_review" };
}
function passedResult(): RunnerE2EResult {
const retained = result();
retained.status = "passed";
delete retained.error;
retained.runIds = ["run-1"];
Object.assign(retained.productJourney!, { outcome: "passed", campaignId: "campaign", attemptId: "attempt",
phases: [{ phase: "journey", outcome: "passed" }], resourceDisposition: "released",
checks: ["fresh-workspace", "ui-connection-and-native-agent", "first-task-complete", "downloadable-artifact-verified", "native-trace-verified"].map(id => ({ id, passed: true })),
deployment: { kind: "cloud", targetId: "qa", origin: "https://qa.example.test", commit: "a".repeat(40), cloudRelease: "release", runtimeImage: "image", configurationHash: "config" },
operations: { campaignId: "campaign", attemptId: "attempt", modelCostUsd: null, setupCostUsd: null, requestCount: 1,
feedback: { collection: "complete", complaints: 0, suggestions: 0 }, traceIds: ["trace-1"] } });
return retained;
}
describe("onboarding pre-acceptance retained evidence", () => {
it("accepts historical results without pre-acceptance evidence and preserves optional receipts in reports", () => {
const retained = result();
expect(() => validateRetainedRunnerResult(retained)).not.toThrow();
expect(renderCaseOutcome(retained, true, [])).not.toContain("Onboarding pre-acceptance evidence");
Object.assign(retained.productJourney!, { onboardingPreAcceptance: preAcceptanceReceipt() });
const roundTrip = JSON.parse(JSON.stringify(retained));
expect(() => validateRetainedRunnerResult(roundTrip)).not.toThrow();
const output = renderCaseOutcome(roundTrip, true, []);
for (const text of ["Onboarding pre-acceptance evidence", "awaiting-proposal", "attachment_present", "First task status", "Canonical parent status", "in_review", "Attachments", "Non-planning documents", "onboarding_work_before_acceptance"]) expect(output).toContain(text);
expect(roundTrip.status).toBe("failed");
expect(roundTrip.productJourney.outcome).toBe("failed");
});
it.each([
["a passed result", passedResult],
["another case", (): RunnerE2EResult => ({ ...result(), caseId: "task-properties",
executionId: "product-journeys.qa.browser.local.task-properties", profileId: "browser", provider: null, model: null, runtimeMode: "none" })],
["another suite", (): RunnerE2EResult => ({ ...result(), suiteId: "first-task", executionId: "first-task.runner-codex.local.cloud-onboarding" })],
] as const)("rejects a valid receipt on %s after accepting the same result without it", (_label, makeResult) => {
const retained = makeResult();
expect(() => validateRetainedRunnerResult(retained)).not.toThrow();
retained.productJourney!.onboardingPreAcceptance = preAcceptanceReceipt();
expect(() => validateRetainedRunnerResult(retained)).toThrow("result.productJourney.onboardingPreAcceptance");
});
it.each(["passed", "blocked", "unsupported"] as const)("rejects a valid receipt on a %s journey outcome after accepting the same result without it", outcome => {
const retained = result();
retained.productJourney!.outcome = outcome;
expect(() => validateRetainedRunnerResult(retained)).not.toThrow();
retained.productJourney!.onboardingPreAcceptance = preAcceptanceReceipt();
expect(() => validateRetainedRunnerResult(retained)).toThrow("result.productJourney.onboardingPreAcceptance");
});
it.each(["onboarding_artifact_evidence_missing", undefined])("rejects a valid receipt for an unrelated or absent failure code (%s)", error => {
const retained = result();
retained.error = error;
expect(() => validateRetainedRunnerResult(retained)).not.toThrow();
retained.productJourney!.onboardingPreAcceptance = preAcceptanceReceipt();
expect(() => validateRetainedRunnerResult(retained)).toThrow("result.productJourney.onboardingPreAcceptance");
});
it.each([
["raw bodies", (): unknown => ({ ...preAcceptanceReceipt(), body: "private-body-must-not-render" })],
["task identifiers", (): unknown => ({ ...preAcceptanceReceipt(), taskIds: ["private-task"] })],
["document keys", (): unknown => ({ ...preAcceptanceReceipt(), documentKeys: ["private-key"] })],
["unknown version", (): unknown => ({ ...preAcceptanceReceipt(), version: 2 })],
["unknown checkpoint", (): unknown => ({ ...preAcceptanceReceipt(), checkpoint: "private text" })],
["unknown violation", (): unknown => ({ ...preAcceptanceReceipt(), violations: ["private text"] })],
["duplicate violations", (): unknown => ({ ...preAcceptanceReceipt(), violations: ["attachment_present", "attachment_present"] })],
["empty violations", (): unknown => ({ ...preAcceptanceReceipt(), violations: [] })],
["inconsistent violations", (): unknown => ({ ...preAcceptanceReceipt(), violations: ["first_task_done"] })],
["negative counts", (): unknown => ({ ...preAcceptanceReceipt(), attachmentCount: -1 })],
["fractional counts", (): unknown => ({ ...preAcceptanceReceipt(), taskCount: 1.5 })],
["unsafe counts", (): unknown => ({ ...preAcceptanceReceipt(), documentCount: Number.MAX_SAFE_INTEGER + 1 })],
["invalid document counts", (): unknown => ({ ...preAcceptanceReceipt(), nonPlanningDocumentCount: 1 })],
["raw task status", (): unknown => ({ ...preAcceptanceReceipt(), firstTaskStatus: "private status" })],
["raw parent status", (): unknown => ({ ...preAcceptanceReceipt(), canonicalParentStatus: "private status" })],
["nonboolean match", (): unknown => ({ ...preAcceptanceReceipt(), firstTaskMatchesCanonicalParent: "true" })],
["inconsistent task match", (): unknown => ({ ...preAcceptanceReceipt(), firstTaskMatchesCanonicalParent: false })],
["inconsistent matched statuses", (): unknown => ({ ...preAcceptanceReceipt(), canonicalParentStatus: "done" })],
["missing fields", (): unknown => ({ version: 1 })],
["null", (): unknown => null],
] as const)("rejects pre-acceptance evidence containing %s", (_label, makeReceipt) => {
const retained = result();
Object.assign(retained.productJourney!, { onboardingPreAcceptance: makeReceipt() });
expect(() => validateRetainedRunnerResult(retained)).toThrow("result.productJourney.onboardingPreAcceptance");
});
it("renders a fixed fallback for malformed pre-acceptance evidence without exposing its fields", () => {
const retained = result();
Object.assign(retained.productJourney!, { onboardingPreAcceptance: { ...preAcceptanceReceipt(), body: "private-body-must-not-render" } });
const output = renderCaseOutcome(retained, false, []);
expect(output).toContain("Onboarding pre-acceptance evidence unavailable: invalid receipt");
expect(output).not.toContain("private-body-must-not-render");
expect(output).not.toContain("attachment_present");
});
});
@@ -0,0 +1,59 @@
import { ISSUE_STATUSES } from "../../../packages/shared/src/constants.js";
const checkpoints = ["before-opening-answer", "awaiting-proposal"] as const;
const violations = ["unexpected_task_set", "attachment_present", "non_planning_document_present", "first_task_done"] as const;
const statuses = [...ISSUE_STATUSES, "missing", "unknown"] as const;
/** Scalar observations from the first failing pre-acceptance guard; never raw rows. */
export interface OnboardingPreAcceptanceEvidence {
version: 1;
checkpoint: typeof checkpoints[number];
violations: Array<typeof violations[number]>;
taskCount: number;
attachmentCount: number;
documentCount: number;
nonPlanningDocumentCount: number;
firstTaskMatchesCanonicalParent: boolean;
firstTaskStatus: typeof statuses[number];
canonicalParentStatus: typeof statuses[number];
}
const receiptKeys = {
version: true, checkpoint: true, violations: true, taskCount: true, attachmentCount: true,
documentCount: true, nonPlanningDocumentCount: true, firstTaskMatchesCanonicalParent: true,
firstTaskStatus: true, canonicalParentStatus: true,
} satisfies Record<keyof OnboardingPreAcceptanceEvidence, true>;
export function preAcceptanceTaskStatus(task: { status?: unknown; [key: string]: unknown } | undefined): OnboardingPreAcceptanceEvidence["firstTaskStatus"] {
if (!task) return "missing";
return ISSUE_STATUSES.find(status => status === task.status) ?? "unknown";
}
export function preAcceptanceViolations(observed: Pick<OnboardingPreAcceptanceEvidence,
"taskCount" | "attachmentCount" | "nonPlanningDocumentCount" | "firstTaskMatchesCanonicalParent" | "firstTaskStatus">): OnboardingPreAcceptanceEvidence["violations"] {
return [
...(observed.taskCount !== 1 || !observed.firstTaskMatchesCanonicalParent ? ["unexpected_task_set" as const] : []),
...(observed.attachmentCount ? ["attachment_present" as const] : []),
...(observed.nonPlanningDocumentCount ? ["non_planning_document_present" as const] : []),
...(observed.firstTaskStatus === "done" ? ["first_task_done" as const] : []),
];
}
/** Closed validation is shared by normalized-result readers and explicit report rendering. */
export function validateOnboardingPreAcceptanceEvidence(value: unknown): asserts value is OnboardingPreAcceptanceEvidence {
const invalid = (): never => { throw new Error("invalid_onboarding_pre_acceptance_evidence"); };
const keys = Object.keys(receiptKeys);
if (!value || typeof value !== "object" || Array.isArray(value) || Object.keys(value).length !== keys.length || keys.some(key => !Object.hasOwn(value, key))) invalid();
const receipt = value as OnboardingPreAcceptanceEvidence;
if (receipt.version !== 1 || !checkpoints.includes(receipt.checkpoint) ||
!Array.isArray(receipt.violations) || !receipt.violations.length || receipt.violations.length > violations.length ||
receipt.violations.some(code => !violations.includes(code)) || new Set(receipt.violations).size !== receipt.violations.length ||
[receipt.taskCount, receipt.attachmentCount, receipt.documentCount, receipt.nonPlanningDocumentCount].some(count => !Number.isSafeInteger(count) || count < 0) ||
receipt.nonPlanningDocumentCount > receipt.documentCount || typeof receipt.firstTaskMatchesCanonicalParent !== "boolean" ||
!statuses.includes(receipt.firstTaskStatus) || !statuses.includes(receipt.canonicalParentStatus)) invalid();
if (receipt.taskCount === 0 && (receipt.firstTaskStatus !== "missing" || receipt.canonicalParentStatus !== "missing" || receipt.firstTaskMatchesCanonicalParent) ||
receipt.taskCount > 0 && receipt.firstTaskStatus === "missing" ||
receipt.firstTaskMatchesCanonicalParent && receipt.firstTaskStatus !== receipt.canonicalParentStatus ||
receipt.taskCount === 1 && !receipt.firstTaskMatchesCanonicalParent && receipt.canonicalParentStatus !== "missing" ||
preAcceptanceViolations(receipt).join(",") !== receipt.violations.join(",")) invalid();
}
@@ -29,6 +29,7 @@ export interface JourneyEvidence {
operations?: JourneyOperations;
artifact?: OnboardingArtifactEvidence;
onboardingConnect?: import("./onboarding-connect.js").OnboardingConnectEvidence;
onboardingPreAcceptance?: import("./onboarding-pre-acceptance.js").OnboardingPreAcceptanceEvidence;
}
export interface OnboardingArtifactEvidence {
schema: "paperclip.product-journey.artifact/v1";
+6
View File
@@ -0,0 +1,6 @@
{
"inputSha256": "893496dddc21c31bd812ea106ac30d1515dda305eb1dd183007d89cf5b1bab4d",
"outputSha256": "c7f23021335b60da0cff1dd4c75cf8b948333ef3165d7d1703eb79a589edb798",
"patchSha256": "c7c11e899c12e6a030fe1554a8ceee5f21689a3a4f19956746e852050049cc49",
"basis": "Reviewed corrections to existing patch hashes and the already declared yaml workspace dependency. Existing package tarball integrity entries are unchanged. The only added package is yaml@2.9.1 with its exact integrity pinned by this reviewed correction."
}
+612
View File
@@ -0,0 +1,612 @@
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
@@ -16,7 +16,7 @@
patchedDependencies:
'@agentclientprotocol/claude-agent-acp@0.73.0':
- hash: riqgguivxjxydoydnzpedjy5au
+ hash: xjlrz3rllh2ztafbdiywpfh2rq
path: patches/@agentclientprotocol__claude-agent-acp@0.73.0.patch
'@agentclientprotocol/codex-acp@1.6.2':
hash: 2iynuypg76nxmydiylgtffhr7e
@@ -49,7 +49,7 @@
hash: 55uhvnotpqyiy37rn3pqpukhei
path: patches/embedded-postgres@18.1.0-beta.16.patch
postgres@3.4.9:
- hash: sryle6qd5qfmnxvoebal4gashi
+ hash: 6q4cic6qawywsyeo2spgnjamda
path: patches/postgres@3.4.9.patch
importers:
@@ -73,7 +73,10 @@
version: 7.0.2
vitest:
specifier: ^5.0.3
- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
+ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
+ yaml:
+ specifier: ^2.8.1
+ version: 2.9.1
cli:
dependencies:
@@ -133,7 +136,7 @@
version: 17.4.2
drizzle-orm:
specifier: 0.45.2
- version: 0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi))
+ version: 0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda))
embedded-postgres:
specifier: ^18.1.0-beta.16
version: 18.1.0-beta.16(patch_hash=55uhvnotpqyiy37rn3pqpukhei)
@@ -180,7 +183,7 @@
dependencies:
'@agentclientprotocol/claude-agent-acp':
specifier: ^0.73.0
- version: 0.73.0(patch_hash=riqgguivxjxydoydnzpedjy5au)(@anthropic-ai/sdk@0.129.0(zod@4.6.5))(@modelcontextprotocol/sdk@1.31.0(zod@4.6.5))
+ version: 0.73.0(patch_hash=xjlrz3rllh2ztafbdiywpfh2rq)(@anthropic-ai/sdk@0.129.0(zod@4.6.5))(@modelcontextprotocol/sdk@1.31.0(zod@4.6.5))
'@anthropic-ai/sdk':
specifier: 0.129.0
version: 0.129.0(zod@4.6.5)
@@ -310,7 +313,7 @@
version: 7.0.2
vitest:
specifier: ^5.0.3
- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
+ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
packages/adapters/hermes-gateway:
dependencies:
@@ -326,7 +329,7 @@
version: 7.0.2
vitest:
specifier: ^5.0.3
- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
+ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
packages/adapters/kimi-local:
dependencies:
@@ -405,13 +408,13 @@
version: link:../shared
drizzle-orm:
specifier: ^0.45.2
- version: 0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi))
+ version: 0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda))
embedded-postgres:
specifier: ^18.1.0-beta.16
version: 18.1.0-beta.16(patch_hash=55uhvnotpqyiy37rn3pqpukhei)
postgres:
specifier: ^3.4.9
- version: 3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)
+ version: 3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda)
devDependencies:
'@types/node':
specifier: ^24.0.0
@@ -427,7 +430,7 @@
version: 7.0.2
vitest:
specifier: ^5.0.3
- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
+ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
packages/google-sheets-mcp-server:
dependencies:
@@ -449,7 +452,7 @@
version: 7.0.2
vitest:
specifier: ^5.0.3
- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
+ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
packages/kv-demo-mcp-server:
dependencies:
@@ -468,7 +471,7 @@
version: 7.0.2
vitest:
specifier: ^5.0.3
- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
+ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
packages/mcp-server:
dependencies:
@@ -490,7 +493,7 @@
version: 7.0.2
vitest:
specifier: ^5.0.3
- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
+ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
packages/paperclip-eval-kernel:
devDependencies:
@@ -505,7 +508,7 @@
dependencies:
'@agentclientprotocol/claude-agent-acp':
specifier: 0.73.0
- version: 0.73.0(patch_hash=riqgguivxjxydoydnzpedjy5au)(@anthropic-ai/sdk@0.129.0(zod@4.6.5))(@modelcontextprotocol/sdk@1.31.0(zod@4.6.5))
+ version: 0.73.0(patch_hash=xjlrz3rllh2ztafbdiywpfh2rq)(@anthropic-ai/sdk@0.129.0(zod@4.6.5))(@modelcontextprotocol/sdk@1.31.0(zod@4.6.5))
'@agentclientprotocol/codex-acp':
specifier: 1.6.2
version: 1.6.2(patch_hash=2iynuypg76nxmydiylgtffhr7e)
@@ -561,7 +564,7 @@
version: 19.2.5(@types/react@19.2.18)
'@vitejs/plugin-react':
specifier: ^4.7.0
- version: 4.7.0(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15))
+ version: 4.7.0(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)(yaml@2.9.1))
axe-core:
specifier: ^4.12.1
version: 4.13.0
@@ -579,10 +582,10 @@
version: 7.0.2
vite:
specifier: ^6.1.0
- version: 6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)
+ version: 6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)(yaml@2.9.1)
vitest:
specifier: ^5.0.3
- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15))
+ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)(yaml@2.9.1))
packages/plugins/create-paperclip-plugin:
dependencies:
@@ -629,7 +632,7 @@
version: 7.0.2
vitest:
specifier: ^5.0.3
- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
+ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
packages/plugins/examples/plugin-file-browser-example:
dependencies:
@@ -747,7 +750,7 @@
version: 7.0.2
vitest:
specifier: ^5.0.3
- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
+ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
packages/plugins/plugin-llm-wiki:
devDependencies:
@@ -786,7 +789,7 @@
version: 7.0.2
vitest:
specifier: ^5.0.3
- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
+ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
packages/plugins/plugin-workspace-diff:
dependencies:
@@ -820,7 +823,7 @@
version: 7.0.2
vitest:
specifier: ^5.0.3
- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
+ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
packages/plugins/sdk:
dependencies:
@@ -880,7 +883,7 @@
version: 7.0.2
vitest:
specifier: ^5.0.3
- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
+ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
packages/teams-catalog:
devDependencies:
@@ -895,7 +898,7 @@
version: 0.5.0
'@agentclientprotocol/claude-agent-acp':
specifier: 0.73.0
- version: 0.73.0(patch_hash=riqgguivxjxydoydnzpedjy5au)(@anthropic-ai/sdk@0.129.0(zod@4.4.3))(@modelcontextprotocol/sdk@1.31.0(zod@4.4.3))
+ version: 0.73.0(patch_hash=xjlrz3rllh2ztafbdiywpfh2rq)(@anthropic-ai/sdk@0.129.0(zod@4.4.3))(@modelcontextprotocol/sdk@1.31.0(zod@4.4.3))
'@agentclientprotocol/codex-acp':
specifier: 1.6.2
version: 1.6.2(patch_hash=2iynuypg76nxmydiylgtffhr7e)
@@ -1006,7 +1009,7 @@
version: 2.1.0(@grpc/grpc-js@1.14.5)(nice-grpc-common@2.0.4)(nice-grpc@2.1.17)
'@vercel/connect':
specifier: 2.3.3
- version: 2.3.3(@chat-adapter/slack@4.39.0(patch_hash=226jj24akljccsfbw7bcxujz4u)(ai@7.0.130(zod@4.4.3))(zod@4.4.3))(@modelcontextprotocol/sdk@1.31.0(zod@4.4.3))(ai@7.0.130(zod@4.4.3))(better-auth@1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))))
+ version: 2.3.3(@chat-adapter/slack@4.39.0(patch_hash=226jj24akljccsfbw7bcxujz4u)(ai@7.0.130(zod@4.4.3))(zod@4.4.3))(@modelcontextprotocol/sdk@1.31.0(zod@4.4.3))(ai@7.0.130(zod@4.4.3))(better-auth@1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))))
acorn:
specifier: 8.18.0
version: 8.18.0
@@ -1024,7 +1027,7 @@
version: 3.0.1(ajv@8.20.0)
better-auth:
specifier: 1.7.2
- version: 1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)))
+ version: 1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)))
chat:
specifier: 4.39.0
version: 4.39.0(ai@7.0.130(zod@4.4.3))(zod@4.4.3)
@@ -1045,7 +1048,7 @@
version: 17.4.2
drizzle-orm:
specifier: ^0.45.2
- version: 0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi))
+ version: 0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda))
embedded-postgres:
specifier: ^18.1.0-beta.16
version: 18.1.0-beta.16(patch_hash=55uhvnotpqyiy37rn3pqpukhei)
@@ -1167,10 +1170,10 @@
version: 7.0.2
vite:
specifier: ^8.2.2
- version: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)
+ version: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)
vitest:
specifier: ^5.0.3
- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
+ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
ui:
dependencies:
@@ -1321,10 +1324,10 @@
version: 10.6.0(@types/react-dom@19.2.5(@types/react@19.2.18))(@types/react@19.2.18)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))
'@storybook/react-vite':
specifier: 10.5.10
- version: 10.5.10(@types/react-dom@19.2.5(@types/react@19.2.18))(@types/react@19.2.18)(esbuild@0.28.2)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(typescript@7.0.2)(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
+ version: 10.5.10(@types/react-dom@19.2.5(@types/react@19.2.18))(@types/react@19.2.18)(esbuild@0.28.2)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(typescript@7.0.2)(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
'@tailwindcss/vite':
specifier: ^4.3.3
- version: 4.3.3(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
+ version: 4.3.3(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
'@types/node':
specifier: ^24.0.0
version: 24.13.3
@@ -1336,7 +1339,7 @@
version: 19.2.5(@types/react@19.2.18)
'@vitejs/plugin-react':
specifier: ^6.1.1
- version: 6.1.1(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
+ version: 6.1.1(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
storybook:
specifier: 10.6.0
version: 10.6.0(@types/react@19.2.18)(react@19.2.8)
@@ -1348,10 +1351,10 @@
version: 7.0.2
vite:
specifier: ^8.2.2
- version: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)
+ version: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)
vitest:
specifier: ^5.0.3
- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
+ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
packages:
@@ -9256,6 +9259,11 @@
yallist@4.0.0:
resolution: {integrity: sha512-3wdGidZyq5PB084XLES5TpOSRA3wjXAlIWMhum2kRcv/41Sn2emQ0dycQW4uZXLejwKvg6EsvbdlVL+FYEct7A==}
+ yaml@2.9.1:
+ resolution: {integrity: sha512-3NxN8+78OdzbT7C/WjGsyfPAtJaN3FNDsWxv7Y7mcDsT/oOmgW8BpyQQFFBnvZE3j9Y2Sdz1ULFLezL7Eb2yFw==}
+ engines: {node: '>= 14.6'}
+ hasBin: true
+
yargs-parser@21.1.1:
resolution: {integrity: sha512-tVpsJW7DdjecAiFpbIB1e3qxIQsE6NoPc5/eTdrbbIC4h0LVsWhnoa3g+m2HclBIujHzsxZ4VJVA+GUuc2/LBw==}
engines: {node: '>=12'}
@@ -9316,7 +9324,7 @@
'@adobe/css-tools@4.5.0': {}
- '@agentclientprotocol/claude-agent-acp@0.73.0(patch_hash=riqgguivxjxydoydnzpedjy5au)(@anthropic-ai/sdk@0.129.0(zod@4.4.3))(@modelcontextprotocol/sdk@1.31.0(zod@4.4.3))':
+ '@agentclientprotocol/claude-agent-acp@0.73.0(patch_hash=xjlrz3rllh2ztafbdiywpfh2rq)(@anthropic-ai/sdk@0.129.0(zod@4.4.3))(@modelcontextprotocol/sdk@1.31.0(zod@4.4.3))':
dependencies:
'@agentclientprotocol/sdk': 1.4.0(zod@4.4.3)
'@anthropic-ai/claude-agent-sdk': 0.3.286(@anthropic-ai/sdk@0.129.0(zod@4.4.3))(@modelcontextprotocol/sdk@1.31.0(zod@4.4.3))(zod@4.4.3)
@@ -9325,7 +9333,7 @@
- '@anthropic-ai/sdk'
- '@modelcontextprotocol/sdk'
- '@agentclientprotocol/claude-agent-acp@0.73.0(patch_hash=riqgguivxjxydoydnzpedjy5au)(@anthropic-ai/sdk@0.129.0(zod@4.6.5))(@modelcontextprotocol/sdk@1.31.0(zod@4.6.5))':
+ '@agentclientprotocol/claude-agent-acp@0.73.0(patch_hash=xjlrz3rllh2ztafbdiywpfh2rq)(@anthropic-ai/sdk@0.129.0(zod@4.6.5))(@modelcontextprotocol/sdk@1.31.0(zod@4.6.5))':
dependencies:
'@agentclientprotocol/sdk': 1.4.0(zod@4.4.3)
'@anthropic-ai/claude-agent-sdk': 0.3.286(@anthropic-ai/sdk@0.129.0(zod@4.6.5))(@modelcontextprotocol/sdk@1.31.0(zod@4.6.5))(zod@4.4.3)
@@ -9907,12 +9915,12 @@
optionalDependencies:
'@opentelemetry/api': 1.9.1
- '@better-auth/drizzle-adapter@1.7.2(@better-auth/core@1.7.2(@better-auth/utils@0.4.2)(@better-fetch/fetch@1.3.1)(@opentelemetry/api@1.9.1)(better-call@1.4.0(zod@4.4.3))(jose@6.2.10)(kysely@0.29.5)(nanostores@1.5.3))(@better-auth/utils@0.4.2)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)))':
+ '@better-auth/drizzle-adapter@1.7.2(@better-auth/core@1.7.2(@better-auth/utils@0.4.2)(@better-fetch/fetch@1.3.1)(@opentelemetry/api@1.9.1)(better-call@1.4.0(zod@4.4.3))(jose@6.2.10)(kysely@0.29.5)(nanostores@1.5.3))(@better-auth/utils@0.4.2)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda)))':
dependencies:
'@better-auth/core': 1.7.2(@better-auth/utils@0.4.2)(@better-fetch/fetch@1.3.1)(@opentelemetry/api@1.9.1)(better-call@1.4.0(zod@4.4.3))(jose@6.2.10)(kysely@0.29.5)(nanostores@1.5.3)
'@better-auth/utils': 0.4.2
optionalDependencies:
- drizzle-orm: 0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi))
+ drizzle-orm: 0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda))
'@better-auth/kysely-adapter@1.7.2(@better-auth/core@1.7.2(@better-auth/utils@0.4.2)(@better-fetch/fetch@1.3.1)(@opentelemetry/api@1.9.1)(better-call@1.4.0(zod@4.4.3))(jose@6.2.10)(kysely@0.29.5)(nanostores@1.5.3))(@better-auth/utils@0.4.2)(kysely@0.29.5)':
dependencies:
@@ -10974,11 +10982,11 @@
'@img/sharp-win32-x64@0.35.4':
optional: true
- '@joshwooding/vite-plugin-react-docgen-typescript@0.7.0(typescript@7.0.2)(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))':
+ '@joshwooding/vite-plugin-react-docgen-typescript@0.7.0(typescript@7.0.2)(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))':
dependencies:
glob: 13.0.6
react-docgen-typescript: 2.4.0(typescript@7.0.2)
- vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)
+ vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)
optionalDependencies:
typescript: 7.0.2
@@ -13184,25 +13192,25 @@
transitivePeerDependencies:
- '@types/react-dom'
- '@storybook/builder-vite@10.5.10(esbuild@0.28.2)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))':
- dependencies:
- '@storybook/csf-plugin': 10.5.10(esbuild@0.28.2)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
+ '@storybook/builder-vite@10.5.10(esbuild@0.28.2)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))':
+ dependencies:
+ '@storybook/csf-plugin': 10.5.10(esbuild@0.28.2)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
storybook: 10.6.0(@types/react@19.2.18)(react@19.2.8)
ts-dedent: 2.3.0
- vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)
+ vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)
transitivePeerDependencies:
- esbuild
- rollup
- webpack
- '@storybook/csf-plugin@10.5.10(esbuild@0.28.2)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))':
+ '@storybook/csf-plugin@10.5.10(esbuild@0.28.2)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))':
dependencies:
storybook: 10.6.0(@types/react@19.2.18)(react@19.2.8)
unplugin: 2.3.11
optionalDependencies:
esbuild: 0.28.2
rollup: 4.63.1
- vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)
+ vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)
'@storybook/global@5.0.0': {}
@@ -13228,11 +13236,11 @@
'@types/react': 19.2.18
'@types/react-dom': 19.2.5(@types/react@19.2.18)
- '@storybook/react-vite@10.5.10(@types/react-dom@19.2.5(@types/react@19.2.18))(@types/react@19.2.18)(esbuild@0.28.2)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(typescript@7.0.2)(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))':
- dependencies:
- '@joshwooding/vite-plugin-react-docgen-typescript': 0.7.0(typescript@7.0.2)(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
+ '@storybook/react-vite@10.5.10(@types/react-dom@19.2.5(@types/react@19.2.18))(@types/react@19.2.18)(esbuild@0.28.2)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(typescript@7.0.2)(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))':
+ dependencies:
+ '@joshwooding/vite-plugin-react-docgen-typescript': 0.7.0(typescript@7.0.2)(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
'@rollup/pluginutils': 5.4.0(rollup@4.63.1)
- '@storybook/builder-vite': 10.5.10(esbuild@0.28.2)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
+ '@storybook/builder-vite': 10.5.10(esbuild@0.28.2)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
'@storybook/react': 10.5.10(@types/react-dom@19.2.5(@types/react@19.2.18))(@types/react@19.2.18)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(typescript@7.0.2)
empathic: 2.0.1
magic-string: 0.30.21
@@ -13242,7 +13250,7 @@
resolve: 1.22.12
storybook: 10.6.0(@types/react@19.2.18)(react@19.2.8)
tsconfig-paths: 4.2.0
- vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)
+ vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)
optionalDependencies:
typescript: 7.0.2
transitivePeerDependencies:
@@ -13335,12 +13343,12 @@
postcss-selector-parser: 6.0.10
tailwindcss: 4.3.3
- '@tailwindcss/vite@4.3.3(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))':
+ '@tailwindcss/vite@4.3.3(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))':
dependencies:
'@tailwindcss/node': 4.3.3
'@tailwindcss/oxide': 4.3.3
tailwindcss: 4.3.3
- vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)
+ vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)
'@tanstack/query-core@5.102.8': {}
@@ -13758,14 +13766,14 @@
dependencies:
execa: 5.1.1
- '@vercel/connect@2.3.3(@chat-adapter/slack@4.39.0(patch_hash=226jj24akljccsfbw7bcxujz4u)(ai@7.0.130(zod@4.4.3))(zod@4.4.3))(@modelcontextprotocol/sdk@1.31.0(zod@4.4.3))(ai@7.0.130(zod@4.4.3))(better-auth@1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))))':
+ '@vercel/connect@2.3.3(@chat-adapter/slack@4.39.0(patch_hash=226jj24akljccsfbw7bcxujz4u)(ai@7.0.130(zod@4.4.3))(zod@4.4.3))(@modelcontextprotocol/sdk@1.31.0(zod@4.4.3))(ai@7.0.130(zod@4.4.3))(better-auth@1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))))':
dependencies:
'@vercel/oidc': 3.8.9
optionalDependencies:
'@chat-adapter/slack': 4.39.0(patch_hash=226jj24akljccsfbw7bcxujz4u)(ai@7.0.130(zod@4.4.3))(zod@4.4.3)
'@modelcontextprotocol/sdk': 1.31.0(zod@4.4.3)
ai: 7.0.130(zod@4.4.3)
- better-auth: 1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)))
+ better-auth: 1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)))
'@vercel/oidc@3.2.0': {}
@@ -13775,7 +13783,7 @@
'@vercel/cli-exec': 1.0.1
jose: 5.10.0
- '@vitejs/plugin-react@4.7.0(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15))':
+ '@vitejs/plugin-react@4.7.0(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)(yaml@2.9.1))':
dependencies:
'@babel/core': 7.29.7
'@babel/plugin-transform-react-jsx-self': 7.29.7(@babel/core@7.29.7)
@@ -13783,14 +13791,14 @@
'@rolldown/pluginutils': 1.0.0-beta.27
'@types/babel__core': 7.20.5
react-refresh: 0.17.0
- vite: 6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)
+ vite: 6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)(yaml@2.9.1)
transitivePeerDependencies:
- supports-color
- '@vitejs/plugin-react@6.1.1(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))':
+ '@vitejs/plugin-react@6.1.1(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))':
dependencies:
'@rolldown/pluginutils': 1.0.1
- vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)
+ vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)
'@vitest/expect@3.2.4':
dependencies:
@@ -13800,23 +13808,23 @@
chai: 5.3.3
tinyrainbow: 2.0.0
- '@vitest/mocker@5.0.3(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15))':
+ '@vitest/mocker@5.0.3(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)(yaml@2.9.1))':
dependencies:
'@jridgewell/trace-mapping': 0.3.31
'@vitest/spy': 5.0.3
estree-walker: 3.0.3
magic-string: 1.4.3
optionalDependencies:
- vite: 6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)
-
- '@vitest/mocker@5.0.3(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))':
+ vite: 6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)(yaml@2.9.1)
+
+ '@vitest/mocker@5.0.3(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))':
dependencies:
'@jridgewell/trace-mapping': 0.3.31
'@vitest/spy': 5.0.3
estree-walker: 3.0.3
magic-string: 1.4.3
optionalDependencies:
- vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)
+ vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)
'@vitest/pretty-format@3.2.4':
dependencies:
@@ -14042,10 +14050,10 @@
before-after-hook@4.0.0: {}
- better-auth@1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))):
+ better-auth@1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))):
dependencies:
'@better-auth/core': 1.7.2(@better-auth/utils@0.4.2)(@better-fetch/fetch@1.3.1)(@opentelemetry/api@1.9.1)(better-call@1.4.0(zod@4.4.3))(jose@6.2.10)(kysely@0.29.5)(nanostores@1.5.3)
- '@better-auth/drizzle-adapter': 1.7.2(@better-auth/core@1.7.2(@better-auth/utils@0.4.2)(@better-fetch/fetch@1.3.1)(@opentelemetry/api@1.9.1)(better-call@1.4.0(zod@4.4.3))(jose@6.2.10)(kysely@0.29.5)(nanostores@1.5.3))(@better-auth/utils@0.4.2)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)))
+ '@better-auth/drizzle-adapter': 1.7.2(@better-auth/core@1.7.2(@better-auth/utils@0.4.2)(@better-fetch/fetch@1.3.1)(@opentelemetry/api@1.9.1)(better-call@1.4.0(zod@4.4.3))(jose@6.2.10)(kysely@0.29.5)(nanostores@1.5.3))(@better-auth/utils@0.4.2)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda)))
'@better-auth/kysely-adapter': 1.7.2(@better-auth/core@1.7.2(@better-auth/utils@0.4.2)(@better-fetch/fetch@1.3.1)(@opentelemetry/api@1.9.1)(better-call@1.4.0(zod@4.4.3))(jose@6.2.10)(kysely@0.29.5)(nanostores@1.5.3))(@better-auth/utils@0.4.2)(kysely@0.29.5)
'@better-auth/memory-adapter': 1.7.2(@better-auth/core@1.7.2(@better-auth/utils@0.4.2)(@better-fetch/fetch@1.3.1)(@opentelemetry/api@1.9.1)(better-call@1.4.0(zod@4.4.3))(jose@6.2.10)(kysely@0.29.5)(nanostores@1.5.3))(@better-auth/utils@0.4.2)
'@better-auth/mongo-adapter': 1.7.2(@better-auth/core@1.7.2(@better-auth/utils@0.4.2)(@better-fetch/fetch@1.3.1)(@opentelemetry/api@1.9.1)(better-call@1.4.0(zod@4.4.3))(jose@6.2.10)(kysely@0.29.5)(nanostores@1.5.3))(@better-auth/utils@0.4.2)
@@ -14063,11 +14071,11 @@
zod: 4.4.3
optionalDependencies:
drizzle-kit: 0.31.10
- drizzle-orm: 0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi))
+ drizzle-orm: 0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda))
pg: 8.18.0
react: 19.2.8
react-dom: 19.2.8(react@19.2.8)
- vitest: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
+ vitest: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
transitivePeerDependencies:
- '@cloudflare/workers-types'
- '@opentelemetry/api'
@@ -14654,12 +14662,12 @@
esbuild: 0.25.12
tsx: 4.23.15
- drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)):
+ drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda)):
optionalDependencies:
'@opentelemetry/api': 1.9.1
kysely: 0.29.5
pg: 8.18.0
- postgres: 3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)
+ postgres: 3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda)
dunder-proto@1.0.1:
dependencies:
@@ -16542,7 +16550,7 @@
dependencies:
xtend: 4.0.2
- postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi): {}
+ postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda): {}
powershell-utils@0.1.0: {}
@@ -17587,7 +17595,7 @@
'@types/unist': 3.0.3
vfile-message: 4.0.3
- vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15):
+ vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)(yaml@2.9.1):
dependencies:
esbuild: 0.25.12
fdir: 6.5.0(picomatch@4.0.7)
@@ -17601,8 +17609,9 @@
jiti: 2.7.0
lightningcss: 1.33.0
tsx: 4.23.15
-
- vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15):
+ yaml: 2.9.1
+
+ vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1):
dependencies:
lightningcss: 1.33.0
picomatch: 4.0.7
@@ -17615,11 +17624,12 @@
fsevents: 2.3.3
jiti: 2.7.0
tsx: 4.23.15
-
- vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)):
+ yaml: 2.9.1
+
+ vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)(yaml@2.9.1)):
dependencies:
'@types/chai': 5.2.3
- '@vitest/mocker': 5.0.3(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15))
+ '@vitest/mocker': 5.0.3(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)(yaml@2.9.1))
chai: 6.3.0
es-module-lexer: 2.3.2
expect-type: 1.4.0
@@ -17630,7 +17640,7 @@
tinybench: 6.2.0
tinyexec: 1.3.1
tinyglobby: 0.2.17
- vite: 6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)
+ vite: 6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)(yaml@2.9.1)
why-is-node-running: 3.2.1
optionalDependencies:
'@opentelemetry/api': 1.9.1
@@ -17639,10 +17649,10 @@
transitivePeerDependencies:
- msw
- vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)):
+ vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)):
dependencies:
'@types/chai': 5.2.3
- '@vitest/mocker': 5.0.3(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
+ '@vitest/mocker': 5.0.3(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
chai: 6.3.0
es-module-lexer: 2.3.2
expect-type: 1.4.0
@@ -17653,7 +17663,7 @@
tinybench: 6.2.0
tinyexec: 1.3.1
tinyglobby: 0.2.17
- vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)
+ vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)
why-is-node-running: 3.2.1
optionalDependencies:
'@opentelemetry/api': 1.9.1
@@ -17748,6 +17758,8 @@
yallist@3.1.1: {}
yallist@4.0.0: {}
+
+ yaml@2.9.1: {}
yargs-parser@21.1.1: {}
+5
View File
@@ -7,6 +7,7 @@ import type { RunnerE2ECampaignAccounting, RunnerE2EResult } from "./types.js";
import { DEPENDABLE_CELLS } from "./dependable-qualification.js";
import { validateOnboardingConnectEvidence } from "./product-journeys/onboarding-connect.js";
import { validateOnboardingArtifact } from "./product-journeys/artifact-evidence.js";
import { validateOnboardingPreAcceptanceEvidence } from "./product-journeys/onboarding-pre-acceptance.js";
type Rule = (value: unknown, at: string) => void;
const invalid = (at: string): never => {
@@ -355,6 +356,7 @@ const fields = {
targetId: optional(string),
campaignId: optional(string), attemptId: optional(string),
onboardingConnect: optional((value, at) => { try { validateOnboardingConnectEvidence(value); } catch { invalid(at); } }),
onboardingPreAcceptance: optional((value, at) => { try { validateOnboardingPreAcceptanceEvidence(value); } catch { invalid(at); } }),
artifact: optional((value, at) => { try { validateOnboardingArtifact(value); } catch { invalid(at); } }),
operations: optional(closedShape({ campaignId: string, attemptId: string, modelCostUsd: nullable(number), setupCostUsd: nullable(number), requestCount: integer,
feedback: closedShape({ collection: oneOf("complete", "partial", "unavailable"), complaints: integer, suggestions: integer }), traceIds: array(string) })),
@@ -414,6 +416,9 @@ export function validateRetainedRunnerResult(
const checked = value as RunnerE2EResult;
if (checked.databaseIdentity && checked.status === "passed" && (!checked.databaseIdentity.artifactVerified || !checked.databaseIdentity.contextSha256 || !checked.databaseIdentity.engineSha256 || !checked.databaseIdentity.containerId || !checked.databaseIdentity.cluster || !["ready", "removed"].includes(checked.databaseIdentity.status))) invalid("result.databaseIdentity");
const artifact = checked.productJourney?.artifact;
if (checked.productJourney?.onboardingPreAcceptance && (checked.suiteId !== "product-journeys" || checked.caseId !== "cloud-onboarding" ||
checked.status !== "failed" || checked.productJourney.outcome !== "failed" ||
checked.error !== "onboarding_work_before_acceptance")) invalid("result.productJourney.onboardingPreAcceptance");
if (artifact && (checked.suiteId !== "product-journeys" || checked.caseId !== "cloud-onboarding" ||
checked.runIds && !checked.runIds.includes(artifact.originatingRunId))) invalid("result.productJourney.artifact");
if (checked.workerTiming !== undefined) {
+1 -1
View File
@@ -1031,7 +1031,7 @@ for (const execution of executions) {
daytonaImage: process.env.PAPERCLIP_E2E_DAYTONA_IMAGE,
onCompanyCreated,
});
await finishBudgetedEvalSetup({ executionId: execution.id, agentId: fixtures.agent.id,
await finishBudgetedEvalSetup({ executionId: execution.id, environmentId: execution.environment.id, agentId: fixtures.agent.id,
companyId: fixtures.company.id, deadlineAt: startedAtMs + deadlineMs, api,
observe: observations => writeSanitizedJson(snapshotsDir, "budgeted-setup.json", { observations }, secrets) });
await workerTiming.enter("execution");
@@ -1,3 +1,9 @@
import { mkdtemp, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { openEvalCostLedger } from "./cost-ledger.js";
import { readEvalOperationsReport } from "./operations-report.js";
import { EVAL_OVERHEAD_COMPONENTS } from "./operating-overhead.js";
import { expect, it } from "vitest";
import { buildRunnerCampaign } from "./history.js";
import { runnerMatrix } from "./catalog.js";
@@ -52,3 +58,31 @@ it("keeps incomplete accounting and trace export visible independently of a pass
campaign.feedbackReview.decisions[0]!.disposition = "needs_investigation";
expect(scheduledCampaignFindings(campaign)).toContain("feedback_needs_review");
});
it("qualifies scheduled model receipts while retaining operating holds, but rejects missing usage and overruns", async () => {
const { campaign } = fixture();
const directory = await mkdtemp(path.join(tmpdir(), "scheduled-model-readiness-"));
const file = path.join(directory, "ledger.jsonl"), now = Date.now();
const ledger = await openEvalCostLedger(file, { campaignId: "scheduled-fixture", maxUsd: "20",
deadlineAt: new Date(now + 60_000).toISOString(), maxRequests: 10, maxConcurrent: 1,
overhead: { version: 1, accountingReadiness: "model_requests", reference: "synthetic-fixture", runner: "github-hosted",
verifiedAt: new Date(now - 1000).toISOString(), expiresAt: new Date(now + 86_400_000).toISOString(),
components: EVAL_OVERHEAD_COMPONENTS.map(component => ({ component, maxUsd: "1", reference: "synthetic-fixture" })) },
maxAttemptsPerExecution: 1 });
try {
await ledger.admitAttempt(campaign.expected[0]!, "one");
await ledger.reserve("model", { executionId: campaign.expected[0]!, attemptId: "one", phase: "setup" }, "2");
await ledger.finish("model", "cancelled");
campaign.operations = await readEvalOperationsReport(file);
expect(scheduledCampaignFindings(campaign)).toContain("accounting_incomplete");
await ledger.observe({ observationId: "model-final", requestId: "model", source: "provider", provider: "anthropic",
currency: "USD", route: "direct", classification: "reported", costUsd: "0.5", final: true });
campaign.operations = await readEvalOperationsReport(file);
expect(scheduledCampaignFindings(campaign)).not.toContain("accounting_incomplete");
expect(campaign.operations).toMatchObject({ heldUsd: "4.000000000", committedUsd: "4.500000000" });
await ledger.observe({ observationId: "overrun", requestId: "overhead:ci_compute", source: "invoice", provider: "aws",
currency: "USD", route: "infrastructure", classification: "reconciled", costUsd: "21", final: true });
campaign.operations = await readEvalOperationsReport(file);
expect(scheduledCampaignFindings(campaign)).toEqual(expect.arrayContaining(["accounting_incomplete", "accounting_over_budget"]));
} finally { await ledger.close(); await rm(directory, { recursive: true, force: true }); }
});
+3 -2
View File
@@ -1,5 +1,5 @@
import { usdToUnits } from "../../packages/shared/src/money.js";
import { evalAccountingStatus } from "./operations-report.js";
import { evalAccountingStatus, evalReadinessStatus } from "./operations-report.js";
import { readFile, writeFile } from "node:fs/promises";
import path from "node:path";
import { pathToFileURL } from "node:url";
@@ -49,7 +49,7 @@ export function scheduledCampaignFindings(campaign: RunnerE2ECampaign): string[]
const operations = campaign.operations;
if (!operations || operations.status === "unavailable") findings.push("accounting_unavailable");
else {
if (evalAccountingStatus(operations) === "blocked") findings.push("accounting_incomplete");
if (evalReadinessStatus(operations) === "blocked") findings.push("accounting_incomplete");
if (usdToUnits(operations.committedUsd) > usdToUnits(operations.limitUsd)) findings.push("accounting_over_budget");
}
const feedback = summarizeCampaignFeedback(campaign.accounting?.attempts ?? campaign.results);
@@ -131,6 +131,7 @@ export async function finalizeScheduledCampaign(env: NodeJS.ProcessEnv) {
await writeFile(path.join(directory, "findings.json"), `${JSON.stringify({ version: 1, findings,
nativeIndexVerification: indexing, gatewayIndexVerification: gatewayIndexing, invoiceReconciliation: "pending",
accountingStatus: campaign.operations ? evalAccountingStatus(campaign.operations) : "unavailable",
readinessStatus: campaign.operations ? evalReadinessStatus(campaign.operations) : "unavailable",
feedbackReviewMethod: "Only exact synthetic fixture bodies with all corroborating graders are automatically dismissed; all other feedback requires review" }, null, 2)}\n`, { mode: 0o600 });
for (const finding of findings) console.error(`::error title=Eval operations::${finding}`);
return findings;
+9
View File
@@ -175,6 +175,7 @@ describe("runner E2E local binary resolution", () => {
runnerBinary,
undefined,
"linux",
"x64",
),
).toBe(runnerBinary);
expect(
@@ -183,6 +184,7 @@ describe("runner E2E local binary resolution", () => {
runnerBinary,
undefined,
"linux",
"x64",
),
).toBeUndefined();
expect(
@@ -201,7 +203,14 @@ describe("runner E2E local binary resolution", () => {
"darwin",
),
).toBe("/cross-compiled/paperclip-runnerd");
expect(resolvePaperclipRemoteRunnerBinaryForHarness(
[remoteNativeExecution], runnerBinary, undefined, "linux", "arm64",
)).toBeUndefined();
expect(resolvePaperclipRemoteRunnerBinaryForHarness(
[remoteNativeExecution], runnerBinary, "/cross-compiled/linux-x64-runnerd", "linux", "arm64",
)).toBe("/cross-compiled/linux-x64-runnerd");
});
});
describe("runner E2E provider environment", () => {
+26 -3
View File
@@ -56,6 +56,10 @@ describe("public repository paid workflow security", () => {
expect(manual).toContain('pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile');
expect(manual).toContain('PAPERCLIP_RUNNER_LOCK_SHA256=$lock_sha');
expect(manual).toContain('docker logout ghcr.io');
expect(manual.indexOf('Apply reviewed qualification lock corrections')).toBeLessThan(manual.indexOf('Resolve source dependencies without lifecycle scripts'));
expect(manual).toContain('node tests/runner-e2e/prepare-report-lock.mjs');
expect(manual).toContain('if [ "$TARGET_SHA" = "$GITHUB_SHA" ]; then');
});
it("uses the reviewed master branch for the first-party trusted PR workflow", async () => {
@@ -275,7 +279,9 @@ describe("public repository paid workflow security", () => {
"target_ref: ${{ steps.target.outputs.ref }}",
);
expect(authorizeJob).toContain('echo "ref=refs/heads/$TARGET_BRANCH"');
expect(authorizeJob).not.toContain("actions/checkout@");
expect(authorizeJob.match(/actions\/checkout@/gu)).toHaveLength(1);
expect(authorizeJob).toContain("ref: ${{ github.sha }}");
expect(authorizeJob).not.toContain("ref: ${{ needs.authorize.outputs.target_sha }}");
expect(authorizeJob).not.toContain("pnpm install");
expect(targetLockJob).toContain("name: Resolve target pnpm lockfile");
expect(targetLockJob).toContain("needs: authorize");
@@ -537,8 +543,8 @@ describe("public repository paid workflow security", () => {
/ref: \$\{\{ needs\.authorize\.outputs\.target_sha \}\}/g,
),
).toHaveLength(6);
expect(fullStack.match(/ref: \$\{\{ github\.sha \}\}/g)).toHaveLength(2);
expect(fullStack.match(/persist-credentials: false/g)).toHaveLength(8);
expect(fullStack.match(/ref: \$\{\{ github\.sha \}\}/g)).toHaveLength(3);
expect(fullStack.match(/persist-credentials: false/g)).toHaveLength(9);
expect(fullStack).not.toContain("ref: ${{ inputs.target_branch }}");
expect(fullStack).toContain(
"PAPERCLIP_RUNNER_SOURCE_REVISION=${TARGET_SHA}",
@@ -553,11 +559,22 @@ describe("public repository paid workflow security", () => {
"ref: ${{ needs.authorize.outputs.target_sha }}",
);
expect(reportJob).not.toContain("Download resolved target lockfile");
expect(reportJob).toContain("run: node tests/runner-e2e/prepare-report-lock.mjs");
expect(reportJob.indexOf("Apply reviewed reporting lock corrections")).toBeLessThan(reportJob.indexOf("Install pinned reporting dependencies"));
expect(reportJob).not.toContain("needs.target_lock.outputs");
expect(historyJob).toContain("ref: ${{ github.sha }}");
expect(historyJob).not.toContain(
"ref: ${{ needs.authorize.outputs.target_sha }}",
);
expect(historyJob).not.toContain("Download resolved target lockfile");
expect(historyJob).toContain("run: node tests/runner-e2e/prepare-report-lock.mjs");
const authorization = fullStack.split(" authorize:\n")[1]!.split(" target_lock:\n")[0]!;
expect(authorization).toContain("ref: ${{ github.sha }}");
expect(authorization).toContain("Preflight reviewed report dependencies before paid admission");
expect(authorization).toContain("run: node tests/runner-e2e/prepare-report-lock.mjs");
expect(authorization).not.toContain("environment:");
expect(fullStack).toContain(
"if: always() && !cancelled() && needs.catalog.result == 'success'",
);
@@ -964,6 +981,12 @@ it("keeps dependency resolution and shared caches out of Runner protocol archive
}
const builder = jobs.find(job => job.startsWith(" build_controller:"))!;
expect(builder).not.toMatch(/id-token: write|environment:|secrets\./u);
const upload = builder.split(" - name: Retain the trusted controller code for every worker\n")[1]!.split("\n - ")[0]!;
expect(upload).toContain("include-hidden-files: true");
expect(upload).toContain("id: upload");
expect(builder).toContain("artifact-ids: ${{ steps.upload.outputs.artifact-id }}");
expect(builder).toContain("run: node --test .github/tests/runner-controller-artifact-transport.test.mjs");
expect(builder).toContain("PAPERCLIP_CONTROLLER_RECEIPT_SHA256: ${{ steps.build.outputs.runtime_receipt_sha256 }}");
const preparer = jobs.find(job => job.startsWith(" prepare_controller:"))!;
expect(preparer).not.toMatch(/pnpm install/u);
expect(preparer.indexOf("verify-runtime.mjs")).toBeLessThan(preparer.indexOf('hosted.mjs" prepare'));
+19 -1
View File
@@ -1,5 +1,5 @@
import { spawnSync } from "node:child_process";
import { mkdtemp, readFile, rm } from "node:fs/promises";
import { mkdtemp, readFile, rm, access } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { describe, expect, it } from "vitest";
@@ -47,3 +47,21 @@ describe("paid workflow target authorization", () => {
expect(result.output).toContain("ref=refs/heads/development\n");
});
});
describe("qualification image dependency correction", () => {
it.each([true, false])("applies the workflow correction only to its own source: %s", async sameSource => {
const workflow = await readFile(new URL("../../.github/workflows/docker-runner-check.yml", import.meta.url), "utf8");
const step = workflow.split(" - name: Apply reviewed qualification lock corrections\n")[1]!.split("\n - ")[0]!;
const script = step.split(" run: |\n")[1]!.replace(/^ {10}/gm, "");
const directory = await mkdtemp(path.join(os.tmpdir(), "eval-image-lock-target-"));
try {
const marker = path.join(directory, "called");
const result = spawnSync("bash", ["-c", `node() { touch "$CALLED"; }\n${script}`], {
cwd: directory, encoding: "utf8", env: { ...process.env, GITHUB_SHA: queuedSha,
TARGET_SHA: sameSource ? queuedSha : advancedSha, CALLED: marker },
});
expect(result.status, result.stderr).toBe(0);
expect(await access(marker).then(() => true, () => false)).toBe(sameSource);
} finally { await rm(directory, { recursive: true, force: true }); }
});
});