mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-05 21:02:34 +01:00
Merge pinned production for the post-custody chat pilot
Preserve frozen 244-scenario YAML intent and 191-capability coverage. Production baseline: 39e811c9c5.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
commit
0195fcb016
398 files changed
+90564
-66016
No files matched your search
@@ -223,7 +223,8 @@ stable key-sorted JSON hashing. The model configuration contains `provider`,
|
||||
`mode` (the manifest route), `price`, `maximumInputTokens`,
|
||||
`maximumOutputTokens`, and `reservation` (the nine-decimal value from
|
||||
`evalRequestReservation`). An operating configuration contains allowance
|
||||
`reference`, `runner`, `verifiedAt`, `expiresAt`, and a `component` object with
|
||||
`reference`, `runner`, `verifiedAt`, `expiresAt`, optional `accountingReadiness`
|
||||
(only when present), and a `component` object with
|
||||
its `component`, `maxUsd`, and `reference`. Changing any of those values
|
||||
invalidates the approval hash. Evidence metadata itself is excluded from this
|
||||
configuration hash.
|
||||
@@ -439,8 +440,11 @@ attempt. Installing the ACP package alone does not satisfy this setup path.
|
||||
|
||||
Bounded cases keep the gateway in the setup phase until the created agent's
|
||||
asynchronous lifecycle verification reaches ready. This wait uses the remaining
|
||||
attempt deadline, allows at most 45 seconds, and stops on an observed setup error
|
||||
attempt deadline, allows at most 45 seconds locally and the bounded attempt window
|
||||
for Daytona provisioning, and stops on an observed setup error
|
||||
or hold. The private `budgeted-setup.json` snapshot retains state transitions.
|
||||
Budgeted Daytona fixtures give the saved-account hello probe five minutes inside
|
||||
that attempt window; its former 90-second default could expire during cold startup.
|
||||
Only then does the case begin agent-phase accounting and create its task.
|
||||
Older evidence that advanced after fixture creation may label hello probes as
|
||||
agent cost. Its campaign totals still include those requests; retain that phase
|
||||
@@ -897,3 +901,69 @@ an operations manifest. They use the same metered connection, prices, campaign
|
||||
journal, setup boundary, and durable attempt allowance as core compatibility.
|
||||
Collection and the independent saved-evidence oracle require no additional
|
||||
model call. The synthetic items need review as test data, not product incidents.
|
||||
|
||||
|
||||
### Qualify model spend while operating reconciliation is pending
|
||||
|
||||
Set `accountingReadiness: "model_requests"` on the reviewed allowance to use
|
||||
complete model receipts as the scheduled readiness gate. This is an explicit
|
||||
operator policy. The report retains the full-accounting status, operating
|
||||
reservations and model-readiness status separately. It does not release any
|
||||
historical hold or certify a full-lifecycle vendor cost bound. Missing model
|
||||
receipts, conflicts, active requests, excess commitments, and the existing
|
||||
product, cleanup, feedback and trace checks still fail qualification.
|
||||
|
||||
|
||||
### Explicit bounded Daytona gateway qualification
|
||||
|
||||
A manual manifest can set `remoteGateway: "cloudflare"` for the native Codex
|
||||
or Claude marker/feedback Daytona cells. Install `cloudflared` on the controller
|
||||
PATH, or set `PAPERCLIP_EVAL_CLOUDFLARED` to its executable path. Empty overrides
|
||||
use the PATH default. The parent starts its existing model
|
||||
gateway and native OTLP relay, then exposes only their authenticated endpoints
|
||||
through one owned HTTPS quick tunnel. It verifies a fresh health nonce before
|
||||
fixture setup, waits up to 120 seconds for registration and 120 seconds for
|
||||
public readiness, and does not retry tunnel creation or provider work.
|
||||
|
||||
Only attempt-scoped tokens leave the controller. Provider and Braintrust keys
|
||||
stay in the parent; the fixture/server retains the existing Daytona provisioning
|
||||
credential separately. The model gateway still enforces its campaign allowance,
|
||||
request count, concurrency, deadline and receipt journal. The bridge streams
|
||||
responses and preserves terminal-usage draining after a client disconnect.
|
||||
The tunnel receives no ambient provider credentials or Cloudflare config. Its
|
||||
exposure closes at the campaign deadline or four hours after startup, whichever
|
||||
comes first, and on attempt cleanup. Owned gateway,
|
||||
relay and tunnel cleanup failures remain failures. A configured remote route
|
||||
is not live acceptance; retain the actual Daytona result, cleanup, feedback,
|
||||
native traces and complete model receipts.
|
||||
|
||||
The Daytona image runs Linux/x64. ARM or macOS controllers use the image's
|
||||
verified daemon by default; only an explicit cross-compiled remote override
|
||||
replaces it.
|
||||
|
||||
Daily and weekly selections remain local. This explicit remote option adds no
|
||||
scheduled coverage or public report publication.
|
||||
|
||||
The worker generates the attempt's trace parent. The relay validates the project
|
||||
in both the header and payload, but does not pin that worker-generated parent.
|
||||
Treat the attempt OTLP token as authority to submit trace attribution within
|
||||
this eval project. Keep it limited to the trusted harness/provider runtime;
|
||||
never expose it to test prompts or unrelated clients. The project check is not
|
||||
an attempt-isolation guarantee. Indexing verification is a separate retained
|
||||
or scheduled check; the manual launcher alone does not perform it.
|
||||
|
||||
### Hosted report dependencies
|
||||
|
||||
The trusted report job applies a reviewed `report-lock.patch` before its frozen
|
||||
install. `report-lock.json` pins the input lock, patch and resulting lock hashes.
|
||||
This updates existing patch identities and the already declared yaml dependency
|
||||
without registry resolution. The worker target and its lock artifact never
|
||||
control report dependencies. An unknown committed lock fails before install;
|
||||
update and review this correction when dependency inputs change. The original
|
||||
failed run remains failed when retained evidence gets a new presentation.
|
||||
|
||||
The trusted authorization job checks this correction before catalog/build and
|
||||
paid work. Unknown dependency locks fail before model admission. Manual history
|
||||
publication applies the same correction before its frozen install. Qualification
|
||||
image builds apply it only when target and workflow revisions match; a different
|
||||
target keeps its own lock and does not require these reporting helper files.
|
||||
@@ -118,7 +118,8 @@ ownership manifests. An expired lease permits cleanup recovery, not reuse.
|
||||
Managed self-hosted instances are removed after every attempt. Existing Cloud
|
||||
stacks retain the QA account and declared baseline while removing verified
|
||||
run-owned fixtures. Successful disposable stacks are deleted. Failed disposable
|
||||
stacks are stopped and retained for 24 hours, then swept. Interrupted cleanup
|
||||
stacks are stopped and retained for up to 24 hours. The deployment provider
|
||||
declares earlier cleanup eligibility to leave time for its sweeper. Interrupted cleanup
|
||||
remains visible and retains exclusive stack ownership. QA identity and inbox
|
||||
never belong to disposable ownership.
|
||||
|
||||
|
||||
@@ -261,6 +261,16 @@ status, known adapters, and allowlisted outcome/check/server codes. The case
|
||||
report renders these fields explicitly; older results without the receipt remain
|
||||
valid. This diagnostic does not change the journey grade or qualify a live run.
|
||||
|
||||
When the canonical first-task journey fails `onboarding_work_before_acceptance`,
|
||||
`productJourney.onboardingPreAcceptance` retains the failed checkpoint and every
|
||||
observed guard violation: unexpected task set, existing attachment, non-planning
|
||||
document, or first task already Done. The closed version-1 receipt contains only
|
||||
counts, whether the first task matches the canonical parent, and closed first-task
|
||||
and canonical-parent statuses. Reports validate and display these observations;
|
||||
they do not infer a product cause. Only the failing boundary is retained, without
|
||||
task/document IDs, document keys, text, or bodies. Historical results may omit the
|
||||
receipt. The original error, failed grade, and dependent blocked outcomes remain.
|
||||
|
||||
## Live provider connection journeys
|
||||
|
||||
See [PROVIDER-CONNECTIONS.md](PROVIDER-CONNECTIONS.md) for the explicit-only
|
||||
@@ -1262,9 +1272,17 @@ intentionally uncommitted workspace patch while keeping every target job on one
|
||||
identical dependency resolution. The shared-build job compiles the selected
|
||||
campaign's TypeScript outputs and native binaries once, then each paid cell
|
||||
verifies and extracts the immutable bundle. Remote native cells similarly reuse
|
||||
one verified provider pack. Report
|
||||
sanitization and AWS history publication do not consume the target lockfile;
|
||||
they explicitly check out and install from the trusted workflow commit. The
|
||||
one verified provider pack. Report sanitization and AWS history publication
|
||||
explicitly check out the trusted workflow commit. The report applies the
|
||||
reviewed, hash-pinned `report-lock.patch` to the committed lock before its
|
||||
frozen install. It does not resolve registry versions at run time or accept a
|
||||
worker target lock. The patch changes existing package patch identities and
|
||||
links the already declared yaml dependency; existing package tarball integrity entries
|
||||
stay unchanged. The correction adds exactly yaml@2.9.1 with pinned integrity. The
|
||||
trusted authorization job rejects unknown locks before paid work. Manual history
|
||||
publication uses the same reviewed correction. Qualification image builds apply
|
||||
it only to a target equal to the workflow revision; other targets keep their own
|
||||
lock. Unknown lock contents fail before installation. The
|
||||
workflow definition, runner-group permission, and protected-environment
|
||||
deployment still come from the default branch. Do not select the target branch
|
||||
in GitHub's **Use workflow from** control.
|
||||
@@ -3051,3 +3069,21 @@ codes/categories. Public pause fences automatic setup retries; failed pause
|
||||
requests the existing owned supervisor's shutdown before further API observations.
|
||||
Multiple probe receipts fail health. Setup-probe usage/cost is unavailable, never
|
||||
zero. This diagnostic cannot qualify a catalog scenario or rewrite prior results.
|
||||
|
||||
### Model-spend readiness and operating reconciliation
|
||||
|
||||
A reviewed operating allowance can opt into `accountingReadiness: "model_requests"`.
|
||||
This policy requires at least one model request and complete final model-cost
|
||||
receipts before scheduled qualification succeeds. Missing or conflicting model
|
||||
receipts, active requests, failed product/cleanup/trace/feedback checks, and a
|
||||
campaign commitment above its limit still fail qualification. The gateway still
|
||||
reserves every request before forwarding it. Attempts, deadlines and concurrency
|
||||
remain bounded.
|
||||
|
||||
The report shows the model subtotal and model-readiness status separately from
|
||||
full accounting status. Unknown operating charges remain unknown. Operating
|
||||
reservations stay in the same campaign admission allowance until reconciled;
|
||||
they are not reported as charges or converted into reviewed lifecycle bounds.
|
||||
This mode therefore does not certify a full-lifecycle operating-cost ceiling.
|
||||
Omitting the policy preserves the stricter full-accounting gate. Historical
|
||||
journals and their holds are not rewritten by changing future campaign policy.
|
||||
@@ -28,6 +28,16 @@ Unknown codes become `unrecognized`; raw bodies, messages, URLs, headers, and
|
||||
browser errors are excluded. Reports validate the receipt before rendering its
|
||||
explicit fields, and historical results may omit it.
|
||||
|
||||
The optional `onboardingPreAcceptance` receipt captures only a failing first-task
|
||||
acceptance guard. Its closed version-1 fields are a known checkpoint, up to four
|
||||
distinct guard-violation codes, nonnegative integer task/attachment/document
|
||||
counts, a canonical-parent match boolean, and closed first-task/canonical-parent
|
||||
statuses (`missing` and `unknown` remain explicit). It excludes all identifiers,
|
||||
task text, document keys and bodies, filenames, URLs, credentials, and raw API
|
||||
responses. Validation rejects extra or inconsistent fields before report use;
|
||||
older results may omit the receipt. It preserves the original failure grade and
|
||||
does not attribute a product cause or authorize acceptance.
|
||||
|
||||
## Attended connection authentication
|
||||
|
||||
See [PROVIDER-CONNECTIONS.md](PROVIDER-CONNECTIONS.md) for the explicit-only
|
||||
|
||||
@@ -4,7 +4,7 @@ import { promisify } from "node:util";
|
||||
import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { buildBraintrustExperiment, importBraintrustExperiment, renderBraintrustExperiment, type BraintrustExperimentReceipt } from "./braintrust-experiment.js";
|
||||
import { buildBraintrustExperiment, experimentEvidenceHash, importBraintrustExperiment, renderBraintrustExperiment, type BraintrustExperimentReceipt } from "./braintrust-experiment.js";
|
||||
import { regenerateRunnerDashboard } from "./dashboard-regenerate.js";
|
||||
import { buildRunnerCampaign } from "./history.js";
|
||||
import { runnerMatrix } from "./catalog.js";
|
||||
@@ -326,3 +326,30 @@ it.each(["public", "wrong-project", "wrong-hash", "partial-insert", "receiver-er
|
||||
expect(JSON.stringify(receipts)).not.toMatch(/private-key|private-upstream/);
|
||||
if (["partial-insert", "receiver-error"].includes(kind)) expect(receipts[0]).toMatchObject({ status: "created", acceptedRows: 0 });
|
||||
});
|
||||
|
||||
|
||||
it("preserves retained experiment identity when adding journal-derived token presentation", async () => {
|
||||
const { openEvalCostLedger } = await import("./cost-ledger.js");
|
||||
const { readEvalOperationsReport } = await import("./operations-report.js");
|
||||
const { campaign, originals } = fixture();
|
||||
const root = await mkdtemp(path.join(tmpdir(), "experiment-token-presentation-")), file = path.join(root, "ledger.jsonl");
|
||||
const ledger = await openEvalCostLedger(file, { campaignId: campaign.campaignId, maxUsd: "1", deadlineAt: "2099-01-01T00:00:00Z", maxRequests: 2, maxConcurrent: 1 });
|
||||
try {
|
||||
await ledger.reserve("setup", { executionId: originals[0]!.executionId, attemptId: "first", phase: "setup" }, "0.2");
|
||||
await ledger.observe({ observationId: "usage", requestId: "setup", provider: "fixture", source: "gateway", currency: "USD", route: "direct", classification: "reported", costUsd: "0.05", final: true,
|
||||
tokens: { input: 1, cachedRead: 2, cachedWrite: 3, output: 4 } });
|
||||
await ledger.finish("setup", "cancelled");
|
||||
const enriched = await readEvalOperationsReport(file);
|
||||
if (enriched.status === "unavailable") throw new Error("Expected token report");
|
||||
campaign.operations = structuredClone(enriched);
|
||||
delete campaign.operations.modelAccounting!.tokenUsage;
|
||||
const legacyHash = experimentEvidenceHash(campaign);
|
||||
const receipt = await importBraintrustExperiment({ projectId, apiKey: "key" }, campaign, originals, async () => {}, receiver(campaign, originals));
|
||||
campaign.braintrustExperiment = receipt;
|
||||
campaign.operations = enriched;
|
||||
expect(experimentEvidenceHash(campaign)).toBe(legacyHash);
|
||||
expect(renderBraintrustExperiment(campaign)).toContain("2/2 rows accepted");
|
||||
campaign.operations.journalHash = "f".repeat(64);
|
||||
expect(() => renderBraintrustExperiment(campaign)).toThrow("Invalid retained");
|
||||
} finally { await ledger.close(); await rm(root, { recursive: true, force: true }); }
|
||||
});
|
||||
@@ -114,13 +114,15 @@ export interface BraintrustExperimentReceipt {
|
||||
}
|
||||
|
||||
export function experimentEvidenceHash(campaign: RunnerE2ECampaign) {
|
||||
const operations = campaign.operations ? structuredClone(campaign.operations) : null;
|
||||
if (operations && operations.status !== "unavailable" && operations.modelAccounting) delete operations.modelAccounting.tokenUsage;
|
||||
return digest({ campaignId: campaign.campaignId, expected: [...campaign.expected].sort(),
|
||||
selected: campaign.results.map(result => ({ executionId: result.executionId, accountingIdentity: result.accountingIdentity,
|
||||
attempt: result.attempt, startedAt: result.attempt === 0 ? null : result.startedAt, sourceRevision: result.source?.sha ?? null,
|
||||
suiteDefinitionHash: result.suiteDefinitionHash ?? null, suiteId: result.suiteId, profileId: result.profileId,
|
||||
caseId: result.caseId, provider: result.provider, model: result.model, environmentId: result.environmentId,
|
||||
runtimeMode: result.runtimeMode })).sort((a, b) => digest(a).localeCompare(digest(b))),
|
||||
accounting: campaign.accounting, operations: campaign.operations ?? null, feedbackReview: campaign.feedbackReview ?? null });
|
||||
accounting: campaign.accounting, operations, feedbackReview: campaign.feedbackReview ?? null });
|
||||
}
|
||||
|
||||
function safeBraintrustUrl(value: unknown): value is string {
|
||||
|
||||
@@ -211,3 +211,13 @@ it("retains finalization export failures without changing the recorded product r
|
||||
}, transport)).toMatchObject({ status: "failed", errorCode: "invalid_finalization_context" });
|
||||
expect(transport).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
|
||||
it("binds remote OTLP to the explicitly assigned eval gateway origin", () => {
|
||||
const origin = "https://synthetic-eval.trycloudflare.com";
|
||||
const env = { PAPERCLIP_E2E_BRAINTRUST_PROJECT_ID: projectId, BRAINTRUST_API_KEY: "synthetic-token",
|
||||
PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN: origin, PAPERCLIP_E2E_OTLP_RELAY_URL: `${origin}/otel` };
|
||||
expect(evalBraintrustConfig(env)).toMatchObject({ remoteOrigin: origin, endpointBaseUrl: `${origin}/otel` });
|
||||
expect(() => evalBraintrustConfig({ ...env, PAPERCLIP_E2E_OTLP_RELAY_URL: "https://different-eval.trycloudflare.com/otel" })).toThrow();
|
||||
expect(buildPaperclipServerEnvironment(env).PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN).toBeUndefined();
|
||||
});
|
||||
@@ -1,13 +1,15 @@
|
||||
import { randomBytes } from "node:crypto";
|
||||
import { createRequire } from "node:module";
|
||||
import { isEvalRemoteOrigin } from "./eval-remote-gateway.js";
|
||||
import type { RunnerApi } from "./api.js";
|
||||
import type { FailureClass, RunnerE2EResult } from "./types.js";
|
||||
|
||||
export const EVAL_OBSERVABILITY_SECRET_NAMES = ["BRAINTRUST_API_KEY"] as const;
|
||||
const uuid = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
|
||||
export interface EvalBraintrustConfig { projectId: string; apiKey: string; endpointBaseUrl?: string }
|
||||
export interface EvalBraintrustConfig { projectId: string; apiKey: string; endpointBaseUrl?: string; remoteOrigin?: string }
|
||||
function evalOtlpEndpoint(config: EvalBraintrustConfig) {
|
||||
if (config.endpointBaseUrl !== undefined && !/^http:\/\/127\.0\.0\.1:[1-9][0-9]{0,4}\/otel$/.test(config.endpointBaseUrl)) {
|
||||
if (config.remoteOrigin !== undefined && !isEvalRemoteOrigin(config.remoteOrigin)) throw new Error("Invalid eval remote OTLP origin");
|
||||
if (config.endpointBaseUrl !== undefined && !(config.remoteOrigin && config.endpointBaseUrl === `${config.remoteOrigin}/otel`) && !/^http:\/\/127\.0\.0\.1:[1-9][0-9]{0,4}\/otel$/.test(config.endpointBaseUrl)) {
|
||||
throw new Error("Eval OTLP relay must use the parent-owned loopback endpoint");
|
||||
}
|
||||
if (config.endpointBaseUrl && Number(new URL(config.endpointBaseUrl).port) > 65535) throw new Error("Invalid eval OTLP relay port");
|
||||
@@ -20,6 +22,7 @@ export function evalBraintrustConfig(env: NodeJS.ProcessEnv): EvalBraintrustConf
|
||||
throw new Error("Eval Braintrust export requires an explicit project UUID and API key");
|
||||
}
|
||||
const config = { projectId, apiKey: env.BRAINTRUST_API_KEY.trim(),
|
||||
...(env.PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN ? { remoteOrigin: env.PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN } : {}),
|
||||
...(env.PAPERCLIP_E2E_OTLP_RELAY_URL ? { endpointBaseUrl: env.PAPERCLIP_E2E_OTLP_RELAY_URL } : {}) };
|
||||
evalOtlpEndpoint(config);
|
||||
return config;
|
||||
|
||||
@@ -21,7 +21,7 @@ it("keeps asynchronous hello probes in setup until the same agent is ready", asy
|
||||
const get = vi.fn().mockResolvedValueOnce({ ...ready, status: "paused", lifecycleState: "verifying", adapterConfig: { private: true } }).mockResolvedValue(ready);
|
||||
let saved: EvalSetupObservation[] = [];
|
||||
try {
|
||||
await finishBudgetedEvalSetup({ executionId: identity.executionId, agentId: "agent", companyId: "company",
|
||||
await finishBudgetedEvalSetup({ executionId: identity.executionId, environmentId: "local", agentId: "agent", companyId: "company",
|
||||
deadlineAt: Date.now() + 5_000, api: { get } as unknown as RunnerApi,
|
||||
observe: async observations => {
|
||||
expect((await readEvalAttemptPhase(f.file, identity)).phase).toBe("setup");
|
||||
@@ -43,7 +43,7 @@ it.each([
|
||||
] as const)("preserves setup attribution and the failed state when readiness is rejected: %s", async (state, reason) => {
|
||||
const f = await fixture(), observe = vi.fn(async () => {}), get = vi.fn().mockResolvedValue(state);
|
||||
try {
|
||||
await expect(finishBudgetedEvalSetup({ executionId: identity.executionId, agentId: "agent", companyId: "company",
|
||||
await expect(finishBudgetedEvalSetup({ executionId: identity.executionId, environmentId: "local", agentId: "agent", companyId: "company",
|
||||
deadlineAt: Date.now() + 5_000, api: { get } as unknown as RunnerApi, observe }, f.env)).rejects.toThrow(reason);
|
||||
expect(get).toHaveBeenCalledOnce();
|
||||
expect(observe).toHaveBeenCalledOnce();
|
||||
@@ -54,7 +54,7 @@ it.each([
|
||||
it("does not start a readiness request after the attempt deadline", async () => {
|
||||
const f = await fixture(), get = vi.fn();
|
||||
try {
|
||||
await expect(finishBudgetedEvalSetup({ executionId: identity.executionId, agentId: "agent", companyId: "company",
|
||||
await expect(finishBudgetedEvalSetup({ executionId: identity.executionId, environmentId: "local", agentId: "agent", companyId: "company",
|
||||
deadlineAt: Date.now() - 1, api: { get } as unknown as RunnerApi, observe: async () => {} }, f.env)).rejects.toThrow("Timed out");
|
||||
expect(get).not.toHaveBeenCalled();
|
||||
expect((await readEvalAttemptPhase(f.file, identity)).phase).toBe("setup");
|
||||
@@ -66,10 +66,32 @@ it("does not advance when saving readiness crosses the deadline", async () => {
|
||||
const deadlineAt = Date.now() + 5_000;
|
||||
const now = vi.spyOn(Date, "now");
|
||||
try {
|
||||
await expect(finishBudgetedEvalSetup({ executionId: identity.executionId, agentId: "agent", companyId: "company",
|
||||
await expect(finishBudgetedEvalSetup({ executionId: identity.executionId, environmentId: "local", agentId: "agent", companyId: "company",
|
||||
deadlineAt, api: { get } as unknown as RunnerApi,
|
||||
observe: async () => { now.mockReturnValue(deadlineAt); } }, f.env)).rejects.toThrow("Timed out saving");
|
||||
expect(get).toHaveBeenCalledOnce();
|
||||
expect((await readEvalAttemptPhase(f.file, identity)).phase).toBe("setup");
|
||||
} finally { now.mockRestore(); await rm(f.directory, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
it.each(["local", "daytona"] as const)("uses the bounded attempt window for remote setup while keeping the local limit: %s", async environmentId => {
|
||||
const f = await fixture();
|
||||
let time = Date.now();
|
||||
const now = vi.spyOn(Date, "now").mockImplementation(() => time);
|
||||
const get = vi.fn().mockResolvedValueOnce({ ...ready, status: "paused", lifecycleState: "verifying" }).mockResolvedValue(ready);
|
||||
let observations = 0;
|
||||
try {
|
||||
const result = finishBudgetedEvalSetup({ executionId: identity.executionId, environmentId, agentId: "agent", companyId: "company",
|
||||
deadlineAt: time + 15 * 60_000, api: { get } as unknown as RunnerApi,
|
||||
observe: async () => { if (++observations === 1) time += 46_000; } }, f.env);
|
||||
if (environmentId === "daytona") {
|
||||
await result;
|
||||
expect(get).toHaveBeenCalledTimes(2);
|
||||
expect((await readEvalAttemptPhase(f.file, identity)).phase).toBe("agent");
|
||||
} else {
|
||||
await expect(result).rejects.toThrow("Timed out");
|
||||
expect(get).toHaveBeenCalledOnce();
|
||||
expect((await readEvalAttemptPhase(f.file, identity)).phase).toBe("setup");
|
||||
}
|
||||
} finally { now.mockRestore(); await rm(f.directory, { recursive: true, force: true }); }
|
||||
});
|
||||
@@ -1,3 +1,4 @@
|
||||
import type { RunnerEnvironmentId } from "./types.js";
|
||||
import type { Agent } from "../../packages/shared/src/types/agent.js";
|
||||
import { pollUntil, type RunnerApi } from "./api.js";
|
||||
import { beginEvalAgentPhase } from "./attempt-phase.js";
|
||||
@@ -7,6 +8,7 @@ export type EvalSetupObservation = SetupState & { observedAt: string };
|
||||
|
||||
export async function finishBudgetedEvalSetup(input: {
|
||||
executionId: string;
|
||||
environmentId: RunnerEnvironmentId;
|
||||
agentId: string;
|
||||
companyId: string;
|
||||
deadlineAt: number;
|
||||
@@ -14,7 +16,7 @@ export async function finishBudgetedEvalSetup(input: {
|
||||
observe: (observations: EvalSetupObservation[]) => Promise<void>;
|
||||
}, env: NodeJS.ProcessEnv = process.env) {
|
||||
if (env.PAPERCLIP_E2E_GATEWAY_BASE_URL) {
|
||||
const deadlineAt = Math.min(input.deadlineAt, Date.now() + 45_000);
|
||||
const deadlineAt = input.environmentId === "daytona" ? input.deadlineAt : Math.min(input.deadlineAt, Date.now() + 45_000);
|
||||
const observations: EvalSetupObservation[] = [];
|
||||
let previous: string | undefined;
|
||||
const scoped = (agent: SetupState) => agent.id === input.agentId && agent.companyId === input.companyId;
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import type { RunnerE2EResult } from "./types.js";
|
||||
import { isBlockedConnectionResult } from "./connection-evidence.js";
|
||||
import { validateOnboardingConnectEvidence } from "./product-journeys/onboarding-connect.js";
|
||||
import { validateOnboardingPreAcceptanceEvidence } from "./product-journeys/onboarding-pre-acceptance.js";
|
||||
const html = (value: unknown) =>
|
||||
String(value ?? "").replace(
|
||||
/[&<>"']/g,
|
||||
@@ -31,6 +32,20 @@ function renderOnboardingConnect(result: RunnerE2EResult): string {
|
||||
${response.truncatedChecks ? "<p>Only the first 32 validation checks were retained.</p>" : ""}`;
|
||||
}).join("")}</details>`;
|
||||
}
|
||||
function renderOnboardingPreAcceptance(result: RunnerE2EResult): string {
|
||||
const receipt = result.productJourney?.onboardingPreAcceptance;
|
||||
if (receipt === undefined) return "";
|
||||
try { validateOnboardingPreAcceptanceEvidence(receipt); }
|
||||
catch { return "<p>Onboarding pre-acceptance evidence unavailable: invalid receipt.</p>"; }
|
||||
return `<details class="case-context"><summary>Onboarding pre-acceptance evidence</summary>
|
||||
<p>Checkpoint: <code>${html(receipt.checkpoint)}</code></p>
|
||||
<p>Observed guard violations:</p><ul>${receipt.violations.map(code => `<li><code>${html(code)}</code></li>`).join("")}</ul>
|
||||
<dl>${[
|
||||
["Tasks", receipt.taskCount], ["Attachments", receipt.attachmentCount], ["Documents", receipt.documentCount],
|
||||
["Non-planning documents", receipt.nonPlanningDocumentCount], ["First task matches canonical parent", receipt.firstTaskMatchesCanonicalParent],
|
||||
["First task status", receipt.firstTaskStatus], ["Canonical parent status", receipt.canonicalParentStatus],
|
||||
].map(([label, value]) => `<div><dt>${html(label)}</dt><dd>${html(value)}</dd></div>`).join("")}</dl></details>`;
|
||||
}
|
||||
/** An interrupted first-task journey is not evidence of failed downstream behavior. */
|
||||
export function isIncompleteFirstTaskResult(
|
||||
result: RunnerE2EResult,
|
||||
@@ -93,6 +108,7 @@ export function renderCaseOutcome(
|
||||
${result.productJourney?.operations ? `<p>Eval operations: ${html(result.productJourney.operations.campaignId)} · ${result.productJourney.operations.requestCount} requests · model cost $${html(result.productJourney.operations.modelCostUsd ?? "unknown")} (setup $${html(result.productJourney.operations.setupCostUsd ?? "unknown")}, counted once). Feedback: ${html(result.productJourney.operations.feedback.collection)} · ${result.productJourney.operations.feedback.complaints} complaints · ${result.productJourney.operations.feedback.suggestions} suggestions.</p>` : ""}
|
||||
${result.productJourney?.operations?.traceIds.length ? `<details><summary>Trace correlation IDs</summary><ul>${result.productJourney.operations.traceIds.map(id => `<li><code>${html(id)}</code></li>`).join("")}</ul></details>` : ""}
|
||||
${renderOnboardingConnect(result)}
|
||||
${renderOnboardingPreAcceptance(result)}
|
||||
${result.providerConnection ? `<p>Target: ${html(result.providerConnection.target.origin)} · ${html(result.providerConnection.target.commit ?? "revision unavailable")} · ${result.providerConnection.assisted ? "Assisted login" : "Unassisted"} · ${html(result.providerConnection.authFreshness)} browser · ${html(result.providerConnection.entry)} / ${html(result.providerConnection.method)} · stopped at ${html(result.providerConnection.phase)}</p>` : ""}
|
||||
${failures.length ? `<p>Failed checks:</p><ul>${failures.map((c) => `<li><strong>${html(c.id)}</strong> — ${html(c.detail)}</li>`).join("")}</ul>` : checks.length ? `<p>No behavioral matcher failed.${notReached.length ? " The journey is incomplete; unexercised checks are not passes." : failed ? " The overall failure came from a separate run, cleanup, or evidence check." : ""}</p>` : ""}
|
||||
${notReached.length ? `<p>Not reached:</p><ul>${notReached.map((c) => `<li><strong>${html(c.id)}</strong> — ${html(c.notReached)}</li>`).join("")}</ul>` : ""}
|
||||
|
||||
@@ -66,6 +66,8 @@ export const DAYTONA_IMAGE_INPUT_PATHS = [
|
||||
"packages/paperclip-runner/test/fixtures/copilot-profile-v57-identity.json",
|
||||
"packages/paperclip-runner/test/fixtures/copilot-profile-v58-identity.json",
|
||||
"packages/paperclip-runner/test/fixtures/copilot-profile-v61-identity.json",
|
||||
"packages/paperclip-runner/test/fixtures/copilot-profile-v65-identity.json",
|
||||
"packages/paperclip-runner/test/fixtures/copilot-profile-v66-identity.json",
|
||||
"packages/paperclip-runner/scripts/materialize-cursor-distribution.mjs",
|
||||
"packages/paperclip-runner/scripts/cursor-runtime-patch.mjs",
|
||||
"packages/paperclip-runner/scripts/build-verified-provider-entrypoints.mjs",
|
||||
|
||||
@@ -3,6 +3,7 @@ import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterEach, expect, it, vi } from "vitest";
|
||||
import { openEvalCostLedger, readEvalCostLedger } from "./cost-ledger.js";
|
||||
import { startEvalRemoteBridge } from "./eval-remote-gateway.js";
|
||||
import { forwardEvalRequest, startEvalGateway, type EvalGatewayReceipt, type EvalGatewayRoute } from "./eval-gateway.js";
|
||||
|
||||
const cleanups: Array<() => Promise<void>> = [];
|
||||
@@ -181,3 +182,19 @@ it.each(["persist", "observe", "finish"] as const)("records a disconnect during
|
||||
expect(f.ledger.snapshot()).toMatchObject({ knownUsd: "0.000070000", heldUsd: "0.000000000", unknownRequests: 0 });
|
||||
expect(transport).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
it("retains terminal usage when a remote bridge client disconnects during shutdown", async () => {
|
||||
const f = await fixture(), started = deferred<void>(), release = deferred<void>();
|
||||
const transport = vi.fn<typeof fetch>(async () => { started.resolve(); await release.promise; return finalResponse("openai"); });
|
||||
const gateway = await startEvalGateway({ ...f, transport }); cleanups.push(() => gateway.close());
|
||||
const bridge = await startEvalRemoteBridge({ gateway, deadlineAt: f.deadlineAt }); cleanups.push(() => bridge.close());
|
||||
const client = new AbortController();
|
||||
const result = fetch(bridge.baseUrl + "/responses", { method: "POST", signal: client.signal,
|
||||
headers: { authorization: `Bearer ${gateway.token}` }, body: JSON.stringify({ model: "fixture", input: "hello" }) }).catch(error => error);
|
||||
await started.promise; client.abort(); await bridge.close();
|
||||
const closing = gateway.close(); release.resolve(); await closing;
|
||||
expect(await result).not.toBeInstanceOf(Response);
|
||||
expect(transport).toHaveBeenCalledOnce();
|
||||
expect(f.receipts.at(-1)).toMatchObject({ clientDisconnected: true, final: true, usageComplete: true, costUsd: "0.000070000" });
|
||||
expect(f.ledger.snapshot()).toMatchObject({ knownUsd: "0.000070000", heldUsd: "0.000000000", unknownRequests: 0, inflight: 0 });
|
||||
});
|
||||
@@ -0,0 +1,101 @@
|
||||
import { mkdtemp, readFile, writeFile, rm, readdir } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { createServer } from "node:http";
|
||||
import { once } from "node:events";
|
||||
import { expect, it, vi } from "vitest";
|
||||
import { isEvalRemoteOrigin, startEvalRemoteBridge, startEvalRemoteGateway } from "./eval-remote-gateway.js";
|
||||
|
||||
it("forwards only authenticated model/OTLP routes and closes owned exposure", async () => {
|
||||
const calls: Array<{ url: string; auth?: string; body: string }> = [];
|
||||
const upstream = createServer(async (request, response) => {
|
||||
let body = ""; for await (const chunk of request) body += chunk;
|
||||
calls.push({ url: request.url!, auth: request.headers.authorization, body });
|
||||
response.writeHead(200, { "content-type": "text/event-stream" }); response.end("data: synthetic\n\n");
|
||||
});
|
||||
upstream.listen(0, "127.0.0.1"); await once(upstream, "listening");
|
||||
const address = upstream.address(); if (!address || typeof address === "string") throw new Error("Fixture did not bind");
|
||||
const url = `http://127.0.0.1:${address.port}`, modelToken = "a".repeat(64), otlpToken = "b".repeat(64);
|
||||
const bridge = await startEvalRemoteBridge({ gateway: { baseUrl: url, token: modelToken },
|
||||
observability: { baseUrl: url + "/otel", token: otlpToken }, deadlineAt: Date.now() + 60_000 });
|
||||
try {
|
||||
expect(await (await fetch(bridge.baseUrl + "/health")).json()).toEqual({ nonce: bridge.nonce });
|
||||
for (const path of ["/v1/responses", "/v1/messages?beta=true", "/otel/v1/traces", "/otel/v1/logs"]) {
|
||||
expect((await fetch(bridge.baseUrl + path, { method: "POST" })).status).toBe(401);
|
||||
const token = path.startsWith("/otel") ? otlpToken : modelToken;
|
||||
const response = await fetch(bridge.baseUrl + path, { method: "POST", headers: { authorization: `Bearer ${token}`, "content-type": "application/json" }, body: "{}" });
|
||||
expect(response.status).toBe(200); expect(await response.text()).toBe("data: synthetic\n\n");
|
||||
}
|
||||
expect(calls).toHaveLength(4);
|
||||
expect(calls.map(call => call.url)).toEqual(["/v1/responses", "/v1/messages?beta=true", "/otel/v1/traces", "/otel/v1/logs"]);
|
||||
expect(calls.map(call => call.auth)).toEqual([`Bearer ${modelToken}`, `Bearer ${modelToken}`, `Bearer ${otlpToken}`, `Bearer ${otlpToken}`]);
|
||||
expect((await fetch(bridge.baseUrl + "/admin", { method: "POST", headers: { authorization: `Bearer ${modelToken}` } })).status).toBe(404);
|
||||
expect((await fetch(bridge.baseUrl + "/otel/v1/traces", { method: "POST", headers: { authorization: `Bearer ${modelToken}` } })).status).toBe(401);
|
||||
await bridge.close(); await expect(fetch(bridge.baseUrl + "/health")).rejects.toThrow();
|
||||
} finally { await bridge.close(); upstream.closeAllConnections(); await new Promise<void>(resolve => upstream.close(() => resolve())); }
|
||||
});
|
||||
|
||||
it("admits only HTTPS quick-tunnel origins and bounded loopback destinations", async () => {
|
||||
expect(isEvalRemoteOrigin("https://synthetic-fixture.trycloudflare.com")).toBe(true);
|
||||
for (const value of ["https://foreign.invalid", "http://synthetic.trycloudflare.com", "https://synthetic.trycloudflare.com/path", "https://synthetic.trycloudflare.com?token=private"]) expect(isEvalRemoteOrigin(value)).toBe(false);
|
||||
await expect(startEvalRemoteBridge({ gateway: { baseUrl: "https://foreign.invalid", token: "a".repeat(64) }, deadlineAt: Date.now() + 60_000 })).rejects.toThrow("policy");
|
||||
});
|
||||
|
||||
|
||||
it("fails before remote admission when the tunnel executable is unavailable", async () => {
|
||||
await expect(startEvalRemoteGateway({ gateway: { baseUrl: "http://127.0.0.1:1", token: "a".repeat(64) },
|
||||
deadlineAt: Date.now() + 5000 }, { PAPERCLIP_EVAL_CLOUDFLARED: "/missing/synthetic-cloudflared", PATH: "" }))
|
||||
.rejects.toThrow("Remote eval tunnel executable unavailable");
|
||||
});
|
||||
|
||||
it("closes remote exposure at its original campaign deadline", async () => {
|
||||
const bridge = await startEvalRemoteBridge({ gateway: { baseUrl: "http://127.0.0.1:1", token: "a".repeat(64) }, deadlineAt: Date.now() + 50 });
|
||||
try {
|
||||
await new Promise(resolve => setTimeout(resolve, 100));
|
||||
await expect(fetch(bridge.baseUrl + "/health")).rejects.toThrow();
|
||||
} finally { await bridge.close(); }
|
||||
});
|
||||
|
||||
|
||||
it.each(["explicit", "empty-override"])("waits for registration and the correct public nonce, then terminates the owned tunnel: %s", async mode => {
|
||||
const root = await mkdtemp(path.join(tmpdir(), "eval-fake-tunnel-"));
|
||||
const executable = path.join(root, "cloudflared"), state = path.join(root, "state.json"), stopped = path.join(root, "stopped");
|
||||
const origin = "https://synthetic-owned.trycloudflare.com";
|
||||
const script = `#!${process.execPath}
|
||||
const fs = require("node:fs");
|
||||
fs.writeFileSync(${JSON.stringify(state)}, JSON.stringify({baseUrl:process.argv.at(-1),pid:process.pid,env:process.env}));
|
||||
process.stderr.write("${origin}\\n");
|
||||
setTimeout(() => process.stderr.write("Registered tunnel connection\\n"), 20);
|
||||
process.on("SIGTERM", () => { fs.writeFileSync(${JSON.stringify(stopped)}, "stopped"); process.exit(0); });
|
||||
setInterval(() => {}, 1000);
|
||||
`;
|
||||
await writeFile(executable, script, { mode: 0o700 });
|
||||
const realFetch = globalThis.fetch; let healthChecks = 0;
|
||||
const fetchSpy = vi.spyOn(globalThis, "fetch").mockImplementation(async url => {
|
||||
expect(String(url)).toBe(origin + "/health");
|
||||
if (++healthChecks === 1) return new Response(JSON.stringify({ nonce: "foreign" }), { status: 200 });
|
||||
const { baseUrl } = JSON.parse(await readFile(state, "utf8"));
|
||||
return realFetch(baseUrl + "/health");
|
||||
});
|
||||
let remote: Awaited<ReturnType<typeof startEvalRemoteGateway>> | undefined;
|
||||
try {
|
||||
remote = await startEvalRemoteGateway({ gateway: { baseUrl: "http://127.0.0.1:1", token: "a".repeat(64) },
|
||||
deadlineAt: Date.now() + 10_000 }, { PAPERCLIP_EVAL_CLOUDFLARED: mode === "explicit" ? executable : "", PATH: root, OPENAI_API_KEY: "must-not-leave" });
|
||||
expect(remote.baseUrl).toBe(origin); expect(healthChecks).toBe(2);
|
||||
const { baseUrl, pid, env } = JSON.parse(await readFile(state, "utf8"));
|
||||
expect(env.OPENAI_API_KEY).toBeUndefined(); expect(env.HOME).not.toBe(process.env.HOME);
|
||||
await remote.close();
|
||||
expect(await readFile(stopped, "utf8")).toBe("stopped");
|
||||
expect(() => process.kill(pid, 0)).toThrow();
|
||||
await expect(realFetch(baseUrl + "/health")).rejects.toThrow();
|
||||
await expect(readFile(path.join(env.HOME, "anything"))).rejects.toThrow();
|
||||
} finally { fetchSpy.mockRestore(); await remote?.close(); await rm(root, { recursive: true, force: true }); }
|
||||
}, 15_000);
|
||||
|
||||
|
||||
it("cleans the private tunnel home after synchronous spawn rejection", async () => {
|
||||
const before = new Set((await readdir(tmpdir())).filter(name => name.startsWith("eval-cloudflare-")));
|
||||
await expect(startEvalRemoteGateway({ gateway: { baseUrl: "http://127.0.0.1:1", token: "a".repeat(64) },
|
||||
deadlineAt: Date.now() + 60_000 }, { PAPERCLIP_EVAL_CLOUDFLARED: "invalid\0executable", PATH: "" })).rejects.toThrow();
|
||||
expect((await readdir(tmpdir())).filter(name => name.startsWith("eval-cloudflare-") && !before.has(name))).toEqual([]);
|
||||
});
|
||||
@@ -0,0 +1,119 @@
|
||||
import { spawn } from "node:child_process";
|
||||
import { randomBytes, timingSafeEqual } from "node:crypto";
|
||||
import { once } from "node:events";
|
||||
import { mkdtemp, rm } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { createServer, request as httpRequest } from "node:http";
|
||||
|
||||
export const isEvalRemoteOrigin = (value: string) => /^https:\/\/[a-z0-9-]+\.trycloudflare\.com$/.test(value);
|
||||
|
||||
export async function startEvalRemoteBridge(input: {
|
||||
gateway: { baseUrl: string; token: string };
|
||||
observability?: { baseUrl: string; token: string };
|
||||
deadlineAt: number;
|
||||
}) {
|
||||
if (!Number.isFinite(input.deadlineAt) || input.deadlineAt <= Date.now() || input.deadlineAt > Date.now() + 4 * 3_600_000
|
||||
|| !/^http:\/\/127\.0\.0\.1:[1-9][0-9]{0,4}$/.test(input.gateway.baseUrl)
|
||||
|| input.observability && !/^http:\/\/127\.0\.0\.1:[1-9][0-9]{0,4}\/otel$/.test(input.observability.baseUrl)
|
||||
|| !/^[a-f0-9]{64}$/.test(input.gateway.token)
|
||||
|| input.observability && !/^[a-f0-9]{64}$/.test(input.observability.token)) throw new Error("Invalid remote eval bridge policy");
|
||||
try { new URL(input.gateway.baseUrl); if (input.observability) new URL(input.observability.baseUrl); }
|
||||
catch { throw new Error("Invalid remote eval bridge policy"); }
|
||||
const nonce = randomBytes(32).toString("hex");
|
||||
let closed = false;
|
||||
const server = createServer((incoming, outgoing) => {
|
||||
if (closed || Date.now() >= input.deadlineAt) { outgoing.writeHead(503).end(); return; }
|
||||
if (incoming.method === "GET" && incoming.url === "/health") {
|
||||
outgoing.writeHead(200, { "content-type": "application/json", "cache-control": "no-store" }).end(JSON.stringify({ nonce })); return;
|
||||
}
|
||||
const model = ["/responses", "/v1/responses", "/messages", "/messages?beta=true", "/v1/messages", "/v1/messages?beta=true"].includes(incoming.url ?? "");
|
||||
const otlp = ["/otel/v1/traces", "/otel/v1/logs"].includes(incoming.url ?? "");
|
||||
const destination = model ? input.gateway : otlp ? input.observability : undefined;
|
||||
if (incoming.method !== "POST" || !destination) { outgoing.writeHead(404).end(); return; }
|
||||
const auth = String(incoming.headers.authorization ?? "").replace(/^Bearer /, "") || String(incoming.headers["x-api-key"] ?? "");
|
||||
const presented = Buffer.from(auth), expected = Buffer.from(destination.token);
|
||||
if (presented.length !== expected.length || !timingSafeEqual(presented, expected)) { outgoing.writeHead(401).end(); return; }
|
||||
const suffix = otlp ? incoming.url!.slice("/otel".length) : incoming.url!;
|
||||
const headers: Record<string, string> = { authorization: `Bearer ${destination.token}` };
|
||||
for (const name of ["content-type", "content-encoding", "anthropic-beta", "anthropic-version", "x-bt-parent"]) {
|
||||
const value = incoming.headers[name]; if (typeof value === "string") headers[name] = value;
|
||||
}
|
||||
const upstream = httpRequest(destination.baseUrl + suffix, { method: "POST", headers }, response => {
|
||||
if (outgoing.destroyed) { response.destroy(); return; }
|
||||
outgoing.writeHead(response.statusCode ?? 502, response.headers); response.pipe(outgoing);
|
||||
response.on("error", () => outgoing.destroy());
|
||||
});
|
||||
upstream.on("error", () => { if (!outgoing.headersSent) outgoing.writeHead(502).end(); else outgoing.destroy(); });
|
||||
incoming.on("aborted", () => upstream.destroy());
|
||||
incoming.on("error", () => upstream.destroy());
|
||||
outgoing.on("error", () => upstream.destroy());
|
||||
outgoing.on("close", () => { if (!outgoing.writableEnded) upstream.destroy(); });
|
||||
incoming.pipe(upstream);
|
||||
});
|
||||
server.requestTimeout = 15_000; server.headersTimeout = 10_000;
|
||||
server.listen(0, "127.0.0.1"); await once(server, "listening");
|
||||
const address = server.address();
|
||||
if (!address || typeof address === "string") throw new Error("Remote eval bridge did not bind loopback");
|
||||
let timer: ReturnType<typeof setTimeout>;
|
||||
let closing: Promise<void> | undefined;
|
||||
const close = () => closing ??= (async () => {
|
||||
closed = true; clearTimeout(timer);
|
||||
server.closeAllConnections(); await new Promise<void>(resolve => server.close(() => resolve()));
|
||||
})();
|
||||
timer = setTimeout(() => { void close().catch(() => {}); }, input.deadlineAt - Date.now()); timer.unref();
|
||||
return { baseUrl: `http://127.0.0.1:${address.port}`, nonce, close };
|
||||
}
|
||||
|
||||
export async function startEvalRemoteGateway(input: Parameters<typeof startEvalRemoteBridge>[0], env: NodeJS.ProcessEnv) {
|
||||
const bridge = await startEvalRemoteBridge(input);
|
||||
let home: string;
|
||||
try { home = await mkdtemp(path.join(tmpdir(), "eval-cloudflare-")); }
|
||||
catch (error) { await bridge.close(); throw error; }
|
||||
let tunnel: ReturnType<typeof spawn> | undefined;
|
||||
let closing: Promise<void> | undefined, deadlineTimer: ReturnType<typeof setTimeout> | undefined;
|
||||
const close = () => closing ??= (async () => {
|
||||
clearTimeout(deadlineTimer);
|
||||
const errors: unknown[] = [];
|
||||
try { await bridge.close(); } catch (error) { errors.push(error); }
|
||||
try {
|
||||
if (tunnel?.pid && tunnel.exitCode === null && tunnel.signalCode === null) {
|
||||
const exited = once(tunnel, "exit").catch(() => {});
|
||||
tunnel.kill("SIGTERM"); const timer = setTimeout(() => tunnel?.kill("SIGKILL"), 5000);
|
||||
try { await exited; } finally { clearTimeout(timer); }
|
||||
}
|
||||
} catch (error) { errors.push(error); }
|
||||
try { await rm(home, { recursive: true, force: true }); } catch (error) { errors.push(error); }
|
||||
if (errors.length) throw new AggregateError(errors, "Remote eval exposure cleanup failed");
|
||||
})();
|
||||
try {
|
||||
tunnel = spawn(env.PAPERCLIP_EVAL_CLOUDFLARED?.trim() || "cloudflared",
|
||||
["tunnel", "--no-autoupdate", "--protocol", "http2", "--url", bridge.baseUrl],
|
||||
{ stdio: ["ignore", "pipe", "pipe"], env: { PATH: env.PATH, HOME: home, TMPDIR: home } });
|
||||
deadlineTimer = setTimeout(() => { void close().catch(() => {}); }, input.deadlineAt - Date.now()); deadlineTimer.unref();
|
||||
const child = tunnel;
|
||||
const origin = await new Promise<string>((resolve, reject) => {
|
||||
let output = "";
|
||||
const timer = setTimeout(() => finish(new Error("Remote eval tunnel startup timed out")), Math.min(120_000, input.deadlineAt - Date.now()));
|
||||
const finish = (error?: Error, value?: string) => { clearTimeout(timer); if (error) reject(error); else resolve(value!); };
|
||||
child.once("error", () => finish(new Error("Remote eval tunnel executable unavailable")));
|
||||
child.once("exit", () => finish(new Error("Remote eval tunnel exited")));
|
||||
const read = (chunk: Buffer) => {
|
||||
output = (output + chunk.toString()).slice(-32768);
|
||||
const value = output.match(/https:\/\/[a-z0-9-]+\.trycloudflare\.com/)?.[0];
|
||||
if (value && output.includes("Registered tunnel connection")) finish(undefined, value);
|
||||
};
|
||||
child.stdout!.on("data", read); child.stderr!.on("data", read);
|
||||
});
|
||||
const readyBy = Math.min(Date.now() + 120_000, input.deadlineAt);
|
||||
while (Date.now() < readyBy) {
|
||||
try {
|
||||
const response = await fetch(origin + "/health", { redirect: "error", signal: AbortSignal.timeout(Math.min(5000, readyBy - Date.now())) });
|
||||
if (response.ok && (await response.json() as { nonce?: string }).nonce === bridge.nonce) return { baseUrl: origin, close };
|
||||
await response.body?.cancel();
|
||||
} catch {}
|
||||
await new Promise(resolve => setTimeout(resolve, 1000));
|
||||
}
|
||||
throw new Error("Remote eval tunnel not publicly reachable before admission");
|
||||
} catch (error) { await close(); throw error; }
|
||||
}
|
||||
@@ -5,6 +5,7 @@ import { installedDaytonaPluginKeys } from "./installed-daytona-plugin.js";
|
||||
import { QUALIFIED_ACPX_PROFILES } from "../../packages/paperclip-runner/src/drivers/acpx/qualified-profiles.js";
|
||||
import { assertInstalledCliSelection, installedCliKeys, usesInstalledCli } from "./installed-cli.js";
|
||||
import path from "node:path";
|
||||
import { isEvalRemoteOrigin } from "./eval-remote-gateway.js";
|
||||
import { chatNeedsApiTools, isManagedHiringCase } from "./chat-cases.js";
|
||||
import { CREDENTIAL_NAMES } from "./types.js";
|
||||
import type { MatrixExecution } from "./types.js";
|
||||
@@ -17,6 +18,7 @@ const AMBIENT_PAPERCLIP_CREDENTIAL_KEYS = [
|
||||
"PAPERCLIP_E2E_GATEWAY_TOKEN",
|
||||
"PAPERCLIP_E2E_GATEWAY_PHASE_PATH",
|
||||
"PAPERCLIP_E2E_OTLP_RELAY_URL",
|
||||
"PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN",
|
||||
"PAPERCLIP_SETUP_TOKEN",
|
||||
"PAPERCLIP_SECRETS_MASTER_KEY",
|
||||
"PAPERCLIP_SECRETS_MASTER_KEY_FILE",
|
||||
@@ -51,6 +53,8 @@ export function applyEvalGatewayEnvironment(env: NodeJS.ProcessEnv, credential:
|
||||
|| PROVIDER_SECRET_KEY.test(name) && /KEY|TOKEN|SECRET|PASSWORD/.test(name)) delete env[name];
|
||||
}
|
||||
delete env.PAPERCLIP_E2E_OTLP_RELAY_URL;
|
||||
delete env.PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN;
|
||||
if (isEvalRemoteOrigin(gateway.baseUrl)) env.PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN = gateway.baseUrl;
|
||||
env[credential] = gateway.token;
|
||||
env.PAPERCLIP_E2E_GATEWAY_BASE_URL = gateway.baseUrl;
|
||||
env.PAPERCLIP_E2E_GATEWAY_TOKEN = gateway.token;
|
||||
@@ -113,13 +117,13 @@ export function resolvePaperclipRemoteRunnerBinaryForHarness(
|
||||
runnerBinary: string | undefined,
|
||||
configuredPath = process.env.PAPERCLIP_RUNNER_REMOTE_BINARY_PATH,
|
||||
platform: NodeJS.Platform = process.platform,
|
||||
architecture: NodeJS.Architecture = process.arch,
|
||||
): string | undefined {
|
||||
if (configuredPath?.trim()) return configuredPath;
|
||||
if (!runnerBinary) return undefined;
|
||||
// Daytona runs Linux. A default debug binary built by a macOS developer is
|
||||
// Mach-O and cannot be staged into that sandbox. Leave the remote override
|
||||
// unset so the pinned Daytona image's verified runnerd is discovered instead.
|
||||
if (platform !== "linux") return undefined;
|
||||
// The pinned Daytona runtime is Linux/x64. Other controller binaries cannot
|
||||
// run there; discover its verified runnerd unless an explicit cross-build is set.
|
||||
if (platform !== "linux" || architecture !== "x64") return undefined;
|
||||
return executions.some(
|
||||
(execution) =>
|
||||
execution.profile.generation === "native" &&
|
||||
@@ -147,6 +151,7 @@ export function buildRunnerE2EProcessEnvironment(
|
||||
else scrubChatPrivateEnvironment(result, []);
|
||||
delete result.PAPERCLIP_E2E_GATEWAY_PHASE_PATH;
|
||||
delete result.PAPERCLIP_E2E_OTLP_RELAY_URL;
|
||||
delete result.PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN;
|
||||
// Announcements are unrelated to the scenarios and obscure screenshot evidence.
|
||||
result.PAPERCLIP_ANNOUNCEMENTS_ENABLED = "false";
|
||||
delete result.OPENCODE_ALLOW_ALL_MODELS;
|
||||
|
||||
@@ -631,6 +631,9 @@ async function runAttempt(input: {
|
||||
};
|
||||
if (gateway) {
|
||||
applyEvalGatewayEnvironment(childEnv, execution.profile.credential, gateway);
|
||||
// The fixture/server controller needs Daytona provisioning access. The
|
||||
// remote provider still receives only the scoped model and OTLP tokens.
|
||||
if (execution.environment.id === "daytona") childEnv.DAYTONA_API_KEY = process.env.DAYTONA_API_KEY;
|
||||
}
|
||||
// The database URLs are stripped here and again at the Playwright web-server
|
||||
// boundary so a developer's shell can never redirect this paid test.
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { CHAT_SETUP_HEALTH_SUITE } from "./chat-setup-health.js";
|
||||
import { isEvalRemoteOrigin } from "./eval-remote-gateway.js";
|
||||
import type { AiConnectionBinding } from "../../packages/shared/src/ai-connections.js";
|
||||
import { verifyInstalledDaytonaPlugin } from "./installed-daytona-plugin.js";
|
||||
import { NATIVE_COMPLETION_BUDGET_CENTS } from "./native-completion-defaults.js";
|
||||
@@ -283,8 +284,9 @@ export async function setupLiveFixtures(input: {
|
||||
const gatewayUrl = process.env.PAPERCLIP_E2E_GATEWAY_BASE_URL;
|
||||
const gatewayToken = process.env.PAPERCLIP_E2E_GATEWAY_TOKEN;
|
||||
const budgetedGateway = Boolean(gatewayUrl || gatewayToken);
|
||||
if (budgetedGateway && (!gatewayUrl || !gatewayToken || !/^http:\/\/127\.0\.0\.1:[0-9]+$/.test(gatewayUrl)
|
||||
|| execution.environment.id !== "local" || !["core-compatibility", "feedback-persistence"].includes(execution.suite.id))) throw new Error("Unqualified fixture gateway route");
|
||||
if (budgetedGateway && (!gatewayUrl || !gatewayToken || !(execution.environment.id === "local" ? /^http:\/\/127\.0\.0\.1:[0-9]+$/.test(gatewayUrl)
|
||||
: isEvalRemoteOrigin(gatewayUrl) && gatewayUrl === process.env.PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN)
|
||||
|| !["local", "daytona"].includes(execution.environment.id) || !["core-compatibility", "feedback-persistence"].includes(execution.suite.id))) throw new Error("Unqualified fixture gateway route");
|
||||
if (managedHiring || savedCopilot || budgetedGateway) {
|
||||
registry.register<ManagedAccountFixture>({
|
||||
id: "ai-connection",
|
||||
@@ -386,6 +388,7 @@ export async function setupLiveFixtures(input: {
|
||||
if (managedHiring || savedCopilot || budgetedGateway) {
|
||||
const account = value<ManagedAccountFixture>(resolved, "ai-connection");
|
||||
const config = agent.adapterConfig as Record<string, unknown>;
|
||||
if (budgetedGateway && execution.environment.id === "daytona") config.helloProbeTimeoutSec = 300;
|
||||
delete config.env;
|
||||
agent.runtimeConfig = {
|
||||
...(agent.runtimeConfig as Record<string, unknown>),
|
||||
|
||||
@@ -3,8 +3,10 @@ import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { expect, it } from "vitest";
|
||||
import { EvalSpendState, openEvalCostLedger, readEvalCostLedger, type EvalSpendLimits } from "./cost-ledger.js";
|
||||
import { EVAL_OVERHEAD_COMPONENTS, scheduledOperatingOverhead, validateOperatingOverhead, type EvalOperatingOverhead } from "./operating-overhead.js";
|
||||
import { readEvalOperationsReport, renderEvalOperationsReport } from "./operations-report.js";
|
||||
import { EVAL_OVERHEAD_COMPONENTS, operatingReservationBound, scheduledOperatingOverhead, validateOperatingOverhead, type EvalOperatingOverhead } from "./operating-overhead.js";
|
||||
import { evalAccountingStatus, evalReadinessStatus, readEvalOperationsReport, renderEvalOperationsReport } from "./operations-report.js";
|
||||
|
||||
import { costBoundConfigurationSha256 } from "./cost-bound-evidence.js";
|
||||
|
||||
const now = Date.parse("2026-10-21T00:00:00Z");
|
||||
const allowance = (): EvalOperatingOverhead => ({ version: 1, reference: "fixture-only", runner: "aws-fleet",
|
||||
@@ -117,3 +119,51 @@ it("stops new admission after allowance expiry while preserving late invoice rec
|
||||
expect(() => state.validate({ type: "observe", observation: { observationId: "late-invoice", requestId: "overhead:ci_compute",
|
||||
source: "invoice", provider: "aws", currency: "USD", route: "infrastructure", classification: "reconciled", costUsd: "0.5", final: true } }, expiredAt)).not.toThrow();
|
||||
});
|
||||
|
||||
|
||||
it("separates model readiness from unresolved overhead without releasing holds or masking model gaps", async () => {
|
||||
const directory = await mkdtemp(path.join(tmpdir(), "eval-model-readiness-"));
|
||||
const file = path.join(directory, "ledger.jsonl");
|
||||
const overhead = { ...currentAllowance(), accountingReadiness: "model_requests" as const };
|
||||
const ledger = await openEvalCostLedger(file, { campaignId: "model-readiness", maxUsd: "20",
|
||||
deadlineAt: "2099-01-01T00:00:00Z", maxRequests: 10, maxConcurrent: 1, maxAttemptsPerExecution: 1, overhead });
|
||||
try {
|
||||
expect(evalReadinessStatus(await readEvalOperationsReport(file))).toBe("blocked");
|
||||
await ledger.admitAttempt(modelIdentity.executionId, modelIdentity.attemptId);
|
||||
await ledger.reserve("model", modelIdentity, "2");
|
||||
expect(evalReadinessStatus(await readEvalOperationsReport(file))).toBe("blocked");
|
||||
await ledger.finish("model", "cancelled");
|
||||
expect(evalReadinessStatus(await readEvalOperationsReport(file))).toBe("blocked");
|
||||
await ledger.observe({ observationId: "model-final", requestId: "model", source: "provider", provider: "anthropic",
|
||||
currency: "USD", route: "direct", classification: "reported", costUsd: "0.5", final: true });
|
||||
const report = await readEvalOperationsReport(file);
|
||||
expect(evalAccountingStatus(report)).toBe("blocked");
|
||||
expect(evalReadinessStatus(report)).toBe("receipts_complete");
|
||||
expect(evalReadinessStatus({ ...report, accountingReadiness: undefined } as never)).toBe("blocked");
|
||||
expect(report).toMatchObject({ accountingReadiness: "model_requests", heldUsd: "4.000000000",
|
||||
committedUsd: "4.500000000", modelAccounting: { requests: 1, knownUsd: "0.500000000", unresolvedRequests: 0 } });
|
||||
expect(renderEvalOperationsReport(report)).toContain("Operating costs remain separately unreconciled");
|
||||
await ledger.observe({ observationId: "overrun", requestId: "overhead:ci_compute", source: "invoice", provider: "aws",
|
||||
currency: "USD", route: "infrastructure", classification: "reconciled", costUsd: "21", final: true });
|
||||
expect(evalReadinessStatus(await readEvalOperationsReport(file))).toBe("blocked");
|
||||
} finally { await ledger.close(); await rm(directory, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
it("rejects unknown readiness policies instead of silently loosening accounting", () => {
|
||||
expect(() => validateOperatingOverhead({ ...allowance(), accountingReadiness: "ignore_costs" } as never, "20"))
|
||||
.toThrow("Invalid accounting readiness policy");
|
||||
});
|
||||
|
||||
it("binds optional readiness policy to reviewed operating evidence without changing legacy hash recipes", () => {
|
||||
for (const accountingReadiness of [undefined, "model_requests"] as const) {
|
||||
const value = { ...allowance(), ...(accountingReadiness ? { accountingReadiness } : {}) };
|
||||
const component = value.components[0]!;
|
||||
const calculation = { reference: value.reference, runner: value.runner, verifiedAt: value.verifiedAt,
|
||||
expiresAt: value.expiresAt, component: { ...component }, ...(accountingReadiness ? { accountingReadiness } : {}) };
|
||||
component.provisionalBound = { version: 1, scope: "full_lifecycle", evidenceSha256: "a".repeat(64),
|
||||
configurationSha256: costBoundConfigurationSha256(calculation) };
|
||||
expect(operatingReservationBound(value, component)?.calculationSha256).toBe(component.provisionalBound.configurationSha256);
|
||||
const changed = { ...value, accountingReadiness: accountingReadiness ? undefined : "model_requests" as const };
|
||||
expect(() => operatingReservationBound(changed, component)).toThrow("Reviewed cost bound configuration changed");
|
||||
}
|
||||
});
|
||||
@@ -4,6 +4,7 @@ import { usdToUnits } from "../../packages/shared/src/money.js";
|
||||
export const EVAL_OVERHEAD_COMPONENTS = ["ci_compute", "artifact_storage", "observability", "shared_subscription"] as const;
|
||||
export interface EvalOperatingOverhead {
|
||||
version: 1;
|
||||
accountingReadiness?: "model_requests";
|
||||
reference: string;
|
||||
runner: "github-hosted" | "aws-fleet";
|
||||
verifiedAt: string;
|
||||
@@ -20,13 +21,15 @@ const reference = (value: unknown) => typeof value === "string" && /^[A-Za-z0-9]
|
||||
export function operatingReservationBound(allowance: EvalOperatingOverhead, component: EvalOperatingOverhead["components"][number]) {
|
||||
if (component.provisionalBound === undefined) return undefined;
|
||||
return reservationBound("operating_allowance", component.provisionalBound, {
|
||||
...(allowance.accountingReadiness ? { accountingReadiness: allowance.accountingReadiness } : {}),
|
||||
reference: allowance.reference, runner: allowance.runner, verifiedAt: allowance.verifiedAt, expiresAt: allowance.expiresAt,
|
||||
component: { component: component.component, maxUsd: component.maxUsd, reference: component.reference },
|
||||
});
|
||||
}
|
||||
|
||||
export function validateOperatingOverhead(value: EvalOperatingOverhead, totalUsd: string): bigint {
|
||||
object(value, ["version", "reference", "runner", "verifiedAt", "expiresAt", "components"]);
|
||||
object(value, ["version", "accountingReadiness", "reference", "runner", "verifiedAt", "expiresAt", "components"]);
|
||||
if (value.accountingReadiness !== undefined && value.accountingReadiness !== "model_requests") throw new Error("Invalid accounting readiness policy");
|
||||
if (value.version !== 1 || !reference(value.reference) || !["github-hosted", "aws-fleet"].includes(value.runner)
|
||||
|| typeof value.verifiedAt !== "string" || typeof value.expiresAt !== "string"
|
||||
|| !Number.isFinite(Date.parse(value.verifiedAt)) || !Number.isFinite(Date.parse(value.expiresAt))
|
||||
|
||||
@@ -2,7 +2,7 @@ import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { expect, it } from "vitest";
|
||||
import { openEvalCostLedger } from "./cost-ledger.js";
|
||||
import { openEvalCostLedger, type EvalCostObservation } from "./cost-ledger.js";
|
||||
import { readEvalOperationsReport, renderEvalOperationsReport, validateEvalOperationsReport } from "./operations-report.js";
|
||||
import { buildRunnerCampaign } from "./history.js";
|
||||
import { regenerateRunnerDashboard } from "./dashboard-regenerate.js";
|
||||
@@ -91,3 +91,71 @@ it("keeps an in-flight final receipt reserved until the forwarding request finis
|
||||
expect(report).toMatchObject({ inflight: 1, knownUsd: "0.050000000", heldUsd: "0.150000000", committedUsd: "0.200000000", unresolvedRequests: 0 });
|
||||
} finally { await ledger.close(); await rm(root, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
it("reports final setup and canceled-request tokens once while exposing missing and conflicting usage", async () => {
|
||||
const root = await mkdtemp(path.join(tmpdir(), "eval-token-report-")), file = path.join(root, "ledger.jsonl");
|
||||
const ledger = await openEvalCostLedger(file, { campaignId: "tokens", maxUsd: "10", deadlineAt: "2099-01-01T00:00:00Z", maxRequests: 10, maxConcurrent: 1 });
|
||||
const observation = { provider: "fixture", currency: "USD" as const, route: "direct" as const, classification: "reported" as const, costUsd: "0.05", final: true };
|
||||
try {
|
||||
for (const [requestId, phase] of [["setup", "setup"], ["canceled", "agent"], ["missing", "agent"], ["conflict", "agent"], ["full-conflict", "agent"], ["reasoning-conflict", "agent"], ["overhead", "infrastructure"]] as const) {
|
||||
await ledger.reserve(requestId, { executionId: "cell", attemptId: "first", phase }, "1");
|
||||
if (requestId !== "missing") {
|
||||
const tokens = { input: 10, cachedRead: 20, cachedWrite: 30, output: 5, ...(requestId === "reasoning-conflict" ? {} : { reasoning: 2 }) };
|
||||
await ledger.observe({ ...observation, observationId: requestId + "-partial", requestId, source: "gateway", tokens: { ...tokens, input: 1 }, final: false });
|
||||
await ledger.observe({ ...observation, observationId: requestId + "-gateway", requestId, source: "gateway", tokens });
|
||||
await ledger.observe({ ...observation, observationId: requestId + "-provider", requestId, source: "provider", tokens });
|
||||
if (requestId === "setup") await ledger.observe({ ...observation, observationId: "native-telemetry", requestId, source: "native_otlp", tokens: { ...tokens, input: 999 } });
|
||||
if (requestId === "conflict") await ledger.observe({ ...observation, observationId: "conflicting-usage", requestId, source: "provider", tokens: { input: 11 } });
|
||||
if (requestId === "reasoning-conflict") await ledger.observe({ ...observation, observationId: "partial-reasoning", requestId, source: "provider", tokens: { reasoning: 9 } });
|
||||
if (requestId === "full-conflict") await ledger.observe({ ...observation, observationId: "full-conflicting-usage", requestId, source: "provider", tokens: { ...tokens, input: 11 } });
|
||||
}
|
||||
await ledger.finish(requestId, requestId === "canceled" ? "cancelled" : "succeeded");
|
||||
}
|
||||
const report = await readEvalOperationsReport(file);
|
||||
if (report.status === "unavailable") throw new Error("Expected readable token report");
|
||||
expect(report.modelAccounting?.tokenUsage).toEqual({ requests: 2, conflictingRequests: 3,
|
||||
totals: { input: 20, cachedRead: 40, cachedWrite: 60, output: 10, reasoning: 4 },
|
||||
setup: { requests: 1, totals: { input: 10, cachedRead: 20, cachedWrite: 30, output: 5, reasoning: 2 } } });
|
||||
const rendered = renderEvalOperationsReport(report);
|
||||
expect(rendered).toContain("2/6 model requests have unambiguous final usage; 3 conflicting; 1 missing");
|
||||
expect(rendered).toContain("Reported reasoning: 4, included in output");
|
||||
expect(rendered).toContain("Setup token usage (1 requests, included above)");
|
||||
const emptySetup = structuredClone(report);
|
||||
emptySetup.modelAccounting!.tokenUsage!.setup.requests = 0;
|
||||
expect(() => validateEvalOperationsReport(emptySetup)).toThrow("Invalid retained");
|
||||
const emptyOverall = structuredClone(report);
|
||||
emptyOverall.modelAccounting!.tokenUsage!.requests = 0;
|
||||
emptyOverall.modelAccounting!.tokenUsage!.setup = { requests: 0, totals: { input: 0, cachedRead: 0, cachedWrite: 0, output: 0, reasoning: 0 } };
|
||||
expect(() => validateEvalOperationsReport(emptyOverall)).toThrow("Invalid retained");
|
||||
const changed = structuredClone(report);
|
||||
changed.modelAccounting!.tokenUsage!.setup.totals.input = 21;
|
||||
expect(() => validateEvalOperationsReport(changed)).toThrow("Invalid retained");
|
||||
changed.modelAccounting!.tokenUsage!.setup.totals.input = Number.MAX_SAFE_INTEGER + 1;
|
||||
expect(() => validateEvalOperationsReport(changed)).toThrow("Invalid retained");
|
||||
} finally { await ledger.close(); await rm(root, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
|
||||
it("preserves agreed final usage across every partial-field subset and observation order", async () => {
|
||||
const root = await mkdtemp(path.join(tmpdir(), "eval-token-subsets-")), file = path.join(root, "ledger.jsonl");
|
||||
const ledger = await openEvalCostLedger(file, { campaignId: "subsets", maxUsd: "10", deadlineAt: "2099-01-01T00:00:00Z", maxRequests: 64, maxConcurrent: 1 });
|
||||
const values = { input: 10, cachedRead: 20, cachedWrite: 30, output: 5, reasoning: 2 };
|
||||
const keys = ["input", "cachedRead", "cachedWrite", "output", "reasoning"] as const;
|
||||
try {
|
||||
for (let mask = 0; mask < 32; mask++) for (const reverse of [false, true]) {
|
||||
const requestId = `request-${mask}-${reverse}`, partial: EvalCostObservation["tokens"] = {};
|
||||
keys.forEach((key, index) => { if (mask & 1 << index) partial[key] = values[key]; });
|
||||
await ledger.reserve(requestId, { executionId: "cell", attemptId: "attempt", phase: "agent" }, "0.2");
|
||||
const sources = [{ source: "gateway" as const, tokens: { input: 10, cachedRead: 20, cachedWrite: 30, output: 5 } }, { source: "provider" as const, tokens: partial }];
|
||||
if (reverse) sources.reverse();
|
||||
for (const observation of sources) await ledger.observe({ ...observation, observationId: requestId + observation.source, requestId,
|
||||
provider: "fixture", currency: "USD", route: "direct", classification: "reported", costUsd: "0.05", final: true });
|
||||
await ledger.finish(requestId, "cancelled");
|
||||
}
|
||||
const report = await readEvalOperationsReport(file);
|
||||
if (report.status === "unavailable") throw new Error("Expected usable accounting for agreeing partial receipts");
|
||||
expect(report.modelAccounting?.tokenUsage).toMatchObject({ requests: 64, conflictingRequests: 0,
|
||||
totals: { input: 640, cachedRead: 1280, cachedWrite: 1920, output: 320, reasoning: 64 } });
|
||||
expect(report.knownUsd).toBe("3.200000000");
|
||||
} finally { await ledger.close(); await rm(root, { recursive: true, force: true }); }
|
||||
});
|
||||
@@ -3,10 +3,17 @@ import { usdToUnits } from "../../packages/shared/src/money.js";
|
||||
import { html, isPublicReportTimestamp } from "./report-format.js";
|
||||
|
||||
const money = (n: bigint) => `${n / 1_000_000_000n}.${String(n % 1_000_000_000n).padStart(9, "0")}`;
|
||||
interface ModelTokenTotals { input: number; cachedRead: number; cachedWrite: number; output: number; reasoning: number }
|
||||
interface ModelTokenUsage { requests: number; conflictingRequests: number; totals: ModelTokenTotals; setup: { requests: number; totals: ModelTokenTotals } }
|
||||
const tokenFields = ["input", "cachedRead", "cachedWrite", "output", "reasoning"] as const;
|
||||
const emptyTokenTotals = (): ModelTokenTotals => ({ input: 0, cachedRead: 0, cachedWrite: 0, output: 0, reasoning: 0 });
|
||||
|
||||
interface EvalOperationsAvailableReport {
|
||||
version: 1;
|
||||
scope: "shared_gateway_ledger";
|
||||
status?: never;
|
||||
accountingReadiness?: "model_requests";
|
||||
modelAccounting?: { requests: number; unresolvedRequests: number; conflictingRequests: number; inflight: number; knownUsd: string; heldUsd: string; tokenUsage?: ModelTokenUsage };
|
||||
ledgerCampaignId: string;
|
||||
journalHash: string;
|
||||
records: number;
|
||||
@@ -39,7 +46,7 @@ export function validateEvalOperationsReport(report: EvalOperationsReport): void
|
||||
}
|
||||
const dollars = (value: unknown): bigint => typeof value === "string" && /^(?:0|[1-9][0-9]*)\.[0-9]{9}$/.test(value) ? usdToUnits(value) : fail();
|
||||
const count = (value: unknown) => { if (!Number.isSafeInteger(value) || Number(value) < 0) fail(); };
|
||||
fields(report, ["version", "scope", "ledgerCampaignId", "journalHash", "records", "deadlineAt", "limitUsd", "knownUsd", "heldUsd", "committedUsd", "requests", "unresolvedRequests", "boundedUnresolvedRequests", "conflictingRequests", "inflight", "breakdown", "attempts"]);
|
||||
fields(report, ["version", "scope", "accountingReadiness", "modelAccounting", "ledgerCampaignId", "journalHash", "records", "deadlineAt", "limitUsd", "knownUsd", "heldUsd", "committedUsd", "requests", "unresolvedRequests", "boundedUnresolvedRequests", "conflictingRequests", "inflight", "breakdown", "attempts"]);
|
||||
if (report.version !== 1 || report.scope !== "shared_gateway_ledger" || !validEvalLedgerId(report.ledgerCampaignId)
|
||||
|| !/^[a-f0-9]{64}$/.test(report.journalHash) || !isPublicReportTimestamp(report.deadlineAt) || !Array.isArray(report.attempts)) fail();
|
||||
[report.records, report.requests, report.unresolvedRequests, report.conflictingRequests, report.inflight].forEach(count);
|
||||
@@ -48,6 +55,29 @@ export function validateEvalOperationsReport(report: EvalOperationsReport): void
|
||||
count(report.boundedUnresolvedRequests);
|
||||
if (report.boundedUnresolvedRequests > report.unresolvedRequests - report.conflictingRequests) fail();
|
||||
}
|
||||
if (report.accountingReadiness !== undefined && (report.accountingReadiness !== "model_requests" || !report.modelAccounting)) fail();
|
||||
if (report.modelAccounting !== undefined) {
|
||||
const model = report.modelAccounting;
|
||||
fields(model, ["requests", "unresolvedRequests", "conflictingRequests", "inflight", "knownUsd", "heldUsd", "tokenUsage"]);
|
||||
if (model.tokenUsage !== undefined) {
|
||||
const usage = model.tokenUsage;
|
||||
fields(usage, ["requests", "conflictingRequests", "totals", "setup"]);
|
||||
fields(usage.setup, ["requests", "totals"]);
|
||||
[usage.requests, usage.conflictingRequests, usage.setup.requests].forEach(count);
|
||||
if (usage.requests + usage.conflictingRequests > model.requests || usage.setup.requests > usage.requests) fail();
|
||||
for (const [requests, totals] of [[usage.requests, usage.totals], [usage.setup.requests, usage.setup.totals]] as const) {
|
||||
fields(totals, [...tokenFields]);
|
||||
tokenFields.forEach(key => count(totals[key]));
|
||||
if (totals.reasoning > totals.output || requests === 0 && tokenFields.some(key => totals[key] !== 0)) fail();
|
||||
}
|
||||
if (tokenFields.some(key => usage.setup.totals[key] > usage.totals[key])) fail();
|
||||
}
|
||||
[model.requests, model.unresolvedRequests, model.conflictingRequests, model.inflight].forEach(count);
|
||||
if (model.requests > report.requests || model.unresolvedRequests > model.requests || model.unresolvedRequests > report.unresolvedRequests
|
||||
|| model.conflictingRequests > model.unresolvedRequests || model.conflictingRequests > report.conflictingRequests
|
||||
|| model.inflight > model.requests || model.inflight > report.inflight
|
||||
|| dollars(model.knownUsd) > dollars(report.knownUsd) || dollars(model.heldUsd) > dollars(report.heldUsd)) fail();
|
||||
}
|
||||
dollars(report.limitUsd);
|
||||
const known = dollars(report.knownUsd), held = dollars(report.heldUsd);
|
||||
if (known + held !== dollars(report.committedUsd)) fail();
|
||||
@@ -86,7 +116,35 @@ async function readAvailableEvalOperationsReport(file: string): Promise<EvalOper
|
||||
group.heldUsd = money(usdToUnits(group.heldUsd) + held);
|
||||
groups.set(key, group);
|
||||
}
|
||||
const modelRequests = snapshot.requests.filter(request => request.phase !== "infrastructure");
|
||||
const modelKnown = modelRequests.reduce((sum, request) => sum + usdToUnits(request.costUsd ?? "0"), 0n);
|
||||
const modelHeld = modelRequests.reduce((sum, request) => {
|
||||
const cost = usdToUnits(request.costUsd ?? "0"), reserved = usdToUnits(request.reservedUsd);
|
||||
return sum + ((!request.finished || !request.receiptComplete) && reserved > cost ? reserved - cost : 0n);
|
||||
}, 0n);
|
||||
const tokenUsage: ModelTokenUsage = { requests: 0, conflictingRequests: 0, totals: emptyTokenTotals(), setup: { requests: 0, totals: emptyTokenTotals() } };
|
||||
for (const request of loaded.state.requests.values()) {
|
||||
if (request.identity.phase === "infrastructure") continue;
|
||||
const candidates = request.observations.filter(observation => observation.final && observation.tokens
|
||||
&& ["gateway", "provider"].includes(observation.source));
|
||||
const values = tokenFields.map(key => new Set(candidates.flatMap(observation => observation.tokens![key] === undefined ? [] : [observation.tokens![key]!] )));
|
||||
const complete = candidates.find(observation => ["input", "cachedRead", "cachedWrite", "output"].every(key => observation.tokens![key as keyof ModelTokenTotals] !== undefined));
|
||||
if (values.some(value => value.size > 1) || complete && candidates.some(observation => (observation.tokens!.reasoning ?? 0) > complete.tokens!.output!)) { tokenUsage.conflictingRequests++; continue; }
|
||||
if (!complete) continue;
|
||||
const tokens = Object.fromEntries(tokenFields.map(key => [key, complete.tokens![key] ?? 0])) as unknown as ModelTokenTotals;
|
||||
tokens.reasoning = candidates.find(observation => observation.tokens!.reasoning !== undefined)?.tokens?.reasoning ?? 0;
|
||||
tokenUsage.requests++;
|
||||
if (request.identity.phase === "setup") tokenUsage.setup.requests++;
|
||||
for (const key of tokenFields) {
|
||||
tokenUsage.totals[key] += tokens[key];
|
||||
if (request.identity.phase === "setup") tokenUsage.setup.totals[key] += tokens[key];
|
||||
}
|
||||
}
|
||||
const report: EvalOperationsAvailableReport = { version: 1, scope: "shared_gateway_ledger", ledgerCampaignId: snapshot.campaignId,
|
||||
...(loaded.state.limits.overhead?.accountingReadiness ? { accountingReadiness: loaded.state.limits.overhead.accountingReadiness } : {}),
|
||||
modelAccounting: { requests: modelRequests.length, unresolvedRequests: modelRequests.filter(request => !request.receiptComplete).length,
|
||||
conflictingRequests: modelRequests.filter(request => request.conflict).length, inflight: modelRequests.filter(request => !request.finished).length,
|
||||
knownUsd: money(modelKnown), heldUsd: money(modelHeld), tokenUsage },
|
||||
journalHash: loaded.previous, records: loaded.sequence, deadlineAt: new Date(loaded.state.limits.deadlineAt).toISOString(),
|
||||
limitUsd: snapshot.limitUsd, knownUsd: snapshot.knownUsd, heldUsd: snapshot.heldUsd, committedUsd: snapshot.committedUsd,
|
||||
requests: snapshot.requests.length, unresolvedRequests: snapshot.unknownRequests, boundedUnresolvedRequests: snapshot.boundedUnresolvedRequests, conflictingRequests: snapshot.conflictingRequests,
|
||||
@@ -103,12 +161,28 @@ export function evalAccountingStatus(report: EvalOperationsReport): "unavailable
|
||||
return report.unresolvedRequests ? "provisional" : "receipts_complete";
|
||||
}
|
||||
|
||||
export function evalReadinessStatus(report: EvalOperationsReport): ReturnType<typeof evalAccountingStatus> {
|
||||
const full = evalAccountingStatus(report);
|
||||
if (report.status === "unavailable" || report.accountingReadiness !== "model_requests") return full;
|
||||
const model = report.modelAccounting!;
|
||||
if (!model.requests || model.unresolvedRequests || model.inflight || report.conflictingRequests || report.inflight
|
||||
|| usdToUnits(report.committedUsd) > usdToUnits(report.limitUsd)) return "blocked";
|
||||
return "receipts_complete";
|
||||
}
|
||||
|
||||
export function operationsReportText(report: EvalOperationsReport): string[] {
|
||||
if (report.status === "unavailable") return [
|
||||
"Request ledger accounting unavailable: the journal could not be read or validated. Request amounts, counts, and held reservations are unknown.",
|
||||
"Retained test results are preserved. Recover and validate the original journal before resuming paid work; do not reset the allowance or treat missing amounts as zero.",
|
||||
];
|
||||
return [
|
||||
...(report.modelAccounting ? [`Model requests: ${report.modelAccounting.requests}; $${report.modelAccounting.knownUsd} known token cost; $${report.modelAccounting.heldUsd} additional held; ${report.modelAccounting.unresolvedRequests} unresolved.`] : []),
|
||||
...(report.modelAccounting?.tokenUsage ? (() => {
|
||||
const model = report.modelAccounting!, usage = model.tokenUsage!, total = usage.totals, setup = usage.setup.totals;
|
||||
return [`Ledger token usage: ${usage.requests}/${model.requests} model requests have unambiguous final usage; ${usage.conflictingRequests} conflicting; ${model.requests - usage.requests - usage.conflictingRequests} missing. Totals: ${total.input} uncached input / ${total.cachedRead} cache-read input / ${total.cachedWrite} cache-write input / ${total.output} output. Reported reasoning: ${total.reasoning}, included in output.`,
|
||||
`Setup token usage (${usage.setup.requests} requests, included above): ${setup.input} uncached input / ${setup.cachedRead} cache-read input / ${setup.cachedWrite} cache-write input / ${setup.output} output. Completed, failed and canceled requests are included once; token coverage is separate from invoice reconciliation.`];
|
||||
})() : []),
|
||||
...(report.accountingReadiness === "model_requests" ? [`Readiness scope: model requests; status: ${evalReadinessStatus(report)}. Operating costs remain separately unreconciled unless final receipts establish otherwise. Their reservations remain inside campaign admission; this does not establish a full-lifecycle operating-cost bound.`] : []),
|
||||
`Accounting status: ${evalAccountingStatus(report)}. ${report.boundedUnresolvedRequests ?? 0} pending charges have reviewed bounds; ${report.unresolvedRequests - (report.boundedUnresolvedRequests ?? 0)} do not. Provisional completion preserves holds and requires later reconciliation; it does not establish invoice completeness.`,
|
||||
`Shared request ledger ${report.ledgerCampaignId}: $${report.knownUsd} known subtotal; $${report.heldUsd} additional held reservations; $${report.committedUsd} committed against $${report.limitUsd}.`,
|
||||
`Charges: $${report.breakdown.reconciled} reconciled; $${report.breakdown.reported} reported; $${report.breakdown.estimated} estimated; $${report.breakdown.allocated} allocated.`,
|
||||
|
||||
@@ -3,6 +3,7 @@ import { access, mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promise
|
||||
import { createHash } from "node:crypto";
|
||||
import path from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import * as remoteGateway from "./eval-remote-gateway.js";
|
||||
import { runnerMatrix } from "./catalog.js";
|
||||
import { validateEvalOperationsManifest, openEvalOperations, type EvalOperationsManifest } from "./operations.js";
|
||||
import { setupLiveFixtures } from "./live-fixtures.js";
|
||||
@@ -17,7 +18,7 @@ function manifest(): EvalOperationsManifest {
|
||||
models: [{ provider: "openai", maximumInputTokens: 1000, maximumOutputTokens: 100, timeoutMs: 1000,
|
||||
price: { provider: "openai", model: cell.profile.model, version: "fixture-only", sourceUrl: "https://provider.invalid/fixture", inputUsdPerMillion: "2", cachedReadUsdPerMillion: "1", outputUsdPerMillion: "10", cacheWrite30mUsdPerMillion: "2", cacheWrite12hUsdPerMillion: "2" } }] };
|
||||
}
|
||||
afterEach(() => { vi.unstubAllEnvs(); vi.unstubAllGlobals(); });
|
||||
afterEach(() => { vi.restoreAllMocks(); vi.unstubAllEnvs(); vi.unstubAllGlobals(); });
|
||||
it("admits only the exact bounded local selection and explicitly priced models", () => {
|
||||
expect(validateEvalOperationsManifest(manifest(), [cell], 1)).toMatchObject({ route: "direct" });
|
||||
expect(() => validateEvalOperationsManifest({ ...manifest(), executionIds: [] }, [cell], 1)).toThrow("selection");
|
||||
@@ -163,16 +164,21 @@ it.each([
|
||||
cell.id,
|
||||
"feedback-persistence.runner-codex.local.complaint-and-suggestion",
|
||||
"feedback-persistence.runner-acpx-claude.local.complaint-and-suggestion",
|
||||
"core-compatibility.runner-acpx-claude.daytona.message-marker",
|
||||
])("routes %s through a shared connection without ambient provider bindings", async executionId => {
|
||||
const selected = runnerMatrix.find(execution => execution.id === executionId)!;
|
||||
const anthropic = selected.profile.credential === "ANTHROPIC_API_KEY";
|
||||
const admitted = manifest(); admitted.executionIds = [selected.id];
|
||||
const remote = selected.environment.id === "daytona";
|
||||
const origin = remote ? "https://fixture.trycloudflare.com" : "http://127.0.0.1:43123";
|
||||
const admitted = manifest();
|
||||
if (remote) admitted.remoteGateway = "cloudflare"; admitted.executionIds = [selected.id];
|
||||
admitted.models[0]!.provider = anthropic ? "anthropic" : "openai";
|
||||
admitted.models[0]!.price.provider = admitted.models[0]!.provider;
|
||||
admitted.models[0]!.price.model = selected.profile.model;
|
||||
if (anthropic) Object.assign(admitted.models[0]!.price, { cacheWrite5mUsdPerMillion: "2", cacheWrite1hUsdPerMillion: "2" });
|
||||
expect(validateEvalOperationsManifest(admitted, [selected], 0)).toBe(admitted);
|
||||
vi.stubEnv("PAPERCLIP_E2E_GATEWAY_BASE_URL", "http://127.0.0.1:43123");
|
||||
vi.stubEnv("PAPERCLIP_E2E_GATEWAY_BASE_URL", origin);
|
||||
if (remote) vi.stubEnv("PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN", origin);
|
||||
vi.stubEnv("PAPERCLIP_E2E_GATEWAY_TOKEN", "ephemeral-fixture-token");
|
||||
const id = "11111111-1111-4111-8111-111111111111", grantId = "22222222-2222-4222-8222-222222222222";
|
||||
let connection: unknown, agent: Record<string, any> | undefined, company: Record<string, unknown> | undefined;
|
||||
@@ -180,14 +186,58 @@ it.each([
|
||||
async get() { return [{ id, driver: "local" }]; },
|
||||
async postSensitive(url: string, body: unknown) { if (url.endsWith("/ai-connections")) { connection = body; return { connectionId: id, grantId }; } return { id }; },
|
||||
async post(url: string, body: Record<string, unknown>) { if (url.endsWith("/agents")) agent = body; if (url === "/api/companies") company = body; return { id, name: "fixture" }; },
|
||||
async delete() {},
|
||||
} as unknown as RunnerApi;
|
||||
const fixtures = await setupLiveFixtures({ api, execution: selected, executionNonce: "fixture", workspacePath: "/tmp/fixture", credentials: { [selected.profile.credential]: "ephemeral-fixture-token" } });
|
||||
const fixtures = await setupLiveFixtures({ api, execution: selected, executionNonce: "fixture", workspacePath: "/tmp/fixture", credentials: { [selected.profile.credential]: "ephemeral-fixture-token", ...(remote ? { DAYTONA_API_KEY: "fixture-daytona" } : {}) },
|
||||
...(remote ? { daytonaImage: `ghcr.io/paperclip/image@sha256:${"a".repeat(64)}` } : {}) });
|
||||
await fixtures.teardown();
|
||||
expect(createAiConnectionSchema.parse(connection)).toMatchObject({ ownership: "shared", apiKey: "ephemeral-fixture-token", routing: { kind: "gateway", protocol: anthropic ? "messages" : "responses", baseUrl: `http://127.0.0.1:43123${anthropic ? "" : "/v1"}`, auth: "bearer" } });
|
||||
expect(createAiConnectionSchema.parse(connection)).toMatchObject({ ownership: "shared", apiKey: "ephemeral-fixture-token", routing: { kind: "gateway", protocol: anthropic ? "messages" : "responses", baseUrl: `${origin}${anthropic ? "" : "/v1"}`, auth: "bearer" } });
|
||||
expect(aiConnectionBindingSchema.parse(agent!.runtimeConfig.aiConnection)).toEqual({ provider: anthropic ? "anthropic" : "openai", method: "api_key", mode: "shared", connectionId: id, grantId });
|
||||
expect(agent!.adapterConfig.env).toBeUndefined();
|
||||
expect(agent!.adapterConfig.helloProbeTimeoutSec).toBe(remote ? 300 : undefined);
|
||||
if (selected.suite.id === "feedback-persistence") {
|
||||
expect(company!.budgetMonthlyCents).toBe(1_000);
|
||||
expect(agent!.budgetMonthlyCents).toBe(1_000);
|
||||
}
|
||||
});
|
||||
|
||||
|
||||
it("requires explicit remote gateway admission and infrastructure credentials before opening a journal", async () => {
|
||||
const remote = runnerMatrix.find(e => e.suite.id === cell.suite.id && e.profile.id === cell.profile.id && e.environment.id === "daytona")!;
|
||||
const value = { ...manifest(), executionIds: [remote.id], remoteGateway: "cloudflare" as const };
|
||||
expect(validateEvalOperationsManifest(value, [remote], 0)).toBe(value);
|
||||
const root = await mkdtemp(path.join(tmpdir(), "eval-remote-preflight-"));
|
||||
try {
|
||||
value.ledgerPath = path.join(root, "ledger.jsonl");
|
||||
const file = path.join(root, "manifest.json"); await writeFile(file, JSON.stringify(value));
|
||||
await expect(openEvalOperations(file, [remote], 0, { OPENAI_API_KEY: "synthetic" })).rejects.toThrow("credential");
|
||||
await expect(access(value.ledgerPath)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
} finally { await rm(root, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
it("gives remote workers scoped gateway/relay URLs and closes both local listeners and the tunnel", async () => {
|
||||
const remote = runnerMatrix.find(e => e.suite.id === cell.suite.id && e.profile.id === cell.profile.id && e.environment.id === "daytona")!;
|
||||
const root = await mkdtemp(path.join(tmpdir(), "eval-remote-operation-"));
|
||||
const close = vi.fn(async () => {}), origin = "https://synthetic-owned.trycloudflare.com";
|
||||
const start = vi.spyOn(remoteGateway, "startEvalRemoteGateway").mockResolvedValue({ baseUrl: origin, close });
|
||||
const value = { ...manifest(), executionIds: [remote.id], remoteGateway: "cloudflare" as const, ledgerPath: path.join(root, "ledger.jsonl") };
|
||||
value.limits.deadlineAt = new Date(Date.now() + 24 * 3_600_000).toISOString();
|
||||
const file = path.join(root, "manifest.json"), env = { OPENAI_API_KEY: "provider-private", DAYTONA_API_KEY: "infra-private",
|
||||
BRAINTRUST_API_KEY: "trace-private", PAPERCLIP_E2E_BRAINTRUST_PROJECT_ID: "11111111-1111-4111-8111-111111111111" };
|
||||
let operations: Awaited<ReturnType<typeof openEvalOperations>> | undefined;
|
||||
try {
|
||||
await writeFile(file, JSON.stringify(value)); operations = await openEvalOperations(file, [remote], 0, env);
|
||||
const endpoint = await operations.start(remote, "remote-one", 1);
|
||||
const worker: NodeJS.ProcessEnv = { ...env }; applyEvalGatewayEnvironment(worker, remote.profile.credential, endpoint);
|
||||
expect(worker).toMatchObject({ PAPERCLIP_E2E_GATEWAY_BASE_URL: origin, PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN: origin,
|
||||
PAPERCLIP_E2E_OTLP_RELAY_URL: `${origin}/otel`, OPENAI_API_KEY: endpoint.token, BRAINTRUST_API_KEY: endpoint.observability!.token });
|
||||
expect(JSON.stringify(worker)).not.toMatch(/provider-private|infra-private|trace-private/);
|
||||
const local = start.mock.calls[0]![0];
|
||||
expect(local.gateway.baseUrl).toMatch(/^http:\/\/127\.0\.0\.1:/);
|
||||
expect(local.deadlineAt).toBeLessThanOrEqual(Date.now() + 4 * 3_600_000);
|
||||
expect(local.deadlineAt).toBeLessThan(Date.parse(value.limits.deadlineAt));
|
||||
await operations.close(); operations = undefined; expect(close).toHaveBeenCalledOnce();
|
||||
await expect(fetch(local.gateway.baseUrl)).rejects.toThrow();
|
||||
await expect(fetch(local.observability!.baseUrl)).rejects.toThrow();
|
||||
} finally { await operations?.close(); await rm(root, { recursive: true, force: true }); }
|
||||
});
|
||||
@@ -4,6 +4,7 @@ import { createHash } from "node:crypto";
|
||||
import { createEvalAttemptPhase, readEvalAttemptPhase } from "./attempt-phase.js";
|
||||
import { EvalSpendState, openEvalCostLedger, type EvalSpendLimits } from "./cost-ledger.js";
|
||||
import { evalRequestReservation, startEvalGateway, type EvalGatewayRoute, type EvalGatewayReceipt } from "./eval-gateway.js";
|
||||
import { startEvalRemoteGateway } from "./eval-remote-gateway.js";
|
||||
import type { MatrixExecution } from "./types.js";
|
||||
import { evalBraintrustConfig } from "./braintrust.js";
|
||||
import { startEvalOtlpRelay } from "./eval-otlp-relay.js";
|
||||
@@ -17,13 +18,15 @@ export interface EvalOperationsManifest {
|
||||
maxAttemptsPerExecution: number;
|
||||
route: "direct" | "braintrust";
|
||||
projectId?: string;
|
||||
remoteGateway?: "cloudflare";
|
||||
models: Array<Omit<EvalGatewayRoute, "apiKey" | "projectId" | "endpointName" | "mode">>;
|
||||
}
|
||||
const fields = (value: object, names: string[]) => {
|
||||
if (!value || typeof value !== "object" || Array.isArray(value) || Object.keys(value).some(name => !names.includes(name))) throw new Error("Invalid eval operations manifest fields");
|
||||
};
|
||||
export function validateEvalOperationsManifest(value: EvalOperationsManifest, executions: readonly MatrixExecution[], maxRetries: number) {
|
||||
fields(value, ["version", "limits", "ledgerPath", "executionIds", "maxAttemptsPerExecution", "route", "projectId", "models"]);
|
||||
fields(value, ["version", "limits", "ledgerPath", "executionIds", "maxAttemptsPerExecution", "route", "projectId", "remoteGateway", "models"]);
|
||||
if (value.remoteGateway !== undefined && value.remoteGateway !== "cloudflare") throw new Error("Invalid remote eval gateway policy");
|
||||
const initial = new EvalSpendState(value.limits).snapshot();
|
||||
const initialModelAllowance = usdToUnits(value.limits.maxUsd) - usdToUnits(initial.committedUsd);
|
||||
if (value.version !== 1 || !["direct", "braintrust"].includes(value.route) || !path.isAbsolute(value.ledgerPath)
|
||||
@@ -46,9 +49,11 @@ export function validateEvalOperationsManifest(value: EvalOperationsManifest, ex
|
||||
models.set(key, usdToUnits(evalRequestReservation({ ...model, mode: value.route, apiKey: "validation" })));
|
||||
}
|
||||
for (const execution of executions) {
|
||||
if (execution.environment.id !== "local" || !["core-compatibility", "feedback-persistence"].includes(execution.suite.id)
|
||||
if (!(execution.environment.id === "local" || execution.environment.id === "daytona" && value.remoteGateway === "cloudflare"
|
||||
&& ["runner-codex", "runner-acpx-claude"].includes(execution.profile.id)) || !["core-compatibility", "feedback-persistence"].includes(execution.suite.id)
|
||||
|| !["runner-codex", "runner-acpx-claude", "legacy-codex", "legacy-claude"].includes(execution.profile.id)
|
||||
|| execution.requiredCredentials.length !== 1) throw new Error(`Unqualified budgeted eval route: ${execution.id}`);
|
||||
|| execution.requiredCredentials.length !== (execution.environment.id === "daytona" ? 2 : 1)
|
||||
|| execution.requiredCredentials.filter(name => name !== "DAYTONA_API_KEY").length !== 1) throw new Error(`Unqualified budgeted eval route: ${execution.id}`);
|
||||
const provider = execution.profile.credential === "ANTHROPIC_API_KEY" ? "anthropic" : "openai";
|
||||
if (!models.has(`${provider}:${execution.profile.model}`)) throw new Error(`Missing exact model pricing: ${execution.id}`);
|
||||
if (models.get(`${provider}:${execution.profile.model}`)! > initialModelAllowance) {
|
||||
@@ -65,7 +70,8 @@ export async function openEvalOperations(file: string, executions: readonly Matr
|
||||
|| Date.parse(manifest.limits.deadlineAt) >= Date.parse(manifest.limits.overhead.expiresAt))) {
|
||||
throw new Error("Operating overhead allowance does not cover the campaign window");
|
||||
}
|
||||
const required = manifest.route === "braintrust" ? ["BRAINTRUST_API_KEY"] : executions.map(execution => execution.profile.credential);
|
||||
const required = [...(manifest.route === "braintrust" ? ["BRAINTRUST_API_KEY"] : executions.map(execution => execution.profile.credential)),
|
||||
...(executions.some(execution => execution.environment.id === "daytona") ? ["DAYTONA_API_KEY"] : [])];
|
||||
if (required.some(name => !env[name]?.trim())) throw new Error("Missing eval gateway upstream credential");
|
||||
const observability = evalBraintrustConfig({ ...env, PAPERCLIP_E2E_OTLP_RELAY_URL: undefined });
|
||||
if (observability && manifest.route === "braintrust" && observability.projectId !== manifest.projectId) throw new Error("Eval gateway and OTLP projects differ");
|
||||
@@ -99,14 +105,19 @@ export async function openEvalOperations(file: string, executions: readonly Matr
|
||||
persist: (receipt: EvalGatewayReceipt) => atomicJson(path.join(receiptDirectory, `${receipt.requestId}.json`), receipt),
|
||||
});
|
||||
let relay: Awaited<ReturnType<typeof startEvalOtlpRelay>> | undefined;
|
||||
let remote: Awaited<ReturnType<typeof startEvalRemoteGateway>> | undefined;
|
||||
try {
|
||||
if (observability) relay = await startEvalOtlpRelay({ projectId: observability.projectId, apiKey: observability.apiKey,
|
||||
campaignId: manifest.limits.campaignId, attemptId, deadlineAt: Math.min(Date.parse(manifest.limits.deadlineAt), Date.now() + 4 * 3_600_000),
|
||||
receiptPath: path.join(`${manifest.ledgerPath}.otlp`, `${createHash("sha256").update(attemptId).digest("hex")}.jsonl`) });
|
||||
} catch (error) { await gateway.close(); throw error; }
|
||||
if (execution.environment.id === "daytona") remote = await startEvalRemoteGateway({ gateway, observability: relay,
|
||||
deadlineAt: Math.min(Date.parse(manifest.limits.deadlineAt), Date.now() + 4 * 3_600_000) }, env);
|
||||
} catch (error) { await Promise.allSettled([gateway.close(), relay?.close()]); throw error; }
|
||||
let closing: Promise<void> | undefined;
|
||||
const endpoint = { ...gateway, phasePath, observability: relay, close: () => closing ??= (async () => {
|
||||
const results = await Promise.allSettled([gateway.close(), relay?.close()]);
|
||||
const endpoint = { ...gateway, baseUrl: remote?.baseUrl ?? gateway.baseUrl, phasePath,
|
||||
observability: relay ? { ...relay, baseUrl: remote ? `${remote.baseUrl}/otel` : relay.baseUrl } : undefined, close: () => closing ??= (async () => {
|
||||
const remoteResults = await Promise.allSettled([remote?.close()]);
|
||||
const results = [...remoteResults, ...await Promise.allSettled([gateway.close(), relay?.close()])];
|
||||
active.delete(endpoint);
|
||||
if (results.some(result => result.status === "rejected")) throw new Error("Eval gateway or OTLP relay cleanup failed");
|
||||
})() };
|
||||
|
||||
@@ -188,8 +188,8 @@ describe("Pi controls catalog admission", () => {
|
||||
// Extended v3 states the strict file oracle's task-wide Bash limit.
|
||||
const pi = runnerMatrix.find(c => c.profile.qualificationCandidate === "pi")!.profile;
|
||||
expect(pi.modelQualification?.qualificationId).toBe("pi:0.0.33:1.0.0:openrouter");
|
||||
expect(runnerSuites.find(s => s.id === "pi-native")!.definitionMetadata).toMatchObject({ version: 23, remoteProcExit: "separately-confirmed-absence-after-read-failure", taskCreation: "explicit-title-and-creation-response-id", agentMemoryParent: "public-managed-file-seed-before-admission", incompleteTerminalCleanup: "retirement-retained-with-failed-watch", profileVersion: 29, agentMemoryContent: "utf8-nonce-plus-final-lf", agentMemoryPrompt: "single-json-write-and-content-bound-native-read-both-runs", agentMemoryReadAuthority: "local-withheld-or-exact-remote-agent-run-file", taskPromptTransport: "fenced-markdown-paste-and-multiline-literal-escapes", nativeFinish: "current-contract-objective-evidence-refs", providerFaultExecutable: "stable-preinstalled-runner-link-and-snapshot-node-inode-with-held-bootstrap-fd-3-or-7" });
|
||||
expect(runnerSuites.find(s => s.id === "pi-controls")!.definitionMetadata).toMatchObject({ version: 10, remoteProcExit: "separately-confirmed-absence-after-read-failure", taskCreation: "explicit-title-and-creation-response-id", profileVersion: 29,
|
||||
expect(runnerSuites.find(s => s.id === "pi-native")!.definitionMetadata).toMatchObject({ version: 23, remoteProcExit: "separately-confirmed-absence-after-read-failure", taskCreation: "explicit-title-and-creation-response-id", agentMemoryParent: "public-managed-file-seed-before-admission", incompleteTerminalCleanup: "retirement-retained-with-failed-watch", profileVersion: 30, agentMemoryContent: "utf8-nonce-plus-final-lf", agentMemoryPrompt: "single-json-write-and-content-bound-native-read-both-runs", agentMemoryReadAuthority: "local-withheld-or-exact-remote-agent-run-file", taskPromptTransport: "fenced-markdown-paste-and-multiline-literal-escapes", nativeFinish: "current-contract-objective-evidence-refs", providerFaultExecutable: "stable-preinstalled-runner-link-and-snapshot-node-inode-with-held-bootstrap-fd-3-or-7" });
|
||||
expect(runnerSuites.find(s => s.id === "pi-controls")!.definitionMetadata).toMatchObject({ version: 10, remoteProcExit: "separately-confirmed-absence-after-read-failure", taskCreation: "explicit-title-and-creation-response-id", profileVersion: 30,
|
||||
remoteProcessIdentity: "observer-pid-startTicks-bootId",
|
||||
controlPlaneSettlement: "required-scoped-result-and-terminal-after-runner" });
|
||||
expect(runnerSuites.find(s => s.id === "extended-harnesses")!.definitionMetadata).toMatchObject({ version: 6, piFileArtifactTitle: "exact-filename", piFileCommandTransport: "fenced-bash-markdown-paste" });
|
||||
@@ -199,7 +199,7 @@ describe("Pi controls catalog admission", () => {
|
||||
}
|
||||
const hashes = Object.fromEntries(runnerSuites.filter(s => ["pi-native", "native-active-stop", "extended-harnesses", "rich-acp-warm-continuity"].includes(s.id)).map(s => [s.id, suiteDefinitionHash(s)]));
|
||||
expect(hashes).toEqual({
|
||||
"pi-native": "d7d691088af76d460e353a3982f7732e6a65b2e5a23a66d2ef0d534f9d8dae48",
|
||||
"pi-native": "04d49c6e098c24091d2939108a97722d1b8353e39192b87e98ee33377f183001",
|
||||
"native-active-stop": "4654a2169bff96cf8377fdd62441f5cabea5c613f447e518830f6b1efd82c051",
|
||||
"rich-acp-warm-continuity": "7e8a0012f2f67fba03cd94403b096a07cfc95cdb7aa568b4c3fd18b852d735f3",
|
||||
"extended-harnesses": "d5bcf5de0ade3a54e4e1560e76feb5d8a787039d56d2379cd501626d85ed7418",
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
export function prepareReportLock(root?: string): Promise<'already-reviewed' | 'reviewed-correction'>;
|
||||
@@ -0,0 +1,31 @@
|
||||
import { createHash } from 'node:crypto';
|
||||
import { execFileSync } from 'node:child_process';
|
||||
import { readFile, writeFile, mkdtemp, rm } from 'node:fs/promises';
|
||||
import path from 'node:path';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { pathToFileURL } from 'node:url';
|
||||
|
||||
const sha256 = value => createHash('sha256').update(value).digest('hex');
|
||||
export async function prepareReportLock(root = path.resolve(import.meta.dirname, '../..')) {
|
||||
const policy = JSON.parse(await readFile(path.join(root, 'tests/runner-e2e/report-lock.json'), 'utf8'));
|
||||
const lockPath = path.join(root, 'pnpm-lock.yaml');
|
||||
const bytes = await readFile(lockPath), current = sha256(bytes);
|
||||
if (current === policy.outputSha256) return 'already-reviewed';
|
||||
if (current !== policy.inputSha256) throw new Error('Reporting lock changed; review the dependency correction before install');
|
||||
const patch = await readFile(path.join(root, 'tests/runner-e2e/report-lock.patch'));
|
||||
if (sha256(patch) !== policy.patchSha256) throw new Error('Reviewed reporting lock patch changed');
|
||||
const result = execFileSync('git', ['apply', '--numstat', '-'], { cwd: root, input: patch, encoding: 'utf8' });
|
||||
if (!/^\d+\s+\d+\s+pnpm-lock\.yaml\n$/.test(result)) throw new Error('Reporting correction must change only the lockfile');
|
||||
const temporary = await mkdtemp(path.join(tmpdir(), 'eval-reviewed-lock-'));
|
||||
try {
|
||||
await writeFile(path.join(temporary, 'pnpm-lock.yaml'), bytes);
|
||||
execFileSync('git', ['apply', '-'], { cwd: temporary, input: patch });
|
||||
const corrected = await readFile(path.join(temporary, 'pnpm-lock.yaml'));
|
||||
if (sha256(corrected) !== policy.outputSha256) throw new Error('Reviewed reporting lock result changed');
|
||||
await writeFile(lockPath, corrected);
|
||||
return 'reviewed-correction';
|
||||
} finally { await rm(temporary, { recursive: true, force: true }); }
|
||||
}
|
||||
if (process.argv[1] && import.meta.url === pathToFileURL(path.resolve(process.argv[1])).href) {
|
||||
console.log(await prepareReportLock());
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
import { mkdtemp, mkdir, readFile, writeFile, copyFile, rm } from 'node:fs/promises';
|
||||
import { createHash } from 'node:crypto';
|
||||
import { execFileSync } from 'node:child_process';
|
||||
import { tmpdir } from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { expect, it } from 'vitest';
|
||||
import { parse } from 'yaml';
|
||||
import { prepareReportLock } from './prepare-report-lock.mjs';
|
||||
|
||||
it.each(['valid', 'unknown-lock', 'changed-patch', 'wrong-output'])(
|
||||
'installs only the reviewed dependency correction: %s', async variant => {
|
||||
const root = await mkdtemp(path.join(tmpdir(), 'eval-report-lock-test-'));
|
||||
const here = import.meta.dirname, relative = path.join(root, 'tests/runner-e2e');
|
||||
try {
|
||||
await mkdir(relative, { recursive: true });
|
||||
for (const file of ['report-lock.json', 'report-lock.patch']) await copyFile(path.join(here, file), path.join(relative, file));
|
||||
const input = execFileSync('git', ['show', 'HEAD:pnpm-lock.yaml'], { cwd: path.resolve(here, '../..') });
|
||||
const policy = JSON.parse(await readFile(path.join(relative, 'report-lock.json'), 'utf8'));
|
||||
expect(createHash('sha256').update(input).digest('hex')).toBe(policy.inputSha256);
|
||||
const lock = path.join(root, 'pnpm-lock.yaml');
|
||||
await writeFile(lock, variant === 'unknown-lock' ? 'unknown\n' : input);
|
||||
if (variant === 'changed-patch') await writeFile(path.join(relative, 'report-lock.patch'), 'changed');
|
||||
if (variant === 'wrong-output') { policy.outputSha256 = '0'.repeat(64); await writeFile(path.join(relative, 'report-lock.json'), JSON.stringify(policy)); }
|
||||
if (variant === 'valid') {
|
||||
expect(await prepareReportLock(root)).toBe('reviewed-correction');
|
||||
expect(createHash('sha256').update(await readFile(lock)).digest('hex')).toBe(policy.outputSha256);
|
||||
expect(await prepareReportLock(root)).toBe('already-reviewed');
|
||||
const before = parse(input.toString()).packages;
|
||||
const after = parse(await readFile(lock, 'utf8')).packages;
|
||||
expect(Object.keys(after).filter(key => !Object.hasOwn(before, key))).toEqual(['yaml@2.9.1']);
|
||||
for (const [key, value] of Object.entries(before)) expect(after[key]).toEqual(value);
|
||||
expect(after['yaml@2.9.1'].resolution.integrity).toBe('sha512-3NxN8+78OdzbT7C/WjGsyfPAtJaN3FNDsWxv7Y7mcDsT/oOmgW8BpyQQFFBnvZE3j9Y2Sdz1ULFLezL7Eb2yFw==');
|
||||
|
||||
} else {
|
||||
await expect(prepareReportLock(root)).rejects.toThrow();
|
||||
expect(await readFile(lock)).toEqual(variant === 'unknown-lock' ? Buffer.from('unknown\n') : input);
|
||||
}
|
||||
} finally { await rm(root, { recursive: true, force: true }); }
|
||||
},
|
||||
);
|
||||
@@ -5,6 +5,10 @@ import os from "node:os";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { createConnectionProof } from "../connection-evidence.js";
|
||||
import { retainOnboardingArtifact, validateOnboardingArtifact } from "./artifact-evidence.js";
|
||||
import { validateOnboardingPreAcceptanceEvidence, type OnboardingPreAcceptanceEvidence } from "./onboarding-pre-acceptance.js";
|
||||
const preAcceptanceEvidence: OnboardingPreAcceptanceEvidence = { version: 1, checkpoint: "awaiting-proposal", violations: ["attachment_present"],
|
||||
taskCount: 1, attachmentCount: 1, documentCount: 0, nonPlanningDocumentCount: 0,
|
||||
firstTaskMatchesCanonicalParent: true, firstTaskStatus: "in_review", canonicalParentStatus: "in_review" };
|
||||
const persistenceFault = vi.hoisted(() => ({ kind: null as "write" | "rename" | null, active: false, latestReport: null as any }));
|
||||
vi.mock("node:fs/promises", async importOriginal => {
|
||||
const actual = await importOriginal<typeof import("node:fs/promises")>();
|
||||
@@ -33,7 +37,7 @@ import { regenerateRunnerDashboard } from "../dashboard-regenerate.js";
|
||||
const roots: string[]=[];
|
||||
afterEach(async()=>{vi.useRealTimers(); vi.restoreAllMocks(); persistenceFault.kind=null; persistenceFault.active=false; persistenceFault.latestReport=null; process.exitCode=0; await Promise.all(roots.splice(0).map(root=>rm(root,{recursive:true,force:true})));});
|
||||
type Hook = "collect" | "release" | "stop" | "finalizeEvidence" | "checkpointEvidence";
|
||||
async function run(options:{authFailure?:boolean;onboardingFailure?:boolean;missingArtifact?:boolean;drift?:boolean;cleanupFailure?:boolean;collectionFailure?:boolean;mini?:boolean;missingReceipt?:boolean;zeroRequests?:boolean;missingCheck?:boolean;finalizationFailure?:boolean;persistenceFailure?:"write"|"rename";persistenceAfter?:"collection"|"release";stallHook?:Hook;onHook?:(hook:Hook,signal?:AbortSignal)=>void}={}) {
|
||||
async function run(options:{authFailure?:boolean;onboardingFailure?:boolean;preAcceptanceFailure?:boolean;missingArtifact?:boolean;drift?:boolean;cleanupFailure?:boolean;collectionFailure?:boolean;mini?:boolean;missingReceipt?:boolean;zeroRequests?:boolean;missingCheck?:boolean;finalizationFailure?:boolean;persistenceFailure?:"write"|"rename";persistenceAfter?:"collection"|"release";stallHook?:Hook;onHook?:(hook:Hook,signal?:AbortSignal)=>void}={}) {
|
||||
const root=await mkdtemp(path.join(os.tmpdir(),"journey-campaign-test-")); roots.push(root);
|
||||
const module=path.join(root,"provider.mjs");
|
||||
const proof = createConnectionProof();
|
||||
@@ -44,7 +48,7 @@ let onboarded=false;
|
||||
const context={identity,company:{id:'company-1',issuePrefix:'QA'},api:{post:async()=>({id:'issue-fixture-1',identifier:'QA-1'})},startingState:'onboarding',capabilities:['company','board-user','task-write'],page:{async screenshot(){} }};
|
||||
export const provider={version:1, async acquire(input){return {
|
||||
async authenticate(){${options.authFailure ? "throw {outcome:'failed',code:'login_code_expired'};" : ""}},
|
||||
async onboard(evidence){${options.onboardingFailure ? "throw {outcome:'failed',code:'onboarding_failed'};" : `${options.missingArtifact ? "" : `evidence.artifact=${JSON.stringify(artifact)};`}evidence.checks.push(...['fresh-workspace','ui-connection-and-native-agent','first-task-complete','downloadable-artifact-verified'${options.missingCheck ? "" : ",'native-trace-verified'"}].map(id=>({id,passed:true})));onboarded=true;return context;`}},
|
||||
async onboard(evidence){${options.preAcceptanceFailure ? `evidence.onboardingPreAcceptance=${JSON.stringify(preAcceptanceEvidence)};throw {outcome:'failed',code:'onboarding_work_before_acceptance'};` : options.onboardingFailure ? "throw {outcome:'failed',code:'onboarding_failed'};" : `${options.missingArtifact ? "" : `evidence.artifact=${JSON.stringify(artifact)};`}evidence.checks.push(...['fresh-workspace','ui-connection-and-native-agent','first-task-complete','downloadable-artifact-verified'${options.missingCheck ? "" : ",'native-trace-verified'"}].map(id=>({id,passed:true})));onboarded=true;return context;`}},
|
||||
async prepare(){throw Error('onboarding receipt was not reused')},
|
||||
async identity(){return {...identity,commit:'${(options.drift?"b":"a").repeat(40)}'}},
|
||||
async collect(){${options.collectionFailure ? "throw Error('private collection error');" : options.missingReceipt ? "return [];" : `return onboarded?[{journeyId:'cloud-onboarding',runIds:['run-1'],usage:{runs:[{runId:'run-1',usage:{costUsd:0.01}}]},operations:{campaignId:input.campaignId,attemptId:input.campaignId+':cloud-onboarding:1',modelCostUsd:0.01,setupCostUsd:0.005,requestCount:${options.zeroRequests ? 0 : 2},feedback:{collection:'complete',complaints:0,suggestions:0},traceIds:['a'.repeat(32)]}}]:[];`}}, async own(){}, async release(){${options.cleanupFailure ? "throw Error('private raw error');" : "return 'released';"}}, async stop(){}
|
||||
@@ -109,6 +113,17 @@ describe("product journey campaign boundary outcomes",()=>{
|
||||
expect(campaign.results.map((r:any)=>r.productJourney.outcome)).toEqual(["failed","blocked"]);
|
||||
expect(campaign.results[0].productJourney.phases.find((p:any)=>p.phase==="journey").outcome).toBe("failed");
|
||||
});
|
||||
it("preserves failed pre-acceptance evidence in normalized results and blocks dependent reuse", async () => {
|
||||
const campaign = await run({ preAcceptanceFailure: true });
|
||||
const [onboarding, dependent] = campaign.results;
|
||||
expect(campaign.results.map((row:any) => row.productJourney.outcome)).toEqual(["failed", "blocked"]);
|
||||
expect(onboarding).toMatchObject({ status: "failed", failureClass: "candidate_failure", error: "onboarding_work_before_acceptance" });
|
||||
expect(onboarding.productJourney.onboardingPreAcceptance).toEqual(preAcceptanceEvidence);
|
||||
expect(() => validateOnboardingPreAcceptanceEvidence(onboarding.productJourney.onboardingPreAcceptance)).not.toThrow();
|
||||
expect(onboarding.productJourney.phases).toContainEqual({ phase: "journey", outcome: "failed", code: "onboarding_work_before_acceptance" });
|
||||
expect(dependent.error).toBe("prerequisite_journey_failed");
|
||||
expect(dependent.productJourney.onboardingPreAcceptance).toBeUndefined();
|
||||
});
|
||||
it("reuses a verified onboarding context for the same shared story",async()=>{
|
||||
const campaign=await run();
|
||||
expect(campaign.results.map((r:any)=>r.status)).toEqual(["passed","passed"]);
|
||||
|
||||
@@ -9,7 +9,7 @@ export function preparationPreview(config: TargetConfig) {
|
||||
target: { id: config.id, mode: config.mode, executionEnvironment: config.executionEnvironment },
|
||||
journeys: config.journeys.map(id => productJourneys.find(journey => journey.id === id)!),
|
||||
configuration: config.baseline,
|
||||
lifetime: config.mode === "existing-cloud" ? "persistent; clean run-owned resources only" : config.mode === "new-cloud" ? "disposable; delete on pass, retain failures for 24 hours" : "managed instance; remove after every attempt",
|
||||
lifetime: config.mode === "existing-cloud" ? "persistent; clean run-owned resources only" : config.mode === "new-cloud" ? "disposable; delete on pass, retain failures up to 24 hours" : "managed instance; remove after every attempt",
|
||||
limits: config.limits, authentication: config.mode === "managed-self-hosted" ? "local-trusted" : "normal email-code sign-in",
|
||||
modelProfileId: config.modelProfileId ?? null,
|
||||
maxDisposableStacks: config.mode === "new-cloud" ? 1 : 0,
|
||||
|
||||
@@ -2,6 +2,8 @@ import { describe, expect, it, vi } from "vitest";
|
||||
import { cloudOnboarding, type CloudOnboardingInput } from "./cloud-onboarding.js";
|
||||
import { onboardingFirstTask } from "./onboarding-first-task.js";
|
||||
import { retainOnboardingArtifact } from "./artifact-evidence.js";
|
||||
import { JourneyError } from "./types.js";
|
||||
import type { OnboardingPreAcceptanceEvidence } from "./onboarding-pre-acceptance.js";
|
||||
|
||||
vi.mock("@playwright/test", () => ({ expect: (locator: { isVisible(): Promise<boolean> }) => ({
|
||||
toBeVisible: async () => { expect(await locator.isVisible()).toBe(true); },
|
||||
@@ -118,6 +120,20 @@ function fixture(entry: Entry, options: { intentFailure?: boolean; qualification
|
||||
}
|
||||
|
||||
describe("Cloud onboarding visible controls", () => {
|
||||
it("retains pre-acceptance evidence while preserving the failed error and withholding completion markers", async () => {
|
||||
const f = fixture("portfolio");
|
||||
const receipt: OnboardingPreAcceptanceEvidence = { version: 1, checkpoint: "awaiting-proposal", violations: ["attachment_present"],
|
||||
taskCount: 1, attachmentCount: 1, documentCount: 0, nonPlanningDocumentCount: 0,
|
||||
firstTaskMatchesCanonicalParent: true, firstTaskStatus: "in_review", canonicalParentStatus: "in_review" };
|
||||
vi.mocked(onboardingFirstTask).mockImplementationOnce(async input => {
|
||||
input.onPreAcceptanceFailure?.(receipt);
|
||||
throw new JourneyError("failed", "onboarding_work_before_acceptance");
|
||||
});
|
||||
await expect(cloudOnboarding(f.input)).rejects.toMatchObject({ outcome: "failed", code: "onboarding_work_before_acceptance" });
|
||||
expect(JSON.parse(JSON.stringify(f.input.evidence)).onboardingPreAcceptance).toEqual(receipt);
|
||||
expect(f.input.evidence.checks.map(check => check.id)).toEqual(["fresh-workspace", "ui-connection-and-native-agent"]);
|
||||
expect(f.events).not.toContain("native:verified");
|
||||
});
|
||||
it.each(["first-run", "portfolio", "create-form"] as const)("creates from %s with intent before dispatch and qualification before paid Connect", async entry => {
|
||||
const f = fixture(entry);
|
||||
const result = await cloudOnboarding(f.input);
|
||||
|
||||
@@ -83,7 +83,8 @@ export async function cloudOnboarding(input: CloudOnboardingInput): Promise<Jour
|
||||
await input.phase("agent");
|
||||
const completed = await onboardingFirstTask({ context, agent: { id: agent.id }, connectionId: connection.id,
|
||||
proof: createConnectionProof(), campaignId, maxRuns: input.maxRuns, deadlineAt: input.deadlineAt,
|
||||
own: input.own, observeRun: input.observeRun, onArtifact: artifact => { evidence.artifact = artifact; } });
|
||||
own: input.own, observeRun: input.observeRun, onArtifact: artifact => { evidence.artifact = artifact; },
|
||||
onPreAcceptanceFailure: receipt => { evidence.onboardingPreAcceptance = receipt; } });
|
||||
await input.verifyExecution(context, [completed.artifact.originatingRunId]);
|
||||
evidence.checks.push({ id: "first-task-complete", passed: true },
|
||||
{ id: "downloadable-artifact-verified", passed: true }, { id: "native-trace-verified", passed: true });
|
||||
|
||||
@@ -99,7 +99,7 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri
|
||||
try { sha = execFileSync("git", ["rev-parse", "HEAD"], { cwd: repositoryRoot, encoding: "utf8", stdio: ["ignore", "pipe", "ignore"] }).trim(); } catch { /* Packaged harnesses may not include Git metadata. */ }
|
||||
const source = resolveRunnerE2ESource({ sha, ref: null, workflowRunUrl: null });
|
||||
const definitionHash = createHash("sha256").update(JSON.stringify(productJourneys));
|
||||
for (const name of ["catalog.ts", "task-properties.ts", "cloud-onboarding.ts", "onboarding-connect.ts", "onboarding-first-task.ts", "artifact-evidence.ts", "fixtures.ts", "launch.ts", "types.ts", "collection.ts", "local-target.ts"]) definitionHash.update(await readFile(path.join(import.meta.dirname, name)));
|
||||
for (const name of ["catalog.ts", "task-properties.ts", "cloud-onboarding.ts", "onboarding-connect.ts", "onboarding-first-task.ts", "onboarding-pre-acceptance.ts", "artifact-evidence.ts", "fixtures.ts", "launch.ts", "types.ts", "collection.ts", "local-target.ts"]) definitionHash.update(await readFile(path.join(import.meta.dirname, name)));
|
||||
const suiteDefinitionHash = definitionHash.digest("hex");
|
||||
const expected = config.journeys.map(id => `product-journeys.${config.id}.${id === "cloud-onboarding" ? config.modelProfileId : "browser"}.${config.executionEnvironment}.${id}`);
|
||||
let reportPersisted = false;
|
||||
|
||||
@@ -4,6 +4,7 @@ import { createConnectionProof } from "../connection-evidence.js";
|
||||
import { connectionDelay } from "../connection-target.js";
|
||||
import { onboardingFirstTask, type OnboardingFirstTaskInput } from "./onboarding-first-task.js";
|
||||
import { validateOnboardingArtifact } from "./artifact-evidence.js";
|
||||
import { validateOnboardingPreAcceptanceEvidence } from "./onboarding-pre-acceptance.js";
|
||||
|
||||
vi.mock("../connection-target.js", () => ({ connectionDelay: vi.fn(async () => {}) }));
|
||||
type Row = { id: string; [key: string]: any };
|
||||
@@ -92,6 +93,90 @@ function fixture(options: Options = {}) {
|
||||
afterEach(() => { vi.restoreAllMocks(); vi.mocked(connectionDelay).mockReset().mockResolvedValue(undefined); });
|
||||
|
||||
describe("Cloud canonical onboarding first task", () => {
|
||||
it.each([
|
||||
["unexpected task set", "unexpected_task_set", (state: Parameters<NonNullable<Options["mutate"]>>[1]) => { state.tasks.push({ id: "private-early-task", title: "private task text", status: "todo" }); }, { taskCount: 2 }],
|
||||
["attachment", "attachment_present", (state: Parameters<NonNullable<Options["mutate"]>>[1]) => { state.attachments.push({ id: "private-attachment", issueId: "parent", originalFilename: "private-file.txt" }); }, { attachmentCount: 1 }],
|
||||
["non-planning document", "non_planning_document_present", (state: Parameters<NonNullable<Options["mutate"]>>[1]) => { state.documents.push({ id: "private-document", issueId: "parent", key: "private-result-key", body: "private document body" }); }, { documentCount: 1, nonPlanningDocumentCount: 1 }],
|
||||
["first task done", "first_task_done", (state: Parameters<NonNullable<Options["mutate"]>>[1]) => { state.tasks[0]!.status = "done"; }, { firstTaskStatus: "done", canonicalParentStatus: "done" }],
|
||||
] as const)("retains the observed %s violation at the failing acceptance boundary", async (_label, violation, mutate, observed) => {
|
||||
const f = fixture({ mutate(phase, state) { if (phase === "proposal") mutate(state); } });
|
||||
const capture = vi.fn();
|
||||
f.input.onPreAcceptanceFailure = capture;
|
||||
await expect(onboardingFirstTask(f.input)).rejects.toMatchObject({ outcome: "failed", code: "onboarding_work_before_acceptance" });
|
||||
expect(capture).toHaveBeenCalledExactlyOnceWith({ version: 1, checkpoint: "awaiting-proposal", violations: [violation],
|
||||
taskCount: 1, attachmentCount: 0, documentCount: 0, nonPlanningDocumentCount: 0,
|
||||
firstTaskMatchesCanonicalParent: true, firstTaskStatus: "in_review", canonicalParentStatus: "in_review", ...observed });
|
||||
expect(() => validateOnboardingPreAcceptanceEvidence(JSON.parse(JSON.stringify(capture.mock.calls[0]![0])))).not.toThrow();
|
||||
const retained = JSON.stringify(capture.mock.calls[0]![0]);
|
||||
for (const text of ["private-", "parent", "proposal-run", "qa.example.test", "connection", "campaign-1"]) expect(retained).not.toContain(text);
|
||||
expect(f.events).not.toContain("click:Approve");
|
||||
});
|
||||
it("retains every simultaneous guard violation from the same failed snapshot", async () => {
|
||||
const f = fixture({ plan: true, mutate(phase, state) {
|
||||
if (phase !== "proposal") return;
|
||||
state.tasks[0]!.status = "done";
|
||||
state.tasks.push({ id: "private-early-task" });
|
||||
state.attachments.push({ id: "private-attachment", issueId: "parent" });
|
||||
state.documents.push({ id: "private-document", issueId: "parent", key: "private-result", body: "private body" });
|
||||
} });
|
||||
const capture = vi.fn();
|
||||
f.input.onPreAcceptanceFailure = capture;
|
||||
await expect(onboardingFirstTask(f.input)).rejects.toMatchObject({ code: "onboarding_work_before_acceptance" });
|
||||
expect(capture).toHaveBeenCalledExactlyOnceWith({ version: 1, checkpoint: "awaiting-proposal",
|
||||
violations: ["unexpected_task_set", "attachment_present", "non_planning_document_present", "first_task_done"],
|
||||
taskCount: 2, attachmentCount: 1, documentCount: 2, nonPlanningDocumentCount: 1,
|
||||
firstTaskMatchesCanonicalParent: true, firstTaskStatus: "done", canonicalParentStatus: "done" });
|
||||
expect(f.events).not.toContain("click:Confirm selection");
|
||||
});
|
||||
it.each(["missing", "replaced", "reordered", "unknown-status"] as const)("records closed task observations when the snapshot is %s", async mode => {
|
||||
const f = fixture({ mutate(phase, state) {
|
||||
if (phase !== "proposal") return;
|
||||
if (mode === "missing") state.tasks = [];
|
||||
if (mode === "replaced") state.tasks = [{ id: "private-replacement", status: "done" }];
|
||||
if (mode === "reordered") state.tasks.unshift({ id: "private-first", status: "done" });
|
||||
if (mode === "unknown-status") { state.tasks[0]!.status = "private future status"; state.attachments.push({ id: "private-attachment", issueId: "parent" }); }
|
||||
} });
|
||||
const capture = vi.fn();
|
||||
f.input.onPreAcceptanceFailure = capture;
|
||||
await expect(onboardingFirstTask(f.input)).rejects.toMatchObject({ code: "onboarding_work_before_acceptance" });
|
||||
expect(capture).toHaveBeenCalledOnce();
|
||||
expect(capture.mock.calls[0]![0]).toMatchObject({
|
||||
violations: mode === "unknown-status" ? ["attachment_present"] : mode === "missing" ? ["unexpected_task_set"] : ["unexpected_task_set", "first_task_done"],
|
||||
taskCount: mode === "missing" ? 0 : mode === "reordered" ? 2 : 1,
|
||||
firstTaskMatchesCanonicalParent: mode === "unknown-status",
|
||||
firstTaskStatus: mode === "missing" ? "missing" : mode === "unknown-status" ? "unknown" : "done",
|
||||
canonicalParentStatus: mode === "reordered" ? "in_review" : mode === "unknown-status" ? "unknown" : "missing",
|
||||
});
|
||||
expect(JSON.stringify(capture.mock.calls[0]![0])).not.toContain("private");
|
||||
expect(f.events).not.toContain("click:Approve");
|
||||
});
|
||||
it("retains an opening-boundary failure before the opening answer dispatches work", async () => {
|
||||
const f = fixture({ mutate(phase, state) { if (phase === "opening") state.tasks[0]!.status = "done"; } });
|
||||
const capture = vi.fn();
|
||||
f.input.onPreAcceptanceFailure = capture;
|
||||
await expect(onboardingFirstTask(f.input)).rejects.toMatchObject({ code: "onboarding_work_before_acceptance" });
|
||||
expect(capture).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({ checkpoint: "before-opening-answer", violations: ["first_task_done"] }));
|
||||
expect(f.events).not.toContain("click:Continue");
|
||||
});
|
||||
it("does not claim the first-slot Done guard when only a later canonical parent is Done", async () => {
|
||||
const f = fixture({ mutate(phase, state) {
|
||||
if (phase !== "proposal") return;
|
||||
state.tasks[0]!.status = "done";
|
||||
state.tasks.unshift({ id: "private-first", status: "todo" });
|
||||
} });
|
||||
const capture = vi.fn();
|
||||
f.input.onPreAcceptanceFailure = capture;
|
||||
await expect(onboardingFirstTask(f.input)).rejects.toMatchObject({ code: "onboarding_work_before_acceptance" });
|
||||
expect(capture).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({ violations: ["unexpected_task_set"],
|
||||
firstTaskMatchesCanonicalParent: false, firstTaskStatus: "todo", canonicalParentStatus: "done" }));
|
||||
});
|
||||
it.each([false, true])("does not retain failure snapshots during clean canonical success, plan mode %s", async plan => {
|
||||
const f = fixture({ plan });
|
||||
const capture = vi.fn();
|
||||
f.input.onPreAcceptanceFailure = capture;
|
||||
await expect(onboardingFirstTask(f.input)).resolves.toMatchObject({ onboardingIssueId: "parent", childIssueId: "child" });
|
||||
expect(capture).not.toHaveBeenCalled();
|
||||
});
|
||||
it.each(["reload", "download-link"] as const)("retains the independently verified download when later %s fails", async readbackFailure => {
|
||||
const f = fixture({ readbackFailure });
|
||||
const capture = vi.fn();
|
||||
|
||||
@@ -6,6 +6,7 @@ import { activeRuns, firstTaskCompletionSettled, type Row } from "../first-task-
|
||||
import { isBlockedUnstartedWake, isTerminalUnstartedWake } from "../non-execution-wake.js";
|
||||
import { JourneyError, type JourneyContext, type OnboardingArtifactEvidence, type TargetSession } from "./types.js";
|
||||
import { retainOnboardingArtifact } from "./artifact-evidence.js";
|
||||
import { preAcceptanceTaskStatus, preAcceptanceViolations, type OnboardingPreAcceptanceEvidence } from "./onboarding-pre-acceptance.js";
|
||||
|
||||
export interface OnboardingFirstTaskInput {
|
||||
context: JourneyContext;
|
||||
@@ -18,6 +19,7 @@ export interface OnboardingFirstTaskInput {
|
||||
own: TargetSession["own"];
|
||||
observeRun(run: Row): Promise<void>;
|
||||
onArtifact?(artifact: OnboardingArtifactEvidence): void;
|
||||
onPreAcceptanceFailure?(evidence: OnboardingPreAcceptanceEvidence): void;
|
||||
}
|
||||
export interface OnboardingFirstTaskResult {
|
||||
runIds: string[];
|
||||
@@ -91,9 +93,19 @@ export async function onboardingFirstTask(input: OnboardingFirstTaskInput): Prom
|
||||
fail("onboarding_first_task_failed");
|
||||
}
|
||||
};
|
||||
const assertBeforeAcceptance = (snapshot: Snapshot, parentId: string) => {
|
||||
if (snapshot.tasks.length !== 1 || snapshot.tasks[0]!.id !== parentId || snapshot.attachments.length ||
|
||||
snapshot.documents.some(document => !isPlanningDocument(document)) || snapshot.tasks[0]!.status === "done") fail("onboarding_work_before_acceptance");
|
||||
const assertBeforeAcceptance = (snapshot: Snapshot, parentId: string, checkpoint: OnboardingPreAcceptanceEvidence["checkpoint"]) => {
|
||||
const observed = {
|
||||
taskCount: snapshot.tasks.length, attachmentCount: snapshot.attachments.length,
|
||||
documentCount: snapshot.documents.length, nonPlanningDocumentCount: snapshot.documents.filter(document => !isPlanningDocument(document)).length,
|
||||
firstTaskMatchesCanonicalParent: snapshot.tasks[0]?.id === parentId,
|
||||
firstTaskStatus: preAcceptanceTaskStatus(snapshot.tasks[0]),
|
||||
canonicalParentStatus: preAcceptanceTaskStatus(snapshot.tasks.find(task => task.id === parentId)),
|
||||
};
|
||||
const violations = preAcceptanceViolations(observed);
|
||||
if (violations.length) {
|
||||
input.onPreAcceptanceFailure?.({ version: 1, checkpoint, violations, ...observed });
|
||||
fail("onboarding_work_before_acceptance");
|
||||
}
|
||||
assertRuns(snapshot, [parentId]);
|
||||
};
|
||||
|
||||
@@ -105,7 +117,7 @@ export async function onboardingFirstTask(input: OnboardingFirstTaskInput): Prom
|
||||
await input.own({ kind: "issue", name: parent.title, id: parent.id });
|
||||
snapshot = await load(parent.id);
|
||||
if (seen.size) fail("onboarding_opening_started_model_work");
|
||||
assertBeforeAcceptance(snapshot, parent.id);
|
||||
assertBeforeAcceptance(snapshot, parent.id, "before-opening-answer");
|
||||
const opening = snapshot.interactions.filter(row => row.kind === "ask_user_questions" && row.status === "pending");
|
||||
const questions = opening[0]?.payload?.questionSet?.questions ?? opening[0]?.payload?.questions ?? [];
|
||||
const question = questions.find((row: Row) => row.id === "first-task-opening");
|
||||
@@ -129,7 +141,7 @@ export async function onboardingFirstTask(input: OnboardingFirstTaskInput): Prom
|
||||
let proposal: Row | undefined;
|
||||
while (!proposal) {
|
||||
snapshot = await load(parent.id);
|
||||
assertBeforeAcceptance(snapshot, parent.id);
|
||||
assertBeforeAcceptance(snapshot, parent.id, "awaiting-proposal");
|
||||
const pending = snapshot.interactions.filter(row => row.status === "pending");
|
||||
const proposals = pending.filter(isProposal);
|
||||
if (proposals.length > 1 || pending.some(row => row.kind === "ask_user_questions" && row.id !== opening[0]!.id)) fail("onboarding_proposal_scope_unclear");
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { validateRetainedRunnerResult } from "../result-validation.js";
|
||||
import { renderCaseOutcome } from "../case-outcome.js";
|
||||
import type { RunnerE2EResult } from "../types.js";
|
||||
import type { OnboardingPreAcceptanceEvidence } from "./onboarding-pre-acceptance.js";
|
||||
|
||||
function result(): RunnerE2EResult {
|
||||
return { schema: "paperclip.runner-e2e.result/v2", suiteId: "product-journeys", suiteDefinitionHash: "test",
|
||||
executionId: "product-journeys.qa.runner-codex.local.cloud-onboarding", profileId: "runner-codex", environmentId: "local", caseId: "cloud-onboarding",
|
||||
provider: "codex", model: "synthetic-model", runtimeMode: "native", status: "failed", error: "onboarding_work_before_acceptance", cleanup: "passed", attempt: 1,
|
||||
startedAt: "2026-01-01T00:00:00Z", finishedAt: "2026-01-01T00:00:01Z", durationMs: 1000,
|
||||
productJourney: { outcome: "failed", startingState: "onboarding", phases: [{ phase: "journey", outcome: "failed", code: "onboarding_work_before_acceptance" }],
|
||||
checks: [], privacy: "private", resourceDisposition: "retained" } };
|
||||
}
|
||||
function preAcceptanceReceipt(): OnboardingPreAcceptanceEvidence {
|
||||
return { version: 1, checkpoint: "awaiting-proposal", violations: ["attachment_present"],
|
||||
taskCount: 1, attachmentCount: 1, documentCount: 0, nonPlanningDocumentCount: 0,
|
||||
firstTaskMatchesCanonicalParent: true, firstTaskStatus: "in_review", canonicalParentStatus: "in_review" };
|
||||
}
|
||||
function passedResult(): RunnerE2EResult {
|
||||
const retained = result();
|
||||
retained.status = "passed";
|
||||
delete retained.error;
|
||||
retained.runIds = ["run-1"];
|
||||
Object.assign(retained.productJourney!, { outcome: "passed", campaignId: "campaign", attemptId: "attempt",
|
||||
phases: [{ phase: "journey", outcome: "passed" }], resourceDisposition: "released",
|
||||
checks: ["fresh-workspace", "ui-connection-and-native-agent", "first-task-complete", "downloadable-artifact-verified", "native-trace-verified"].map(id => ({ id, passed: true })),
|
||||
deployment: { kind: "cloud", targetId: "qa", origin: "https://qa.example.test", commit: "a".repeat(40), cloudRelease: "release", runtimeImage: "image", configurationHash: "config" },
|
||||
operations: { campaignId: "campaign", attemptId: "attempt", modelCostUsd: null, setupCostUsd: null, requestCount: 1,
|
||||
feedback: { collection: "complete", complaints: 0, suggestions: 0 }, traceIds: ["trace-1"] } });
|
||||
return retained;
|
||||
}
|
||||
describe("onboarding pre-acceptance retained evidence", () => {
|
||||
it("accepts historical results without pre-acceptance evidence and preserves optional receipts in reports", () => {
|
||||
const retained = result();
|
||||
expect(() => validateRetainedRunnerResult(retained)).not.toThrow();
|
||||
expect(renderCaseOutcome(retained, true, [])).not.toContain("Onboarding pre-acceptance evidence");
|
||||
Object.assign(retained.productJourney!, { onboardingPreAcceptance: preAcceptanceReceipt() });
|
||||
const roundTrip = JSON.parse(JSON.stringify(retained));
|
||||
expect(() => validateRetainedRunnerResult(roundTrip)).not.toThrow();
|
||||
const output = renderCaseOutcome(roundTrip, true, []);
|
||||
for (const text of ["Onboarding pre-acceptance evidence", "awaiting-proposal", "attachment_present", "First task status", "Canonical parent status", "in_review", "Attachments", "Non-planning documents", "onboarding_work_before_acceptance"]) expect(output).toContain(text);
|
||||
expect(roundTrip.status).toBe("failed");
|
||||
expect(roundTrip.productJourney.outcome).toBe("failed");
|
||||
});
|
||||
it.each([
|
||||
["a passed result", passedResult],
|
||||
["another case", (): RunnerE2EResult => ({ ...result(), caseId: "task-properties",
|
||||
executionId: "product-journeys.qa.browser.local.task-properties", profileId: "browser", provider: null, model: null, runtimeMode: "none" })],
|
||||
["another suite", (): RunnerE2EResult => ({ ...result(), suiteId: "first-task", executionId: "first-task.runner-codex.local.cloud-onboarding" })],
|
||||
] as const)("rejects a valid receipt on %s after accepting the same result without it", (_label, makeResult) => {
|
||||
const retained = makeResult();
|
||||
expect(() => validateRetainedRunnerResult(retained)).not.toThrow();
|
||||
retained.productJourney!.onboardingPreAcceptance = preAcceptanceReceipt();
|
||||
expect(() => validateRetainedRunnerResult(retained)).toThrow("result.productJourney.onboardingPreAcceptance");
|
||||
});
|
||||
it.each(["passed", "blocked", "unsupported"] as const)("rejects a valid receipt on a %s journey outcome after accepting the same result without it", outcome => {
|
||||
const retained = result();
|
||||
retained.productJourney!.outcome = outcome;
|
||||
expect(() => validateRetainedRunnerResult(retained)).not.toThrow();
|
||||
retained.productJourney!.onboardingPreAcceptance = preAcceptanceReceipt();
|
||||
expect(() => validateRetainedRunnerResult(retained)).toThrow("result.productJourney.onboardingPreAcceptance");
|
||||
});
|
||||
it.each(["onboarding_artifact_evidence_missing", undefined])("rejects a valid receipt for an unrelated or absent failure code (%s)", error => {
|
||||
const retained = result();
|
||||
retained.error = error;
|
||||
expect(() => validateRetainedRunnerResult(retained)).not.toThrow();
|
||||
retained.productJourney!.onboardingPreAcceptance = preAcceptanceReceipt();
|
||||
expect(() => validateRetainedRunnerResult(retained)).toThrow("result.productJourney.onboardingPreAcceptance");
|
||||
});
|
||||
it.each([
|
||||
["raw bodies", (): unknown => ({ ...preAcceptanceReceipt(), body: "private-body-must-not-render" })],
|
||||
["task identifiers", (): unknown => ({ ...preAcceptanceReceipt(), taskIds: ["private-task"] })],
|
||||
["document keys", (): unknown => ({ ...preAcceptanceReceipt(), documentKeys: ["private-key"] })],
|
||||
["unknown version", (): unknown => ({ ...preAcceptanceReceipt(), version: 2 })],
|
||||
["unknown checkpoint", (): unknown => ({ ...preAcceptanceReceipt(), checkpoint: "private text" })],
|
||||
["unknown violation", (): unknown => ({ ...preAcceptanceReceipt(), violations: ["private text"] })],
|
||||
["duplicate violations", (): unknown => ({ ...preAcceptanceReceipt(), violations: ["attachment_present", "attachment_present"] })],
|
||||
["empty violations", (): unknown => ({ ...preAcceptanceReceipt(), violations: [] })],
|
||||
["inconsistent violations", (): unknown => ({ ...preAcceptanceReceipt(), violations: ["first_task_done"] })],
|
||||
["negative counts", (): unknown => ({ ...preAcceptanceReceipt(), attachmentCount: -1 })],
|
||||
["fractional counts", (): unknown => ({ ...preAcceptanceReceipt(), taskCount: 1.5 })],
|
||||
["unsafe counts", (): unknown => ({ ...preAcceptanceReceipt(), documentCount: Number.MAX_SAFE_INTEGER + 1 })],
|
||||
["invalid document counts", (): unknown => ({ ...preAcceptanceReceipt(), nonPlanningDocumentCount: 1 })],
|
||||
["raw task status", (): unknown => ({ ...preAcceptanceReceipt(), firstTaskStatus: "private status" })],
|
||||
["raw parent status", (): unknown => ({ ...preAcceptanceReceipt(), canonicalParentStatus: "private status" })],
|
||||
["nonboolean match", (): unknown => ({ ...preAcceptanceReceipt(), firstTaskMatchesCanonicalParent: "true" })],
|
||||
["inconsistent task match", (): unknown => ({ ...preAcceptanceReceipt(), firstTaskMatchesCanonicalParent: false })],
|
||||
["inconsistent matched statuses", (): unknown => ({ ...preAcceptanceReceipt(), canonicalParentStatus: "done" })],
|
||||
["missing fields", (): unknown => ({ version: 1 })],
|
||||
["null", (): unknown => null],
|
||||
] as const)("rejects pre-acceptance evidence containing %s", (_label, makeReceipt) => {
|
||||
const retained = result();
|
||||
Object.assign(retained.productJourney!, { onboardingPreAcceptance: makeReceipt() });
|
||||
expect(() => validateRetainedRunnerResult(retained)).toThrow("result.productJourney.onboardingPreAcceptance");
|
||||
});
|
||||
it("renders a fixed fallback for malformed pre-acceptance evidence without exposing its fields", () => {
|
||||
const retained = result();
|
||||
Object.assign(retained.productJourney!, { onboardingPreAcceptance: { ...preAcceptanceReceipt(), body: "private-body-must-not-render" } });
|
||||
const output = renderCaseOutcome(retained, false, []);
|
||||
expect(output).toContain("Onboarding pre-acceptance evidence unavailable: invalid receipt");
|
||||
expect(output).not.toContain("private-body-must-not-render");
|
||||
expect(output).not.toContain("attachment_present");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,59 @@
|
||||
import { ISSUE_STATUSES } from "../../../packages/shared/src/constants.js";
|
||||
|
||||
const checkpoints = ["before-opening-answer", "awaiting-proposal"] as const;
|
||||
const violations = ["unexpected_task_set", "attachment_present", "non_planning_document_present", "first_task_done"] as const;
|
||||
const statuses = [...ISSUE_STATUSES, "missing", "unknown"] as const;
|
||||
|
||||
/** Scalar observations from the first failing pre-acceptance guard; never raw rows. */
|
||||
export interface OnboardingPreAcceptanceEvidence {
|
||||
version: 1;
|
||||
checkpoint: typeof checkpoints[number];
|
||||
violations: Array<typeof violations[number]>;
|
||||
taskCount: number;
|
||||
attachmentCount: number;
|
||||
documentCount: number;
|
||||
nonPlanningDocumentCount: number;
|
||||
firstTaskMatchesCanonicalParent: boolean;
|
||||
firstTaskStatus: typeof statuses[number];
|
||||
canonicalParentStatus: typeof statuses[number];
|
||||
}
|
||||
|
||||
const receiptKeys = {
|
||||
version: true, checkpoint: true, violations: true, taskCount: true, attachmentCount: true,
|
||||
documentCount: true, nonPlanningDocumentCount: true, firstTaskMatchesCanonicalParent: true,
|
||||
firstTaskStatus: true, canonicalParentStatus: true,
|
||||
} satisfies Record<keyof OnboardingPreAcceptanceEvidence, true>;
|
||||
|
||||
export function preAcceptanceTaskStatus(task: { status?: unknown; [key: string]: unknown } | undefined): OnboardingPreAcceptanceEvidence["firstTaskStatus"] {
|
||||
if (!task) return "missing";
|
||||
return ISSUE_STATUSES.find(status => status === task.status) ?? "unknown";
|
||||
}
|
||||
|
||||
export function preAcceptanceViolations(observed: Pick<OnboardingPreAcceptanceEvidence,
|
||||
"taskCount" | "attachmentCount" | "nonPlanningDocumentCount" | "firstTaskMatchesCanonicalParent" | "firstTaskStatus">): OnboardingPreAcceptanceEvidence["violations"] {
|
||||
return [
|
||||
...(observed.taskCount !== 1 || !observed.firstTaskMatchesCanonicalParent ? ["unexpected_task_set" as const] : []),
|
||||
...(observed.attachmentCount ? ["attachment_present" as const] : []),
|
||||
...(observed.nonPlanningDocumentCount ? ["non_planning_document_present" as const] : []),
|
||||
...(observed.firstTaskStatus === "done" ? ["first_task_done" as const] : []),
|
||||
];
|
||||
}
|
||||
|
||||
/** Closed validation is shared by normalized-result readers and explicit report rendering. */
|
||||
export function validateOnboardingPreAcceptanceEvidence(value: unknown): asserts value is OnboardingPreAcceptanceEvidence {
|
||||
const invalid = (): never => { throw new Error("invalid_onboarding_pre_acceptance_evidence"); };
|
||||
const keys = Object.keys(receiptKeys);
|
||||
if (!value || typeof value !== "object" || Array.isArray(value) || Object.keys(value).length !== keys.length || keys.some(key => !Object.hasOwn(value, key))) invalid();
|
||||
const receipt = value as OnboardingPreAcceptanceEvidence;
|
||||
if (receipt.version !== 1 || !checkpoints.includes(receipt.checkpoint) ||
|
||||
!Array.isArray(receipt.violations) || !receipt.violations.length || receipt.violations.length > violations.length ||
|
||||
receipt.violations.some(code => !violations.includes(code)) || new Set(receipt.violations).size !== receipt.violations.length ||
|
||||
[receipt.taskCount, receipt.attachmentCount, receipt.documentCount, receipt.nonPlanningDocumentCount].some(count => !Number.isSafeInteger(count) || count < 0) ||
|
||||
receipt.nonPlanningDocumentCount > receipt.documentCount || typeof receipt.firstTaskMatchesCanonicalParent !== "boolean" ||
|
||||
!statuses.includes(receipt.firstTaskStatus) || !statuses.includes(receipt.canonicalParentStatus)) invalid();
|
||||
if (receipt.taskCount === 0 && (receipt.firstTaskStatus !== "missing" || receipt.canonicalParentStatus !== "missing" || receipt.firstTaskMatchesCanonicalParent) ||
|
||||
receipt.taskCount > 0 && receipt.firstTaskStatus === "missing" ||
|
||||
receipt.firstTaskMatchesCanonicalParent && receipt.firstTaskStatus !== receipt.canonicalParentStatus ||
|
||||
receipt.taskCount === 1 && !receipt.firstTaskMatchesCanonicalParent && receipt.canonicalParentStatus !== "missing" ||
|
||||
preAcceptanceViolations(receipt).join(",") !== receipt.violations.join(",")) invalid();
|
||||
}
|
||||
@@ -29,6 +29,7 @@ export interface JourneyEvidence {
|
||||
operations?: JourneyOperations;
|
||||
artifact?: OnboardingArtifactEvidence;
|
||||
onboardingConnect?: import("./onboarding-connect.js").OnboardingConnectEvidence;
|
||||
onboardingPreAcceptance?: import("./onboarding-pre-acceptance.js").OnboardingPreAcceptanceEvidence;
|
||||
}
|
||||
export interface OnboardingArtifactEvidence {
|
||||
schema: "paperclip.product-journey.artifact/v1";
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"inputSha256": "893496dddc21c31bd812ea106ac30d1515dda305eb1dd183007d89cf5b1bab4d",
|
||||
"outputSha256": "c7f23021335b60da0cff1dd4c75cf8b948333ef3165d7d1703eb79a589edb798",
|
||||
"patchSha256": "c7c11e899c12e6a030fe1554a8ceee5f21689a3a4f19956746e852050049cc49",
|
||||
"basis": "Reviewed corrections to existing patch hashes and the already declared yaml workspace dependency. Existing package tarball integrity entries are unchanged. The only added package is yaml@2.9.1 with its exact integrity pinned by this reviewed correction."
|
||||
}
|
||||
@@ -0,0 +1,612 @@
|
||||
--- a/pnpm-lock.yaml
|
||||
+++ b/pnpm-lock.yaml
|
||||
@@ -16,7 +16,7 @@
|
||||
|
||||
patchedDependencies:
|
||||
'@agentclientprotocol/claude-agent-acp@0.73.0':
|
||||
- hash: riqgguivxjxydoydnzpedjy5au
|
||||
+ hash: xjlrz3rllh2ztafbdiywpfh2rq
|
||||
path: patches/@agentclientprotocol__claude-agent-acp@0.73.0.patch
|
||||
'@agentclientprotocol/codex-acp@1.6.2':
|
||||
hash: 2iynuypg76nxmydiylgtffhr7e
|
||||
@@ -49,7 +49,7 @@
|
||||
hash: 55uhvnotpqyiy37rn3pqpukhei
|
||||
path: patches/embedded-postgres@18.1.0-beta.16.patch
|
||||
postgres@3.4.9:
|
||||
- hash: sryle6qd5qfmnxvoebal4gashi
|
||||
+ hash: 6q4cic6qawywsyeo2spgnjamda
|
||||
path: patches/postgres@3.4.9.patch
|
||||
|
||||
importers:
|
||||
@@ -73,7 +73,10 @@
|
||||
version: 7.0.2
|
||||
vitest:
|
||||
specifier: ^5.0.3
|
||||
- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
|
||||
+ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
|
||||
+ yaml:
|
||||
+ specifier: ^2.8.1
|
||||
+ version: 2.9.1
|
||||
|
||||
cli:
|
||||
dependencies:
|
||||
@@ -133,7 +136,7 @@
|
||||
version: 17.4.2
|
||||
drizzle-orm:
|
||||
specifier: 0.45.2
|
||||
- version: 0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi))
|
||||
+ version: 0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda))
|
||||
embedded-postgres:
|
||||
specifier: ^18.1.0-beta.16
|
||||
version: 18.1.0-beta.16(patch_hash=55uhvnotpqyiy37rn3pqpukhei)
|
||||
@@ -180,7 +183,7 @@
|
||||
dependencies:
|
||||
'@agentclientprotocol/claude-agent-acp':
|
||||
specifier: ^0.73.0
|
||||
- version: 0.73.0(patch_hash=riqgguivxjxydoydnzpedjy5au)(@anthropic-ai/sdk@0.129.0(zod@4.6.5))(@modelcontextprotocol/sdk@1.31.0(zod@4.6.5))
|
||||
+ version: 0.73.0(patch_hash=xjlrz3rllh2ztafbdiywpfh2rq)(@anthropic-ai/sdk@0.129.0(zod@4.6.5))(@modelcontextprotocol/sdk@1.31.0(zod@4.6.5))
|
||||
'@anthropic-ai/sdk':
|
||||
specifier: 0.129.0
|
||||
version: 0.129.0(zod@4.6.5)
|
||||
@@ -310,7 +313,7 @@
|
||||
version: 7.0.2
|
||||
vitest:
|
||||
specifier: ^5.0.3
|
||||
- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
|
||||
+ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
|
||||
|
||||
packages/adapters/hermes-gateway:
|
||||
dependencies:
|
||||
@@ -326,7 +329,7 @@
|
||||
version: 7.0.2
|
||||
vitest:
|
||||
specifier: ^5.0.3
|
||||
- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
|
||||
+ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
|
||||
|
||||
packages/adapters/kimi-local:
|
||||
dependencies:
|
||||
@@ -405,13 +408,13 @@
|
||||
version: link:../shared
|
||||
drizzle-orm:
|
||||
specifier: ^0.45.2
|
||||
- version: 0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi))
|
||||
+ version: 0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda))
|
||||
embedded-postgres:
|
||||
specifier: ^18.1.0-beta.16
|
||||
version: 18.1.0-beta.16(patch_hash=55uhvnotpqyiy37rn3pqpukhei)
|
||||
postgres:
|
||||
specifier: ^3.4.9
|
||||
- version: 3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)
|
||||
+ version: 3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda)
|
||||
devDependencies:
|
||||
'@types/node':
|
||||
specifier: ^24.0.0
|
||||
@@ -427,7 +430,7 @@
|
||||
version: 7.0.2
|
||||
vitest:
|
||||
specifier: ^5.0.3
|
||||
- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
|
||||
+ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
|
||||
|
||||
packages/google-sheets-mcp-server:
|
||||
dependencies:
|
||||
@@ -449,7 +452,7 @@
|
||||
version: 7.0.2
|
||||
vitest:
|
||||
specifier: ^5.0.3
|
||||
- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
|
||||
+ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
|
||||
|
||||
packages/kv-demo-mcp-server:
|
||||
dependencies:
|
||||
@@ -468,7 +471,7 @@
|
||||
version: 7.0.2
|
||||
vitest:
|
||||
specifier: ^5.0.3
|
||||
- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
|
||||
+ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
|
||||
|
||||
packages/mcp-server:
|
||||
dependencies:
|
||||
@@ -490,7 +493,7 @@
|
||||
version: 7.0.2
|
||||
vitest:
|
||||
specifier: ^5.0.3
|
||||
- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
|
||||
+ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
|
||||
|
||||
packages/paperclip-eval-kernel:
|
||||
devDependencies:
|
||||
@@ -505,7 +508,7 @@
|
||||
dependencies:
|
||||
'@agentclientprotocol/claude-agent-acp':
|
||||
specifier: 0.73.0
|
||||
- version: 0.73.0(patch_hash=riqgguivxjxydoydnzpedjy5au)(@anthropic-ai/sdk@0.129.0(zod@4.6.5))(@modelcontextprotocol/sdk@1.31.0(zod@4.6.5))
|
||||
+ version: 0.73.0(patch_hash=xjlrz3rllh2ztafbdiywpfh2rq)(@anthropic-ai/sdk@0.129.0(zod@4.6.5))(@modelcontextprotocol/sdk@1.31.0(zod@4.6.5))
|
||||
'@agentclientprotocol/codex-acp':
|
||||
specifier: 1.6.2
|
||||
version: 1.6.2(patch_hash=2iynuypg76nxmydiylgtffhr7e)
|
||||
@@ -561,7 +564,7 @@
|
||||
version: 19.2.5(@types/react@19.2.18)
|
||||
'@vitejs/plugin-react':
|
||||
specifier: ^4.7.0
|
||||
- version: 4.7.0(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15))
|
||||
+ version: 4.7.0(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)(yaml@2.9.1))
|
||||
axe-core:
|
||||
specifier: ^4.12.1
|
||||
version: 4.13.0
|
||||
@@ -579,10 +582,10 @@
|
||||
version: 7.0.2
|
||||
vite:
|
||||
specifier: ^6.1.0
|
||||
- version: 6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)
|
||||
+ version: 6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)(yaml@2.9.1)
|
||||
vitest:
|
||||
specifier: ^5.0.3
|
||||
- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15))
|
||||
+ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)(yaml@2.9.1))
|
||||
|
||||
packages/plugins/create-paperclip-plugin:
|
||||
dependencies:
|
||||
@@ -629,7 +632,7 @@
|
||||
version: 7.0.2
|
||||
vitest:
|
||||
specifier: ^5.0.3
|
||||
- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
|
||||
+ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
|
||||
|
||||
packages/plugins/examples/plugin-file-browser-example:
|
||||
dependencies:
|
||||
@@ -747,7 +750,7 @@
|
||||
version: 7.0.2
|
||||
vitest:
|
||||
specifier: ^5.0.3
|
||||
- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
|
||||
+ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
|
||||
|
||||
packages/plugins/plugin-llm-wiki:
|
||||
devDependencies:
|
||||
@@ -786,7 +789,7 @@
|
||||
version: 7.0.2
|
||||
vitest:
|
||||
specifier: ^5.0.3
|
||||
- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
|
||||
+ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
|
||||
|
||||
packages/plugins/plugin-workspace-diff:
|
||||
dependencies:
|
||||
@@ -820,7 +823,7 @@
|
||||
version: 7.0.2
|
||||
vitest:
|
||||
specifier: ^5.0.3
|
||||
- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
|
||||
+ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
|
||||
|
||||
packages/plugins/sdk:
|
||||
dependencies:
|
||||
@@ -880,7 +883,7 @@
|
||||
version: 7.0.2
|
||||
vitest:
|
||||
specifier: ^5.0.3
|
||||
- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
|
||||
+ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
|
||||
|
||||
packages/teams-catalog:
|
||||
devDependencies:
|
||||
@@ -895,7 +898,7 @@
|
||||
version: 0.5.0
|
||||
'@agentclientprotocol/claude-agent-acp':
|
||||
specifier: 0.73.0
|
||||
- version: 0.73.0(patch_hash=riqgguivxjxydoydnzpedjy5au)(@anthropic-ai/sdk@0.129.0(zod@4.4.3))(@modelcontextprotocol/sdk@1.31.0(zod@4.4.3))
|
||||
+ version: 0.73.0(patch_hash=xjlrz3rllh2ztafbdiywpfh2rq)(@anthropic-ai/sdk@0.129.0(zod@4.4.3))(@modelcontextprotocol/sdk@1.31.0(zod@4.4.3))
|
||||
'@agentclientprotocol/codex-acp':
|
||||
specifier: 1.6.2
|
||||
version: 1.6.2(patch_hash=2iynuypg76nxmydiylgtffhr7e)
|
||||
@@ -1006,7 +1009,7 @@
|
||||
version: 2.1.0(@grpc/grpc-js@1.14.5)(nice-grpc-common@2.0.4)(nice-grpc@2.1.17)
|
||||
'@vercel/connect':
|
||||
specifier: 2.3.3
|
||||
- version: 2.3.3(@chat-adapter/slack@4.39.0(patch_hash=226jj24akljccsfbw7bcxujz4u)(ai@7.0.130(zod@4.4.3))(zod@4.4.3))(@modelcontextprotocol/sdk@1.31.0(zod@4.4.3))(ai@7.0.130(zod@4.4.3))(better-auth@1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))))
|
||||
+ version: 2.3.3(@chat-adapter/slack@4.39.0(patch_hash=226jj24akljccsfbw7bcxujz4u)(ai@7.0.130(zod@4.4.3))(zod@4.4.3))(@modelcontextprotocol/sdk@1.31.0(zod@4.4.3))(ai@7.0.130(zod@4.4.3))(better-auth@1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))))
|
||||
acorn:
|
||||
specifier: 8.18.0
|
||||
version: 8.18.0
|
||||
@@ -1024,7 +1027,7 @@
|
||||
version: 3.0.1(ajv@8.20.0)
|
||||
better-auth:
|
||||
specifier: 1.7.2
|
||||
- version: 1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)))
|
||||
+ version: 1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)))
|
||||
chat:
|
||||
specifier: 4.39.0
|
||||
version: 4.39.0(ai@7.0.130(zod@4.4.3))(zod@4.4.3)
|
||||
@@ -1045,7 +1048,7 @@
|
||||
version: 17.4.2
|
||||
drizzle-orm:
|
||||
specifier: ^0.45.2
|
||||
- version: 0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi))
|
||||
+ version: 0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda))
|
||||
embedded-postgres:
|
||||
specifier: ^18.1.0-beta.16
|
||||
version: 18.1.0-beta.16(patch_hash=55uhvnotpqyiy37rn3pqpukhei)
|
||||
@@ -1167,10 +1170,10 @@
|
||||
version: 7.0.2
|
||||
vite:
|
||||
specifier: ^8.2.2
|
||||
- version: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)
|
||||
+ version: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)
|
||||
vitest:
|
||||
specifier: ^5.0.3
|
||||
- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
|
||||
+ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
|
||||
|
||||
ui:
|
||||
dependencies:
|
||||
@@ -1321,10 +1324,10 @@
|
||||
version: 10.6.0(@types/react-dom@19.2.5(@types/react@19.2.18))(@types/react@19.2.18)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))
|
||||
'@storybook/react-vite':
|
||||
specifier: 10.5.10
|
||||
- version: 10.5.10(@types/react-dom@19.2.5(@types/react@19.2.18))(@types/react@19.2.18)(esbuild@0.28.2)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(typescript@7.0.2)(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
|
||||
+ version: 10.5.10(@types/react-dom@19.2.5(@types/react@19.2.18))(@types/react@19.2.18)(esbuild@0.28.2)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(typescript@7.0.2)(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
|
||||
'@tailwindcss/vite':
|
||||
specifier: ^4.3.3
|
||||
- version: 4.3.3(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
|
||||
+ version: 4.3.3(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
|
||||
'@types/node':
|
||||
specifier: ^24.0.0
|
||||
version: 24.13.3
|
||||
@@ -1336,7 +1339,7 @@
|
||||
version: 19.2.5(@types/react@19.2.18)
|
||||
'@vitejs/plugin-react':
|
||||
specifier: ^6.1.1
|
||||
- version: 6.1.1(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
|
||||
+ version: 6.1.1(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
|
||||
storybook:
|
||||
specifier: 10.6.0
|
||||
version: 10.6.0(@types/react@19.2.18)(react@19.2.8)
|
||||
@@ -1348,10 +1351,10 @@
|
||||
version: 7.0.2
|
||||
vite:
|
||||
specifier: ^8.2.2
|
||||
- version: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)
|
||||
+ version: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)
|
||||
vitest:
|
||||
specifier: ^5.0.3
|
||||
- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
|
||||
+ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
|
||||
|
||||
packages:
|
||||
|
||||
@@ -9256,6 +9259,11 @@
|
||||
yallist@4.0.0:
|
||||
resolution: {integrity: sha512-3wdGidZyq5PB084XLES5TpOSRA3wjXAlIWMhum2kRcv/41Sn2emQ0dycQW4uZXLejwKvg6EsvbdlVL+FYEct7A==}
|
||||
|
||||
+ yaml@2.9.1:
|
||||
+ resolution: {integrity: sha512-3NxN8+78OdzbT7C/WjGsyfPAtJaN3FNDsWxv7Y7mcDsT/oOmgW8BpyQQFFBnvZE3j9Y2Sdz1ULFLezL7Eb2yFw==}
|
||||
+ engines: {node: '>= 14.6'}
|
||||
+ hasBin: true
|
||||
+
|
||||
yargs-parser@21.1.1:
|
||||
resolution: {integrity: sha512-tVpsJW7DdjecAiFpbIB1e3qxIQsE6NoPc5/eTdrbbIC4h0LVsWhnoa3g+m2HclBIujHzsxZ4VJVA+GUuc2/LBw==}
|
||||
engines: {node: '>=12'}
|
||||
@@ -9316,7 +9324,7 @@
|
||||
|
||||
'@adobe/css-tools@4.5.0': {}
|
||||
|
||||
- '@agentclientprotocol/claude-agent-acp@0.73.0(patch_hash=riqgguivxjxydoydnzpedjy5au)(@anthropic-ai/sdk@0.129.0(zod@4.4.3))(@modelcontextprotocol/sdk@1.31.0(zod@4.4.3))':
|
||||
+ '@agentclientprotocol/claude-agent-acp@0.73.0(patch_hash=xjlrz3rllh2ztafbdiywpfh2rq)(@anthropic-ai/sdk@0.129.0(zod@4.4.3))(@modelcontextprotocol/sdk@1.31.0(zod@4.4.3))':
|
||||
dependencies:
|
||||
'@agentclientprotocol/sdk': 1.4.0(zod@4.4.3)
|
||||
'@anthropic-ai/claude-agent-sdk': 0.3.286(@anthropic-ai/sdk@0.129.0(zod@4.4.3))(@modelcontextprotocol/sdk@1.31.0(zod@4.4.3))(zod@4.4.3)
|
||||
@@ -9325,7 +9333,7 @@
|
||||
- '@anthropic-ai/sdk'
|
||||
- '@modelcontextprotocol/sdk'
|
||||
|
||||
- '@agentclientprotocol/claude-agent-acp@0.73.0(patch_hash=riqgguivxjxydoydnzpedjy5au)(@anthropic-ai/sdk@0.129.0(zod@4.6.5))(@modelcontextprotocol/sdk@1.31.0(zod@4.6.5))':
|
||||
+ '@agentclientprotocol/claude-agent-acp@0.73.0(patch_hash=xjlrz3rllh2ztafbdiywpfh2rq)(@anthropic-ai/sdk@0.129.0(zod@4.6.5))(@modelcontextprotocol/sdk@1.31.0(zod@4.6.5))':
|
||||
dependencies:
|
||||
'@agentclientprotocol/sdk': 1.4.0(zod@4.4.3)
|
||||
'@anthropic-ai/claude-agent-sdk': 0.3.286(@anthropic-ai/sdk@0.129.0(zod@4.6.5))(@modelcontextprotocol/sdk@1.31.0(zod@4.6.5))(zod@4.4.3)
|
||||
@@ -9907,12 +9915,12 @@
|
||||
optionalDependencies:
|
||||
'@opentelemetry/api': 1.9.1
|
||||
|
||||
- '@better-auth/drizzle-adapter@1.7.2(@better-auth/core@1.7.2(@better-auth/utils@0.4.2)(@better-fetch/fetch@1.3.1)(@opentelemetry/api@1.9.1)(better-call@1.4.0(zod@4.4.3))(jose@6.2.10)(kysely@0.29.5)(nanostores@1.5.3))(@better-auth/utils@0.4.2)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)))':
|
||||
+ '@better-auth/drizzle-adapter@1.7.2(@better-auth/core@1.7.2(@better-auth/utils@0.4.2)(@better-fetch/fetch@1.3.1)(@opentelemetry/api@1.9.1)(better-call@1.4.0(zod@4.4.3))(jose@6.2.10)(kysely@0.29.5)(nanostores@1.5.3))(@better-auth/utils@0.4.2)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda)))':
|
||||
dependencies:
|
||||
'@better-auth/core': 1.7.2(@better-auth/utils@0.4.2)(@better-fetch/fetch@1.3.1)(@opentelemetry/api@1.9.1)(better-call@1.4.0(zod@4.4.3))(jose@6.2.10)(kysely@0.29.5)(nanostores@1.5.3)
|
||||
'@better-auth/utils': 0.4.2
|
||||
optionalDependencies:
|
||||
- drizzle-orm: 0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi))
|
||||
+ drizzle-orm: 0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda))
|
||||
|
||||
'@better-auth/kysely-adapter@1.7.2(@better-auth/core@1.7.2(@better-auth/utils@0.4.2)(@better-fetch/fetch@1.3.1)(@opentelemetry/api@1.9.1)(better-call@1.4.0(zod@4.4.3))(jose@6.2.10)(kysely@0.29.5)(nanostores@1.5.3))(@better-auth/utils@0.4.2)(kysely@0.29.5)':
|
||||
dependencies:
|
||||
@@ -10974,11 +10982,11 @@
|
||||
'@img/sharp-win32-x64@0.35.4':
|
||||
optional: true
|
||||
|
||||
- '@joshwooding/vite-plugin-react-docgen-typescript@0.7.0(typescript@7.0.2)(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))':
|
||||
+ '@joshwooding/vite-plugin-react-docgen-typescript@0.7.0(typescript@7.0.2)(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))':
|
||||
dependencies:
|
||||
glob: 13.0.6
|
||||
react-docgen-typescript: 2.4.0(typescript@7.0.2)
|
||||
- vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)
|
||||
+ vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)
|
||||
optionalDependencies:
|
||||
typescript: 7.0.2
|
||||
|
||||
@@ -13184,25 +13192,25 @@
|
||||
transitivePeerDependencies:
|
||||
- '@types/react-dom'
|
||||
|
||||
- '@storybook/builder-vite@10.5.10(esbuild@0.28.2)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))':
|
||||
- dependencies:
|
||||
- '@storybook/csf-plugin': 10.5.10(esbuild@0.28.2)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
|
||||
+ '@storybook/builder-vite@10.5.10(esbuild@0.28.2)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))':
|
||||
+ dependencies:
|
||||
+ '@storybook/csf-plugin': 10.5.10(esbuild@0.28.2)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
|
||||
storybook: 10.6.0(@types/react@19.2.18)(react@19.2.8)
|
||||
ts-dedent: 2.3.0
|
||||
- vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)
|
||||
+ vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)
|
||||
transitivePeerDependencies:
|
||||
- esbuild
|
||||
- rollup
|
||||
- webpack
|
||||
|
||||
- '@storybook/csf-plugin@10.5.10(esbuild@0.28.2)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))':
|
||||
+ '@storybook/csf-plugin@10.5.10(esbuild@0.28.2)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))':
|
||||
dependencies:
|
||||
storybook: 10.6.0(@types/react@19.2.18)(react@19.2.8)
|
||||
unplugin: 2.3.11
|
||||
optionalDependencies:
|
||||
esbuild: 0.28.2
|
||||
rollup: 4.63.1
|
||||
- vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)
|
||||
+ vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)
|
||||
|
||||
'@storybook/global@5.0.0': {}
|
||||
|
||||
@@ -13228,11 +13236,11 @@
|
||||
'@types/react': 19.2.18
|
||||
'@types/react-dom': 19.2.5(@types/react@19.2.18)
|
||||
|
||||
- '@storybook/react-vite@10.5.10(@types/react-dom@19.2.5(@types/react@19.2.18))(@types/react@19.2.18)(esbuild@0.28.2)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(typescript@7.0.2)(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))':
|
||||
- dependencies:
|
||||
- '@joshwooding/vite-plugin-react-docgen-typescript': 0.7.0(typescript@7.0.2)(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
|
||||
+ '@storybook/react-vite@10.5.10(@types/react-dom@19.2.5(@types/react@19.2.18))(@types/react@19.2.18)(esbuild@0.28.2)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(typescript@7.0.2)(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))':
|
||||
+ dependencies:
|
||||
+ '@joshwooding/vite-plugin-react-docgen-typescript': 0.7.0(typescript@7.0.2)(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
|
||||
'@rollup/pluginutils': 5.4.0(rollup@4.63.1)
|
||||
- '@storybook/builder-vite': 10.5.10(esbuild@0.28.2)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
|
||||
+ '@storybook/builder-vite': 10.5.10(esbuild@0.28.2)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
|
||||
'@storybook/react': 10.5.10(@types/react-dom@19.2.5(@types/react@19.2.18))(@types/react@19.2.18)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(typescript@7.0.2)
|
||||
empathic: 2.0.1
|
||||
magic-string: 0.30.21
|
||||
@@ -13242,7 +13250,7 @@
|
||||
resolve: 1.22.12
|
||||
storybook: 10.6.0(@types/react@19.2.18)(react@19.2.8)
|
||||
tsconfig-paths: 4.2.0
|
||||
- vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)
|
||||
+ vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)
|
||||
optionalDependencies:
|
||||
typescript: 7.0.2
|
||||
transitivePeerDependencies:
|
||||
@@ -13335,12 +13343,12 @@
|
||||
postcss-selector-parser: 6.0.10
|
||||
tailwindcss: 4.3.3
|
||||
|
||||
- '@tailwindcss/vite@4.3.3(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))':
|
||||
+ '@tailwindcss/vite@4.3.3(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))':
|
||||
dependencies:
|
||||
'@tailwindcss/node': 4.3.3
|
||||
'@tailwindcss/oxide': 4.3.3
|
||||
tailwindcss: 4.3.3
|
||||
- vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)
|
||||
+ vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)
|
||||
|
||||
'@tanstack/query-core@5.102.8': {}
|
||||
|
||||
@@ -13758,14 +13766,14 @@
|
||||
dependencies:
|
||||
execa: 5.1.1
|
||||
|
||||
- '@vercel/connect@2.3.3(@chat-adapter/slack@4.39.0(patch_hash=226jj24akljccsfbw7bcxujz4u)(ai@7.0.130(zod@4.4.3))(zod@4.4.3))(@modelcontextprotocol/sdk@1.31.0(zod@4.4.3))(ai@7.0.130(zod@4.4.3))(better-auth@1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))))':
|
||||
+ '@vercel/connect@2.3.3(@chat-adapter/slack@4.39.0(patch_hash=226jj24akljccsfbw7bcxujz4u)(ai@7.0.130(zod@4.4.3))(zod@4.4.3))(@modelcontextprotocol/sdk@1.31.0(zod@4.4.3))(ai@7.0.130(zod@4.4.3))(better-auth@1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))))':
|
||||
dependencies:
|
||||
'@vercel/oidc': 3.8.9
|
||||
optionalDependencies:
|
||||
'@chat-adapter/slack': 4.39.0(patch_hash=226jj24akljccsfbw7bcxujz4u)(ai@7.0.130(zod@4.4.3))(zod@4.4.3)
|
||||
'@modelcontextprotocol/sdk': 1.31.0(zod@4.4.3)
|
||||
ai: 7.0.130(zod@4.4.3)
|
||||
- better-auth: 1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)))
|
||||
+ better-auth: 1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)))
|
||||
|
||||
'@vercel/oidc@3.2.0': {}
|
||||
|
||||
@@ -13775,7 +13783,7 @@
|
||||
'@vercel/cli-exec': 1.0.1
|
||||
jose: 5.10.0
|
||||
|
||||
- '@vitejs/plugin-react@4.7.0(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15))':
|
||||
+ '@vitejs/plugin-react@4.7.0(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)(yaml@2.9.1))':
|
||||
dependencies:
|
||||
'@babel/core': 7.29.7
|
||||
'@babel/plugin-transform-react-jsx-self': 7.29.7(@babel/core@7.29.7)
|
||||
@@ -13783,14 +13791,14 @@
|
||||
'@rolldown/pluginutils': 1.0.0-beta.27
|
||||
'@types/babel__core': 7.20.5
|
||||
react-refresh: 0.17.0
|
||||
- vite: 6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)
|
||||
+ vite: 6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)(yaml@2.9.1)
|
||||
transitivePeerDependencies:
|
||||
- supports-color
|
||||
|
||||
- '@vitejs/plugin-react@6.1.1(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))':
|
||||
+ '@vitejs/plugin-react@6.1.1(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))':
|
||||
dependencies:
|
||||
'@rolldown/pluginutils': 1.0.1
|
||||
- vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)
|
||||
+ vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)
|
||||
|
||||
'@vitest/expect@3.2.4':
|
||||
dependencies:
|
||||
@@ -13800,23 +13808,23 @@
|
||||
chai: 5.3.3
|
||||
tinyrainbow: 2.0.0
|
||||
|
||||
- '@vitest/mocker@5.0.3(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15))':
|
||||
+ '@vitest/mocker@5.0.3(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)(yaml@2.9.1))':
|
||||
dependencies:
|
||||
'@jridgewell/trace-mapping': 0.3.31
|
||||
'@vitest/spy': 5.0.3
|
||||
estree-walker: 3.0.3
|
||||
magic-string: 1.4.3
|
||||
optionalDependencies:
|
||||
- vite: 6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)
|
||||
-
|
||||
- '@vitest/mocker@5.0.3(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))':
|
||||
+ vite: 6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)(yaml@2.9.1)
|
||||
+
|
||||
+ '@vitest/mocker@5.0.3(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))':
|
||||
dependencies:
|
||||
'@jridgewell/trace-mapping': 0.3.31
|
||||
'@vitest/spy': 5.0.3
|
||||
estree-walker: 3.0.3
|
||||
magic-string: 1.4.3
|
||||
optionalDependencies:
|
||||
- vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)
|
||||
+ vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)
|
||||
|
||||
'@vitest/pretty-format@3.2.4':
|
||||
dependencies:
|
||||
@@ -14042,10 +14050,10 @@
|
||||
|
||||
before-after-hook@4.0.0: {}
|
||||
|
||||
- better-auth@1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))):
|
||||
+ better-auth@1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))):
|
||||
dependencies:
|
||||
'@better-auth/core': 1.7.2(@better-auth/utils@0.4.2)(@better-fetch/fetch@1.3.1)(@opentelemetry/api@1.9.1)(better-call@1.4.0(zod@4.4.3))(jose@6.2.10)(kysely@0.29.5)(nanostores@1.5.3)
|
||||
- '@better-auth/drizzle-adapter': 1.7.2(@better-auth/core@1.7.2(@better-auth/utils@0.4.2)(@better-fetch/fetch@1.3.1)(@opentelemetry/api@1.9.1)(better-call@1.4.0(zod@4.4.3))(jose@6.2.10)(kysely@0.29.5)(nanostores@1.5.3))(@better-auth/utils@0.4.2)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)))
|
||||
+ '@better-auth/drizzle-adapter': 1.7.2(@better-auth/core@1.7.2(@better-auth/utils@0.4.2)(@better-fetch/fetch@1.3.1)(@opentelemetry/api@1.9.1)(better-call@1.4.0(zod@4.4.3))(jose@6.2.10)(kysely@0.29.5)(nanostores@1.5.3))(@better-auth/utils@0.4.2)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda)))
|
||||
'@better-auth/kysely-adapter': 1.7.2(@better-auth/core@1.7.2(@better-auth/utils@0.4.2)(@better-fetch/fetch@1.3.1)(@opentelemetry/api@1.9.1)(better-call@1.4.0(zod@4.4.3))(jose@6.2.10)(kysely@0.29.5)(nanostores@1.5.3))(@better-auth/utils@0.4.2)(kysely@0.29.5)
|
||||
'@better-auth/memory-adapter': 1.7.2(@better-auth/core@1.7.2(@better-auth/utils@0.4.2)(@better-fetch/fetch@1.3.1)(@opentelemetry/api@1.9.1)(better-call@1.4.0(zod@4.4.3))(jose@6.2.10)(kysely@0.29.5)(nanostores@1.5.3))(@better-auth/utils@0.4.2)
|
||||
'@better-auth/mongo-adapter': 1.7.2(@better-auth/core@1.7.2(@better-auth/utils@0.4.2)(@better-fetch/fetch@1.3.1)(@opentelemetry/api@1.9.1)(better-call@1.4.0(zod@4.4.3))(jose@6.2.10)(kysely@0.29.5)(nanostores@1.5.3))(@better-auth/utils@0.4.2)
|
||||
@@ -14063,11 +14071,11 @@
|
||||
zod: 4.4.3
|
||||
optionalDependencies:
|
||||
drizzle-kit: 0.31.10
|
||||
- drizzle-orm: 0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi))
|
||||
+ drizzle-orm: 0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda))
|
||||
pg: 8.18.0
|
||||
react: 19.2.8
|
||||
react-dom: 19.2.8(react@19.2.8)
|
||||
- vitest: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
|
||||
+ vitest: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
|
||||
transitivePeerDependencies:
|
||||
- '@cloudflare/workers-types'
|
||||
- '@opentelemetry/api'
|
||||
@@ -14654,12 +14662,12 @@
|
||||
esbuild: 0.25.12
|
||||
tsx: 4.23.15
|
||||
|
||||
- drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)):
|
||||
+ drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda)):
|
||||
optionalDependencies:
|
||||
'@opentelemetry/api': 1.9.1
|
||||
kysely: 0.29.5
|
||||
pg: 8.18.0
|
||||
- postgres: 3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)
|
||||
+ postgres: 3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda)
|
||||
|
||||
dunder-proto@1.0.1:
|
||||
dependencies:
|
||||
@@ -16542,7 +16550,7 @@
|
||||
dependencies:
|
||||
xtend: 4.0.2
|
||||
|
||||
- postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi): {}
|
||||
+ postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda): {}
|
||||
|
||||
powershell-utils@0.1.0: {}
|
||||
|
||||
@@ -17587,7 +17595,7 @@
|
||||
'@types/unist': 3.0.3
|
||||
vfile-message: 4.0.3
|
||||
|
||||
- vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15):
|
||||
+ vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)(yaml@2.9.1):
|
||||
dependencies:
|
||||
esbuild: 0.25.12
|
||||
fdir: 6.5.0(picomatch@4.0.7)
|
||||
@@ -17601,8 +17609,9 @@
|
||||
jiti: 2.7.0
|
||||
lightningcss: 1.33.0
|
||||
tsx: 4.23.15
|
||||
-
|
||||
- vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15):
|
||||
+ yaml: 2.9.1
|
||||
+
|
||||
+ vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1):
|
||||
dependencies:
|
||||
lightningcss: 1.33.0
|
||||
picomatch: 4.0.7
|
||||
@@ -17615,11 +17624,12 @@
|
||||
fsevents: 2.3.3
|
||||
jiti: 2.7.0
|
||||
tsx: 4.23.15
|
||||
-
|
||||
- vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)):
|
||||
+ yaml: 2.9.1
|
||||
+
|
||||
+ vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)(yaml@2.9.1)):
|
||||
dependencies:
|
||||
'@types/chai': 5.2.3
|
||||
- '@vitest/mocker': 5.0.3(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15))
|
||||
+ '@vitest/mocker': 5.0.3(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)(yaml@2.9.1))
|
||||
chai: 6.3.0
|
||||
es-module-lexer: 2.3.2
|
||||
expect-type: 1.4.0
|
||||
@@ -17630,7 +17640,7 @@
|
||||
tinybench: 6.2.0
|
||||
tinyexec: 1.3.1
|
||||
tinyglobby: 0.2.17
|
||||
- vite: 6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)
|
||||
+ vite: 6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)(yaml@2.9.1)
|
||||
why-is-node-running: 3.2.1
|
||||
optionalDependencies:
|
||||
'@opentelemetry/api': 1.9.1
|
||||
@@ -17639,10 +17649,10 @@
|
||||
transitivePeerDependencies:
|
||||
- msw
|
||||
|
||||
- vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)):
|
||||
+ vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)):
|
||||
dependencies:
|
||||
'@types/chai': 5.2.3
|
||||
- '@vitest/mocker': 5.0.3(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))
|
||||
+ '@vitest/mocker': 5.0.3(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))
|
||||
chai: 6.3.0
|
||||
es-module-lexer: 2.3.2
|
||||
expect-type: 1.4.0
|
||||
@@ -17653,7 +17663,7 @@
|
||||
tinybench: 6.2.0
|
||||
tinyexec: 1.3.1
|
||||
tinyglobby: 0.2.17
|
||||
- vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)
|
||||
+ vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)
|
||||
why-is-node-running: 3.2.1
|
||||
optionalDependencies:
|
||||
'@opentelemetry/api': 1.9.1
|
||||
@@ -17748,6 +17758,8 @@
|
||||
yallist@3.1.1: {}
|
||||
|
||||
yallist@4.0.0: {}
|
||||
+
|
||||
+ yaml@2.9.1: {}
|
||||
|
||||
yargs-parser@21.1.1: {}
|
||||
|
||||
@@ -7,6 +7,7 @@ import type { RunnerE2ECampaignAccounting, RunnerE2EResult } from "./types.js";
|
||||
import { DEPENDABLE_CELLS } from "./dependable-qualification.js";
|
||||
import { validateOnboardingConnectEvidence } from "./product-journeys/onboarding-connect.js";
|
||||
import { validateOnboardingArtifact } from "./product-journeys/artifact-evidence.js";
|
||||
import { validateOnboardingPreAcceptanceEvidence } from "./product-journeys/onboarding-pre-acceptance.js";
|
||||
|
||||
type Rule = (value: unknown, at: string) => void;
|
||||
const invalid = (at: string): never => {
|
||||
@@ -355,6 +356,7 @@ const fields = {
|
||||
targetId: optional(string),
|
||||
campaignId: optional(string), attemptId: optional(string),
|
||||
onboardingConnect: optional((value, at) => { try { validateOnboardingConnectEvidence(value); } catch { invalid(at); } }),
|
||||
onboardingPreAcceptance: optional((value, at) => { try { validateOnboardingPreAcceptanceEvidence(value); } catch { invalid(at); } }),
|
||||
artifact: optional((value, at) => { try { validateOnboardingArtifact(value); } catch { invalid(at); } }),
|
||||
operations: optional(closedShape({ campaignId: string, attemptId: string, modelCostUsd: nullable(number), setupCostUsd: nullable(number), requestCount: integer,
|
||||
feedback: closedShape({ collection: oneOf("complete", "partial", "unavailable"), complaints: integer, suggestions: integer }), traceIds: array(string) })),
|
||||
@@ -414,6 +416,9 @@ export function validateRetainedRunnerResult(
|
||||
const checked = value as RunnerE2EResult;
|
||||
if (checked.databaseIdentity && checked.status === "passed" && (!checked.databaseIdentity.artifactVerified || !checked.databaseIdentity.contextSha256 || !checked.databaseIdentity.engineSha256 || !checked.databaseIdentity.containerId || !checked.databaseIdentity.cluster || !["ready", "removed"].includes(checked.databaseIdentity.status))) invalid("result.databaseIdentity");
|
||||
const artifact = checked.productJourney?.artifact;
|
||||
if (checked.productJourney?.onboardingPreAcceptance && (checked.suiteId !== "product-journeys" || checked.caseId !== "cloud-onboarding" ||
|
||||
checked.status !== "failed" || checked.productJourney.outcome !== "failed" ||
|
||||
checked.error !== "onboarding_work_before_acceptance")) invalid("result.productJourney.onboardingPreAcceptance");
|
||||
if (artifact && (checked.suiteId !== "product-journeys" || checked.caseId !== "cloud-onboarding" ||
|
||||
checked.runIds && !checked.runIds.includes(artifact.originatingRunId))) invalid("result.productJourney.artifact");
|
||||
if (checked.workerTiming !== undefined) {
|
||||
|
||||
@@ -1031,7 +1031,7 @@ for (const execution of executions) {
|
||||
daytonaImage: process.env.PAPERCLIP_E2E_DAYTONA_IMAGE,
|
||||
onCompanyCreated,
|
||||
});
|
||||
await finishBudgetedEvalSetup({ executionId: execution.id, agentId: fixtures.agent.id,
|
||||
await finishBudgetedEvalSetup({ executionId: execution.id, environmentId: execution.environment.id, agentId: fixtures.agent.id,
|
||||
companyId: fixtures.company.id, deadlineAt: startedAtMs + deadlineMs, api,
|
||||
observe: observations => writeSanitizedJson(snapshotsDir, "budgeted-setup.json", { observations }, secrets) });
|
||||
await workerTiming.enter("execution");
|
||||
|
||||
@@ -1,3 +1,9 @@
|
||||
import { mkdtemp, rm } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { openEvalCostLedger } from "./cost-ledger.js";
|
||||
import { readEvalOperationsReport } from "./operations-report.js";
|
||||
import { EVAL_OVERHEAD_COMPONENTS } from "./operating-overhead.js";
|
||||
import { expect, it } from "vitest";
|
||||
import { buildRunnerCampaign } from "./history.js";
|
||||
import { runnerMatrix } from "./catalog.js";
|
||||
@@ -52,3 +58,31 @@ it("keeps incomplete accounting and trace export visible independently of a pass
|
||||
campaign.feedbackReview.decisions[0]!.disposition = "needs_investigation";
|
||||
expect(scheduledCampaignFindings(campaign)).toContain("feedback_needs_review");
|
||||
});
|
||||
|
||||
it("qualifies scheduled model receipts while retaining operating holds, but rejects missing usage and overruns", async () => {
|
||||
const { campaign } = fixture();
|
||||
const directory = await mkdtemp(path.join(tmpdir(), "scheduled-model-readiness-"));
|
||||
const file = path.join(directory, "ledger.jsonl"), now = Date.now();
|
||||
const ledger = await openEvalCostLedger(file, { campaignId: "scheduled-fixture", maxUsd: "20",
|
||||
deadlineAt: new Date(now + 60_000).toISOString(), maxRequests: 10, maxConcurrent: 1,
|
||||
overhead: { version: 1, accountingReadiness: "model_requests", reference: "synthetic-fixture", runner: "github-hosted",
|
||||
verifiedAt: new Date(now - 1000).toISOString(), expiresAt: new Date(now + 86_400_000).toISOString(),
|
||||
components: EVAL_OVERHEAD_COMPONENTS.map(component => ({ component, maxUsd: "1", reference: "synthetic-fixture" })) },
|
||||
maxAttemptsPerExecution: 1 });
|
||||
try {
|
||||
await ledger.admitAttempt(campaign.expected[0]!, "one");
|
||||
await ledger.reserve("model", { executionId: campaign.expected[0]!, attemptId: "one", phase: "setup" }, "2");
|
||||
await ledger.finish("model", "cancelled");
|
||||
campaign.operations = await readEvalOperationsReport(file);
|
||||
expect(scheduledCampaignFindings(campaign)).toContain("accounting_incomplete");
|
||||
await ledger.observe({ observationId: "model-final", requestId: "model", source: "provider", provider: "anthropic",
|
||||
currency: "USD", route: "direct", classification: "reported", costUsd: "0.5", final: true });
|
||||
campaign.operations = await readEvalOperationsReport(file);
|
||||
expect(scheduledCampaignFindings(campaign)).not.toContain("accounting_incomplete");
|
||||
expect(campaign.operations).toMatchObject({ heldUsd: "4.000000000", committedUsd: "4.500000000" });
|
||||
await ledger.observe({ observationId: "overrun", requestId: "overhead:ci_compute", source: "invoice", provider: "aws",
|
||||
currency: "USD", route: "infrastructure", classification: "reconciled", costUsd: "21", final: true });
|
||||
campaign.operations = await readEvalOperationsReport(file);
|
||||
expect(scheduledCampaignFindings(campaign)).toEqual(expect.arrayContaining(["accounting_incomplete", "accounting_over_budget"]));
|
||||
} finally { await ledger.close(); await rm(directory, { recursive: true, force: true }); }
|
||||
});
|
||||
@@ -1,5 +1,5 @@
|
||||
import { usdToUnits } from "../../packages/shared/src/money.js";
|
||||
import { evalAccountingStatus } from "./operations-report.js";
|
||||
import { evalAccountingStatus, evalReadinessStatus } from "./operations-report.js";
|
||||
import { readFile, writeFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { pathToFileURL } from "node:url";
|
||||
@@ -49,7 +49,7 @@ export function scheduledCampaignFindings(campaign: RunnerE2ECampaign): string[]
|
||||
const operations = campaign.operations;
|
||||
if (!operations || operations.status === "unavailable") findings.push("accounting_unavailable");
|
||||
else {
|
||||
if (evalAccountingStatus(operations) === "blocked") findings.push("accounting_incomplete");
|
||||
if (evalReadinessStatus(operations) === "blocked") findings.push("accounting_incomplete");
|
||||
if (usdToUnits(operations.committedUsd) > usdToUnits(operations.limitUsd)) findings.push("accounting_over_budget");
|
||||
}
|
||||
const feedback = summarizeCampaignFeedback(campaign.accounting?.attempts ?? campaign.results);
|
||||
@@ -131,6 +131,7 @@ export async function finalizeScheduledCampaign(env: NodeJS.ProcessEnv) {
|
||||
await writeFile(path.join(directory, "findings.json"), `${JSON.stringify({ version: 1, findings,
|
||||
nativeIndexVerification: indexing, gatewayIndexVerification: gatewayIndexing, invoiceReconciliation: "pending",
|
||||
accountingStatus: campaign.operations ? evalAccountingStatus(campaign.operations) : "unavailable",
|
||||
readinessStatus: campaign.operations ? evalReadinessStatus(campaign.operations) : "unavailable",
|
||||
feedbackReviewMethod: "Only exact synthetic fixture bodies with all corroborating graders are automatically dismissed; all other feedback requires review" }, null, 2)}\n`, { mode: 0o600 });
|
||||
for (const finding of findings) console.error(`::error title=Eval operations::${finding}`);
|
||||
return findings;
|
||||
|
||||
@@ -175,6 +175,7 @@ describe("runner E2E local binary resolution", () => {
|
||||
runnerBinary,
|
||||
undefined,
|
||||
"linux",
|
||||
"x64",
|
||||
),
|
||||
).toBe(runnerBinary);
|
||||
expect(
|
||||
@@ -183,6 +184,7 @@ describe("runner E2E local binary resolution", () => {
|
||||
runnerBinary,
|
||||
undefined,
|
||||
"linux",
|
||||
"x64",
|
||||
),
|
||||
).toBeUndefined();
|
||||
expect(
|
||||
@@ -201,7 +203,14 @@ describe("runner E2E local binary resolution", () => {
|
||||
"darwin",
|
||||
),
|
||||
).toBe("/cross-compiled/paperclip-runnerd");
|
||||
expect(resolvePaperclipRemoteRunnerBinaryForHarness(
|
||||
[remoteNativeExecution], runnerBinary, undefined, "linux", "arm64",
|
||||
)).toBeUndefined();
|
||||
expect(resolvePaperclipRemoteRunnerBinaryForHarness(
|
||||
[remoteNativeExecution], runnerBinary, "/cross-compiled/linux-x64-runnerd", "linux", "arm64",
|
||||
)).toBe("/cross-compiled/linux-x64-runnerd");
|
||||
});
|
||||
|
||||
});
|
||||
|
||||
describe("runner E2E provider environment", () => {
|
||||
|
||||
@@ -56,6 +56,10 @@ describe("public repository paid workflow security", () => {
|
||||
expect(manual).toContain('pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile');
|
||||
expect(manual).toContain('PAPERCLIP_RUNNER_LOCK_SHA256=$lock_sha');
|
||||
expect(manual).toContain('docker logout ghcr.io');
|
||||
expect(manual.indexOf('Apply reviewed qualification lock corrections')).toBeLessThan(manual.indexOf('Resolve source dependencies without lifecycle scripts'));
|
||||
expect(manual).toContain('node tests/runner-e2e/prepare-report-lock.mjs');
|
||||
expect(manual).toContain('if [ "$TARGET_SHA" = "$GITHUB_SHA" ]; then');
|
||||
|
||||
});
|
||||
|
||||
it("uses the reviewed master branch for the first-party trusted PR workflow", async () => {
|
||||
@@ -275,7 +279,9 @@ describe("public repository paid workflow security", () => {
|
||||
"target_ref: ${{ steps.target.outputs.ref }}",
|
||||
);
|
||||
expect(authorizeJob).toContain('echo "ref=refs/heads/$TARGET_BRANCH"');
|
||||
expect(authorizeJob).not.toContain("actions/checkout@");
|
||||
expect(authorizeJob.match(/actions\/checkout@/gu)).toHaveLength(1);
|
||||
expect(authorizeJob).toContain("ref: ${{ github.sha }}");
|
||||
expect(authorizeJob).not.toContain("ref: ${{ needs.authorize.outputs.target_sha }}");
|
||||
expect(authorizeJob).not.toContain("pnpm install");
|
||||
expect(targetLockJob).toContain("name: Resolve target pnpm lockfile");
|
||||
expect(targetLockJob).toContain("needs: authorize");
|
||||
@@ -537,8 +543,8 @@ describe("public repository paid workflow security", () => {
|
||||
/ref: \$\{\{ needs\.authorize\.outputs\.target_sha \}\}/g,
|
||||
),
|
||||
).toHaveLength(6);
|
||||
expect(fullStack.match(/ref: \$\{\{ github\.sha \}\}/g)).toHaveLength(2);
|
||||
expect(fullStack.match(/persist-credentials: false/g)).toHaveLength(8);
|
||||
expect(fullStack.match(/ref: \$\{\{ github\.sha \}\}/g)).toHaveLength(3);
|
||||
expect(fullStack.match(/persist-credentials: false/g)).toHaveLength(9);
|
||||
expect(fullStack).not.toContain("ref: ${{ inputs.target_branch }}");
|
||||
expect(fullStack).toContain(
|
||||
"PAPERCLIP_RUNNER_SOURCE_REVISION=${TARGET_SHA}",
|
||||
@@ -553,11 +559,22 @@ describe("public repository paid workflow security", () => {
|
||||
"ref: ${{ needs.authorize.outputs.target_sha }}",
|
||||
);
|
||||
expect(reportJob).not.toContain("Download resolved target lockfile");
|
||||
expect(reportJob).toContain("run: node tests/runner-e2e/prepare-report-lock.mjs");
|
||||
expect(reportJob.indexOf("Apply reviewed reporting lock corrections")).toBeLessThan(reportJob.indexOf("Install pinned reporting dependencies"));
|
||||
expect(reportJob).not.toContain("needs.target_lock.outputs");
|
||||
|
||||
expect(historyJob).toContain("ref: ${{ github.sha }}");
|
||||
expect(historyJob).not.toContain(
|
||||
"ref: ${{ needs.authorize.outputs.target_sha }}",
|
||||
);
|
||||
expect(historyJob).not.toContain("Download resolved target lockfile");
|
||||
expect(historyJob).toContain("run: node tests/runner-e2e/prepare-report-lock.mjs");
|
||||
const authorization = fullStack.split(" authorize:\n")[1]!.split(" target_lock:\n")[0]!;
|
||||
expect(authorization).toContain("ref: ${{ github.sha }}");
|
||||
expect(authorization).toContain("Preflight reviewed report dependencies before paid admission");
|
||||
expect(authorization).toContain("run: node tests/runner-e2e/prepare-report-lock.mjs");
|
||||
expect(authorization).not.toContain("environment:");
|
||||
|
||||
expect(fullStack).toContain(
|
||||
"if: always() && !cancelled() && needs.catalog.result == 'success'",
|
||||
);
|
||||
@@ -964,6 +981,12 @@ it("keeps dependency resolution and shared caches out of Runner protocol archive
|
||||
}
|
||||
const builder = jobs.find(job => job.startsWith(" build_controller:"))!;
|
||||
expect(builder).not.toMatch(/id-token: write|environment:|secrets\./u);
|
||||
const upload = builder.split(" - name: Retain the trusted controller code for every worker\n")[1]!.split("\n - ")[0]!;
|
||||
expect(upload).toContain("include-hidden-files: true");
|
||||
expect(upload).toContain("id: upload");
|
||||
expect(builder).toContain("artifact-ids: ${{ steps.upload.outputs.artifact-id }}");
|
||||
expect(builder).toContain("run: node --test .github/tests/runner-controller-artifact-transport.test.mjs");
|
||||
expect(builder).toContain("PAPERCLIP_CONTROLLER_RECEIPT_SHA256: ${{ steps.build.outputs.runtime_receipt_sha256 }}");
|
||||
const preparer = jobs.find(job => job.startsWith(" prepare_controller:"))!;
|
||||
expect(preparer).not.toMatch(/pnpm install/u);
|
||||
expect(preparer.indexOf("verify-runtime.mjs")).toBeLessThan(preparer.indexOf('hosted.mjs" prepare'));
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { mkdtemp, readFile, rm } from "node:fs/promises";
|
||||
import { mkdtemp, readFile, rm, access } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { describe, expect, it } from "vitest";
|
||||
@@ -47,3 +47,21 @@ describe("paid workflow target authorization", () => {
|
||||
expect(result.output).toContain("ref=refs/heads/development\n");
|
||||
});
|
||||
});
|
||||
|
||||
describe("qualification image dependency correction", () => {
|
||||
it.each([true, false])("applies the workflow correction only to its own source: %s", async sameSource => {
|
||||
const workflow = await readFile(new URL("../../.github/workflows/docker-runner-check.yml", import.meta.url), "utf8");
|
||||
const step = workflow.split(" - name: Apply reviewed qualification lock corrections\n")[1]!.split("\n - ")[0]!;
|
||||
const script = step.split(" run: |\n")[1]!.replace(/^ {10}/gm, "");
|
||||
const directory = await mkdtemp(path.join(os.tmpdir(), "eval-image-lock-target-"));
|
||||
try {
|
||||
const marker = path.join(directory, "called");
|
||||
const result = spawnSync("bash", ["-c", `node() { touch "$CALLED"; }\n${script}`], {
|
||||
cwd: directory, encoding: "utf8", env: { ...process.env, GITHUB_SHA: queuedSha,
|
||||
TARGET_SHA: sameSource ? queuedSha : advancedSha, CALLED: marker },
|
||||
});
|
||||
expect(result.status, result.stderr).toBe(0);
|
||||
expect(await access(marker).then(() => true, () => false)).toBe(sameSource);
|
||||
} finally { await rm(directory, { recursive: true, force: true }); }
|
||||
});
|
||||
});
|
||||
Reference in new issue
Block a user