mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-24 18:19:05 +02:00
fix(hermes): settle direct API usage with complete wire receipts
Export image-owned qualification assets from cloud builds and cover Claude/OpenAI image, question and billing journeys. Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
8b5d075f0c
commit
e025ff2286
38 files changed
+1347
-120
No files matched your search
@@ -1571,7 +1571,9 @@ candidate image and the matching controller-owned provider pack described in
|
||||
The explicit-only `hermes-api-connections` suite adds ten one-turn completion
|
||||
cells: managed OpenRouter, Anthropic, OpenAI, xAI and Google API accounts on
|
||||
local and Daytona. Select one exact execution ID. The October 7 authenticated
|
||||
catalog choices are pending inference qualification, and no production default
|
||||
catalog choices are pending inference qualification; OpenAI's API cell now
|
||||
uses `gpt-6-luna`, confirmed by authenticated catalog discovery on October 9
|
||||
and covered by the reviewed direct-API rate card. No production default
|
||||
changes. Each fixture creates its personal account through the public
|
||||
Connections API, removes agent credential overrides, and independently checks
|
||||
the native run's company/task/agent scope, responsible user, selected account,
|
||||
@@ -1585,7 +1587,11 @@ The OpenRouter cell additionally waits for its public run's reported wire cost
|
||||
to settle and checks that the company remains active and the agent idle with
|
||||
no pause reason before cleanup. Its separate settlement snapshot rejects missing
|
||||
price evidence, estimates and an unpriced budget pause even when the answer is
|
||||
correct. Other providers retain completion-only coverage.
|
||||
correct. Claude and OpenAI API cells require a complete selected-account wire
|
||||
token receipt, the correct model's explicitly estimated rate-card cost and
|
||||
healthy post-settlement budgets. Their `hermes-api-settlement.json` rejects
|
||||
reported-dollar claims, missing tokens, wrong model/biller and pending or
|
||||
unpriced accounting. xAI, Google and routed Bedrock retain completion-only coverage.
|
||||
Before creating the task, public account readback must identify the selected
|
||||
connected personal account, its company and owner, and the authenticated caller.
|
||||
Both run attribution fields must match that independently recorded owner.
|
||||
@@ -1623,7 +1629,8 @@ pnpm test:e2e:runner -- --id hermes-bedrock-connections.runner-acpx-hermes-bedro
|
||||
```
|
||||
|
||||
The explicit-only `hermes-native-interactions` suite declares question/reconnect
|
||||
cells on local and Daytona, plus a separate local native Stop cell. One native Hermes
|
||||
cells on local and Daytona, plus a separate local native Stop cell, for managed
|
||||
OpenRouter, Claude API and OpenAI API profiles (nine explicit cells). One native Hermes
|
||||
`clarify` call asks an ordered batch of single choice, multiple choice with a
|
||||
custom answer, and free text. The browser reloads while the original run remains
|
||||
active, checks the same durable request and complete form, then submits all
|
||||
@@ -1634,8 +1641,9 @@ original succeeded run without a continuation. The final answer must contain
|
||||
the reviewer's undisclosed free text returned by the native callback; a
|
||||
semantic `request_human_input` call or a guessed completion cannot pass.
|
||||
|
||||
This suite uses the same managed OpenRouter account/model, public bounded
|
||||
budgets, settled reported billing, pre-credential source admission and cleanup
|
||||
This suite uses the same managed accounts/models, public bounded
|
||||
budgets, settled reported OpenRouter or estimated direct-API billing,
|
||||
pre-credential source admission and cleanup
|
||||
pipeline. It allows one attempt and records its harness source digest. Its
|
||||
browser-reconnect scope does not qualify controller restart, cancellation,
|
||||
Stop, subscriptions, or Daytona until each corresponding live check passes.
|
||||
@@ -1658,11 +1666,13 @@ mock control plane. Neither suite substitutes for the other.
|
||||
|
||||
### Hermes native image input (explicit-only)
|
||||
|
||||
`hermes-image-input` declares one local and one Daytona `image-code-complete`
|
||||
cell. Its separate candidate profile uses `google/gemini-2.5-flash-lite` through
|
||||
`hermes-image-input` declares local and Daytona `image-code-complete`
|
||||
cells for OpenRouter, Claude API and OpenAI API (six explicit cells).
|
||||
Its separate OpenRouter candidate profile uses `google/gemini-2.5-flash-lite` through
|
||||
the selected managed OpenRouter account. The public model catalog reports image
|
||||
input and tools; this metadata is preparation, not inference qualification.
|
||||
The existing DeepSeek profile remains text-only.
|
||||
The existing DeepSeek profile remains text-only. Direct API profiles use their
|
||||
selected Claude/OpenAI models; successful inference is required to qualify image input.
|
||||
|
||||
The browser uploads a generated PNG in the ordinary task-creation flow. Its
|
||||
eight-character hexadecimal code is present only in pixels, absent from the
|
||||
@@ -1677,7 +1687,7 @@ can pass. It evaluates this gate after collecting account and billing evidence,
|
||||
so wrong image bytes or forbidden tools cannot qualify on a correct final code.
|
||||
|
||||
The suite reuses managed account attribution, pre-credential source/runtime
|
||||
admission, bounded company/agent budgets, per-run reported billing and cleanup.
|
||||
admission, bounded company/agent budgets, per-run reported or estimated billing and cleanup.
|
||||
It admits one attempt with no automatic retry. Remote selection still requires
|
||||
the verified immutable image and authorized Daytona credential. Neither target
|
||||
is qualified by the fixture's unit tests or public model discovery.
|
||||
|
||||
@@ -146,10 +146,10 @@ describe("runner E2E catalog", () => {
|
||||
expect(localIntegrityTasks).toHaveLength(2);
|
||||
expect(openRouterBreadthTasks).toHaveLength(3);
|
||||
expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
|
||||
63, 12, 6, 8, 2, 2, 2, 40, 3, 2, 10, 2, 3, 16, 16, 2, 6, 8, 46, 23, 15, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1, 116,
|
||||
63, 12, 6, 8, 2, 2, 2, 40, 9, 6, 10, 2, 3, 16, 16, 2, 6, 8, 46, 23, 15, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1, 116,
|
||||
]);
|
||||
expect(validateRunnerCatalog()).toHaveLength(725);
|
||||
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(725);
|
||||
expect(validateRunnerCatalog()).toHaveLength(735);
|
||||
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(735);
|
||||
expect(
|
||||
runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
|
||||
).toHaveLength(48);
|
||||
|
||||
+18
-11
@@ -371,11 +371,18 @@ export const hermesApiConnectionProfiles: readonly RunnerProfileFixture[] = [
|
||||
credential: choice.credential, model: choice.model,
|
||||
modelQualification: {
|
||||
source: "candidate_runner_profile",
|
||||
qualificationId: `hermes:v2026.9.24:${choice.provider}:api:catalog-2026-10-07:pending`,
|
||||
qualificationId: `hermes:v2026.9.24:${choice.provider}:api:catalog-${choice.provider === "openai" ? "2026-10-09" : "2026-10-07"}:pending`,
|
||||
},
|
||||
})),
|
||||
];
|
||||
|
||||
/** API-billed Claude/OpenAI milestone; broader connection qualification stays pending. */
|
||||
export const hermesBasicApiProfiles = hermesApiConnectionProfiles.filter(profile =>
|
||||
profile.credential === "ANTHROPIC_API_KEY" || profile.credential === "OPENAI_API_KEY");
|
||||
const hermesNativeInteractionProfiles = [
|
||||
extendedHarnessProfiles.find(profile => profile.qualificationCandidate === "hermes")!, ...hermesBasicApiProfiles,
|
||||
];
|
||||
|
||||
/** Region-bound Bedrock credentials enter only through a managed connection. */
|
||||
export const hermesBedrockConnectionProfile = nativeProfile({
|
||||
id: "runner-acpx-hermes-bedrock", label: "Hermes Bedrock (candidate)",
|
||||
@@ -1216,14 +1223,14 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
|
||||
{
|
||||
id: HERMES_NATIVE_INTERACTION_SUITE, label: "Hermes native interactions", manualOnly: true,
|
||||
description: "A native Hermes question batch receives exact answers through reconnect; browser Stop cancels an unanswered local callback and retires its owned process tree.",
|
||||
groups: ["native"], profiles: [extendedHarnessProfiles.find(profile => profile.qualificationCandidate === "hermes")!],
|
||||
groups: ["native"], profiles: hermesNativeInteractionProfiles,
|
||||
environments: runnerEnvironments, tasks: [hermesNativeQuestionTask, hermesNativeQuestionStopTask],
|
||||
excludedExecutionIds: [`${HERMES_NATIVE_INTERACTION_SUITE}.runner-acpx-hermes.daytona.native-question-batch-stop`], expectedMatrixSize: 3,
|
||||
excludedExecutionIds: hermesNativeInteractionProfiles.map(profile => `${HERMES_NATIVE_INTERACTION_SUITE}.${profile.id}.daytona.native-question-batch-stop`), expectedMatrixSize: 9,
|
||||
definitionMetadata: {
|
||||
version: 8, qualification: "pending", scheduling: "explicit-only", accountMethod: "api_key", accountMode: "responsible_user",
|
||||
version: 9, qualification: "pending", scheduling: "explicit-only", accountMethod: "api_key", accountMode: "responsible_user",
|
||||
providerTurns: 1, lifecycle: "per-turn", nativeMethod: "_hermes/ask_questions", maximumAttemptsPerCell: 1,
|
||||
objectiveAdmission: "production-delivery-guard-question-only",
|
||||
budgetMonthlyCents: HERMES_API_CONNECTION_BUDGET_CENTS, billing: "reported-cost-and-budget-health",
|
||||
budgetMonthlyCents: HERMES_API_CONNECTION_BUDGET_CENTS, billing: "reported-openrouter-or-estimated-direct-api-cost-and-budget-health",
|
||||
coverage: "native-question-batch-browser-reconnect-exact-delivery-and-local-stop",
|
||||
stopBoundary: "retained-unanswered-native-question-before-browser-click", stopProcessEvidence: "local-public-per-turn-owner-and-descendants-through-cleanup",
|
||||
remoteStopQualification: "pending-separate-remote-retirement-observer", sourceDigest: hermesApiConnectionDefinitionDigest,
|
||||
@@ -1232,16 +1239,16 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
|
||||
{
|
||||
id: HERMES_IMAGE_INPUT_SUITE, label: "Hermes native image input", manualOnly: true,
|
||||
description: "An undisclosed image-only code passes through browser upload and the native Hermes prompt, with independent content and account checks.",
|
||||
groups: ["native"], environments: runnerEnvironments, tasks: [hermesImageInputTask], expectedMatrixSize: 2,
|
||||
groups: ["native"], environments: runnerEnvironments, tasks: [hermesImageInputTask], expectedMatrixSize: 6,
|
||||
profiles: [nativeProfile({
|
||||
id: "runner-acpx-hermes-vision", label: "Hermes vision (candidate)", provider: "acpx", acpxAgent: "hermes",
|
||||
qualificationCandidate: "hermes", credential: "OPENROUTER_API_KEY", model: HERMES_IMAGE_INPUT_MODEL,
|
||||
modelQualification: { source: "candidate_runner_profile", qualificationId: "hermes:v2026.9.24:openrouter:gemini-2.5-flash-lite:vision:catalog-2026-10-08:pending" },
|
||||
})],
|
||||
definitionMetadata: { version: 2, qualification: "pending", scheduling: "explicit-only", providerTurns: 1,
|
||||
}), ...hermesBasicApiProfiles],
|
||||
definitionMetadata: { version: 3, qualification: "pending", scheduling: "explicit-only", providerTurns: 1,
|
||||
maximumAttemptsPerCell: 1, budgetMonthlyCents: HERMES_API_CONNECTION_BUDGET_CENTS, lifecycle: "per-turn",
|
||||
input: "browser-upload-authorized-native-image", oracle: "undisclosed-pixel-code-exact-bytes-no-file-tools",
|
||||
billing: "per-run-reported-cost-and-budget-health", sourceDigest: hermesApiConnectionDefinitionDigest },
|
||||
billing: "reported-openrouter-or-estimated-direct-api-cost-and-budget-health", sourceDigest: hermesApiConnectionDefinitionDigest },
|
||||
},
|
||||
{
|
||||
id: "hermes-api-connections", label: "Hermes managed API connections", manualOnly: true,
|
||||
@@ -1250,10 +1257,10 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
|
||||
tasks: [{ ...openRouterBreadthTasks.find(task => task.id === "hello-complete")!, automaticRetryPolicy: "single_attempt" }],
|
||||
expectedMatrixSize: 10,
|
||||
definitionMetadata: {
|
||||
version: 4, qualification: "pending", scheduling: "explicit-only", authenticatedDiscoveryDate: "2026-10-07",
|
||||
version: 5, qualification: "pending", scheduling: "explicit-only", authenticatedDiscoveryDate: "2026-10-07", openaiDiscoveryDate: "2026-10-09",
|
||||
accountMethod: "api_key", accountMode: "responsible_user", providerTurns: 1, expectedUserSource: "public-account-owner-before-task",
|
||||
budgetMonthlyCents: HERMES_API_CONNECTION_BUDGET_CENTS, maximumAttemptsPerCell: 1,
|
||||
coverage: "api-account-native-completion-only", sourceDigest: hermesApiConnectionDefinitionDigest,
|
||||
coverage: "api-account-native-completion-only", basicApiBilling: "complete-wire-tokens-rate-card-estimate-and-budget-health", sourceDigest: hermesApiConnectionDefinitionDigest,
|
||||
},
|
||||
},
|
||||
{
|
||||
|
||||
@@ -3,25 +3,46 @@ import { runnerMatrix, runnerSuites, suiteDefinitionHash } from "./catalog.js";
|
||||
import { buildMatrixJobs, parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js";
|
||||
import { buildRunnerE2EProcessEnvironment } from "./harness-env.js";
|
||||
import { explicitlyRequestsFileOutput, explicitlyRequestsTaskDocumentOutput } from "../../server/src/services/native-runtime/native-deliverable-feedback.js";
|
||||
import { captureHermesApiAccountOwner, captureHermesApiBudgets, captureHermesOpenRouterSettlement, gradeHermesApiConnection, isHermesOpenRouterWorkflow, isHermesConnectionSuite, HERMES_NATIVE_INTERACTION_SUITE, hasExactHermesNativeQuestionResponse, hasHermesNativeQuestionBatch, hermesNativeAnswerText, hasHermesNativeQuestionStop, hasHermesNativeQuestionStopCard, resolveHermesQualificationBudgetCents, HERMES_IMAGE_INPUT_SUITE, HERMES_IMAGE_INPUT_MODEL, hermesImageChallenge, gradeHermesImageInput } from "./hermes-api-connections.js";
|
||||
import { captureHermesApiAccountOwner, captureHermesApiBudgets, captureHermesApiSettlement, captureHermesOpenRouterSettlement, gradeHermesApiConnection, isHermesOpenRouterWorkflow, isHermesConnectionSuite, HERMES_NATIVE_INTERACTION_SUITE, hasExactHermesNativeQuestionResponse, hasHermesNativeQuestionBatch, hermesNativeAnswerText, hasHermesNativeQuestionStop, hasHermesNativeQuestionStopCard, resolveHermesQualificationBudgetCents, HERMES_IMAGE_INPUT_SUITE, HERMES_IMAGE_INPUT_MODEL, hermesImageChallenge, gradeHermesImageInput } from "./hermes-api-connections.js";
|
||||
import { inflateSync } from "node:zlib";
|
||||
|
||||
describe("Hermes direct API billing oracle", () => {
|
||||
it.each(["anthropic", "openai"] as const)("requires a scoped complete %s estimate and healthy post-run budgets", async biller => {
|
||||
const model = biller === "anthropic" ? "claude-haiku-4-5-20251001" : "gpt-6-luna";
|
||||
for (const fault of ["valid", "reported", "partial", "model", "biller", "provenance", "paused", "pending"]) {
|
||||
const company = { id: "company", status: "active", budgetMonthlyCents: 200 };
|
||||
const agent = { id: "agent", companyId: "company", status: fault === "paused" ? "paused" : "idle", pauseReason: null, budgetMonthlyCents: 200 };
|
||||
const usage = { provider: biller, biller: fault === "biller" ? "unknown" : biller, model: fault === "model" ? "foreign" : model,
|
||||
billingType: "metered_api", costStatus: fault === "reported" ? "reported" : "estimated", costUsd: 0.0042, costUsdExact: "0.004200000",
|
||||
inputTokens: 40, outputTokens: 10, accountingReceiptReady: true, accountingUsageComplete: fault !== "partial",
|
||||
pricingProvenance: { source: fault === "provenance" ? "agent_claim" : "rate_card",
|
||||
version: biller === "anthropic" ? "anthropic-standard-2026-10-09" : "openai-standard-2026-09-30" } };
|
||||
const run = { id: "run", companyId: "company", agentId: "agent", issueId: "task", status: "succeeded", usageJson: usage,
|
||||
costAccountingPending: fault === "pending", costAccountedAt: "2026-10-09T14:00:00Z" };
|
||||
const receipt = await captureHermesApiSettlement({ companyId: "company", agentId: "agent", issueId: "task", runId: "run", expectedBiller: biller, model,
|
||||
api: { async get<T>(path: string) { return (path === "/api/companies/company" ? company : path === "/api/agents/agent" ? agent : run) as T; } } });
|
||||
expect(receipt.checks.every(check => check.passed), `${biller}: ${fault}`).toBe(fault === "valid");
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("Hermes native image input", () => {
|
||||
const suite = runnerSuites.find(s => s.id === HERMES_IMAGE_INPUT_SUITE)!;
|
||||
const cells = runnerMatrix.filter(e => e.suite.id === suite.id);
|
||||
it("declares one bounded image journey per execution target on a separate vision model", () => {
|
||||
expect(cells).toHaveLength(2);
|
||||
it("declares bounded OpenRouter, Claude API and OpenAI API image journeys on both targets", () => {
|
||||
expect(cells).toHaveLength(6);
|
||||
expect(new Set(cells.map(c => c.environment.id))).toEqual(new Set(["local", "daytona"]));
|
||||
expect(suite.manualOnly).toBe(true);
|
||||
expect(isHermesConnectionSuite(suite.id)).toBe(true);
|
||||
expect(cells.every(c => c.profile.model === HERMES_IMAGE_INPUT_MODEL && c.profile.credential === "OPENROUTER_API_KEY"
|
||||
&& c.profile.qualificationCandidate === "hermes" && c.task.expectedRunCount === 1 && c.task.automaticRetryPolicy === "single_attempt")).toBe(true);
|
||||
expect(new Set(cells.map(c => c.profile.credential))).toEqual(new Set(["OPENROUTER_API_KEY", "ANTHROPIC_API_KEY", "OPENAI_API_KEY"]));
|
||||
expect(cells.every(c => c.profile.qualificationCandidate === "hermes" && c.task.expectedRunCount === 1 && c.task.automaticRetryPolicy === "single_attempt")).toBe(true);
|
||||
expect(cells.filter(c => c.profile.credential === "OPENROUTER_API_KEY").every(c => c.profile.model === HERMES_IMAGE_INPUT_MODEL)).toBe(true);
|
||||
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(c => c.suite.id === suite.id)).toBe(false);
|
||||
const env = buildRunnerE2EProcessEnvironment({}, [cells[0]!]);
|
||||
expect(JSON.parse(env.PAPERCLIP_RUNNER_ACPX_QUALIFICATION!)).toEqual([{ agent: "hermes", model: HERMES_IMAGE_INPUT_MODEL }]);
|
||||
expect(() => buildRunnerE2EProcessEnvironment({}, [{ ...cells[0]!, suite: { ...suite, manualOnly: false } }])).toThrow("explicit");
|
||||
expect(() => buildRunnerE2EProcessEnvironment({}, [{ ...cells[0]!, profile: { ...cells[0]!.profile, qualificationCandidate: "pi" } }])).toThrow("explicit");
|
||||
expect(suite.definitionMetadata).toMatchObject({ version: 2, qualification: "pending", providerTurns: 1,
|
||||
expect(suite.definitionMetadata).toMatchObject({ version: 3, qualification: "pending", providerTurns: 1,
|
||||
maximumAttemptsPerCell: 1, budgetMonthlyCents: 200 });
|
||||
});
|
||||
it("keeps the expected code out of prompt, filename and PNG metadata", () => {
|
||||
@@ -116,15 +137,15 @@ describe("Hermes native browser questions", () => {
|
||||
event("runtime_request.resolved", 3, { requestId: "request", turnId: "turn", action: "submit", response }),
|
||||
];
|
||||
const grade = (events: unknown[]) => hasExactHermesNativeQuestionResponse({ events, runId: "run", turnId: "turn", requestId: "request", questionSet, response });
|
||||
it("declares three explicit bounded single-attempt cells with local-only Stop ownership proof", () => {
|
||||
expect(cells).toHaveLength(3);
|
||||
it("declares nine explicit bounded API cells with local-only Stop ownership proof", () => {
|
||||
expect(cells).toHaveLength(9);
|
||||
expect(suite.manualOnly).toBe(true);
|
||||
expect(cells.filter(cell => cell.task.flow === "native_question_stop").map(cell => cell.environment.id)).toEqual(["local"]);
|
||||
expect(cells.filter(cell => cell.task.flow === "native_question_stop").map(cell => cell.environment.id)).toEqual(["local", "local", "local"]);
|
||||
expect(new Set(cells.map(cell => cell.environment.id))).toEqual(new Set(["local", "daytona"]));
|
||||
expect(cells.every(cell => cell.profile.qualificationCandidate === "hermes" && ["native_question_completion", "native_question_stop"].includes(cell.task.flow)
|
||||
&& cell.task.expectedRunCount === 1 && cell.task.automaticRetryPolicy === "single_attempt")).toBe(true);
|
||||
expect(suite.definitionMetadata).toMatchObject({ qualification: "pending", providerTurns: 1, maximumAttemptsPerCell: 1, budgetMonthlyCents: 200,
|
||||
lifecycle: "per-turn", nativeMethod: "_hermes/ask_questions", billing: "reported-cost-and-budget-health" });
|
||||
lifecycle: "per-turn", nativeMethod: "_hermes/ask_questions", billing: "reported-openrouter-or-estimated-direct-api-cost-and-budget-health" });
|
||||
expect(isHermesConnectionSuite(suite.id)).toBe(true);
|
||||
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(cell => cell.suite.id === suite.id)).toBe(false);
|
||||
expect(selectRunnerExecutions(parseRunnerSelectors(["--id", cells[0]!.id]))).toEqual([cells[0]]);
|
||||
|
||||
@@ -451,12 +451,14 @@ export async function runHermesNativeQuestionStop(input: {
|
||||
issueId: issue.id, connectionId: fixtures.aiConnection.connectionId, provider: fixtures.aiConnection.binding.provider,
|
||||
expectedResponsibleUserId: input.callerUserId, model: execution.profile.model, runs, expectedRunStatus: "cancelled" });
|
||||
connectionChecks.forEach(value => check(value.id, value.passed, "Public native model and account attribution remain correct after Stop."));
|
||||
let billing: Awaited<ReturnType<typeof captureHermesOpenRouterSettlement>> | undefined;
|
||||
const expectedBiller = fixtures.aiConnection.binding.provider;
|
||||
if (expectedBiller !== "openrouter" && expectedBiller !== "anthropic" && expectedBiller !== "openai") throw new Error("Unqualified Hermes Stop accounting provider");
|
||||
let billing: Awaited<ReturnType<typeof captureHermesApiSettlement>> | undefined;
|
||||
await pollUntil({ label: "cancelled Hermes run billing settlement", deadlineAt: Math.min(deadlineAt, Date.now() + 30_000), intervalMs: 200,
|
||||
load: async () => billing = await captureHermesOpenRouterSettlement({ api, companyId: fixtures.company.id, agentId: fixtures.agent.id,
|
||||
issueId: issue.id, runId: final.run.id, expectedRunStatus: "cancelled" }), accept: receipt => receipt.checks.every(value => value.passed) });
|
||||
billing!.checks.forEach(value => check(value.id, value.passed, "Cancelled native usage settles reported cost while the company and agent retain healthy budgets."));
|
||||
await input.evidence("hermes-openrouter-settlement.json", billing);
|
||||
load: async () => billing = await captureHermesApiSettlement({ api, companyId: fixtures.company.id, agentId: fixtures.agent.id,
|
||||
issueId: issue.id, runId: final.run.id, expectedRunStatus: "cancelled", expectedBiller, model: execution.profile.model }), accept: receipt => receipt.checks.every(value => value.passed) });
|
||||
billing!.checks.forEach(value => check(value.id, value.passed, "Cancelled native usage settles with explicit reported or estimated provenance while budgets remain healthy."));
|
||||
await input.evidence(expectedBiller === "openrouter" ? "hermes-openrouter-settlement.json" : "hermes-api-settlement.json", billing);
|
||||
await input.capture("final-state", "Cancelled native question remains unanswerable", "final-state.png");
|
||||
await input.evidence("api-state.json", { ...final, runs, run: final.run, checks, processes, nativeStop: identity });
|
||||
complete = true;
|
||||
@@ -500,9 +502,10 @@ export async function captureHermesApiBudgets(input: {
|
||||
}
|
||||
|
||||
/** Observe settled billing and budget health before fixture cleanup pauses the agent. */
|
||||
export async function captureHermesOpenRouterSettlement(input: {
|
||||
export async function captureHermesApiSettlement(input: {
|
||||
api: { get<T>(path: string): Promise<T> }; companyId: string; agentId: string; issueId: string; runId: string;
|
||||
expectedRunStatus?: "succeeded" | "cancelled";
|
||||
expectedBiller: "openrouter" | "anthropic" | "openai"; model?: string;
|
||||
}) {
|
||||
const [company, agent, run] = await Promise.all([
|
||||
input.api.get<Record<string, unknown>>(`/api/companies/${input.companyId}`),
|
||||
@@ -510,19 +513,23 @@ export async function captureHermesOpenRouterSettlement(input: {
|
||||
input.api.get<Record<string, unknown>>(`/api/heartbeat-runs/${input.runId}`),
|
||||
]);
|
||||
const usage = record(run.usageJson), provenance = record(usage.pricingProvenance);
|
||||
const direct = input.expectedBiller !== "openrouter";
|
||||
const expectedVersion = input.expectedBiller === "anthropic" ? "anthropic-standard-2026-10-09"
|
||||
: input.expectedBiller === "openai" ? "openai-standard-2026-09-30" : "hermes-openrouter-wire/v1";
|
||||
const cost = usage.costUsd, exact = usage.costUsdExact;
|
||||
return { observation: {
|
||||
company: { id: company.id, status: company.status, pauseReason: company.pauseReason, budgetMonthlyCents: company.budgetMonthlyCents },
|
||||
agent: { id: agent.id, companyId: agent.companyId, status: agent.status, pauseReason: agent.pauseReason, budgetMonthlyCents: agent.budgetMonthlyCents },
|
||||
run: { id: run.id, companyId: run.companyId, agentId: run.agentId, issueId: run.issueId, status: run.status,
|
||||
costAccountingPending: run.costAccountingPending, costAccountedAt: run.costAccountedAt,
|
||||
usage: Object.fromEntries(["provider", "biller", "billingType", "costStatus", "costUsd", "costUsdExact", "inputTokens", "outputTokens", "accountingReceiptReady", "pricingProvenance"].map(key => [key, usage[key]])) },
|
||||
usage: Object.fromEntries(["provider", "biller", "billingType", "model", "costStatus", "costUsd", "costUsdExact", "inputTokens", "outputTokens", "accountingReceiptReady", "accountingUsageComplete", "pricingProvenance"].map(key => [key, usage[key]])) },
|
||||
}, checks: [
|
||||
{ id: "billing-observation-scope", passed: company.id === input.companyId && agent.id === input.agentId && agent.companyId === input.companyId
|
||||
&& run.id === input.runId && run.companyId === input.companyId && run.agentId === input.agentId && run.issueId === input.issueId },
|
||||
{ id: "settled-openrouter-reported-cost", passed: run.status === (input.expectedRunStatus ?? "succeeded") && run.costAccountingPending === false && present(run.costAccountedAt)
|
||||
&& usage.accountingReceiptReady === true && usage.biller === "openrouter" && usage.billingType === "metered_api" && usage.costStatus === "reported"
|
||||
&& provenance.source === "provider_reported" && provenance.version === "hermes-openrouter-wire/v1"
|
||||
{ id: direct ? `settled-${input.expectedBiller}-estimated-cost` : "settled-openrouter-reported-cost", passed: run.status === (input.expectedRunStatus ?? "succeeded") && run.costAccountingPending === false && present(run.costAccountedAt)
|
||||
&& usage.accountingReceiptReady === true && usage.biller === input.expectedBiller && usage.billingType === "metered_api" && usage.costStatus === (direct ? "estimated" : "reported")
|
||||
&& provenance.source === (direct ? "rate_card" : "provider_reported") && provenance.version === expectedVersion
|
||||
&& (!direct || usage.provider === input.expectedBiller && usage.accountingUsageComplete === true && present(input.model) && usage.model === input.model)
|
||||
&& typeof cost === "number" && Number.isFinite(cost) && cost >= 0 && typeof exact === "string" && /^(0|[1-9][0-9]{0,6})\.[0-9]{9}$/.test(exact) && Number(exact) === cost
|
||||
&& typeof usage.inputTokens === "number" && Number.isSafeInteger(usage.inputTokens) && usage.inputTokens >= 0
|
||||
&& typeof usage.outputTokens === "number" && Number.isSafeInteger(usage.outputTokens) && usage.outputTokens >= 0 && usage.inputTokens + usage.outputTokens > 0 },
|
||||
@@ -531,6 +538,11 @@ export async function captureHermesOpenRouterSettlement(input: {
|
||||
] };
|
||||
}
|
||||
|
||||
/** Preserve the existing OpenRouter oracle's exact reported-price contract. */
|
||||
export function captureHermesOpenRouterSettlement(input: Omit<Parameters<typeof captureHermesApiSettlement>[0], "expectedBiller" | "model">) {
|
||||
return captureHermesApiSettlement({ ...input, expectedBiller: "openrouter" });
|
||||
}
|
||||
|
||||
/** Grade public run/account/model metadata; a model's completion claim cannot supply it. */
|
||||
export function gradeHermesApiConnection(input: {
|
||||
companyId: string; agentId: string; issueId: string; connectionId: string; provider: string; model: string; expectedResponsibleUserId: string;
|
||||
@@ -561,7 +573,7 @@ export function gradeHermesApiConnection(input: {
|
||||
/** Explicit authenticated catalog choices; none is a production default or live qualification. */
|
||||
export const hermesApiConnectionChoices = [
|
||||
{ provider: "anthropic", credential: "ANTHROPIC_API_KEY", model: "claude-haiku-4-5-20251001" },
|
||||
{ provider: "openai", credential: "OPENAI_API_KEY", model: "gpt-5.6-luna" },
|
||||
{ provider: "openai", credential: "OPENAI_API_KEY", model: "gpt-6-luna" },
|
||||
{ provider: "xai", credential: "XAI_API_KEY", model: "grok-4.7" },
|
||||
// Google restricts 2.5 models to prior users even when they appear in the catalog.
|
||||
// https://ai.google.dev/gemini-api/docs/deprecations
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { runPlanTaskFlow } from "./plan-task-flow.js";
|
||||
import { captureHermesApiAccountOwner, captureHermesApiBudgets, captureHermesOpenRouterSettlement, gradeHermesApiConnection, isHermesConnectionSuite, isHermesOpenRouterWorkflow, hasExactHermesNativeQuestionResponse, hasHermesNativeQuestionBatch, hermesNativeAnswerText, runHermesNativeQuestionStop, HERMES_IMAGE_INPUT_SUITE, hermesImageChallenge, gradeHermesImageInput } from "./hermes-api-connections.js";
|
||||
import { captureHermesApiAccountOwner, captureHermesApiBudgets, captureHermesApiSettlement, captureHermesOpenRouterSettlement, gradeHermesApiConnection, isHermesConnectionSuite, isHermesOpenRouterWorkflow, hasExactHermesNativeQuestionResponse, hasHermesNativeQuestionBatch, hermesNativeAnswerText, runHermesNativeQuestionStop, HERMES_IMAGE_INPUT_SUITE, hermesImageChallenge, gradeHermesImageInput } from "./hermes-api-connections.js";
|
||||
import { hasAcpxNativeOrigin } from "./acpx-native-origin.js";
|
||||
import { isDeepStrictEqual } from "node:util";
|
||||
import { assertNativeCompletionSelection, NATIVE_COMPLETION_PREFLIGHT_ENV, verifyNativeCompletionPreflight } from "./native-completion-admission.js";
|
||||
@@ -3055,20 +3055,21 @@ for (const execution of executions) {
|
||||
{ id: "account-owner-preserved", passed: after.expectedResponsibleUserId === hermesApiAccountOwner.expectedResponsibleUserId });
|
||||
await writeSanitizedJson(snapshotsDir, "hermes-api-owner-after-execution.json", after, secrets);
|
||||
}
|
||||
if (fixtures.aiConnection.binding.provider === "openrouter") {
|
||||
const settlementBiller = fixtures.aiConnection.binding.provider;
|
||||
if (settlementBiller === "openrouter" || !execution.profile.managedConnectionRouting && (settlementBiller === "anthropic" || settlementBiller === "openai")) {
|
||||
const settlementScope = { companyId: fixtures.company.id, agentId: fixtures.agent.id, issueId: issue.id,
|
||||
runId: selectedRuns[0]?.id ?? "missing" };
|
||||
let receipt: Awaited<ReturnType<typeof captureHermesOpenRouterSettlement>> | undefined;
|
||||
let receipt: Awaited<ReturnType<typeof captureHermesApiSettlement>> | undefined;
|
||||
try {
|
||||
await expect.poll(async () => {
|
||||
receipt = await captureHermesOpenRouterSettlement({ api, ...settlementScope, expectedRunStatus });
|
||||
receipt = await captureHermesApiSettlement({ api, ...settlementScope, expectedRunStatus, expectedBiller: settlementBiller, model: execution.profile.model });
|
||||
return receipt.checks.every(check => check.passed);
|
||||
}, { timeout: 30_000, message: "OpenRouter billing must settle without pausing the agent before cleanup" }).toBe(true);
|
||||
}, { timeout: 30_000, message: "API billing must settle without pausing the agent before cleanup" }).toBe(true);
|
||||
} finally {
|
||||
if (receipt) {
|
||||
await writeSanitizedJson(snapshotsDir, "hermes-openrouter-settlement.json", receipt, secrets);
|
||||
await writeSanitizedJson(snapshotsDir, settlementBiller === "openrouter" ? "hermes-openrouter-settlement.json" : "hermes-api-settlement.json", receipt, secrets);
|
||||
matcherResults.push(...receipt.checks.map(check => ({ matcher: { kind: "json_path" as const, path: `hermesBilling.${check.id}`, expected: true },
|
||||
passed: check.passed, detail: "Public reported cost and company/agent budget health must pass before teardown." })));
|
||||
passed: check.passed, detail: "Public cost attribution, its reported/estimated provenance and company/agent budget health must pass before teardown." })));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -143,6 +143,11 @@ describe("public repository paid workflow security", () => {
|
||||
expect(manual).toContain('pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile');
|
||||
expect(manual).toContain('PAPERCLIP_RUNNER_LOCK_SHA256=$lock_sha');
|
||||
expect(manual).toContain('docker logout ghcr.io');
|
||||
expect(manual).toContain('docker create --platform linux/amd64 --network none --entrypoint /bin/true "$immutable"');
|
||||
expect(manual).toContain('docker cp "$container_id:/opt/paperclip-runner/provider-pack" "$runtime_dir/provider-pack"');
|
||||
expect(manual).toContain('docker cp "$container_id:/usr/local/bin/paperclip-runnerd" "$runtime_dir/paperclip-runnerd"');
|
||||
expect(manual).toContain('sha256sum paperclip-runnerd provider-pack/provider-pack.json');
|
||||
expect(manual).toContain('sha256sum controller-runtime.tar.gz > controller-runtime.tar.gz.sha256');
|
||||
});
|
||||
|
||||
it("uses the reviewed master branch for the first-party trusted PR workflow", async () => {
|
||||
|
||||
Reference in new issue
Block a user