mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-20 06:02:18 +02:00
fix(hermes): retain the owned billing receipt during Stop
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
d4eee95165
commit
ec40b1eb32
6 files changed
+100
-41
No files matched your search
@@ -617,14 +617,6 @@ fingerprint; secrets travel through the ephemeral credential channel. Implemente
|
||||
projections remain subject to per-method live qualification. See the
|
||||
[Hermes runner contract](../../packages/paperclip-runner/docs/hermes.md).
|
||||
|
||||
The Hermes runner's optional per-turn billing receipt currently recognizes
|
||||
provider-reported OpenRouter `usage.cost` only on its selected managed Chat
|
||||
Completions route. Account attribution still comes from the authoritative
|
||||
connection projection. Token totals, displayed native price estimates, and
|
||||
cumulative session costs do not prove billed spending. Incomplete wire attempts
|
||||
retain known positive subtotals and leave missing charges unpriced; other Hermes
|
||||
providers and protocols remain pending billing qualification.
|
||||
|
||||
OpenRouter connections without an explicit model list automatically load its public
|
||||
[model catalog](https://openrouter.ai/docs/api/api-reference/models/list-all-models-and-their-properties),
|
||||
ordered with `sort=most-popular`. New-agent setup and agent settings share this
|
||||
|
||||
@@ -4,31 +4,9 @@ The explicit-only [live provider connection suite](../tests/runner-e2e/PROVIDER-
|
||||
is a Product E2E workflow for fresh subscription/API-key/gateway connections,
|
||||
with attended login and independent artifact checks against local or staging targets.
|
||||
|
||||
The explicit-only [Hermes managed API suite](../tests/runner-e2e/README.md#extended-acp-harnesses-explicit-only)
|
||||
checks native completion through five selected managed API accounts on local
|
||||
and Daytona, with independent account/model attribution. Its registered cells
|
||||
remain pending until their live evidence passes; its API-only scope leaves the
|
||||
subscription and lifecycle gates separate.
|
||||
The launcher records clean Git source before credential loading for Hermes
|
||||
candidates and rejects a mismatched requested SHA. Its source receipt and normal
|
||||
result metadata identify the controller; they do not qualify runtime behavior.
|
||||
|
||||
The same guide registers the separate explicit-only `hermes-bedrock-connections`
|
||||
suite for one exact `us-east-1` inference profile on local and Daytona. Its two
|
||||
cells remain pending. They verify the explicitly selected personal grant and
|
||||
public region/protocol/model routing before and after native completion, using
|
||||
a short-lived AWS bearer credential and the existing budget/evidence pipeline.
|
||||
|
||||
Paperclip has two live eval families with different questions, owners, and
|
||||
evidence. Choose the family before selecting a model, profile, or case.
|
||||
|
||||
The explicit-only `hermes-native-interactions` Product E2E suite adds a native
|
||||
three-question batch on local and Daytona. It checks browser reconnect within
|
||||
the active run, exact native callback delivery, late duplicate rejection and a
|
||||
final reply derived from the returned reviewer text. The registered cells remain
|
||||
pending; assigned Paperclip question tools do not satisfy this native oracle.
|
||||
See the [Hermes fixture contract](../tests/runner-e2e/README.md#extended-acp-harnesses-explicit-only).
|
||||
|
||||
The explicit-only [native instruction consolidation comparison](plans/2026-10-03-native-completion-consolidation.md)
|
||||
uses six Product E2E cells per source variant. It measures the completion
|
||||
constraint reduction separately from the earlier native tool-description
|
||||
@@ -447,13 +425,3 @@ See the [suite contract](../tests/runner-e2e/README.md#conversational-confirmati
|
||||
Hiring notification accounting now also requires exact completed action attribution. Missing native/provider ID mapping is uncomparable evidence; it must not be reported as a model task regression or waived through name/order matching. The fixture waits for both known completion callbacks and settled bracketed observations, including the gap before pending outbox work becomes a wake. Strict action replay and original machine verdicts are retained separately.
|
||||
|
||||
The explicit-only [planning guidance utility comparison](../tests/runner-e2e/PLAN-TASK-GUIDANCE.md) measures task decomposition and handoffs with current, short, and disabled skills.
|
||||
|
||||
The native interaction suite also declares one local `native-question-batch-stop`
|
||||
cell. It leaves a complete native callback unanswered, retains the pending
|
||||
request, clicks the browser Stop control, and independently requires the exact
|
||||
cancelled callback/turn and audited public Stop acknowledgement. The task must
|
||||
remain In Progress, a late answer must receive HTTP 409, and the public per-turn
|
||||
process owner and observed descendants must retire before cleanup and remain
|
||||
retired through cleanup. Cancelled-run account attribution and complete reported
|
||||
OpenRouter billing are required. This is a separate pending release gate; remote
|
||||
Stop still requires its own retirement observer and actual Daytona proof.
|
||||
@@ -1416,3 +1416,40 @@ fixture attempts remain separate: one checked the late answer prematurely and
|
||||
failed during shutdown; another placed the workspace inside its isolated host
|
||||
home and was rejected before launch. Neither is live-provider proof. A fresh
|
||||
paid browser Stop campaign and the remaining release gates are still required.
|
||||
|
||||
|
||||
### 2026-10-08 native Stop projection and terminal receipt fixes
|
||||
|
||||
The original `759f7bfccd` paid Stop attempt remains failed and is not regraded.
|
||||
Its native callback and turn cancelled, its audited browser Stop returned HTTP
|
||||
200, and all nine observed process identities retired. The card stayed pending,
|
||||
generic stranded-work recovery later blocked the task, and the cancelled run's
|
||||
cost stayed unpriced. No held reservation is released from these fixes.
|
||||
|
||||
The server now projects the exact native input cancellation into its question
|
||||
card, checking company, task, agent, run, session, runner, turn, item and the
|
||||
committed creation. The pending-card compare-and-update cannot overwrite an
|
||||
answer, replay cannot create another expiration, and the transaction locks the
|
||||
task before the card. Both local and durable PRP observation use this projection.
|
||||
Unrelated historical questions and provider-loss handoffs retain their lifecycle.
|
||||
Recovery checks deliberate operator Stop before escalating an uninvokable
|
||||
assignee, so a later agent pause cannot turn that Stop into stranded work.
|
||||
|
||||
The ACP boundary had discarded Hermes's final usage notification once Stop
|
||||
aborted its control signal. Only that admitted prompt's final usage can now
|
||||
cross the cancelled boundary, with its exact native session and negotiated turn
|
||||
token, before prompt settlement. Requests and other activity remain closed.
|
||||
Stream/process closure ends the receipt window and cannot reopen it. The native
|
||||
wire ledger and completeness rules are unchanged; missing charges stay unknown.
|
||||
|
||||
Credential-free regression evidence: the paused-agent recovery test first failed,
|
||||
then its full suite passed 79 tests. Native card cancellation first failed seven
|
||||
checks, then the final suite passed 38 tests, including nine identity denials,
|
||||
replay, late-answer rejection, preserved historical answers and a concurrent
|
||||
answer/Stop race. The billing-boundary regression first failed because no receipt
|
||||
reached its owner; the affected ACP/usage suites passed all 110 tests after repair.
|
||||
Direct server and Runner TypeScript checks passed. An intermediate implementation
|
||||
scope error is retained as a failed test attempt; it is not qualification proof.
|
||||
No local Docker, Rust build, secret access or paid provider call was used for this
|
||||
repair. A fresh committed-source browser Stop attempt and complete release gates
|
||||
are still required. The PR stack remains draft and Hermes remains pending.
|
||||
@@ -116,6 +116,10 @@ extend the task's execution deadline.
|
||||
selected OpenRouter billing identity and the current native turn before
|
||||
using it for accounting. Cumulative session cost and native model-price
|
||||
estimates cannot supply this receipt.
|
||||
- During Stop, requests and steering close immediately. The same admitted
|
||||
Hermes prompt may still deliver its final usage notification with the exact
|
||||
native session and turn token before prompt settlement. Stream or process
|
||||
closure ends that receipt window. Missing charges remain unpriced.
|
||||
- Failed, interrupted, unsupported asynchronous and background delegated work
|
||||
leave measurement totals incomplete. Known positive reported subtotals survive, but
|
||||
missing charges remain unpriced. Explicit reported zero is accepted only for
|
||||
|
||||
@@ -29,6 +29,45 @@ const HANDLE: AcpRuntimeHandle = {
|
||||
};
|
||||
|
||||
describe("Codex ACPX runtime adapter", () => {
|
||||
it("accepts only the owning Hermes terminal usage after Stop and before prompt settlement", async () => {
|
||||
const pending = pendingExtensionTurn("turn-1");
|
||||
const runtime = fakeRuntime(); vi.mocked(runtime.startTurn).mockReturnValue(pending.turn);
|
||||
let created!: AcpRuntimeOptions;
|
||||
const options = openOptions(fakeCommand());
|
||||
options.profile = { ...options.profile, agent: "hermes" };
|
||||
const port = await openCodexAcpxRuntime(options, {
|
||||
createRegistry: () => registry(), createStore: () => store(), createRuntime: value => { created = value; return runtime; },
|
||||
});
|
||||
const notification = vi.fn();
|
||||
const request = vi.fn();
|
||||
const turn = port.startTurn({ text: "Work", requestId: "turn-1", onExtensionRequest: request, onExtensionNotification: notification });
|
||||
const identity = { version: 1, sessionId: "backend-1", turnToken: "00000000-0000-0000-0000-000000000001" };
|
||||
created.onExtensionNotification!("_hermes/turn_started", identity);
|
||||
await turn.cancel();
|
||||
created.onExtensionNotification!("_hermes/delegation", identity);
|
||||
await expect(created.onExtensionRequest!("_hermes/ask_questions", identity,
|
||||
{ requestId: 0, signal: new AbortController().signal })).rejects.toThrow("admitted active turn");
|
||||
for (const foreign of [{ ...identity, sessionId: "foreign" }, { ...identity, sessionId: "agent-1" }, { ...identity, turnToken: "00000000-0000-0000-0000-000000000002" }]) {
|
||||
expect(() => created.onExtensionNotification!("_hermes/usage", foreign)).not.toThrow();
|
||||
}
|
||||
expect(notification).not.toHaveBeenCalled();
|
||||
const receipt = { ...identity, tokens: "reported", cost: "unavailable", billing: {
|
||||
schema: "paperclip.usage.billing/v1", source: "provider_reported", biller: "openrouter", currency: "USD",
|
||||
complete: true, requestCount: 1, reportedRequestCount: 1, amountUsd: 0.0042, amountUsdExact: "0.004200000",
|
||||
} };
|
||||
created.onExtensionNotification!("_hermes/usage", receipt);
|
||||
expect(notification).toHaveBeenCalledExactlyOnceWith("_hermes/usage", receipt);
|
||||
expect(request).not.toHaveBeenCalled();
|
||||
await turn.closeStream();
|
||||
await turn.cancel();
|
||||
created.onExtensionNotification!("_hermes/usage", receipt);
|
||||
expect(notification).toHaveBeenCalledTimes(1);
|
||||
pending.settle(); await turn.result;
|
||||
created.onExtensionNotification!("_hermes/usage", receipt);
|
||||
expect(notification).toHaveBeenCalledTimes(1);
|
||||
await port.close({ reason: "terminal billing boundary verified" });
|
||||
});
|
||||
|
||||
it("rejects forged permission session identifiers before delegating or applying full-auto policy", async () => {
|
||||
const pending = pendingExtensionTurn("turn-1");
|
||||
const runtime = fakeRuntime(); vi.mocked(runtime.startTurn).mockReturnValue(pending.turn);
|
||||
|
||||
@@ -101,6 +101,8 @@ interface AcpxRuntimeExtensionBoundary {
|
||||
active: AcpxRuntimeExtensionTurn | null;
|
||||
sessionIds: Set<string>;
|
||||
steering: AcpxRuntimeSteeringCapability | null;
|
||||
cancelledReceiptOwner: AcpxRuntimeExtensionTurn | null;
|
||||
receiptAdmissionClosed: boolean;
|
||||
}
|
||||
|
||||
class AcpxRuntimeCloseTimeoutError extends Error {
|
||||
@@ -278,6 +280,7 @@ export async function openQualifiedAcpxRuntime(
|
||||
const extensionNotifications = new Set(extensionProfile.extensionNotifications);
|
||||
const extensionBoundary: AcpxRuntimeExtensionBoundary = {
|
||||
agent: options.profile.agent, active: null, sessionIds: new Set(), steering: null,
|
||||
cancelledReceiptOwner: null, receiptAdmissionClosed: false,
|
||||
};
|
||||
const ownsExtensionTurn = (active: AcpxRuntimeExtensionTurn, params: Record<string, unknown>): boolean =>
|
||||
extensionBoundary.active === active && !active.signal.aborted
|
||||
@@ -354,7 +357,15 @@ export async function openQualifiedAcpxRuntime(
|
||||
active.nativeTurnToken = params.turnToken;
|
||||
return;
|
||||
}
|
||||
if (!extensionNotifications.has(method) || !active?.onNotification || !ownsExtensionTurn(active, params)) return;
|
||||
// Stop revokes requests and activity immediately, but the native prompt
|
||||
// still settles its owned work and billing. Admit only its final usage
|
||||
// notification, with the exact negotiated session/token, until result
|
||||
// settlement. Stream/process closure never opens this receipt window.
|
||||
const cancelledUsage = options.profile.agent === "hermes" && method === "_hermes/usage"
|
||||
&& active !== null && extensionBoundary.cancelledReceiptOwner === active && extensionBoundary.active === active
|
||||
&& params.sessionId === active.sessionId && params.version === 1
|
||||
&& !!active.nativeTurnToken && params.turnToken === active.nativeTurnToken;
|
||||
if (!extensionNotifications.has(method) || !active?.onNotification || (!ownsExtensionTurn(active, params) && !cancelledUsage)) return;
|
||||
if (options.profile.agent === "hermes" && (params.version !== 1 || !active.nativeTurnToken || params.turnToken !== active.nativeTurnToken)) throw new Error("Hermes activity belongs to a stale turn");
|
||||
active.onNotification(method, params.sessionId === undefined ? { ...params, sessionId: active.sessionId } : params);
|
||||
},
|
||||
@@ -1204,6 +1215,8 @@ function runtimePort(
|
||||
attemptNumber: number;
|
||||
};
|
||||
}): Promise<void> {
|
||||
extensionBoundary.receiptAdmissionClosed = true;
|
||||
extensionBoundary.cancelledReceiptOwner = null;
|
||||
extensionBoundary.active?.controller.abort(new Error("ACPX runtime is closing"));
|
||||
if (runtimeClosed) return;
|
||||
if (
|
||||
@@ -1399,7 +1412,10 @@ function runtimePort(
|
||||
onRequest: input.onExtensionRequest, onNotification: input.onExtensionNotification,
|
||||
};
|
||||
extensionBoundary.active = extensionTurn;
|
||||
let receiptWindowClosed = false;
|
||||
const releaseExtensionTurn = (): void => {
|
||||
receiptWindowClosed = true;
|
||||
if (extensionBoundary.cancelledReceiptOwner === extensionTurn) extensionBoundary.cancelledReceiptOwner = null;
|
||||
controller.abort(new Error("ACPX extension turn expired"));
|
||||
if (extensionBoundary.active === extensionTurn) extensionBoundary.active = null;
|
||||
};
|
||||
@@ -1450,10 +1466,13 @@ function runtimePort(
|
||||
...guarded,
|
||||
result,
|
||||
cancel: (input) => {
|
||||
if (!extensionBoundary.receiptAdmissionClosed && !receiptWindowClosed && extensionBoundary.active === extensionTurn) extensionBoundary.cancelledReceiptOwner = extensionTurn;
|
||||
controller.abort(new Error("ACPX extension turn cancelled"));
|
||||
return guarded.cancel(input);
|
||||
},
|
||||
closeStream: (input) => {
|
||||
receiptWindowClosed = true;
|
||||
if (extensionBoundary.cancelledReceiptOwner === extensionTurn) extensionBoundary.cancelledReceiptOwner = null;
|
||||
controller.abort(new Error("ACPX extension stream closed"));
|
||||
return guarded.closeStream(input);
|
||||
},
|
||||
|
||||
Reference in new issue
Block a user