fix(evals): correlate canceled gateway requests with their ledger (#17354)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Product E2E evaluations use a budget ledger and a local model
gateway.
> - The gateway writes each request reservation before it calls a
provider.
> - A client can cancel before the gateway returns response headers.
> - Project-only logging context cannot then bind a late gateway log to
its reservation.
> - This change sends the reserved request identity in Braintrust log
metadata before dispatch.
> - Operators can use that identity to inspect late records without
guessing from time or session order.

## Linked Issues or Issue Description

Refs: #17009

**What happened?**

A hosted evaluation passed its product checks but retained two canceled
requests with no response headers. Later gateway logs had partial usage.
Their time and session matched, but they had no ledger request identity.

**Expected behavior**

A gateway log must carry the ledger request ID even when the client
receives no headers. A log match must not imply final billing.

**Steps to reproduce**

Run the gateway regression with a transport that receives a request and
then cancels before response headers. Before this change, both requests
have the same project-only parent header.

## What Changed

- Add a small Braintrust context encoder owned by the eval gateway
integration. It uses the existing SpanComponentsV4 project-log format
and an allowlist of request, campaign, execution, attempt and phase
metadata.
- Save the exact context in the private request receipt before dispatch.
Send it as the gateway parent header.
- Test both provider routes, repeated identical requests, cancellation
holds and direct-provider isolation.
- Document the lookup field and the limits of historical and partial
evidence.

## Verification

- The new cancellation regressions failed before the change: 2 failed,
37 passed.
- Gateway, operations and tracing checks: 82 passed.
- `pnpm test:e2e:runner:typecheck`: passed after rebuilding stale
generated plugin-SDK types from current master. The initial stale-type
failure remains recorded locally.
- Official Braintrust SDK 3.38.0 decoded the actual generated context.
Its `startSpan` implementation emitted the reserved identity in the
initial LLM log. The check captured logs in memory and forbade all
network calls.
- `git diff --check`: passed.
- Full repository CI passed on
`529446a9295f2dc6fa6499a2f976480d74626097`, including typecheck, tests,
build and both required gates. Greptile is 5/5 on that head; its sole
finding is fixed and resolved.
- The optional Paperclip review turn ended without a final response. Its
check remains queued and provides no review assessment. No passing
review is inferred from it.
- Live hosted acceptance of this header change remains unqualified; no
paid test was dispatched for this fix.

## Risks

- The encoder follows Braintrust's versioned span format. A future
vendor change can require an update. It adds no package dependency.
- A late log can have partial usage. Existing reservations remain held
until valid cost evidence resolves them.
- Historical project-only requests cannot gain the new identity
retroactively.
- Only synthetic eval identifiers enter the new metadata. Credentials,
prompts and responses are excluded. Direct provider routes keep their
current headers.

## Model Used

OpenAI Codex, GPT-6 family. The exact deployment ID and context-window
size are not exposed in this session. Used reasoning, repository tools
and code execution. No paid evaluation model calls were made for this
change.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

tests/runner-e2e/OPERATIONS.md

@@ -156,7 +156,17 @@ keys. The server environment strips the gateway token and upstream credentials.
 
 Every upstream attempt reserves its maximum estimated charge in an fsynced
 journal before dispatch. Requests, retries, and cancelled calls retain distinct
-IDs. Unknown usage keeps the reservation held. One shared writer serializes
+IDs. Braintrust gateway requests propagate the reserved request ID, campaign,
+execution, attempt and phase as allowlisted `paperclip_eval_*` metadata in the
+SDK-compatible parent context. The exact context is retained in the private
+receipt before dispatch, so a request canceled before response headers can
+still be looked up by `metadata.paperclip_eval_request_id` in project logs.
+This context contains no provider credential, prompt or response, and does not
+create an extra model call or a synthetic parent span. Direct provider routes
+receive no Braintrust context. Older project-only receipts cannot acquire this
+identity retroactively; time or session matches remain candidate evidence.
+A matching logged span does not itself prove complete usage or final billing.
+Unknown usage keeps the reservation held. One shared writer serializes
 admission across cells, so concurrent requests cannot each spend the same remaining
 allowance. Campaign deadline, request count, and in-flight count are also enforced.
 Private request receipts retain transport IDs, response IDs, usage and cache TTLs,

tests/runner-e2e/braintrust-gateway-context.ts

@@ -0,0 +1,17 @@
+import { validEvalLedgerId, type EvalChargeIdentity } from "./cost-ledger.js";
+
+export function braintrustGatewayRequestParent(projectId: string, requestId: string, campaignId: string, identity: EvalChargeIdentity): string {
+  if (!/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i.test(projectId)
+      || ![requestId, campaignId, identity.executionId, identity.attemptId].every(validEvalLedgerId)
+      || !["setup", "agent", "judge", "feedback_review", "tool", "infrastructure"].includes(identity.phase)) {
+    throw new Error("Invalid eval gateway logging identity");
+  }
+  return Buffer.concat([Buffer.from([4, 2, 0]), Buffer.from(JSON.stringify({
+    object_id: projectId,
+    propagated_event: { metadata: {
+      paperclip_eval_request_id: requestId, paperclip_eval_campaign_id: campaignId,
+      paperclip_eval_execution_id: identity.executionId, paperclip_eval_attempt_id: identity.attemptId,
+      paperclip_eval_phase: identity.phase,
+    } },
+  }))]).toString("base64");
+}

tests/runner-e2e/eval-gateway.test.ts

@@ -31,12 +31,54 @@ it("fsyncs reservation before dispatch, pins gateway routing, and replaces reser
   const result = await forwardEvalRequest({ ...f, transport });
   expect(reservedBeforeDispatch).toBe(true);
   expect(transport.mock.calls[0]![0]).toBe("https://gateway.braintrust.dev/responses");
-  expect(captured).toMatchObject({ redirect: "error", headers: { authorization: `Bearer ${route.apiKey}`, "x-bt-use-cache": "never", "cache-control": "no-cache, no-store", "x-bt-project-id": route.projectId, "x-bt-endpoint-name": route.endpointName, "x-bt-parent": `project_id:${route.projectId}` } });
+  expect(captured).toMatchObject({ redirect: "error", headers: { authorization: `Bearer ${route.apiKey}`, "x-bt-use-cache": "never", "cache-control": "no-cache, no-store", "x-bt-project-id": route.projectId, "x-bt-endpoint-name": route.endpointName, "x-bt-parent": result.braintrustParent } });
   expect(JSON.parse(captured!.body as string)).toMatchObject({ max_output_tokens: 100, service_tier: "default" });
   expect(result).toMatchObject({ outcome: "succeeded", costUsd: "0.000068000", final: true, usageComplete: true, gatewaySpanId: "gateway-span" });
   expect(f.ledger.snapshot()).toMatchObject({ knownUsd: "0.000068000", heldUsd: "0.000000000", inflight: 0 });
   expect(await readFile(f.file, "utf8")).not.toContain(route.apiKey);
 });
+it.each(["openai", "anthropic"] as const)("correlates %s gateway logs before response headers even when the client cancels", async provider => {
+  const f = await fixture();
+  const selectedRoute: EvalGatewayRoute = provider === "openai" ? route : { ...route, provider, endpointName: "ANTHROPIC_API_KEY",
+    price: { ...route.price, provider, cacheWrite5mUsdPerMillion: "2", cacheWrite1hUsdPerMillion: "4" } };
+  const parents: string[] = [];
+  const initialReceipts: EvalGatewayReceipt[] = [];
+  for (let attempt = 0; attempt < 2; attempt++) {
+    const controller = new AbortController();
+    const receipt = await forwardEvalRequest({ ...f, route: selectedRoute, signal: controller.signal,
+      transport: async (_url, init) => {
+        parents.push(new Headers(init?.headers).get("x-bt-parent")!);
+        initialReceipts.push(structuredClone(f.receipts.at(-1)!));
+        controller.abort();
+        throw new DOMException("client disconnected", "AbortError");
+      },
+    });
+    expect(receipt).toMatchObject({ outcome: "cancelled", errorCode: "request_cancelled", costUsd: null, final: false });
+    expect(receipt.gatewaySpanId).toBeUndefined();
+  }
+  expect(parents[0]).not.toBe(parents[1]);
+  for (const [index, encoded] of parents.entries()) {
+    const bytes = Buffer.from(encoded, "base64");
+    expect([...bytes.subarray(0, 3)]).toEqual([4, 2, 0]);
+    expect(JSON.parse(bytes.subarray(3).toString())).toEqual({ object_id: route.projectId, propagated_event: { metadata: {
+      paperclip_eval_request_id: initialReceipts[index]!.requestId, paperclip_eval_campaign_id: "campaign",
+      paperclip_eval_execution_id: identity.executionId, paperclip_eval_attempt_id: identity.attemptId, paperclip_eval_phase: "agent",
+    } } });
+    expect(initialReceipts[index]).toMatchObject({ braintrustParent: encoded, final: false, costUsd: null });
+  }
+  expect(f.ledger.snapshot()).toMatchObject({ inflight: 0, unknownRequests: 2, knownUsd: "0.000000000" });
+  expect(f.ledger.snapshot().heldUsd).not.toBe("0.000000000");
+  expect(JSON.stringify(initialReceipts)).not.toContain(route.apiKey);
+});
+it("keeps Braintrust request metadata out of direct provider requests", async () => {
+  const f = await fixture();
+  let headers: Headers | undefined;
+  const receipt = await forwardEvalRequest({ ...f, route: { ...route, mode: "direct" }, transport: async (_url, init) => {
+    headers = new Headers(init?.headers); return response();
+  } });
+  expect(headers!.has("x-bt-parent")).toBe(false);
+  expect(receipt).not.toHaveProperty("braintrustParent");
+});
 it("stops at the campaign cap before any upstream request", async () => {
   const f = await fixture("0.002"), transport = vi.fn<typeof fetch>();
   expect(evalRequestReservation(route)).toBe("0.003000000");

tests/runner-e2e/eval-gateway.ts

@@ -1,6 +1,7 @@
 import { randomBytes, randomUUID, timingSafeEqual } from "node:crypto";
 import { createServer, type IncomingMessage, type ServerResponse } from "node:http";
 import { once } from "node:events";
+import { braintrustGatewayRequestParent } from "./braintrust-gateway-context.js";
 import { usdToUnits } from "../../packages/shared/src/money.js";
 import { estimateProviderCost, ProviderUsageAccumulator, type EvalModelPrice, type ProviderUsageReceipt } from "./provider-usage.js";
 import { EvalAdmissionError, validEvalLedgerId, type EvalChargeIdentity, type openEvalCostLedger } from "./cost-ledger.js";
@@ -19,7 +20,7 @@ export interface EvalGatewayRoute {
 }
 export interface EvalGatewayReceipt {
   requestId: string; status?: number; route: "direct" | "braintrust";
-  gatewayRequestId?: string; gatewaySpanId?: string; providerRequestId?: string;
+  gatewayRequestId?: string; gatewaySpanId?: string; providerRequestId?: string; braintrustParent?: string;
   endpoint?: string; cache?: string; failoverFrom?: string; failoverTo?: string; errorOrigin?: string;
   outcome: "succeeded" | "failed" | "cancelled"; errorCode?: string;
   usageComplete: boolean; final: boolean; costUsd: string | null;
@@ -111,7 +112,8 @@ export async function forwardEvalRequest(input: {
   if (input.signal?.aborted) throw new EvalRequestError("request_cancelled");
   if (Date.now() >= input.deadlineAt) throw new EvalAdmissionError("request_deadline_reached");
   const requestId = randomUUID();
-  const receipt: EvalGatewayReceipt = { requestId, route: route.mode, outcome: "failed", usageComplete: false, final: false, costUsd: null };
+  const receipt: EvalGatewayReceipt = { requestId, route: route.mode, outcome: "failed", usageComplete: false, final: false, costUsd: null,
+    ...(route.mode === "braintrust" ? { braintrustParent: braintrustGatewayRequestParent(route.projectId!, requestId, ledger.snapshot().campaignId, input.identity) } : {}) };
   const usage = new ProviderUsageAccumulator(route.provider, body);
   await ledger.reserve(requestId, input.identity, reservation);
   const abort = new AbortController();
@@ -122,7 +124,7 @@ export async function forwardEvalRequest(input: {
     const headers: Record<string, string> = { "content-type": "application/json", "accept-encoding": "identity" };
     if (route.mode === "braintrust") Object.assign(headers, {
       authorization: `Bearer ${route.apiKey}`, "x-bt-project-id": route.projectId!, "x-bt-endpoint-name": route.endpointName!,
-      "x-bt-parent": `project_id:${route.projectId}`, "x-bt-use-cache": "never", "cache-control": "no-cache, no-store",
+      "x-bt-parent": receipt.braintrustParent!, "x-bt-use-cache": "never", "cache-control": "no-cache, no-store",
     });
     else if (route.provider === "openai") headers.authorization = `Bearer ${route.apiKey}`;
     else headers["x-api-key"] = route.apiKey;