Files
PaperClipAI/tests/runner-e2e/braintrust-query.ts
T
DottaandPaperclip 661f33f994 fix(evals): verify stored gateway request coverage (#17432)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Product E2E campaigns retain a request ledger and send native traces
and gateway logs to Braintrust.
> - Native trace verification did not prove that each charged gateway
request had a stored log row.
> - A hosted probe showed that `x-bt-span-id` identifies the row `id`,
while the stored `span_id` differs. Propagated campaign metadata was
absent.
> - This pull request joins retained request receipts to the correct
Braintrust rows during scheduled finalization.
> - Operators can see missing request coverage without changing costs,
releasing holds, or rerunning agents.

## Linked Issues or Issue Description

Refs #17354 and #17371.

**Describe the bug**

A campaign could pass native trace and experiment indexing checks while
gateway request logs remained unverified. The runbook also claimed that
propagated metadata made a request searchable after cancellation before
response headers. The hosted gateway did not preserve that metadata in
retained evidence.

**Steps to reproduce**

Inspect a retained gateway response header and query project logs by its
`x-bt-span-id`. The matching record uses that value as `id`, not
`span_id`. Compare every model reservation with its saved response
receipt, including setup and canceled work.

**Expected behavior**

Verify each request against the stored row and report absent or
conflicting evidence. Keep trace coverage separate from final usage and
invoice reconciliation.

## What Changed

- Added a bounded, read-only gateway index verifier. It matches row IDs,
provider and model, and retains the actual span and trace IDs.
- Included setup, failed and canceled requests in coverage. Missing
receipts or row IDs, duplicate attribution, conflicting records and
evidence changes prevent verification.
- Added a separate scheduled gateway coverage gate and private receipt.
Native trace and experiment checks remain independent.
- Shared the existing bounded Braintrust query reader. Queries select
only identity fields, with at most 100 IDs per batch, three rounds and a
45-second deadline.
- Corrected the runbook's metadata guarantee and documented the
historical `gatewaySpanId` receipt field.

## Verification

- `pnpm test:e2e:runner:unit`: 4,120 passed, two existing skips; all 137
Node contracts passed.
- Final repair: 54 focused tests passed, including
scheduled-finalization coverage and all five optional identity fields.
Each conflicting-field regression failed before the fix; absent or
matching partial metadata remains supported.
- `pnpm test:e2e:runner:typecheck`: passed after rebuilding stale local
plugin SDK declarations. The initial stale-declaration failure remains
recorded in the validation logs.
- `git diff --check`: passed.
- Read-only live verification of retained evidence: all ten local Codex
requests and the one canceled drain-probe request matched stored rows.
The hosted Claude campaign matched eight of ten; its two pre-header
cancellations correctly remained incomplete.
- The first hosted lookup exceeded the 1 MiB read limit when selecting
full metadata. The final identity-only projection passed the same
lookup.
- Original journals stayed byte-identical. These checks made zero model
calls and did not change provider data or release holds.
- Full repository CI passed on
`957ad0b684b80267d230fc4fb56d4f002ef089ad`: [run
40105676630](https://github.com/paperclipai/paperclip/actions/runs/40105676630),
including typecheck, tests and build. Both current-head automated
reviews scored 5/5 with no open findings.
- The superseded initial CI run was canceled after the repair push to
release the queue; its completed results remain available. No failed job
was retried on the final commit.

## Risks

Missing gateway IDs now fail scheduled trace coverage. A failed or
canceled call may have no stored row, so this can require investigation
even if product assertions pass. Indexing can lag; bounded query retries
do not retry model work. Stored identity does not prove final billing or
gateway span nesting under native attempt traces. Private identifiers
remain in private evidence only. Recurrence and billing settings are
unchanged.

## Model Used

OpenAI Codex, GPT-6 family. The runtime does not expose a more specific
model ID or context-window size in this session. Used reasoning,
repository tools, local code execution and read-only API verification.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-24 08:23:57 -07:00

23 lines
1.2 KiB
TypeScript

import type { EvalBraintrustConfig } from "./braintrust.js";
export async function queryBraintrustRows(config: EvalBraintrustConfig, sql: string, transport: typeof fetch, signal: AbortSignal): Promise<Record<string, unknown>[]> {
const response = await transport("https://api.braintrust.dev/btql", {
method: "POST", redirect: "error", signal: AbortSignal.any([signal, AbortSignal.timeout(10_000)]),
headers: { authorization: `Bearer ${config.apiKey}`, "content-type": "application/json" },
body: JSON.stringify({ query: sql, fmt: "json" }),
});
const reader = response.body?.getReader(), chunks: Uint8Array[] = []; let size = 0;
try {
if (!response.ok) throw new Error("query_rejected");
while (reader) {
const next = await reader.read(); if (next.done) break;
size += next.value.length; if (size > 1_048_576) throw new Error("query_oversized");
chunks.push(next.value);
}
const payload = JSON.parse(Buffer.concat(chunks).toString("utf8"));
if (!payload || !Array.isArray(payload.data) || payload.data.some((row: unknown) => !row || typeof row !== "object" || Array.isArray(row))) throw new Error("invalid_query_result");
return payload.data;
} finally { await reader?.cancel().catch(() => {}); }
}