Files
PaperClipAI/tests/runner-e2e/api.ts
T
DottaandPaperclip 3bd709df8b Operationalize Product E2E accounting, tracing, and scheduling (#17009)
## Thinking Path

> - Paperclip manages agents and the work they perform.
> - Product E2E tests exercise real agents, tools, setup and cleanup.
> - Repeated runs need retained costs and traces, including failed
attempts.
> - A passing retry must not hide earlier spending or feedback.
> - Recurring work also needs admission limits, private reports and
failure detection.
> - This change supplies those controls while keeping paid activation
behind existing gates.

## Linked Issues or Issue Description

Refs #16996 and #17158. This is the consolidated landing of the reviewed
eval operations stack: #17019, #17024, #17025, #17027, #17030, #17033,
#17035, #17038, #17043, #17044, #17048, #17050, #17051, #17092, #17096,
#17100, #17107, #17109, #17112, #17115, #17117, #17121, #17123, #17127,
#17130, #17133, #17134, #17137, #17140, #17143, #17154. The repository
permits squash merges only. A local rehearsal showed that squashing each
slice into master causes repeated conflicts in dependent slices. This PR
preserves their commits and lands their combined implementation in one
reviewed integration. Superseded slices will close after this merge.

## What Changed

- Retain every attempt's cost receipt, including setup, failures,
interruptions and retries. Separate reported amounts, estimates, missing
costs and reserved capacity in reports and dashboards.
- Add bounded request admission, exact model pricecards, gateway error
accounting, cache policy and reconciliation evidence. Refuse unsupported
routes and requests that cannot fit the allowance.
- Integrate Braintrust gateway traces and OTLP correlation, with secret
filtering and experiment imports. Record phase timing, including setup,
feedback capture and cleanup.
- Collect complaint and suggestion evidence after each attempt,
including recovery after failures. Retain explicit review dispositions
independently from the product grade.
- Add gated daily and weekly selections, measured forecasts,
setup/readiness checks, wall-time limits and retained monitor
notifications. Include declared non-model overhead in admission while
reporting the existing subscription separately.
- Keep recurring history private, bound uploads and retention, and use
isolated archive environments. Keep public publication an explicit
operation. Report and publication jobs install only the committed
lockfile, with scripts disabled and isolated stores; stale pins fail
closed.
- Integrate current master, preserve its Product E2E journey billing and
privacy checks, and test that shared setup receipts do not double-count
estimated agent usage. Preserve upstream fixture repairs.

## Verification

- Complete integration baseline
`0649a5ce7544ab20825c2183d2f421157db49f24` includes master
`a3e1d915a7110a4e115b46396a3f79e6dfb4d05e` and every original stack head
as ancestors.
- On that integration baseline, `pnpm test:e2e:runner:unit`: 4,012
Vitest tests passed, two skipped, across 193 files; all 137 Node
source-contract tests passed.
- Focused billing and Product E2E journey contract checks: 24 tests
passed. They cover estimates, unknown receipts, single-owner setup
costs, browser-only execution and private publication guards.
- Monitor and workflow wiring: all 96 Node tests passed. `node
scripts/check-token-gates.mjs`: all gates clean.
- `pnpm test:e2e:runner:typecheck` passes after the standard Runner
TypeScript build refreshed local declarations. The first attempt
rejected the new master `runner_prepared_launch` union member because
this checkout retained an older generated declaration; the source union
already contained that member.
- Current head `492b0708a1c531a023d9ac10f0f3dc6383cc1754` removes
dependency re-resolution from archive-authorized reporting jobs after
the security scan flagged it. All 15 workflow-security tests pass after
updating the old two-install assertion.
- Hosted [CI run
39965460180](https://github.com/paperclipai/paperclip/actions/runs/39965460180)
passes on the current repaired head, including typecheck, build, general
tests, Runner checks, browser shards and canary dry run. Greptile is 5/5
on this exact head with no unresolved threads. The security scan passes
after the fixed lockfile-resolution thread was resolved.
- Paperclip Review returned a terminal neutral check but an incomplete
0-file assessment on the repaired head: its platform blocked both diff
responses as prompt-injection content. This is not a successful code
review. No bypass or fabricated pass was attempted; current-head
Greptile, security and hosted CI are independent gates.
- No paid test, reservation release or recurring-run activation
accompanies this landing. Live route qualification and invoice
reconciliation remain separate evidence.

<details>
<summary>Retained earlier validation failures</summary>

Earlier broad local runs had two 15-second test timeouts (CLI poll
cadence and Retry validation), one embedded-database setup timeout, and
later PostgreSQL semaphore exhaustion. The exact timed-out tests passed
separately without larger timeouts; the broad-run causes were not
established. Those local failures remain distinct from subsequent hosted
CI results. The independent #17158 fix records its Muse fixture
collision, its UUID correction and hosted verification. Local Muse
execution was also blocked by macOS embedded-PostgreSQL startup. No
historical failure is treated as a successful paid qualification.

</details>

## Risks

- Admission is a local control, not a provider invoice cap. Missing
provider receipts or resource charges retain unknown amounts and holds;
estimates do not become billed dollars.
- The bounded path admits only documented local suites and provider
routes. Ordinary legacy or separate Runner protocol campaigns have
different controls. Merging this code does not qualify additional
routes.
- Recurrence still needs the existing enable gates and current
qualification/allowance evidence. Archive permissions and resources must
match the documented private-environment contract.
- Legacy reports may lack complete attempt or phase evidence. Reports
preserve that limitation and do not infer zero usage from missing data.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools and code
execution. The exact deployment variant and context-window size are not
exposed by this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (targeted checks;
full-suite host limitation disclosed above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-23 07:18:37 -07:00

174 lines
6.8 KiB
TypeScript

import type { APIRequestContext, APIResponse } from "@playwright/test";
export interface JsonRecord {
[key: string]: unknown;
}
async function failureMessage(response: APIResponse, method: string) {
const text = await response.text().catch(() => "");
return `${method} ${response.url()} returned ${response.status()}${text ? `: ${text}` : ""}`;
}
/** Preserve HTTP status independently of diagnostic response text. */
export class RunnerApiHttpError extends Error {
constructor(readonly status: number, message: string) { super(message); this.name = "RunnerApiHttpError"; }
}
/** Admission-only diagnostics. Never retain the rejected value or its cause. */
export class RemoteAdmissionReadError extends Error {
constructor(
readonly endpoint: "issue" | "run" | "leases",
readonly failureClass: "candidate_failure" | "transient_infrastructure" | "permanent_infrastructure",
) {
super(`Remote admission ${endpoint} read failed (${failureClass}); diagnostics withheld`);
this.name = "RemoteAdmissionReadError";
}
}
export class RunnerApi {
readonly baseURL: string;
private browserCookie?: string;
setBrowserSession(cookie: string) { this.browserCookie = cookie; }
private sessionOptions() {
return this.browserCookie ? { headers: { Cookie: this.browserCookie, Origin: this.baseURL } } : {};
}
constructor(readonly request: APIRequestContext) {
const port = process.env.PAPERCLIP_RUNNER_E2E_PORT?.trim();
if (!port) throw new Error("PAPERCLIP_RUNNER_E2E_PORT is required");
this.baseURL = `http://127.0.0.1:${port}`;
}
async get<T>(path: string, options?: { timeout: number }): Promise<T> {
const response = await this.request.get(path, { ...this.sessionOptions(), ...options });
if (!response.ok()) throw new RunnerApiHttpError(response.status(), await failureMessage(response, "GET"));
return response.json() as Promise<T>;
}
async post<T>(path: string, data?: unknown, options?: { timeout: number }): Promise<T> {
const response = await this.request.post(path, { data, ...this.sessionOptions(), ...options });
if (!response.ok()) throw new Error(await failureMessage(response, "POST"));
return response.json() as Promise<T>;
}
/**
* Playwright traces APIRequestContext request bodies. Secret creation must
* still use the public API, but it goes through Node fetch so plaintext is
* never serialized into trace/blob evidence before Paperclip encrypts it.
*/
async postSensitive<T>(path: string, data: unknown): Promise<T> {
return this.sensitive<T>("POST", path, data);
}
async putSensitive<T>(path: string, data: unknown): Promise<T> {
return this.sensitive<T>("PUT", path, data);
}
private async sensitive<T>(method: "POST" | "PUT", path: string, data: unknown): Promise<T> {
const url = new URL(path, this.baseURL);
if (url.origin !== this.baseURL || !url.pathname.startsWith("/api/")) throw new Error("Sensitive request must use the fixture API");
let response: Response;
try {
response = await fetch(url, {
method, redirect: "error", signal: AbortSignal.timeout(15_000),
headers: { "content-type": "application/json", ...this.sessionOptions().headers },
body: JSON.stringify(data),
});
} catch { throw new Error(`Sensitive ${method} request failed; details withheld`); }
if (!response.ok) {
await response.body?.cancel();
throw new Error(`Sensitive ${method} ${url.pathname} returned ${response.status}; response body withheld`);
}
return response.json() as Promise<T>;
}
async patch<T>(path: string, data: unknown): Promise<T> {
const response = await this.request.patch(path, { data, ...this.sessionOptions() });
if (!response.ok())
throw new Error(await failureMessage(response, "PATCH"));
return response.json() as Promise<T>;
}
/** Use the public revision fence when configuring a fixture's instruction entry. */
async saveAgentInstructions(agentId: string, content: string): Promise<void> {
const path = `/api/agents/${agentId}/instructions-bundle/file`;
const current = await this.request.get(`${path}?path=AGENTS.md`);
let baseHash: string | null = null;
if (current.ok()) {
const detail = await current.json();
if (typeof detail.contentHash !== "string" || !detail.contentHash) {
throw new Error("Existing fixture instructions have no revision hash");
}
baseHash = detail.contentHash;
} else if (current.status() !== 404) {
throw new Error(await failureMessage(current, "GET"));
}
const saved = await this.request.put(path, { data: { path: "AGENTS.md", content, baseHash } });
if (!saved.ok()) throw new Error(await failureMessage(saved, "PUT"));
}
async delete(
path: string,
options?: { allowNotFound?: boolean },
): Promise<void> {
const response = await this.request.delete(path, this.sessionOptions());
if (response.ok() || (options?.allowNotFound && response.status() === 404))
return;
throw new Error(await failureMessage(response, "DELETE"));
}
}
export class ObservedStateTimeout extends Error {
constructor(label: string, readonly failureClass: "candidate_failure" | "transient_infrastructure", detail?: string) {
super(`Timed out waiting for ${label}; ${detail ?? "the observed state did not satisfy the condition"}. See the saved state evidence.`);
this.name = "ObservedStateTimeout";
}
}
export async function pollUntil<T>(input: {
label: string;
deadlineAt: number;
load: () => Promise<T>;
accept: (value: T) => boolean;
reject?: (value: T) => string | undefined;
intervalMs?: number;
retryOnError?: (error: unknown) => boolean;
timeoutFailureClass?: "candidate_failure" | "transient_infrastructure";
timeoutDetail?: (value: T | undefined) => string | undefined;
}): Promise<T> {
let last: T | undefined;
let lastError: unknown;
while (Date.now() < input.deadlineAt) {
try {
last = await input.load();
if (input.accept(last)) return last;
const rejection = input.reject?.(last);
if (rejection) {
throw new Error(`Stopped waiting for ${input.label}: ${rejection}`);
}
lastError = undefined;
} catch (error) {
if (input.retryOnError && !input.retryOnError(error)) throw error;
if (
error instanceof Error &&
error.message.startsWith(`Stopped waiting for ${input.label}:`)
) {
throw error;
}
lastError = error;
}
await new Promise((resolve) =>
setTimeout(resolve, input.intervalMs ?? 2_000),
);
}
if (lastError instanceof Error) {
throw new Error(`Timed out waiting for ${input.label}: ${lastError.message}`, { cause: lastError });
}
throw new ObservedStateTimeout(
input.label,
input.timeoutFailureClass ?? "candidate_failure",
input.timeoutDetail?.(last),
);
}