diff --git a/.agents/skills/add-product-e2e-eval/SKILL.md b/.agents/skills/add-product-e2e-eval/SKILL.md new file mode 100644 index 00000000000..686fa44f136 --- /dev/null +++ b/.agents/skills/add-product-e2e-eval/SKILL.md @@ -0,0 +1,84 @@ +--- +name: add-product-e2e-eval +description: Add or extend a Paperclip full-stack runner E2E workflow, fixture, matcher, or report evidence path for local or Daytona execution. +--- + +# Add a Product E2E Eval + +Use this skill for **Product E2E Evals**: real Chromium, Paperclip server, +database, runner, provider, and optionally Daytona. Everyday Workflows are in +this family even when their packaged results are imported into Evalbook. Runner +protocol cases against the mock control plane belong in +[add-runner-eval](../add-runner-eval/SKILL.md). + +Locate the repository using `PAPERCLIP_ROOT` when supplied, or +`git rev-parse --show-toplevel` from a checkout. From outside Git, inspect +workspace roots such as `~/paperclipai/paperclip`; verify the selected root +contains `tests/runner-e2e` and `packages/paperclip-runner`. Run commands from +that repository root. The copied skill may live outside the checkout. +Read `doc/evals.md`, then the authoritative +`tests/runner-e2e/README.md`, `FIXTURES.md`, `SECURITY.md`, and +`EVERYDAY-WORKFLOWS.md` for the selected area. Inspect the nearest existing +catalog entry, case, harness flow, matcher, evidence writer, and report test +before changing anything. Keep the user journey on production browser/API +surfaces; do not add private runner hooks or direct fixture database writes. + +Define one bounded workflow with a clear user outcome, durable state +assertions, and independent evidence. Declare profile, environment, expected +provider turns, timeout, cleanup, screenshots, artifact checks, and billing +scope. Keep credentials and secrets out of catalog data, screenshots, fixture +metadata, logs, and tracked files. Daytona images must be immutable digest +references. Use the existing result validator, failure classifier, screenshot +policy, and Product E2E report pipeline rather than duplicating them. + +Register the workflow in `tests/runner-e2e/catalog.ts` and the relevant +`fixture-registry.ts` paths. Everyday cases and actions live in +`everyday-cases.ts` and `everyday-flow.ts`; match the existing suite's structure. +Wire its +profile/environment/case IDs through existing selector and matcher tables, and +add report/catalog coverage tests where the surrounding suite does so. Calibrate +new grading assertions with a valid outcome and a plausible wrong outcome; +missing evidence must not produce a pass. Confirm +discovery before running it: + +```sh +pnpm test:e2e:runner -- --list --suite +pnpm test:e2e:runner -- --list --suite everyday-workflows +``` + +`--all` intentionally excludes the manual `everyday-workflows` suite and other +explicit-only cells. Use the exact suite or execution ID for those. The Product +E2E generator is `pnpm test:e2e:runner:report`; an Everyday Workflows result may +also be imported into Evalbook with its canonical importer, but it remains a +Product E2E run and should use its packaged dashboard/report first. + +Run credential-free checks first: + +```sh +pnpm test:e2e:runner:typecheck +pnpm test:e2e:runner:unit +pnpm test:e2e:runner -- --list +``` + +For an authorized live check, use the smallest explicit local `--id` selector. +Select Daytona only with the configured immutable image and credentials. A +full `--all` campaign is paid and is for the governed workflow. Verify results +from the packaged attempt evidence and dashboard; passing model-authored tests +cannot override the independent oracle. + +Classify a completed wrong workflow as product or model/provider behavior as +the assertions warrant. Startup, transport, and timeout symptoms require +evidence-based attribution: they may indicate a Paperclip/Runner product bug, +provider behavior, or infrastructure. Preserve the observed failure and cause +separately, keep the existing machine grade/classifier unchanged, and retain +partial attempts, retries, source SHA, catalog/definition digest, model/profile, +environment, grader version, timing, tokens, runtime estimates, and cost +coverage. Do not claim qualification from a partial/manual selection. + +Update `README.md`, `FIXTURES.md`, `SECURITY.md`, or +`EVERYDAY-WORKFLOWS.md` when their authoritative contract changes, and link +from `doc/evals.md`. Do not recreate Evalbook HTML or publish raw trusted +traces, videos, archives, databases, workspaces, credentials, SVG, provider +session IDs, or hidden reasoning. Public sanitized fixture conversation, marked +screenshots, and allowlisted structured evidence are expected when the existing +publisher permits them. diff --git a/.agents/skills/add-runner-eval/SKILL.md b/.agents/skills/add-runner-eval/SKILL.md new file mode 100644 index 00000000000..111a1b2c7a0 --- /dev/null +++ b/.agents/skills/add-runner-eval/SKILL.md @@ -0,0 +1,72 @@ +--- +name: add-runner-eval +description: Add or extend a Paperclip Runner protocol evaluation definition, roster, assertion, or report fixture with provenance and narrow validation. +--- + +# Add a Runner Eval + +Use this skill for the **Runner Evals** family: a real Runner/provider session +against a seeded mock control plane. Product browser/server/database/Daytona +coverage belongs in [add-product-e2e-eval](../add-product-e2e-eval/SKILL.md). + +Locate the Paperclip checkout using `PAPERCLIP_ROOT` when supplied, or +`git rev-parse --show-toplevel` from a checkout. From outside Git, inspect the +workspace roots (for example `~/paperclipai/paperclip`) and verify that the +selected root contains `packages/paperclip-runner` and `tests/runner-e2e`. +Locate `paperclip-evals` using `PAPERCLIP_EVALS_ROOT` or a discovered sibling; +a worktree's parent directory need not contain that repository. Read +`doc/evals.md` and `packages/paperclip-runner/docs/runner-protocol-live-evals.md`, +then inspect the nearest existing case, roster, schema, and report test before +editing. Definitions and authored cases belong in the sibling +`paperclip-evals/evals/paperclip-runner`; Runner integration, aggregation, +viewer, and publication behavior belongs in `packages/paperclip-runner`. +Keep the control-plane boundary explicit in names and documentation. + +The sibling eval README is the concrete map: cases live under `cases/`, +company fixtures under `fixtures/`, runtime/model settings under `configs/`, +selections under `rosters/`, and maintained campaign membership under +`campaigns/live-direct-full.json`. Update inventory/coverage mappings when the +program requires them; a new file alone does not join the maintained campaign. +From the Evals repository root, adapt these provider-free checks to the case +and roster you changed. Run the reliability-plan validator only when that +separate plan changes: + +```sh +python3 evals/paperclip-runner/tools/eval_program.py validate \ + --case evals/paperclip-runner/cases/get-task-context.json \ + --config evals/paperclip-runner/configs/live-codex-pinned.json +python3 evals/paperclip-runner/tools/run_live_roster.py validate \ + --roster evals/paperclip-runner/rosters/live-mini.json --run-id validate-new-case +python3 evals/paperclip-runner/tools/run_live_campaign.py validate \ + --campaign evals/paperclip-runner/campaigns/live-direct-full.json +python3 evals/paperclip-runner/tools/reliability_campaign.py validate \ + --plan evals/paperclip-runner/campaigns/paperclip-runner-reliability.json +``` + +Use nearby positive and negative grader cases/fixtures to calibrate the new +assertion, including malformed or missing evidence where the grader must fail +closed. Preserve the existing machine disposition and grade; product, +model/provider, grading, and infrastructure labels are analytical annotations, +not instructions to rewrite classifiers. + +Author one bounded case with a deterministic semantic assertion and an +inspectable result. Declare its expected operation, state effect, provider +lane/profile, timeout and retry policy, and any required evidence. Do not grade +hidden reasoning, infer success from a provider terminal message, or invent +conversation/tool evidence. Public output follows the reviewed projection: +sanitized fixture conversation and allowlisted tool outcomes may be published; +raw trusted artifacts, credentials, secrets, private references, and hidden +reasoning may not. + +Validate without provider calls first using the commands above and the relevant +report/render validation documented in the Runner docs. When a live run is +authorized, pin the Paperclip commit and exact 40-character +`paperclip-evals` commit, select the smallest useful roster, and retain the +complete provenance and cost record. + +Update authoritative detailed docs when the contract or command changes, then +link from `doc/evals.md` rather than duplicating the Runner runbook. Keep public +reports immutable and use the reviewed projection; sanitized fixture +conversation and allowlisted tool outcomes may be public, while credentials, +secrets, private references, raw trusted payloads, and hidden reasoning must not +be exposed. diff --git a/.agents/skills/paperclip-evals/SKILL.md b/.agents/skills/paperclip-evals/SKILL.md new file mode 100644 index 00000000000..9735805c806 --- /dev/null +++ b/.agents/skills/paperclip-evals/SKILL.md @@ -0,0 +1,63 @@ +--- +name: paperclip-evals +description: Choose, inspect, validate, and report Paperclip Runner or Product E2E evaluations while preserving evidence, provenance, cost, and failure classification. +--- + +# Paperclip evals + +Use this skill when a request concerns Paperclip evaluation selection, +interpretation, evidence, history, or a live run. Read `doc/evals.md` in the +Paperclip repository first. It defines the two families and their boundaries. + +## Discover the repository + +Do not assume the skill's installed location is inside a checkout. Locate the +repo explicitly with `git rev-parse --show-toplevel` from the current directory, +or inspect likely workspace roots and select the checkout containing +`package.json`, `tests/runner-e2e`, and `packages/paperclip-runner`. Locate the +private sibling `paperclip-evals` only when a Runner Eval needs its definitions; +use an explicit `PAPERCLIP_EVALS_ROOT` or a discovered sibling checkout. Never +invent a relative path from this copied skill into the repository. + +## Route the request + +Choose **Runner Evals** for real runner/provider protocol behavior against the +mock control plane. Authoritative details are in +`packages/paperclip-runner/docs/runner-protocol-live-evals.md` and the sibling +`paperclip-evals/evals/paperclip-runner` definitions. + +Choose **Product E2E Evals** for real browser/server/database/runner/provider +workflows, including local and Daytona environments. Read +`tests/runner-e2e/README.md`, then `FIXTURES.md`, `SECURITY.md`, or +`EVERYDAY-WORKFLOWS.md` as relevant. Everyday Workflows remain Product E2E +even when imported into Evalbook. “Headless” is a browser mode, not a family. + +## Work safely + +Start with read-only catalog inspection and credential-free validation. For +Product E2E use `pnpm test:e2e:runner:typecheck`, +`pnpm test:e2e:runner:unit`, and `pnpm test:e2e:runner -- --list`; run one +explicit cell only when the user has authorized a live/paid run and the needed +credentials and immutable Daytona image are configured. For Runner Evals use +the pinned eval revision and the documented workflow/CLI. Never use a partial +selector as evidence of full coverage. + +Keep source revisions, definition/catalog fingerprints, model/profile, +environment, selected cells, retries, timing, usage/cost coverage, and grader +version attached to every interpretation. Preserve partial attempts and +classify failures as product, model/provider behavior, grading/evidence, or +infrastructure from the observed failure and supported cause. A usable +completed behavior failure is not infrastructure; missing provider/profile, +transport, startup, or evidence requires examining the evidence before choosing +the cause. + +Use the existing family generator and viewer. Public projections may contain +sanitized fixture conversation and allowlisted tool outcomes/evidence; follow +the family's projection and publisher checks. Do not expose raw trusted +artifacts, credentials, secrets, private data, provider session IDs, or hidden +reasoning. A refresh from retained evidence has zero provider calls and remains +the original measurement with a new presentation. Link the public histories +and hub from `doc/evals.md` when reporting results. + +For adding a case or fixture, use the narrower [add-runner-eval](../add-runner-eval/SKILL.md) +or [add-product-e2e-eval](../add-product-e2e-eval/SKILL.md) skill. diff --git a/doc/evals.md b/doc/evals.md new file mode 100644 index 00000000000..88fedec15ed --- /dev/null +++ b/doc/evals.md @@ -0,0 +1,214 @@ +# Paperclip evaluation guide + +Paperclip has two live eval families with different questions, owners, and +evidence. Choose the family before selecting a model, profile, or case. + +- **Runner Evals:** real Runner/provider behavior against a seeded mock control + plane. Definitions live in `paperclip-evals/evals/paperclip-runner`; see the + [direct live protocol evals](../packages/paperclip-runner/docs/runner-protocol-live-evals.md). +- **Product E2E Evals:** real browser, Paperclip server, database, Runner, + provider, and (where selected) Daytona, using an isolated instance and + grading oracle. See [`tests/runner-e2e`](../tests/runner-e2e/README.md) and + [Everyday Workflows](../tests/runner-e2e/EVERYDAY-WORKFLOWS.md). + +Runner Evals answer whether a real runner/provider can perform a bounded +protocol operation against the expected control-plane contract. Product E2E +Evals answer whether a person can complete a product workflow through the real +Paperclip surfaces and whether the resulting artifact and state are usable. +The names describe the system under test; “headless” is an execution option, +not an eval category. + +## Selecting a family + +Use **Runner Evals** for a runner protocol, adapter, transport, native session, +tool grant, or one-turn provider qualification question. The workflow checks +out an exact `paperclip-evals` revision, builds the Runner and viewer, runs a +live roster, and renders the canonical Evalbook report. The control plane is a +seeded test authority, so a passing result does not prove browser UX, production +server behavior, database persistence, Daytona behavior, or a real third-party +mutation. + +Use **Product E2E Evals** for browser interaction, issue/task lifecycle, +approval and clarification UI, project/repository selection, persistence over a +controller restart, artifact delivery, billing/evidence behavior, or runner +continuity in local or Daytona environments. The harness creates a fresh +Paperclip instance per cell and uses public APIs and the production browser +surface. The suite's [Everyday Workflows](../tests/runner-e2e/EVERYDAY-WORKFLOWS.md) +are Product E2E even when their results are imported into Evalbook. + +Do not combine a partial Runner campaign and a partial Product E2E campaign into +one score. A campaign is comparable when its definition/grader, model/profile, +environment, and contract match. The evaluated Paperclip revision may +intentionally differ for a before/after fix comparison; record it as a +comparison axis. + +## Ownership and codepaths + +Runner Evals are owned by the Runner/evals maintainers. Definitions, rosters, +case prompts, and the report program live in the sibling private repository +`paperclipai/paperclip-evals`; Runner integration, viewer, aggregation, and +publication code live under `packages/paperclip-runner` and the +`runner-protocol-live-evals.yml` workflow. The public-facing report uses the +same Evalbook renderer and Runner Lab viewer as the trusted report after +sanitization. + +Product E2E Evals are owned by the runner E2E maintainers. The catalog and +harness are under `tests/runner-e2e`; the package scripts are `test:e2e:runner`, +`test:e2e:runner:unit`, `test:e2e:runner:typecheck`, and +`test:e2e:runner:report`. `README.md`, `FIXTURES.md`, `SECURITY.md`, and +`EVERYDAY-WORKFLOWS.md` are the detailed sources of truth. The harness starts +the server and embedded database, creates the company/agent/task through the +real APIs, drives Chromium, and invokes the selected local or Daytona runner. + +## Validation ladder + +Start with credential-free checks and a catalog listing. For Product E2E: + +```sh +pnpm test:e2e:runner:typecheck +pnpm test:e2e:runner:unit +pnpm test:e2e:runner -- --list +``` + +For one explicitly selected local cell, configure only the credentials named +by that cell in `.env.runner-e2e.local`, then run a narrow ID: + +```sh +pnpm test:e2e:runner -- --id core-compatibility.runner-codex.local.message-marker +``` + +Use the selectors documented in the [runner E2E README](../tests/runner-e2e/README.md) +for a suite, profile, case, group, or environment. Daytona needs the immutable +image digest and `DAYTONA_API_KEY`; follow the README and fixture security guide. +`--all` excludes manual suites such as `everyday-workflows`. Select that suite +explicitly; use a narrow selector while developing a fixture. + +For Runner Evals, the narrowest useful local validation is the report program's +help/validation path and the deterministic Runner checks documented in +[`runner-workflow-evals.md`](../packages/paperclip-runner/docs/runner-workflow-evals.md). +Hosted direct live runs must use the default-branch workflow, an exact 40 +character `evals_sha`, an explicitly selected roster (or the maintained +enabled `all` campaign), and the protected paid environment. The complete +hosted command is intentionally kept in the workflow and +[direct live protocol guide](../packages/paperclip-runner/docs/runner-protocol-live-evals.md). +Live provider runs can spend money; use the existing workflow authorization and +the user's stated scope when selecting them. + +## Failure taxonomy + +Record the primary failure class and preserve the evidence that supports it. + +- **Product failure:** evidence shows Paperclip or Runner behavior violates the + authored case or a hard invariant, such as wrong task state, missing approval + gate, lost persistence, bad artifact, or incorrect protocol operation. +- **Model/provider behavior failure:** the provider turn completed with usable + evidence but the model gave the wrong answer, ignored an interaction, failed + to complete the authored operation, or violated a semantic assertion. It is + scored as behavior, not silently retried as infrastructure. +- **Grading/evidence failure:** the case or matcher cannot establish its claim, + a required recording/screenshot/result is malformed, or the report contract + is invalid. Fix the harness or grader before interpreting the score. +- **Infrastructure failure:** the evidence points to provider/profile + unavailability, transport admission failure, service startup failure, a + missing credential/image, or inability to produce usable evidence. Startup, + transport, and timeout symptoms can instead be product defects when evidence + implicates Paperclip or Runner; classify from the observed failure and + supported cause, rather than the symptom name alone. Preserve the artifact. + +Missing usage or price data means unknown, not free. Keep provider-reported +costs separate from estimates, and include retry costs when available. +Latency, cleanup, billing coverage, and unpriced usage are dimensions of the +result and should remain visible alongside the primary class. A timeout after +successful product state reads can be a product behavior failure; a failed +server-health read may be infrastructure, but inspect its cause. Use the +family-specific classifier and read the attempt evidence before changing an +analytical label. + +## Evidence, provenance, and history + +An Evalbook report is a presentation of immutable attempt records, not the +source of truth. Keep the campaign ID, Paperclip commit, `paperclip-evals` +commit, catalog/roster or definition fingerprint, model/profile, environment, +grader version, selected cells, retries, and provider/runtime usage with the +report. Public projections follow each family's reviewed allowlist and may +include sanitized fixture conversation, named tool outcomes, screenshots, and +structured evidence intended for public history. Credentials, secrets, private +data, raw unredacted records, and hidden reasoning stay out of public +projections. + +Distinguish a complete campaign from a partial campaign. A narrow selector, +manual diagnostic, missing cell, or infrastructure retry can be useful evidence +without being a qualification run. History should retain both, with explicit +coverage and completeness, while trend and latest-green views compare only +compatible complete campaigns. Refreshing an existing report from retained +evidence has zero provider calls and is a new presentation of the old +measurement, not a new model run. + +Existing public histories are available at +[Runner protocol history](https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/index.html) +and [Runner Product E2E history](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/). +The consolidated eval hub is at +[pages.paperclip.ing/evals](https://pages.paperclip.ing/evals/). + +For a repeatable workflow, use the matching skill: [paperclip-evals](../.agents/skills/paperclip-evals/SKILL.md), +[add-runner-eval](../.agents/skills/add-runner-eval/SKILL.md), or +[add-product-e2e-eval](../.agents/skills/add-product-e2e-eval/SKILL.md). + +## Install the authoring skills + +The reviewable sources live in this repository's `.agents/skills`. For a +multi-repository workspace, install the three skills at +`~/paperclipai/.agents/skills` (not `~/paperclipai/skills`). From the Paperclip +checkout, run: + +```sh +for skill in paperclip-evals add-runner-eval add-product-e2e-eval; do + install -d "$HOME/paperclipai/.agents/skills/$skill" + install -m 644 ".agents/skills/$skill/SKILL.md" \ + "$HOME/paperclipai/.agents/skills/$skill/SKILL.md" +done +``` + +This replaces only the three named skill entrypoints. Run it again after +updating their tracked sources. Each skill locates the repository independently +of its installation directory. + +## Maintain the public hub + +The hub is a static directory with two links to the existing history systems. +It displays a dated snapshot, not a live scoreboard. It does not run models, +create another result archive, or change the existing campaign URLs. + +Build from the public history feeds and check its summary logic: + +```sh +python3 -m unittest discover -s scripts/evals-hub -p 'test_*.py' +python3 scripts/evals-hub/build.py --output .paperclip/evals-hub +``` + +The hub checks need Python 3 and do not call model providers. + +For offline checks, pass `--history-dir ` containing +`runner-protocol-evals-history.json` and `runner-e2e-history.json`. +For a pre-merge preview, pass `--docs-ref ` to link the guide +at that revision. The default guide link uses `master`. + +Publish with the [Paperclip page helper](../.agents/skills/paperclip-page/SKILL.md) +and the configured page-uploader credentials. Use Bash 4 or newer; macOS's +system Bash 3 cannot run this helper. On macOS with Homebrew Bash installed, +put `$(brew --prefix bash)/bin` first in `PATH` before these commands: + +```sh +export PAPERCLIP_PAGE_BUCKET=pages.paperclip.ing +export PAPERCLIP_PAGE_BASE_URL=https://pages.paperclip.ing +export AWS_REGION=us-east-1 +bash .agents/skills/paperclip-page/scripts/publish.sh .paperclip/evals-hub --slug evals --dry-run +bash .agents/skills/paperclip-page/scripts/publish.sh .paperclip/evals-hub --slug evals +``` + +For later refreshes, rebuild in the same output directory and publish with +`--update`. Keep its ignored `.paperclip-page/state.json` ownership record; +without that record, the helper will refuse to overwrite an existing prefix. +Verify the public page and its links after publication. This manual refresh +does not add a scheduled workflow. Preserve the measurement date when choosing +a newer rendering of the same campaign. diff --git a/packages/paperclip-runner/docs/runner-protocol-live-evals.md b/packages/paperclip-runner/docs/runner-protocol-live-evals.md index fe9ee06f6ea..98270c3201e 100644 --- a/packages/paperclip-runner/docs/runner-protocol-live-evals.md +++ b/packages/paperclip-runner/docs/runner-protocol-live-evals.md @@ -1,5 +1,10 @@ # Direct live Runner protocol evals +For choosing between Runner Evals and full-stack Product E2E Evals, and for +shared guidance on provenance, history, cost, and failure classification, see +the [Paperclip evaluation guide](../../../doc/evals.md). This document remains +the authoritative Runner Evals live protocol runbook. + ## One Evalbook presentation Every new report uses the canonical Evalbook grid and the existing Runner Lab diff --git a/scripts/evals-hub/.gitignore b/scripts/evals-hub/.gitignore new file mode 100644 index 00000000000..c18dd8d83ce --- /dev/null +++ b/scripts/evals-hub/.gitignore @@ -0,0 +1 @@ +__pycache__/ diff --git a/scripts/evals-hub/build.py b/scripts/evals-hub/build.py new file mode 100644 index 00000000000..982e895ce6f --- /dev/null +++ b/scripts/evals-hub/build.py @@ -0,0 +1,96 @@ +#!/usr/bin/env python3 +"""Build the public eval directory from the two existing public history feeds.""" +import argparse +from datetime import datetime, timezone +import html +import json +from pathlib import Path +from string import Template +from urllib.parse import quote, urlsplit +from urllib.request import urlopen + +ORIGIN = "https://d1p6rlowie26tp.cloudfront.net" +SYSTEMS = { + "runner": ("runner-protocol-evals", "paperclip.runner-protocol-eval.history/v1"), + "product": ("runner-e2e", "paperclip.runner-e2e.history/v1"), +} + + +def latest(history, schema): + if history.get("schema") != schema: + raise ValueError("Unsupported history schema") + campaign_id = history.get("latestCampaignId") + if not campaign_id: + raise ValueError("History has no latest campaign") + candidates = [c for c in history["campaigns"] if + c["campaignId"] == campaign_id or + c.get("reportRevision", {}).get("sourceCampaignId") == campaign_id] + if not candidates: + raise ValueError("Latest campaign is absent from history") + # Report refreshes replace presentation, never the measurement date. + return max(candidates, key=lambda c: c.get("reportRevision", {}).get("renderedAt", c["generatedAt"])) + + +def public_report(url, prefix): + parsed = urlsplit(url) + if (parsed.scheme != "https" or parsed.netloc != urlsplit(ORIGIN).netloc + or not parsed.path.startswith(f"/{prefix}/campaigns/") + or parsed.query or parsed.fragment): + raise ValueError("Unexpected public report URL") + return html.escape(url, quote=True) + + +def display_date(value): + date = datetime.fromisoformat(value.replace("Z", "+00:00")) + if date.tzinfo is None: + raise ValueError("History date must include a timezone") + return date.astimezone(timezone.utc).strftime("%d %b %Y · %H:%M UTC") + + +def summarize(history, kind): + prefix, schema = SYSTEMS[kind] + campaign = latest(history, schema) + totals = campaign["totals"] if kind == "runner" else campaign + passed, selected = totals["passed"], totals["selected"] + if any(type(n) is not int for n in (passed, selected)) or not 0 <= passed <= selected: + raise ValueError("Invalid campaign counts") + if type(campaign.get("complete")) is not bool: + raise ValueError("Missing campaign coverage flag") + revision = campaign.get("reportRevision", {}) + if revision and not isinstance(revision.get("sourceGeneratedAt"), str): + raise ValueError("Report refresh has no reliable measurement date") + measured = revision.get("sourceGeneratedAt", campaign["generatedAt"]) + return { + "url": public_report(campaign["publicUrl"], prefix), + "date": html.escape(display_date(measured)), + "counts": f"{passed} / {selected} selected cases passed", + "coverage": "Complete campaign" if campaign["complete"] else "Partial campaign", + "id": html.escape(revision.get("sourceCampaignId", campaign["campaignId"])), + } + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--history-dir", type=Path, help="Use saved public history JSON files for offline verification") + parser.add_argument("--docs-ref", default="master", help="Git ref containing doc/evals.md") + args = parser.parse_args() + values = {} + for kind, (prefix, _) in SYSTEMS.items(): + if args.history_dir: + history = json.loads((args.history_dir / f"{prefix}-history.json").read_text()) + else: + with urlopen(f"{ORIGIN}/{prefix}/history.json", timeout=30) as response: + history = json.load(response) + values.update({f"{kind}_{k}": v for k, v in summarize(history, kind).items()}) + values["refreshed"] = display_date(datetime.now(timezone.utc).isoformat()) + values["guide_url"] = "https://github.com/paperclipai/paperclip/blob/" + quote(args.docs_ref, safe="/") + "/doc/evals.md" + template = Template(Path(__file__).with_name("template.html").read_text()) + rendered = template.substitute(values) + args.output.mkdir(parents=True, exist_ok=True) + (args.output / "index.html").write_text(rendered) + print(args.output / "index.html") + + +if __name__ == "__main__": + main() diff --git a/scripts/evals-hub/template.html b/scripts/evals-hub/template.html new file mode 100644 index 00000000000..45c94132f4a --- /dev/null +++ b/scripts/evals-hub/template.html @@ -0,0 +1,75 @@ + + + + + + + Paperclip Evals + + + +
PaperclipEvaluation directory
+
+
Evidence for better agents
+

Paperclip Evals

+

Two ways to test Paperclip. Follow each program from a test case to its evidence, results, and run history.

+
+
+
01 / RUNNER
+

Runner Evals

+

Does the runner use tools correctly, respect permissions, and preserve its execution contract?

+
Real runner + model provider
Controlled mock Paperclip control plane
+
Latest recorded measurement
$runner_counts
$runner_date · $runner_coverage
$runner_id
+
Browse run history →Open report
+ Runner operations guide +
+
+
02 / PRODUCT
+

Product E2E Evals

+

Can a user complete work through Paperclip, from the first request to a usable result?

+
Real browser + server + database + runner + model provider
Local and Daytona environments, as selected
+
Latest recorded measurement
$product_counts
$product_date · $product_coverage
$product_id
+
Browse run history →Open report
+ Product E2E operations guide +
+
+

Snapshot refreshed $refreshed. History pages may contain newer runs. A partial campaign covers only its selected cases. These programs measure different boundaries; their pass rates are not combined.

+

Choose a test. Make the result inspectable.

The eval guide explains which program to use, how to add a case, and how to distinguish a product defect from model behavior, a grading error, or a setup failure. Evalbook is a report framework used to present evidence; it is not an execution boundary.

Read the eval guide →
+ +
+ + diff --git a/scripts/evals-hub/test_build.py b/scripts/evals-hub/test_build.py new file mode 100644 index 00000000000..2960efd827a --- /dev/null +++ b/scripts/evals-hub/test_build.py @@ -0,0 +1,82 @@ +import copy +import unittest + +from build import SYSTEMS, summarize + + +class HistorySummaryTests(unittest.TestCase): + def history(self, kind="runner"): + prefix, schema = SYSTEMS[kind] + campaign = { + "campaignId": "run-1", "generatedAt": "2026-09-01T12:00:00Z", + "publicUrl": f"https://d1p6rlowie26tp.cloudfront.net/{prefix}/campaigns/run-1/", + "complete": False, + } + counts = {"selected": 3, "passed": 2} + campaign.update({"totals": counts} if kind == "runner" else counts) + return {"schema": schema, "latestCampaignId": "run-1", "campaigns": [campaign]} + + def test_both_families_keep_partial_coverage(self): + for kind in SYSTEMS: + with self.subTest(kind=kind): + result = summarize(self.history(kind), kind) + self.assertEqual(result["coverage"], "Partial campaign") + self.assertEqual(result["counts"], "2 / 3 selected cases passed") + + def test_refresh_changes_report_but_not_measurement_date(self): + history = self.history() + refreshed = copy.deepcopy(history["campaigns"][0]) + refreshed.update(campaignId="run-1-refresh", generatedAt="2026-09-03T12:00:00Z") + refreshed["publicUrl"] += "refresh/" + refreshed["reportRevision"] = { + "sourceCampaignId": "run-1", "sourceGeneratedAt": "2026-09-01T12:00:00Z", + "renderedAt": "2026-09-03T12:00:00Z", + } + history["campaigns"].append(refreshed) + result = summarize(history, "runner") + self.assertEqual(result["id"], "run-1") + self.assertIn("01 Sep 2026", result["date"]) + self.assertTrue(result["url"].endswith("/refresh/")) + + def test_missing_latest_does_not_fall_back_to_an_arbitrary_run(self): + history = self.history() + history["latestCampaignId"] = "absent" + with self.assertRaises(ValueError): + summarize(history, "runner") + + def test_refresh_without_measurement_date_is_rejected(self): + history = self.history() + campaign = history["campaigns"][0] + campaign["campaignId"] = "run-1-refresh" + campaign["reportRevision"] = { + "sourceCampaignId": "run-1", "renderedAt": "2026-09-03T12:00:00Z", + } + with self.assertRaisesRegex(ValueError, "measurement date"): + summarize(history, "runner") + + def test_unknown_schema_and_missing_coverage_are_rejected(self): + for field in ("schema", "complete"): + history = self.history() + if field == "schema": + history["schema"] = "future-schema" + else: + del history["campaigns"][0]["complete"] + with self.subTest(field=field), self.assertRaises(ValueError): + summarize(history, "runner") + + def test_bad_counts_cannot_be_published(self): + history = self.history() + history["campaigns"][0]["totals"]["passed"] = 4 + with self.assertRaises(ValueError): + summarize(history, "runner") + + def test_external_report_urls_are_rejected(self): + history = self.history() + for url in ("javascript:alert(1)", "https://example.com/", "https://d1p6rlowie26tp.cloudfront.net/other/"): + history["campaigns"][0]["publicUrl"] = url + with self.subTest(url=url), self.assertRaises(ValueError): + summarize(history, "runner") + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index 912497d27df..8f201663021 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -1,5 +1,11 @@ # Paid runner full-stack E2E +For family selection, ownership, provenance, history, and failure taxonomy, +see the [Paperclip evaluation guide](../../doc/evals.md). This README is the +authoritative runbook for Product E2E runner cells; the separate Runner Evals +protocol guide lives at +`packages/paperclip-runner/docs/runner-protocol-live-evals.md`. + This is the billable browser acceptance campaign system for Paperclip runner profiles. It is deliberately separate from `tests/e2e`: every independently scheduled execution gets