docs: add eval guide, authoring skills, and public history hub (#13535)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Its evaluations test both the Runner and complete product workflows.
> - The guides and run histories are in separate places.
> - The shared Evalbook viewer can make the test boundary unclear.
> - This pull request names the two families and adds a guide, authoring
skills, and a public hub.
> - Contributors can choose the correct test and inspect its history.

## Linked Issues or Issue Description

**Issue type**

Missing documentation.

**Where is the issue?**

Runner and Product E2E evaluation guides, case-authoring procedures, and
public result navigation.

**What's wrong?**

There is no single entry point. A report format can be mistaken for an
execution boundary. There are no dedicated case-authoring skills for
these two families.

**Suggested fix**

Add a guide and three skills. Link both existing histories from a public
hub. Keep existing campaign URLs and grading unchanged.

## What Changed

- Add `doc/evals.md` and links from existing guides.
- Add the `paperclip-evals`, `add-runner-eval`, and
`add-product-e2e-eval` skills. Install copies in
`~/paperclipai/.agents/skills`.
- Add a static hub builder that reads the existing public history feeds.
- Show a dated snapshot for each family. Label partial campaigns and
preserve measurement dates across report refreshes.
- Document publication and refresh commands for
https://pages.paperclip.ing/evals/.

## Verification

- Seven Python summary tests pass: `python3 -m unittest discover -s
scripts/evals-hub -p 'test_*.py'`. Run these checks directly; this PR
does not modify package scripts.
- All three skills pass the skill-creator `quick_validate.py` check with
`/usr/bin/python3`.
- Build tested with saved history fixtures and the live public feeds.
- Desktop and mobile browser checks pass. The mobile page has no
horizontal overflow.
- Published https://pages.paperclip.ing/evals/. Browser check: HTTP 200,
no page errors, all eight links return HTTP 200, no mobile overflow.
- Independent skill exercises found the existing Notion-decline case and
a direct Runner permission-denial case. Roster validation with an
explicit run ID passes.
- Missing refresh measurement date: regression fails before the fix and
passes after it.
- `git diff --check` passes.
- No paid evals were run for this documentation and reporting change.
The preceding head passed typecheck, build, server/workspace tests,
runner verification, browser E2E, and the canary dry run. Checks for the
latest commit are pending. Local repository-wide typecheck, test, and
build were not repeated because no product code changed.

## Risks

The hub is a dated static snapshot. It can lag behind the linked
histories until an operator refreshes it. A changed history schema stops
the build. Existing archives and grades are not modified. The published
guide link is pinned to the reviewed commit so branch deletion cannot
break it. Later builds can use master.

## Model Used

OpenAI gpt-6-astra for implementation and review. OpenAI gpt-5.6-luna
for documentation and independent skill checks. Both used repository
tools and code execution. Context window sizes are not exposed by this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green (latest commit pending)
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
(preceding head was 5/5; latest commit pending)
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-09-16 08:41:06 -05:00
1 parent 4577d10029
commit 18989a9e73
10 files changed
+698

No files matched your search

+1
View File
@@ -0,0 +1 @@
__pycache__/
+96
View File
@@ -0,0 +1,96 @@
#!/usr/bin/env python3
"""Build the public eval directory from the two existing public history feeds."""
import argparse
from datetime import datetime, timezone
import html
import json
from pathlib import Path
from string import Template
from urllib.parse import quote, urlsplit
from urllib.request import urlopen
ORIGIN = "https://d1p6rlowie26tp.cloudfront.net"
SYSTEMS = {
"runner": ("runner-protocol-evals", "paperclip.runner-protocol-eval.history/v1"),
"product": ("runner-e2e", "paperclip.runner-e2e.history/v1"),
}
def latest(history, schema):
if history.get("schema") != schema:
raise ValueError("Unsupported history schema")
campaign_id = history.get("latestCampaignId")
if not campaign_id:
raise ValueError("History has no latest campaign")
candidates = [c for c in history["campaigns"] if
c["campaignId"] == campaign_id or
c.get("reportRevision", {}).get("sourceCampaignId") == campaign_id]
if not candidates:
raise ValueError("Latest campaign is absent from history")
# Report refreshes replace presentation, never the measurement date.
return max(candidates, key=lambda c: c.get("reportRevision", {}).get("renderedAt", c["generatedAt"]))
def public_report(url, prefix):
parsed = urlsplit(url)
if (parsed.scheme != "https" or parsed.netloc != urlsplit(ORIGIN).netloc
or not parsed.path.startswith(f"/{prefix}/campaigns/")
or parsed.query or parsed.fragment):
raise ValueError("Unexpected public report URL")
return html.escape(url, quote=True)
def display_date(value):
date = datetime.fromisoformat(value.replace("Z", "+00:00"))
if date.tzinfo is None:
raise ValueError("History date must include a timezone")
return date.astimezone(timezone.utc).strftime("%d %b %Y · %H:%M UTC")
def summarize(history, kind):
prefix, schema = SYSTEMS[kind]
campaign = latest(history, schema)
totals = campaign["totals"] if kind == "runner" else campaign
passed, selected = totals["passed"], totals["selected"]
if any(type(n) is not int for n in (passed, selected)) or not 0 <= passed <= selected:
raise ValueError("Invalid campaign counts")
if type(campaign.get("complete")) is not bool:
raise ValueError("Missing campaign coverage flag")
revision = campaign.get("reportRevision", {})
if revision and not isinstance(revision.get("sourceGeneratedAt"), str):
raise ValueError("Report refresh has no reliable measurement date")
measured = revision.get("sourceGeneratedAt", campaign["generatedAt"])
return {
"url": public_report(campaign["publicUrl"], prefix),
"date": html.escape(display_date(measured)),
"counts": f"{passed} / {selected} selected cases passed",
"coverage": "Complete campaign" if campaign["complete"] else "Partial campaign",
"id": html.escape(revision.get("sourceCampaignId", campaign["campaignId"])),
}
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--output", type=Path, required=True)
parser.add_argument("--history-dir", type=Path, help="Use saved public history JSON files for offline verification")
parser.add_argument("--docs-ref", default="master", help="Git ref containing doc/evals.md")
args = parser.parse_args()
values = {}
for kind, (prefix, _) in SYSTEMS.items():
if args.history_dir:
history = json.loads((args.history_dir / f"{prefix}-history.json").read_text())
else:
with urlopen(f"{ORIGIN}/{prefix}/history.json", timeout=30) as response:
history = json.load(response)
values.update({f"{kind}_{k}": v for k, v in summarize(history, kind).items()})
values["refreshed"] = display_date(datetime.now(timezone.utc).isoformat())
values["guide_url"] = "https://github.com/paperclipai/paperclip/blob/" + quote(args.docs_ref, safe="/") + "/doc/evals.md"
template = Template(Path(__file__).with_name("template.html").read_text())
rendered = template.substitute(values)
args.output.mkdir(parents=True, exist_ok=True)
(args.output / "index.html").write_text(rendered)
print(args.output / "index.html")
if __name__ == "__main__":
main()
+75
View File
@@ -0,0 +1,75 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="Paperclip evaluation results, run histories, and guides for Runner Evals and Product E2E Evals.">
<title>Paperclip Evals</title>
<style>
:root { color-scheme: light dark; --bg:#f8f8f5; --card:#fff; --text:#20221e; --muted:#60645c; --line:#d9ddd3; --accent:#306340; --tint:#edf3ec; }
* { box-sizing:border-box; }
body { margin:0; background:var(--bg); color:var(--text); font:16px/1.6 -apple-system,BlinkMacSystemFont,"Segoe UI",sans-serif; }
a { color:inherit; text-underline-offset:4px; }
a:focus-visible { outline:3px solid var(--accent); outline-offset:5px; }
.bar { border-bottom:1px solid var(--line); }
.bar div { max-width:1160px; margin:auto; padding:20px 28px; display:flex; justify-content:space-between; align-items:center; gap:20px; }
.brand { font-weight:750; letter-spacing:-.04em; font-size:21px; text-decoration:none; }
.bar span, .eyebrow { color:var(--muted); font-size:12px; text-transform:uppercase; letter-spacing:.13em; }
main { max-width:1160px; margin:64px auto; padding:0 28px; }
h1 { font-size:clamp(38px,6vw,62px); line-height:1.05; letter-spacing:-.055em; margin:14px 0 20px; font-weight:650; }
.intro { max-width:690px; font-size:19px; color:var(--muted); margin-bottom:36px; }
.grid { display:grid; grid-template-columns:repeat(2,minmax(0,1fr)); gap:24px; }
article { background:var(--card); border:1px solid var(--line); border-radius:12px; padding:30px; display:flex; flex-direction:column; }
.number { font:13px/1.4 ui-monospace,monospace; color:var(--accent); }
h2 { margin:12px 0 12px; letter-spacing:-.035em; font-size:29px; line-height:1.2; }
article p { color:var(--muted); margin:0 0 22px; }
.boundary { font-size:14px; padding:14px 16px; background:var(--tint); border-radius:6px; margin-bottom:24px; }
.latest { border-top:1px solid var(--line); padding-top:20px; margin-top:auto; }
.latest .label { color:var(--muted); font-size:12px; text-transform:uppercase; letter-spacing:.08em; }
.counts { font-size:19px; font-weight:600; margin:7px 0; }
.meta { font-size:13px; color:var(--muted); }
.id { overflow-wrap:anywhere; margin-top:4px; font:11px/1.6 ui-monospace,monospace; color:var(--muted); }
.links { display:flex; flex-wrap:wrap; align-items:center; gap:18px; margin:24px 0 18px; }
.button { display:inline-block; background:var(--text); color:var(--card); padding:10px 15px; border-radius:6px; text-decoration:none; font-weight:600; font-size:14px; }
.guide { font-size:13px; }
.note { font-size:13px; color:var(--muted); margin:19px 0 38px; }
.start { display:flex; gap:28px; justify-content:space-between; align-items:center; padding:25px 0; border-top:1px solid var(--line); border-bottom:1px solid var(--line); }
.start h2 { font-size:22px; margin:0 0 6px; }
.start p { margin:0; color:var(--muted); font-size:14px; max-width:650px; }
footer { margin-top:25px; color:var(--muted); font-size:12px; }
@media (prefers-color-scheme:dark) { :root { --bg:#151714; --card:#1d201b; --text:#f2f4ed; --muted:#aeb6a6; --line:#363d31; --accent:#a0cda5; --tint:#273024; } }
@media (max-width:700px) { main { margin:36px auto; padding:0 20px; } .grid { grid-template-columns:1fr; } article { padding:24px; } .start { align-items:flex-start; flex-direction:column; gap:18px; } .bar div { padding:16px 20px; } .bar span { font-size:10px; } }
</style>
</head>
<body>
<header class="bar"><div><a class="brand" href="https://paperclip.ing">Paperclip</a><span>Evaluation directory</span></div></header>
<main>
<div class="eyebrow">Evidence for better agents</div>
<h1>Paperclip Evals</h1>
<p class="intro">Two ways to test Paperclip. Follow each program from a test case to its evidence, results, and run history.</p>
<div class="grid">
<article>
<div class="number">01 / RUNNER</div>
<h2>Runner Evals</h2>
<p>Does the runner use tools correctly, respect permissions, and preserve its execution contract?</p>
<div class="boundary">Real runner + model provider<br>Controlled mock Paperclip control plane</div>
<div class="latest"><div class="label">Latest recorded measurement</div><div class="counts">$runner_counts</div><div class="meta">$runner_date · $runner_coverage</div><div class="id">$runner_id</div></div>
<div class="links"><a class="button" href="https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/index.html">Browse run history →</a><a href="$runner_url">Open report</a></div>
<a class="guide" href="https://github.com/paperclipai/paperclip/blob/master/packages/paperclip-runner/docs/runner-protocol-live-evals.md">Runner operations guide</a>
</article>
<article>
<div class="number">02 / PRODUCT</div>
<h2>Product E2E Evals</h2>
<p>Can a user complete work through Paperclip, from the first request to a usable result?</p>
<div class="boundary">Real browser + server + database + runner + model provider<br>Local and Daytona environments, as selected</div>
<div class="latest"><div class="label">Latest recorded measurement</div><div class="counts">$product_counts</div><div class="meta">$product_date · $product_coverage</div><div class="id">$product_id</div></div>
<div class="links"><a class="button" href="https://d1p6rlowie26tp.cloudfront.net/runner-e2e/">Browse run history →</a><a href="$product_url">Open report</a></div>
<a class="guide" href="https://github.com/paperclipai/paperclip/blob/master/tests/runner-e2e/README.md">Product E2E operations guide</a>
</article>
</div>
<p class="note">Snapshot refreshed $refreshed. History pages may contain newer runs. A partial campaign covers only its selected cases. These programs measure different boundaries; their pass rates are not combined.</p>
<section class="start"><div><h2>Choose a test. Make the result inspectable.</h2><p>The eval guide explains which program to use, how to add a case, and how to distinguish a product defect from model behavior, a grading error, or a setup failure. Evalbook is a report framework used to present evidence; it is not an execution boundary.</p></div><a class="button" href="$guide_url">Read the eval guide →</a></section>
<footer>Existing run archives remain the source of truth. Report refreshes do not count as new model runs.</footer>
</main>
</body>
</html>
+82
View File
@@ -0,0 +1,82 @@
import copy
import unittest
from build import SYSTEMS, summarize
class HistorySummaryTests(unittest.TestCase):
def history(self, kind="runner"):
prefix, schema = SYSTEMS[kind]
campaign = {
"campaignId": "run-1", "generatedAt": "2026-09-01T12:00:00Z",
"publicUrl": f"https://d1p6rlowie26tp.cloudfront.net/{prefix}/campaigns/run-1/",
"complete": False,
}
counts = {"selected": 3, "passed": 2}
campaign.update({"totals": counts} if kind == "runner" else counts)
return {"schema": schema, "latestCampaignId": "run-1", "campaigns": [campaign]}
def test_both_families_keep_partial_coverage(self):
for kind in SYSTEMS:
with self.subTest(kind=kind):
result = summarize(self.history(kind), kind)
self.assertEqual(result["coverage"], "Partial campaign")
self.assertEqual(result["counts"], "2 / 3 selected cases passed")
def test_refresh_changes_report_but_not_measurement_date(self):
history = self.history()
refreshed = copy.deepcopy(history["campaigns"][0])
refreshed.update(campaignId="run-1-refresh", generatedAt="2026-09-03T12:00:00Z")
refreshed["publicUrl"] += "refresh/"
refreshed["reportRevision"] = {
"sourceCampaignId": "run-1", "sourceGeneratedAt": "2026-09-01T12:00:00Z",
"renderedAt": "2026-09-03T12:00:00Z",
}
history["campaigns"].append(refreshed)
result = summarize(history, "runner")
self.assertEqual(result["id"], "run-1")
self.assertIn("01 Sep 2026", result["date"])
self.assertTrue(result["url"].endswith("/refresh/"))
def test_missing_latest_does_not_fall_back_to_an_arbitrary_run(self):
history = self.history()
history["latestCampaignId"] = "absent"
with self.assertRaises(ValueError):
summarize(history, "runner")
def test_refresh_without_measurement_date_is_rejected(self):
history = self.history()
campaign = history["campaigns"][0]
campaign["campaignId"] = "run-1-refresh"
campaign["reportRevision"] = {
"sourceCampaignId": "run-1", "renderedAt": "2026-09-03T12:00:00Z",
}
with self.assertRaisesRegex(ValueError, "measurement date"):
summarize(history, "runner")
def test_unknown_schema_and_missing_coverage_are_rejected(self):
for field in ("schema", "complete"):
history = self.history()
if field == "schema":
history["schema"] = "future-schema"
else:
del history["campaigns"][0]["complete"]
with self.subTest(field=field), self.assertRaises(ValueError):
summarize(history, "runner")
def test_bad_counts_cannot_be_published(self):
history = self.history()
history["campaigns"][0]["totals"]["passed"] = 4
with self.assertRaises(ValueError):
summarize(history, "runner")
def test_external_report_urls_are_rejected(self):
history = self.history()
for url in ("javascript:alert(1)", "https://example.com/", "https://d1p6rlowie26tp.cloudfront.net/other/"):
history["campaigns"][0]["publicUrl"] = url
with self.subTest(url=url), self.assertRaises(ValueError):
summarize(history, "runner")
if __name__ == "__main__":
unittest.main()