mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 11:54:35 +01:00
docs: add eval guide, authoring skills, and public history hub (#13535)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its evaluations test both the Runner and complete product workflows. > - The guides and run histories are in separate places. > - The shared Evalbook viewer can make the test boundary unclear. > - This pull request names the two families and adds a guide, authoring skills, and a public hub. > - Contributors can choose the correct test and inspect its history. ## Linked Issues or Issue Description **Issue type** Missing documentation. **Where is the issue?** Runner and Product E2E evaluation guides, case-authoring procedures, and public result navigation. **What's wrong?** There is no single entry point. A report format can be mistaken for an execution boundary. There are no dedicated case-authoring skills for these two families. **Suggested fix** Add a guide and three skills. Link both existing histories from a public hub. Keep existing campaign URLs and grading unchanged. ## What Changed - Add `doc/evals.md` and links from existing guides. - Add the `paperclip-evals`, `add-runner-eval`, and `add-product-e2e-eval` skills. Install copies in `~/paperclipai/.agents/skills`. - Add a static hub builder that reads the existing public history feeds. - Show a dated snapshot for each family. Label partial campaigns and preserve measurement dates across report refreshes. - Document publication and refresh commands for https://pages.paperclip.ing/evals/. ## Verification - Seven Python summary tests pass: `python3 -m unittest discover -s scripts/evals-hub -p 'test_*.py'`. Run these checks directly; this PR does not modify package scripts. - All three skills pass the skill-creator `quick_validate.py` check with `/usr/bin/python3`. - Build tested with saved history fixtures and the live public feeds. - Desktop and mobile browser checks pass. The mobile page has no horizontal overflow. - Published https://pages.paperclip.ing/evals/. Browser check: HTTP 200, no page errors, all eight links return HTTP 200, no mobile overflow. - Independent skill exercises found the existing Notion-decline case and a direct Runner permission-denial case. Roster validation with an explicit run ID passes. - Missing refresh measurement date: regression fails before the fix and passes after it. - `git diff --check` passes. - No paid evals were run for this documentation and reporting change. The preceding head passed typecheck, build, server/workspace tests, runner verification, browser E2E, and the canary dry run. Checks for the latest commit are pending. Local repository-wide typecheck, test, and build were not repeated because no product code changed. ## Risks The hub is a dated static snapshot. It can lag behind the linked histories until an operator refreshes it. A changed history schema stops the build. Existing archives and grades are not modified. The published guide link is pinned to the reviewed commit so branch deletion cannot break it. Later builds can use master. ## Model Used OpenAI gpt-6-astra for implementation and review. OpenAI gpt-5.6-luna for documentation and independent skill checks. Both used repository tools and code execution. Context window sizes are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green (latest commit pending) - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups (preceding head was 5/5; latest commit pending) - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
4577d10029
commit
18989a9e73
10 files changed
+698
No files matched your search
@@ -0,0 +1 @@
|
||||
__pycache__/
|
||||
@@ -0,0 +1,96 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Build the public eval directory from the two existing public history feeds."""
|
||||
import argparse
|
||||
from datetime import datetime, timezone
|
||||
import html
|
||||
import json
|
||||
from pathlib import Path
|
||||
from string import Template
|
||||
from urllib.parse import quote, urlsplit
|
||||
from urllib.request import urlopen
|
||||
|
||||
ORIGIN = "https://d1p6rlowie26tp.cloudfront.net"
|
||||
SYSTEMS = {
|
||||
"runner": ("runner-protocol-evals", "paperclip.runner-protocol-eval.history/v1"),
|
||||
"product": ("runner-e2e", "paperclip.runner-e2e.history/v1"),
|
||||
}
|
||||
|
||||
|
||||
def latest(history, schema):
|
||||
if history.get("schema") != schema:
|
||||
raise ValueError("Unsupported history schema")
|
||||
campaign_id = history.get("latestCampaignId")
|
||||
if not campaign_id:
|
||||
raise ValueError("History has no latest campaign")
|
||||
candidates = [c for c in history["campaigns"] if
|
||||
c["campaignId"] == campaign_id or
|
||||
c.get("reportRevision", {}).get("sourceCampaignId") == campaign_id]
|
||||
if not candidates:
|
||||
raise ValueError("Latest campaign is absent from history")
|
||||
# Report refreshes replace presentation, never the measurement date.
|
||||
return max(candidates, key=lambda c: c.get("reportRevision", {}).get("renderedAt", c["generatedAt"]))
|
||||
|
||||
|
||||
def public_report(url, prefix):
|
||||
parsed = urlsplit(url)
|
||||
if (parsed.scheme != "https" or parsed.netloc != urlsplit(ORIGIN).netloc
|
||||
or not parsed.path.startswith(f"/{prefix}/campaigns/")
|
||||
or parsed.query or parsed.fragment):
|
||||
raise ValueError("Unexpected public report URL")
|
||||
return html.escape(url, quote=True)
|
||||
|
||||
|
||||
def display_date(value):
|
||||
date = datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||
if date.tzinfo is None:
|
||||
raise ValueError("History date must include a timezone")
|
||||
return date.astimezone(timezone.utc).strftime("%d %b %Y · %H:%M UTC")
|
||||
|
||||
|
||||
def summarize(history, kind):
|
||||
prefix, schema = SYSTEMS[kind]
|
||||
campaign = latest(history, schema)
|
||||
totals = campaign["totals"] if kind == "runner" else campaign
|
||||
passed, selected = totals["passed"], totals["selected"]
|
||||
if any(type(n) is not int for n in (passed, selected)) or not 0 <= passed <= selected:
|
||||
raise ValueError("Invalid campaign counts")
|
||||
if type(campaign.get("complete")) is not bool:
|
||||
raise ValueError("Missing campaign coverage flag")
|
||||
revision = campaign.get("reportRevision", {})
|
||||
if revision and not isinstance(revision.get("sourceGeneratedAt"), str):
|
||||
raise ValueError("Report refresh has no reliable measurement date")
|
||||
measured = revision.get("sourceGeneratedAt", campaign["generatedAt"])
|
||||
return {
|
||||
"url": public_report(campaign["publicUrl"], prefix),
|
||||
"date": html.escape(display_date(measured)),
|
||||
"counts": f"{passed} / {selected} selected cases passed",
|
||||
"coverage": "Complete campaign" if campaign["complete"] else "Partial campaign",
|
||||
"id": html.escape(revision.get("sourceCampaignId", campaign["campaignId"])),
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument("--history-dir", type=Path, help="Use saved public history JSON files for offline verification")
|
||||
parser.add_argument("--docs-ref", default="master", help="Git ref containing doc/evals.md")
|
||||
args = parser.parse_args()
|
||||
values = {}
|
||||
for kind, (prefix, _) in SYSTEMS.items():
|
||||
if args.history_dir:
|
||||
history = json.loads((args.history_dir / f"{prefix}-history.json").read_text())
|
||||
else:
|
||||
with urlopen(f"{ORIGIN}/{prefix}/history.json", timeout=30) as response:
|
||||
history = json.load(response)
|
||||
values.update({f"{kind}_{k}": v for k, v in summarize(history, kind).items()})
|
||||
values["refreshed"] = display_date(datetime.now(timezone.utc).isoformat())
|
||||
values["guide_url"] = "https://github.com/paperclipai/paperclip/blob/" + quote(args.docs_ref, safe="/") + "/doc/evals.md"
|
||||
template = Template(Path(__file__).with_name("template.html").read_text())
|
||||
rendered = template.substitute(values)
|
||||
args.output.mkdir(parents=True, exist_ok=True)
|
||||
(args.output / "index.html").write_text(rendered)
|
||||
print(args.output / "index.html")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,75 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<meta name="description" content="Paperclip evaluation results, run histories, and guides for Runner Evals and Product E2E Evals.">
|
||||
<title>Paperclip Evals</title>
|
||||
<style>
|
||||
:root { color-scheme: light dark; --bg:#f8f8f5; --card:#fff; --text:#20221e; --muted:#60645c; --line:#d9ddd3; --accent:#306340; --tint:#edf3ec; }
|
||||
* { box-sizing:border-box; }
|
||||
body { margin:0; background:var(--bg); color:var(--text); font:16px/1.6 -apple-system,BlinkMacSystemFont,"Segoe UI",sans-serif; }
|
||||
a { color:inherit; text-underline-offset:4px; }
|
||||
a:focus-visible { outline:3px solid var(--accent); outline-offset:5px; }
|
||||
.bar { border-bottom:1px solid var(--line); }
|
||||
.bar div { max-width:1160px; margin:auto; padding:20px 28px; display:flex; justify-content:space-between; align-items:center; gap:20px; }
|
||||
.brand { font-weight:750; letter-spacing:-.04em; font-size:21px; text-decoration:none; }
|
||||
.bar span, .eyebrow { color:var(--muted); font-size:12px; text-transform:uppercase; letter-spacing:.13em; }
|
||||
main { max-width:1160px; margin:64px auto; padding:0 28px; }
|
||||
h1 { font-size:clamp(38px,6vw,62px); line-height:1.05; letter-spacing:-.055em; margin:14px 0 20px; font-weight:650; }
|
||||
.intro { max-width:690px; font-size:19px; color:var(--muted); margin-bottom:36px; }
|
||||
.grid { display:grid; grid-template-columns:repeat(2,minmax(0,1fr)); gap:24px; }
|
||||
article { background:var(--card); border:1px solid var(--line); border-radius:12px; padding:30px; display:flex; flex-direction:column; }
|
||||
.number { font:13px/1.4 ui-monospace,monospace; color:var(--accent); }
|
||||
h2 { margin:12px 0 12px; letter-spacing:-.035em; font-size:29px; line-height:1.2; }
|
||||
article p { color:var(--muted); margin:0 0 22px; }
|
||||
.boundary { font-size:14px; padding:14px 16px; background:var(--tint); border-radius:6px; margin-bottom:24px; }
|
||||
.latest { border-top:1px solid var(--line); padding-top:20px; margin-top:auto; }
|
||||
.latest .label { color:var(--muted); font-size:12px; text-transform:uppercase; letter-spacing:.08em; }
|
||||
.counts { font-size:19px; font-weight:600; margin:7px 0; }
|
||||
.meta { font-size:13px; color:var(--muted); }
|
||||
.id { overflow-wrap:anywhere; margin-top:4px; font:11px/1.6 ui-monospace,monospace; color:var(--muted); }
|
||||
.links { display:flex; flex-wrap:wrap; align-items:center; gap:18px; margin:24px 0 18px; }
|
||||
.button { display:inline-block; background:var(--text); color:var(--card); padding:10px 15px; border-radius:6px; text-decoration:none; font-weight:600; font-size:14px; }
|
||||
.guide { font-size:13px; }
|
||||
.note { font-size:13px; color:var(--muted); margin:19px 0 38px; }
|
||||
.start { display:flex; gap:28px; justify-content:space-between; align-items:center; padding:25px 0; border-top:1px solid var(--line); border-bottom:1px solid var(--line); }
|
||||
.start h2 { font-size:22px; margin:0 0 6px; }
|
||||
.start p { margin:0; color:var(--muted); font-size:14px; max-width:650px; }
|
||||
footer { margin-top:25px; color:var(--muted); font-size:12px; }
|
||||
@media (prefers-color-scheme:dark) { :root { --bg:#151714; --card:#1d201b; --text:#f2f4ed; --muted:#aeb6a6; --line:#363d31; --accent:#a0cda5; --tint:#273024; } }
|
||||
@media (max-width:700px) { main { margin:36px auto; padding:0 20px; } .grid { grid-template-columns:1fr; } article { padding:24px; } .start { align-items:flex-start; flex-direction:column; gap:18px; } .bar div { padding:16px 20px; } .bar span { font-size:10px; } }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<header class="bar"><div><a class="brand" href="https://paperclip.ing">Paperclip</a><span>Evaluation directory</span></div></header>
|
||||
<main>
|
||||
<div class="eyebrow">Evidence for better agents</div>
|
||||
<h1>Paperclip Evals</h1>
|
||||
<p class="intro">Two ways to test Paperclip. Follow each program from a test case to its evidence, results, and run history.</p>
|
||||
<div class="grid">
|
||||
<article>
|
||||
<div class="number">01 / RUNNER</div>
|
||||
<h2>Runner Evals</h2>
|
||||
<p>Does the runner use tools correctly, respect permissions, and preserve its execution contract?</p>
|
||||
<div class="boundary">Real runner + model provider<br>Controlled mock Paperclip control plane</div>
|
||||
<div class="latest"><div class="label">Latest recorded measurement</div><div class="counts">$runner_counts</div><div class="meta">$runner_date · $runner_coverage</div><div class="id">$runner_id</div></div>
|
||||
<div class="links"><a class="button" href="https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/index.html">Browse run history →</a><a href="$runner_url">Open report</a></div>
|
||||
<a class="guide" href="https://github.com/paperclipai/paperclip/blob/master/packages/paperclip-runner/docs/runner-protocol-live-evals.md">Runner operations guide</a>
|
||||
</article>
|
||||
<article>
|
||||
<div class="number">02 / PRODUCT</div>
|
||||
<h2>Product E2E Evals</h2>
|
||||
<p>Can a user complete work through Paperclip, from the first request to a usable result?</p>
|
||||
<div class="boundary">Real browser + server + database + runner + model provider<br>Local and Daytona environments, as selected</div>
|
||||
<div class="latest"><div class="label">Latest recorded measurement</div><div class="counts">$product_counts</div><div class="meta">$product_date · $product_coverage</div><div class="id">$product_id</div></div>
|
||||
<div class="links"><a class="button" href="https://d1p6rlowie26tp.cloudfront.net/runner-e2e/">Browse run history →</a><a href="$product_url">Open report</a></div>
|
||||
<a class="guide" href="https://github.com/paperclipai/paperclip/blob/master/tests/runner-e2e/README.md">Product E2E operations guide</a>
|
||||
</article>
|
||||
</div>
|
||||
<p class="note">Snapshot refreshed $refreshed. History pages may contain newer runs. A partial campaign covers only its selected cases. These programs measure different boundaries; their pass rates are not combined.</p>
|
||||
<section class="start"><div><h2>Choose a test. Make the result inspectable.</h2><p>The eval guide explains which program to use, how to add a case, and how to distinguish a product defect from model behavior, a grading error, or a setup failure. Evalbook is a report framework used to present evidence; it is not an execution boundary.</p></div><a class="button" href="$guide_url">Read the eval guide →</a></section>
|
||||
<footer>Existing run archives remain the source of truth. Report refreshes do not count as new model runs.</footer>
|
||||
</main>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,82 @@
|
||||
import copy
|
||||
import unittest
|
||||
|
||||
from build import SYSTEMS, summarize
|
||||
|
||||
|
||||
class HistorySummaryTests(unittest.TestCase):
|
||||
def history(self, kind="runner"):
|
||||
prefix, schema = SYSTEMS[kind]
|
||||
campaign = {
|
||||
"campaignId": "run-1", "generatedAt": "2026-09-01T12:00:00Z",
|
||||
"publicUrl": f"https://d1p6rlowie26tp.cloudfront.net/{prefix}/campaigns/run-1/",
|
||||
"complete": False,
|
||||
}
|
||||
counts = {"selected": 3, "passed": 2}
|
||||
campaign.update({"totals": counts} if kind == "runner" else counts)
|
||||
return {"schema": schema, "latestCampaignId": "run-1", "campaigns": [campaign]}
|
||||
|
||||
def test_both_families_keep_partial_coverage(self):
|
||||
for kind in SYSTEMS:
|
||||
with self.subTest(kind=kind):
|
||||
result = summarize(self.history(kind), kind)
|
||||
self.assertEqual(result["coverage"], "Partial campaign")
|
||||
self.assertEqual(result["counts"], "2 / 3 selected cases passed")
|
||||
|
||||
def test_refresh_changes_report_but_not_measurement_date(self):
|
||||
history = self.history()
|
||||
refreshed = copy.deepcopy(history["campaigns"][0])
|
||||
refreshed.update(campaignId="run-1-refresh", generatedAt="2026-09-03T12:00:00Z")
|
||||
refreshed["publicUrl"] += "refresh/"
|
||||
refreshed["reportRevision"] = {
|
||||
"sourceCampaignId": "run-1", "sourceGeneratedAt": "2026-09-01T12:00:00Z",
|
||||
"renderedAt": "2026-09-03T12:00:00Z",
|
||||
}
|
||||
history["campaigns"].append(refreshed)
|
||||
result = summarize(history, "runner")
|
||||
self.assertEqual(result["id"], "run-1")
|
||||
self.assertIn("01 Sep 2026", result["date"])
|
||||
self.assertTrue(result["url"].endswith("/refresh/"))
|
||||
|
||||
def test_missing_latest_does_not_fall_back_to_an_arbitrary_run(self):
|
||||
history = self.history()
|
||||
history["latestCampaignId"] = "absent"
|
||||
with self.assertRaises(ValueError):
|
||||
summarize(history, "runner")
|
||||
|
||||
def test_refresh_without_measurement_date_is_rejected(self):
|
||||
history = self.history()
|
||||
campaign = history["campaigns"][0]
|
||||
campaign["campaignId"] = "run-1-refresh"
|
||||
campaign["reportRevision"] = {
|
||||
"sourceCampaignId": "run-1", "renderedAt": "2026-09-03T12:00:00Z",
|
||||
}
|
||||
with self.assertRaisesRegex(ValueError, "measurement date"):
|
||||
summarize(history, "runner")
|
||||
|
||||
def test_unknown_schema_and_missing_coverage_are_rejected(self):
|
||||
for field in ("schema", "complete"):
|
||||
history = self.history()
|
||||
if field == "schema":
|
||||
history["schema"] = "future-schema"
|
||||
else:
|
||||
del history["campaigns"][0]["complete"]
|
||||
with self.subTest(field=field), self.assertRaises(ValueError):
|
||||
summarize(history, "runner")
|
||||
|
||||
def test_bad_counts_cannot_be_published(self):
|
||||
history = self.history()
|
||||
history["campaigns"][0]["totals"]["passed"] = 4
|
||||
with self.assertRaises(ValueError):
|
||||
summarize(history, "runner")
|
||||
|
||||
def test_external_report_urls_are_rejected(self):
|
||||
history = self.history()
|
||||
for url in ("javascript:alert(1)", "https://example.com/", "https://d1p6rlowie26tp.cloudfront.net/other/"):
|
||||
history["campaigns"][0]["publicUrl"] = url
|
||||
with self.subTest(url=url), self.assertRaises(ValueError):
|
||||
summarize(history, "runner")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in new issue
Block a user