mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-16 05:17:41 +02:00
Dotta
5716fe907e
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The runner subsystem executes agent work across local and managed provider backends. > - The lower pull requests restore the task runtime, provider backends, and managed-provider control plane. > - The restored system needs repeatable full-stack checks before it can ship safely. > - Paid live checks also need clear access, cost, and secret controls. > - This pull request adds acceptance, live evaluation, chaos, and release gates for the restored runner stack. > - The benefit is measurable runner parity with safer release decisions. ## Linked Issues or Issue Description **Subsystem affected** Cross-cutting. This change covers runner tests, release workflows, server contracts, and evaluation tools. **Problem or motivation** The runner stack did not have one complete acceptance surface for native Codex, ACPX, Claude Managed, and AWS AgentCore. Release checks could miss provider drift, task-view regressions, cost-policy errors, and destructive cleanup errors. **Proposed solution** Add a 57-cell full-stack catalog, a Daytona image, and opt-in paid workflows. Add live evaluation, chaos, cost-limit, redaction, and release contract checks. Add AWS AgentCore infrastructure and guarded provisioning tools. Keep the native runner experimental flag off by default. **Alternatives considered** We considered manual smoke tests only. They do not give repeatable evidence and they do not protect release branches. We also considered one large pull request. The stacked pull requests keep each review below the Greptile file limit. **Roadmap alignment** This work supports the shipped Cloud / Sandbox agents milestone and the shipped Agent evals & feedback milestone in `ROADMAP.md`. Related stack: - #12699 adds managed provider backends and lifecycle support. - #12691 adds qualified OpenCode and ACPX provider backends. - #12685 restores task runtime rendering and steering. ## What Changed - Add the runner full-stack harness with 57 catalog cells and 60 unit tests. - Add a Daytona runner image with digest-pinned base images and base-aware image-content checks. - Add guarded live evaluation and chaos workflows with a fixed 40-execution matrix; live and full-stack paid schedules now run only on Sundays or by manual dispatch. - Add in-flight reported-usage cost stops, post-turn cost caps, exact-threshold failure classification, secret redaction, retry classification, and actor authorization. - Reattach stream and hard-budget listeners before restart-recovery continuations so restored paid sessions cannot bypass in-flight interruption. - Preserve OpenCode usage and cost across tool-loop messages and turns while exposing an explicit current-run delta to durable accounting. - Keep PNG/WebM evidence in access-controlled artifacts only, reject SVG, and publish only pruned inert structured per-attempt evidence. - Add AWS AgentCore infrastructure, provisioning checks, and smoke tools; reject unsafe model identifiers, require exact stack ownership markers, and make failed-stack replacement explicit. - Add evaluation-session contracts and capability reports. - Add release workflow checks for immutable action pins, frozen dependency installs, exact weekly cron shape, paid-run guards, provider-secret isolation, and chaos test paths. - Reauthorize the original and triggering numeric actor IDs as the first step of every provider-secret job, including partial reruns, before checkout or provider access. - Give each full-stack matrix cell only its matching provider credential, expose Daytona only to Daytona cells, and disable shared dependency caches anywhere paid credentials or OIDC write access are present. - Protect the legacy manual E2E workflow with the same default-branch, allowlist, environment, and per-job authorization boundary. - Rotate live-eval candidates by week and retain 120 days of compatible history so the seven-week trend window remains viable. - Restore the root runner-acceptance commands and reconcile reported snapshots, raw receipts, and terminal usage without double counting or losing late usage. - Mark ACPX token deltas exact only when every budget field is present, keep cumulative cost/request authority separate, reject non-USD cost labeling, and include thought tokens in output-token budgets. - Keep `enableNativeRunner` off by default. The acceptance harness enables it only in its isolated test instance. ## Verification Passed locally: - `pnpm --filter @paperclipai/paperclip-runner typecheck` - `pnpm test:runner-acceptance:typecheck` - `pnpm test:runner-acceptance` (19 tests) - focused OpenCode proxy, driver, runnerd transport, live-session, and turn-stream tests (106 tests) - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/live/clean-room-server.test.ts` (22 tests) - `pnpm test:e2e:runner:typecheck` - `pnpm test:e2e:runner:unit` (62 tests) - `node --test scripts/__tests__/release-verify-workflow.test.mjs` - `pnpm --filter @paperclipai/paperclip-runner test:runner-workflow-evals` (22 tests) - `pnpm -r typecheck` - `pnpm build` - `node --test packages/paperclip-runner/scripts/aws-agentcore-provisioning.test.mjs` (6 tests) - `git diff --check` - `cargo test --manifest-path packages/paperclip-runner/runner/Cargo.toml -p paperclip-runner-core --lib --locked` (161 tests) - focused ACPX provider-event tests (10 tests) - The rebased PR changes 92 files. `pnpm-lock.yaml` is unchanged. I did not run paid live provider jobs or provision AWS resources. Those checks need credentials and can create cost. ## Risks The paid workflows can create provider cost. They require an allowlisted original and triggering actor, the protected `runner-e2e-paid` environment, explicit opt-in variables, and cost limits. The four provider credentials exist only in that master-only environment, which requires allowlisted reviewer approval and disables administrator bypass; repository and organization Actions scopes contain no copies. Provider usage arrives after a billable request, so the live guard cannot prevent one request from crossing a threshold. It interrupts immediately on the first reported threshold hit and permits no continuation. Visual evidence can contain secrets rendered as pixels. PNG/WebM remain only in access-controlled workflow artifacts; SVG and per-attempt XML are excluded, and S3/Pages receive a pruned structured dashboard. The AWS scripts can create cloud resources. They use explicit commands, least-privilege roles, KMS encryption, saved nonsecret metadata, and explicit teardown. This pull request does not enable the experimental native runner for existing instances. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex with GPT-5. The model used extended reasoning, tool use, code execution, and parallel subagents. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge
222 lines
12 KiB
TypeScript
222 lines
12 KiB
TypeScript
import type {
|
|
RunnerE2EHistoryCampaign,
|
|
RunnerE2EHistoryIndex,
|
|
} from "./types.js";
|
|
|
|
function html(value: unknown) {
|
|
return String(value ?? "")
|
|
.replaceAll("&", "&")
|
|
.replaceAll("<", "<")
|
|
.replaceAll(">", ">")
|
|
.replaceAll('"', """)
|
|
.replaceAll("'", "'");
|
|
}
|
|
|
|
function number(value: number) {
|
|
return new Intl.NumberFormat("en-US").format(value);
|
|
}
|
|
|
|
function usd(value: number) {
|
|
return `$${value.toFixed(value < 0.01 ? 6 : 2)}`;
|
|
}
|
|
|
|
function duration(durationMs: number) {
|
|
const seconds = Math.round(durationMs / 1_000);
|
|
if (seconds < 60) return `${seconds}s`;
|
|
const minutes = Math.floor(seconds / 60);
|
|
if (minutes < 60) return `${minutes}m ${seconds % 60}s`;
|
|
return `${Math.floor(minutes / 60)}h ${minutes % 60}m`;
|
|
}
|
|
|
|
function date(value: string) {
|
|
return new Intl.DateTimeFormat("en-US", {
|
|
dateStyle: "medium",
|
|
timeStyle: "short",
|
|
timeZone: "UTC",
|
|
}).format(new Date(value));
|
|
}
|
|
|
|
function campaignStatus(campaign: RunnerE2EHistoryCampaign) {
|
|
return campaign.failed === 0 &&
|
|
campaign.passed === campaign.selected &&
|
|
campaign.executed === campaign.selected &&
|
|
campaign.cleanupPassed
|
|
? "passed"
|
|
: "failed";
|
|
}
|
|
|
|
function sourceCell(campaign: RunnerE2EHistoryCampaign) {
|
|
const sha = campaign.source.sha;
|
|
const shortSha = sha?.slice(0, 8) ?? "Unknown";
|
|
const shaLabel = /^[0-9a-f]{40}$/i.test(sha ?? "")
|
|
? `<a href="https://github.com/paperclipai/paperclip/commit/${html(sha)}">${html(shortSha)}</a>`
|
|
: html(shortSha);
|
|
const workflow = campaign.source.workflowRunUrl
|
|
? `<a class="secondary-link" href="${html(campaign.source.workflowRunUrl)}">Workflow run</a>`
|
|
: "";
|
|
return `${shaLabel}<small>${html(campaign.source.ref ?? "Unknown ref")}${workflow}</small>`;
|
|
}
|
|
|
|
function campaignRow(campaign: RunnerE2EHistoryCampaign) {
|
|
const status = campaignStatus(campaign);
|
|
const suites = campaign.suites
|
|
.map(
|
|
(suite) =>
|
|
`<span>${html(suite.suiteId)} · ${suite.passed}/${suite.selected}</span>`,
|
|
)
|
|
.join("");
|
|
const billing = campaign.billing;
|
|
return `<tr>
|
|
<td data-label="Campaign">
|
|
<a class="campaign-link" href="${html(campaign.publicUrl)}">${html(campaign.campaignId)}</a>
|
|
<small>${html(date(campaign.generatedAt))} UTC</small>
|
|
</td>
|
|
<td data-label="Status">
|
|
<span class="status status-${status}">${status}</span>
|
|
<small>${campaign.complete ? "Complete campaign" : "Partial campaign"}${campaign.retries > 0 ? ` · ${campaign.retries} retries` : ""}</small>
|
|
</td>
|
|
<td data-label="Source">${sourceCell(campaign)}</td>
|
|
<td data-label="Suites"><div class="suite-list">${suites}</div></td>
|
|
<td data-label="Tests">
|
|
<strong>${campaign.passed}/${campaign.selected} passed</strong>
|
|
<small>${campaign.executed} executed · ${campaign.failed} failed</small>
|
|
</td>
|
|
<td data-label="Tokens">
|
|
<strong>${html(number(billing.llm.totalTokens))}</strong>
|
|
<small>${html(number(billing.llm.inputTokens))} in · ${html(number(billing.llm.outputTokens))} out · ${html(number(billing.llm.cachedInputTokens))} cached</small>
|
|
</td>
|
|
<td data-label="Cost">
|
|
<strong>${html(usd(billing.observedAndEstimatedCostUsd))}</strong>
|
|
<small>${html(usd(billing.reportedLlmCostUsd))} LLM · ${html(usd(billing.estimatedRuntimeCostUsd))} runtime</small>
|
|
</td>
|
|
<td data-label="Time">
|
|
<strong>${html(duration(billing.agentRunDurationMs))}</strong>
|
|
<small>${html(duration(billing.leaseDurationMs))} Daytona lease</small>
|
|
</td>
|
|
<td class="open-cell"><a href="${html(campaign.publicUrl)}" aria-label="Open campaign ${html(campaign.campaignId)}">Open report →</a></td>
|
|
</tr>`;
|
|
}
|
|
|
|
export function renderRunnerHistoryIndex(history: RunnerE2EHistoryIndex) {
|
|
const campaigns = [...history.campaigns].sort((left, right) =>
|
|
right.generatedAt.localeCompare(left.generatedAt),
|
|
);
|
|
const passed = campaigns.filter(
|
|
(campaign) => campaignStatus(campaign) === "passed",
|
|
).length;
|
|
const latest = campaigns.find(
|
|
(campaign) => campaign.campaignId === history.latestCampaignId,
|
|
);
|
|
const latestGreen = campaigns.find(
|
|
(campaign) => campaign.campaignId === history.latestGreenCampaignId,
|
|
);
|
|
const totalCost = campaigns.reduce(
|
|
(sum, campaign) => sum + campaign.billing.observedAndEstimatedCostUsd,
|
|
0,
|
|
);
|
|
const rows =
|
|
campaigns.length > 0
|
|
? campaigns.map(campaignRow).join("")
|
|
: `<tr><td class="empty" colspan="9">No campaigns have been published yet.</td></tr>`;
|
|
|
|
return `<!doctype html>
|
|
<html lang="en">
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
|
<meta name="color-scheme" content="light dark">
|
|
<meta name="theme-color" content="#ffffff" media="(prefers-color-scheme: light)">
|
|
<meta name="theme-color" content="#141413" media="(prefers-color-scheme: dark)">
|
|
<link rel="icon" href="assets/favicon.svg" type="image/svg+xml">
|
|
<title>Runner E2E Campaigns · Paperclip</title>
|
|
<style>
|
|
@font-face { font-family: "Paperclip Inter"; src: url("assets/InterVariable.woff2") format("woff2"); font-style: normal; font-weight: 100 900; font-display: swap; }
|
|
:root { color-scheme: light dark; --background:#fff; --foreground:#0a0a0a; --muted:#60666a; --border:#e5e5e5; --raised:#fafafa; --pass:#188a3c; --pass-bg:#dcfce7; --fail:#991b1b; --fail-bg:#fee2e2; --font:"Paperclip Inter",Inter,ui-sans-serif,system-ui,-apple-system,sans-serif; --mono:ui-monospace,SFMono-Regular,Menlo,monospace; }
|
|
@media (prefers-color-scheme: dark) { :root { --background:#141413; --foreground:#fafafa; --muted:#a3a3a3; --border:rgb(255 255 255 / 12%); --raised:#1c1c1b; --pass:#34d06f; --pass-bg:#22c55e1f; --fail:#ef4444; --fail-bg:#dc26262e; } }
|
|
* { box-sizing:border-box; }
|
|
body { margin:0; background:var(--background); color:var(--foreground); font:14px/1.45 var(--font); }
|
|
a { color:inherit; text-underline-offset:3px; }
|
|
.brand-bar { min-height:64px; display:flex; align-items:center; justify-content:space-between; gap:24px; padding:0 max(24px,calc((100vw - 1560px)/2)); border-bottom:1px solid var(--border); }
|
|
.brand-lockup { display:inline-flex; align-items:center; gap:10px; font-size:16px; font-weight:650; text-decoration:none; }
|
|
.brand-lockup svg { width:24px; height:24px; }
|
|
.brand-context { color:var(--muted); font-size:12px; }
|
|
main { width:min(1560px,calc(100% - 48px)); margin:48px auto 72px; }
|
|
header { display:flex; align-items:end; justify-content:space-between; gap:32px; margin-bottom:36px; }
|
|
.eyebrow { margin:0 0 10px; color:var(--muted); font:500 11px/1.4 var(--mono); letter-spacing:.08em; text-transform:uppercase; }
|
|
h1 { margin:0; font-size:clamp(34px,4vw,56px); line-height:1.04; letter-spacing:-.04em; }
|
|
.lede { max-width:680px; margin:14px 0 0; color:var(--muted); font-size:16px; }
|
|
.summary { display:flex; border:1px solid var(--border); border-radius:8px; background:var(--raised); }
|
|
.metric { min-width:112px; padding:13px 16px; border-right:1px solid var(--border); }
|
|
.metric:last-child { border:0; }
|
|
.metric strong,.metric span { display:block; }
|
|
.metric strong { font:600 19px/1.2 var(--mono); }
|
|
.metric span { margin-top:4px; color:var(--muted); font-size:10px; letter-spacing:.06em; text-transform:uppercase; }
|
|
.pointers { display:flex; flex-wrap:wrap; gap:10px; margin-bottom:16px; }
|
|
.pointers a { padding:8px 11px; border:1px solid var(--border); border-radius:7px; background:var(--raised); font-size:12px; text-decoration:none; }
|
|
.pointers a:hover,.campaign-link:hover,.open-cell a:hover { text-decoration:underline; }
|
|
.table-wrap { border-top:1px solid var(--border); border-bottom:1px solid var(--border); }
|
|
table { width:100%; border-collapse:collapse; }
|
|
th,td { padding:14px 12px; border-bottom:1px solid var(--border); text-align:left; vertical-align:top; }
|
|
tr:last-child td { border-bottom:0; }
|
|
th { background:var(--raised); color:var(--muted); font-size:10px; letter-spacing:.06em; text-transform:uppercase; white-space:nowrap; }
|
|
td strong,.campaign-link { font-weight:650; }
|
|
td small { display:block; margin-top:4px; color:var(--muted); font-size:10px; }
|
|
td a.secondary-link { display:block; margin-top:3px; }
|
|
.campaign-link { font:600 11px/1.4 var(--mono); overflow-wrap:anywhere; }
|
|
.status { display:inline-block; padding:3px 8px; border-radius:999px; font-size:10px; font-weight:700; letter-spacing:.05em; text-transform:uppercase; }
|
|
.status-passed { color:var(--pass); background:var(--pass-bg); }
|
|
.status-failed { color:var(--fail); background:var(--fail-bg); }
|
|
.suite-list { display:grid; gap:3px; font-size:11px; white-space:nowrap; }
|
|
.open-cell { text-align:right; white-space:nowrap; font-weight:650; }
|
|
.empty { padding:48px; color:var(--muted); text-align:center; }
|
|
footer { display:flex; justify-content:space-between; gap:16px; margin-top:20px; color:var(--muted); font-size:11px; }
|
|
@media (max-width:1100px) { header { align-items:start; flex-direction:column; } .summary { width:100%; } .metric { flex:1; } }
|
|
@media (max-width:860px) {
|
|
.brand-context { display:none; } main { width:min(100% - 28px,1560px); margin-top:32px; }
|
|
.summary { display:grid; grid-template-columns:repeat(3,1fr); } .metric { min-width:0; border-bottom:1px solid var(--border); }
|
|
.table-wrap { border:0; } thead { display:none; } table,tbody,tr,td { display:block; width:100%; }
|
|
tr { margin-bottom:14px; padding:8px 14px; border:1px solid var(--border); border-radius:8px; background:var(--raised); }
|
|
td { display:grid; grid-template-columns:105px minmax(0,1fr); gap:12px; padding:10px 0; }
|
|
td::before { content:attr(data-label); color:var(--muted); font-size:10px; letter-spacing:.06em; text-transform:uppercase; }
|
|
.open-cell { display:block; padding-top:14px; text-align:left; } .open-cell::before { display:none; }
|
|
}
|
|
@media (max-width:520px) { .summary { grid-template-columns:1fr; } .metric { border-right:0; } footer { flex-direction:column; } }
|
|
</style>
|
|
</head>
|
|
<body>
|
|
<div class="brand-bar">
|
|
<a class="brand-lockup" href="https://paperclip.ing" aria-label="Paperclip home">
|
|
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="m16 6-8.414 8.586a2 2 0 0 0 2.829 2.829l8.414-8.586a4 4 0 1 0-5.657-5.657l-8.379 8.551a6 6 0 1 0 8.485 8.485l8.379-8.551"/></svg>
|
|
<span>Paperclip</span>
|
|
</a>
|
|
<span class="brand-context">Quality engineering · Runner acceptance</span>
|
|
</div>
|
|
<main>
|
|
<header>
|
|
<div>
|
|
<p class="eyebrow">Historical test reporting</p>
|
|
<h1>Runner E2E campaigns</h1>
|
|
<p class="lede">Each row is one workflow campaign against a Paperclip revision. Open a report for its configuration matrices, matchers, per-test billing, and sanitized structured evidence. Visual evidence remains in access-controlled workflow artifacts.</p>
|
|
</div>
|
|
<div class="summary" aria-label="History summary">
|
|
<div class="metric"><strong>${campaigns.length}</strong><span>Campaigns</span></div>
|
|
<div class="metric"><strong>${passed}</strong><span>Passed</span></div>
|
|
<div class="metric"><strong>${html(usd(totalCost))}</strong><span>Recorded cost</span></div>
|
|
</div>
|
|
</header>
|
|
<nav class="pointers" aria-label="Campaign pointers">
|
|
${latest ? `<a href="${html(latest.publicUrl)}">Latest run · ${html(latest.campaignId)}</a>` : ""}
|
|
${latestGreen ? `<a href="${html(latestGreen.publicUrl)}">Latest complete green · ${html(latestGreen.campaignId)}</a>` : ""}
|
|
</nav>
|
|
<div class="table-wrap">
|
|
<table>
|
|
<thead><tr><th>Campaign</th><th>Status</th><th>Source</th><th>Suites</th><th>Tests</th><th>Tokens</th><th>Cost</th><th>Agent time</th><th></th></tr></thead>
|
|
<tbody>${rows}</tbody>
|
|
</table>
|
|
</div>
|
|
<footer><span>Updated ${html(date(history.updatedAt))} UTC</span><span>Immutable campaign reports · Inert structured public evidence</span></footer>
|
|
</main>
|
|
</body>
|
|
</html>`;
|
|
}
|