PaperClipAI/tests/runner-e2e/OPERATIONS-COVERAGE.md

Eval coverage and operating costs

Use this with OPERATIONS.md. The bounded daily and weekly selections cover Product E2E only. They do not authorize or cap the separate Runner protocol workflow, manual matrix runs, or other live probes.

Initial cadence

The runner E2E maintainers own these selections and their failure triage. Both profiles use the exact model and limits in the reviewed pricecard.

Journey	Failure detected	Feature-map association	Cadence
Native Claude message marker	A real task cannot complete and persist its expected answer through the metered route	runs-adapters/runs	Daily and weekly
Native Claude complaint and suggestion	Completed tools fail to save attributed feedback or collection loses it	activity/agent-commentary	Daily and weekly
Native Codex message marker	The corresponding Codex task/connection path fails	runs-adapters/runs	Weekly
Native Codex complaint and suggestion	The corresponding Codex feedback/tool path fails	activity/agent-commentary	Weekly

These associations identify partial coverage. They do not qualify every run lifecycle, the Commentary browser page, continuation, or remote execution. The October 21 local pilot measured one successful sample per selected cell: $0.427624800 and 72.757 summed attempt seconds for the daily selection, $0.791539400 and 127.069 seconds for the weekly selection. These are historical gateway estimates from the Linux ARM64 container. They include model setup requests and attempt setup/teardown, but omit queue/build time and non-model charges. Earlier failed and interrupted attempts remain in campaign spending.

scheduled-measurements.json retains the exact attempt, source revision, definition, model, price version, result hash and journal hashes for each sample. forecast.json includes this evidence and a snapshot hash. It separates the historical successful selection from samples matching the current definition, model, environment, runtime, route and pricing. A missing matching cell makes the compatible selection total unknown. The feedback definition changed after these four samples; its earlier results are not current qualification.

Per-cell mean and median are descriptive. P95 stays unknown below 20 samples and then uses nearest rank; this is not a statistical confidence guarantee. Do not sum per-cell percentiles into a campaign percentile. Hosted cost/time and the selection's P95 remain unknown until separately measured. A passing sample does not predict failure frequency or account for infrastructure, storage, observability or subscriptions. Forecasts never reduce reservations or the approved budget limits.

At the October 21, 2026 catalog audit, 47 suites declared 738 cells. --all included 179; the other 559 required explicit selection. The bounded weekly selection included four cells, and daily two. These are dated inventory counts, not live acceptance results or an enduring catalog-size contract. Inspect the current pnpm test:e2e:runner -- --list before choosing work. In particular, Everyday Workflows, provider connections, feedback persistence, interruption, and case handoffs must not be inferred from a passing --all campaign.

Keep the broader rotating lane disabled until its exact cells have a metered route, complete retained attempts, a measured cost/time distribution, and a budget that includes failures. Prefer targeted continuation, approval, and cleanup cases that detect different failures over another model running the same marker. A release lane may broaden coverage only with its own reviewed manifest; the daily/weekly allowance does not authorize it.

Adding an eval or changing its cadence

The change's fixture documentation and PR must declare:

• A responsible maintainer/team and the concrete failure the case detects.
• Feature-map capability IDs, or an explicit unmapped gap. State what the journey does not exercise within a broader capability.
• Exact suite, case, profiles, models, environment, and intended cadence. State whether it is explicit-only and how it differs from existing coverage.
• A deterministic alternative considered, and why paid behavior is necessary.
• Attempt, request, output-token, concurrency, retry, and wall-time limits. Include setup, judges, child/background work, and cleanup in the route audit.
• Estimated cost/time before the first run, then all-attempt measurements with sample count, retained failures, source/definition IDs, and uncertainty. An unmeasured field is unknown, never zero.
• Feedback collection/review, trace verification, missing-receipt handling, and a failure/cleanup owner. Novel complaints are reviewed independently of the product grade.

Use the existing Product E2E authoring skill or the separate Runner authoring skill. Update the existing scheduled selection and forecast only after qualification; adding a catalog entry alone must not enable recurring spending.

Reading campaign time

The report's summed attempt time uses every retained accounting receipt, including failed retries. A missing attempt makes the total unknown; the report keeps the observed subtotal and missing count. The observed attempt window runs from the earliest retained start to the latest retained finish. Parallel attempts overlap in that window, while their individual durations add to the sum. Gaps between attempts remain in the window. Carried-forward evidence can span older runs, so this window is not the current workflow's duration.

Agent execution and Daytona lease time remain separate measurements. Attempt durations include the setup and cleanup inside their recorded boundaries. New browser-worker results retain a separate phase receipt for fixture setup, execution, feedback capture, cleanup and worker finalization. Repeated cleanup segments add together. Atomic checkpoints preserve completed phases if the worker is interrupted; the active phase has no inferred end time. Parent recovery checks identity, interval bounds and file size before retaining that partial receipt. Missing, malformed or unavailable receipts keep phase totals unknown, while closed segments remain visible as observed subtotals.

These phases start inside the test callback, after browser/server startup, and end before result publication. Parent teardown/finalization, CI queue/build time and provider-billed runtime need their own receipts. Historical results and non-browser launchers without a worker receipt remain explicitly unmeasured. Test details show each selected attempt's receipt status and raw timing metadata. Do not infer missing measurements by subtracting summed agent time from the observed window. Refreshing a report computes these views from retained results without making provider calls or changing the original measurements.

Costs outside the model journal

Keep a separate cost component for each resource, with its currency, service, resource/request identity, source window, allocation basis, and status. Join components to campaign/attempt IDs where evidence permits. Do not sum a model estimate with the invoice that later reconciles that same usage.

Component	Evidence and allocation	When evidence is missing
Provider models and paid tools	Dedicated project/key usage and billing exports; match provider request IDs or the narrowest available billing window	Preserve estimates, unknown requests, held reservations and the reconciliation difference
Daytona	Owned sandbox ID, actual resources, billable lifecycle and provider receipt; include failed creates, retries, stopped storage and delayed deletion	Retain the reservation or an explicitly partial estimate; a requested TTL does not prove deletion
CI compute	Workflow/job IDs, runner type, actual billed runtime and invoice rates; include setup, builds and failed jobs	Mark CI cost unknown; a short agent turn does not imply a cheap CI job
Storage and transfer	Artifact size, retention window and billed storage/egress	Report the unpriced component, including retained failure evidence
Braintrust/gateway usage	Project usage export and its window; distinguish included allowance from incremental overage	Do not treat free-looking trace metrics as proof of zero provider charges
Shared subscriptions	Show the existing Braintrust Pro $249/month separately under the October 21 operator policy; allocate newly incurred subscription/capacity charges from invoices with an explicit denominator	Keep new shared charges unallocated rather than assign an invented exact per-test cost

Retain both the service total and the allocation denominator so allocated amounts can be reproduced. A zero incremental usage charge can coexist with a nonzero shared subscription. Never add the entire subscription to every test. The approved daily/weekly budgets cover additional run costs, so they exclude the existing $249/month fee while including new usage and capacity charges. Scheduled admission now reserves the declared CI, storage, observability and subscription allowances inside the same $20/$100 journal limit. Missing or stale allowances stop scheduling before the AWS test runner starts. Missing receipts keep their reservations held. Explicit reviewed full-lifecycle bounds may allow provisional completion while billing remains pending; missing bounds, conflicts and overspending still fail. Legacy reservations gain no qualification from the policy change. This is a local admission control, not a provider billing cap. Actual resource bounds, cost collection and reviewed allowances still need qualification before recurrence is enabled; see the allowance contract.

Recurring evidence and reports use private S3 archives with file/byte/request boundaries, retained transfer receipts and verified access/expiration rules; see private recurring history. The public workflow retains only a content-free availability marker for monitoring. Public history is a separate explicit publication step. Private retention still needs hosted OIDC qualification and actual storage/transfer billing evidence; local archive tests and an infrastructure template do not establish those facts.

Separate Runner Eval audit

The October 21 audit inspected Paperclip source on master 8b873e0112 plus the eval operations stack, and the available paperclip-evals checkout at 0bae4e0947dc7959cae25b73d27533c096f23a14. This is a source audit, not a new live run or proof of the workflow's configured private-repository revision.

Runner reports already retain immutable attempts, retry-inclusive estimated cost and provider list cost as separate views, compatible-cohort comparisons, and missing-usage coverage. Its mock control plane is a different authority from the Product E2E server/database; Product feedback persistence cannot be claimed from a Runner tool assertion.

The private run_live_campaign.py requires an explicit authorization envelope. The hosted runner-protocol-live-evals.yml instead fans out individual run_live_roster.py cells, supplies their direct provider credentials, and uses configured per-attempt limits. It does not use the Product E2E shared gateway journal, Braintrust attempt parent, indexing verifier, or feedback finalizer. The private campaign wrapper's aggregate authorization must not be assumed to protect that different hosted path.

Before adding Runner recurrence, its owners must establish an aggregate allowance across fanout and retries, meter setup/provider bypasses, and qualify native tracing through its actual packaged runner. Preserve direct-provider qualification as its own route. Implement definition/roster changes in the maintained private repository, pin that reviewed revision in the existing workflow, and retain Runner's existing Evalbook renderer and scores. These gaps remain open; Product E2E qualification does not close them.

Missing campaign detection

Workflow failure notifications cover runs that start. The independent observer is scripts/eval-schedule-monitor.mjs. Run it through a separate host with an authenticated GitHub CLI and one private state file:

node scripts/eval-schedule-monitor.mjs "$HOME/.config/paperclip-evals/schedule-monitor.json"

The observer uses only GitHub GET requests. It never starts or retries paid work and does not need provider credentials. It reads the two existing enable gates plus RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED_AT, an ISO timestamp set when recurrence is enabled. Missing activation time fails visibly; occurrences before activation are not treated as missed. Workflow run titles carry the literal cron expression to distinguish daily and weekly runs, including Sunday.

Only the legacy RUNNER_FULL_STACK_E2E_NIGHTLY_ENABLED gate being off establishes disabled recurrence. If that gate is on while the operations gate is absent or off, the observer reports configuration_incomplete and retains one incident notification. Older deployed workflows use only the legacy gate, so a missing new gate must not imply that those paid schedules are disabled. Verify the deployed default-branch workflow and both variables before enabling recurrence.

The observer allows 90 minutes for schedule delay. It checks the most recent due occurrence per cadence, marks a run overdue after four hours, and requires a successful terminal run plus a nonempty, unexpired merged-report artifact. The workflow's finalizer remains responsible for checking report content and accounting. Metadata inspection does not repeat that validation. Unexpected reruns, ambiguous matching runs, incomplete API lists and API failures remain visible. The observer reads at most 100 rows per collection and fails closed if a response is truncated.

Retained state includes the last successful inspection and inspected run IDs. notifications is a durable outbox of problems and recovery transitions. Each entry has an ID. After confirmed delivery, the host acknowledges those IDs:

node scripts/eval-schedule-monitor.mjs "$HOME/.config/paperclip-evals/schedule-monitor.json" --ack <notification-id> ...

Until acknowledgment, later inspections return the same pending entries even if the problem has recovered. The host uses IDs to deduplicate delivery. This survives a process exit between saving state and printing output; it does not claim exactly-once delivery across an external notification service. After acknowledgment, an unchanged problem stays quiet. A new missed occurrence is a new incident. Disabled recurrence and unchanged healthy runs create no new alerts. Use one writer for both checks and acknowledgments; do not reset the state file to clear an incident. Malformed history fails before a query or write. The external host owns notification delivery and its own liveness.

Schedule this observer outside the paid workflow after source and recurrence are ready. It should inspect at least daily, after both scheduled start times and their grace windows. Until that external trigger and notification delivery are configured and exercised, missed-run detection remains an activation gap.