PaperClipAI/tests/runner-e2e/OPERATIONS.md

Bounded eval operations

The request ledger and gateway are opt-in. The ordinary launcher still supports unmetered legacy campaigns. Do not infer a spending limit from a campaign ID. Enable the bounded path with PAPERCLIP_E2E_OPERATIONS_MANIFEST=/absolute/file.json. Keep this manifest, its journal, and request receipts private.

The coverage and operating-cost guide records cadence ownership, suite-growth requirements, non-model allocation, the separate Runner Eval audit, and remaining activation gaps.

Supported admission

The initial path admits local core-compatibility and feedback-persistence cells for runner-codex, runner-acpx-claude, legacy-codex, and legacy-claude. Each cell must have one provider credential and an exact priced model. Other suites, remote environments, subscription profiles, hosted tools, multimodal inputs, premium service modes, and separate judges fail admission. This whitelist is implementation scope, not evidence that every admitted harness has passed live qualification.

Before creating the journal or requesting credentials, manifest validation checks that every selected model's worst-case single-request reservation fits the initial campaign allowance after declared operating holds. Prices for unused models do not affect this selection check. Passing it does not promise the whole campaign will fit: later requests still use the journal's current commitments and limits.

Bounded daily and weekly workflows check source-byte limits before upload: 16 MiB for the resolved lockfile, 1 GiB for the build bundle, and 256 MiB each for cell evidence and the merged report. Build artifacts expire after one day; the others expire after 30 days. These are source-byte and retention controls, not measured compressed storage charges. ZIP framing and platform billing need separate receipts.

Evidence and reports are copied into fresh private staging directories. Up to 4,096 directory entries are inspected. Discovery visits accounting and snapshot directories across attempts before archive and Playwright diagnostic directories. Accounting under scheduled/ and JSON records have copy priority over logs, images and archives. Each retained file has a SHA-256 digest. The staging receipt reserves 8 MiB inside the total and identifies missing, changed, unsafe or oversized files and incomplete traversal. Originals are never deleted or rewritten. Hidden paths remain excluded. The expected regular playwright-output/.last-run.json bookkeeping file is outside the retained evidence scope and does not fail retention. Other hidden paths make the scope incomplete. Skipped paths and unvisited subtrees are named; overlong names use a bounded preview and full-path digest. Partial staging is uploaded before the workflow fails, so a storage limit cannot silently turn missing evidence into success. These upload limits do not cap runtime disk writes or Actions caches. Each service still needs billing evidence.

Private recurring history

Daily and weekly evidence and merged reports are retained in a dedicated private S3 bucket. They are not uploaded as plaintext GitHub artifacts, sent to the public history publisher, or deployed to Pages. The report command's output also stays inside the private report. The public workflow retains only a fixed availability marker for the existing schedule monitor, with no report contents, storage address or object identity. Public-repository GitHub artifacts are readable by signed-in repository readers; authentication alone does not make those reports private.

Deploy .github/infra/runner-e2e-private-history.yml with the existing GitHub OIDC provider and set RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET, RUNNER_E2E_PRIVATE_HISTORY_AWS_ROLE_ARN and RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION. The role trusts only this repository's runner-e2e-private-paid and runner-e2e-private-history environments. Create both with a deployment branch policy allowing only the default branch. Reserve them for this workflow: the shared runner-e2e-paid and runner-e2e-history environments also serve protocol evaluations that execute target-branch code and must not have access to the private archive role. Set the three archive variables in both dedicated environments. This workflow separately requires the default branch before paid access and reporting. Manual target-branch tests have no OIDC permission. A separate recurring job receives that permission only when the target SHA equals the default-branch workflow SHA. Both jobs reuse the same anchored setup and test steps. The role has configuration reads and GetObject/PutObject under runner-e2e-private/bounded-v1/; no deletion, policy changes or access to the public history bucket. AWS credentials are step outputs supplied only to private transfer commands, never exported to paid agents.

Admission verifies retention before paid execution. Every transfer verifies all four public-access blocks, disabled object ACLs, no bucket-policy reader grants, versioning, the 30-day/current and one-day/noncurrent expiration rules, and absence of Object Lock and replication. Inspection failure stops the transfer. The private bucket is not a CDN origin and has no external reader grant.

Evidence and report archives each contain at most 256 MiB of staged files and 4,097 entries including the staging receipt, with an additional 8 MiB ceiling for ZIP framing. Archives use stored entries, without decompression. Extraction checks paths, entry types, duplicate names, directory size, payload totals and ZIP checksums before committing a fresh destination. Transfers use one conditional PutObject or one ranged GetObject. Automatic AWS retries and workflow reruns are refused. The bucket also rejects PutObject requests without an If-None-Match condition. A local transfer receipt cannot be overwritten by rerunning the command in the same directory. Each command retains admission and outcome metadata. After private storage passes inspection, completed and failed transfers each get one independent receipt-upload attempt without repeating the original transfer. Successful receipts include the archive hash and object version; lost responses remain failed or unknown. A failed privacy inspection prevents all uploads, and an unsuccessful receipt upload can leave only the local receipt. Its own upload remains pending in the snapshot, so the snapshot is not proof of complete billing. Operator downloads and STS credential exchanges also need separate cost attribution.

For authenticated retrieval, set the bucket variable and AWS profile, then run:

node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/private-history.ts download report RUN_ID /tmp/eval-RUN_ID

Use evidence instead of report for the original staged accounting and diagnostics. The requested destination must not already exist. Keep the downloaded report private. To publish a selected campaign, review its retained normalized report and invoke the existing trusted history publisher explicitly with the public destination; no agent rerun is needed. Recurring execution never invokes that publication path.

Reporting and publication install only the committed lockfile from the trusted workflow revision, with lifecycle scripts disabled and a job-local package store. They never resolve new dependency versions while the job has OIDC access. If that revision has an outdated lockfile, reporting fails before accessing the archive. Use the normal master lockfile maintenance process; do not bypass the frozen install in the credential-bearing job.

Public history retention

Public history remains an explicit manual operation. The optional bounded publisher uses <history-prefix>/bounded-v1/ when PAPERCLIP_E2E_BOUNDED_HISTORY=true; existing manual history otherwise retains its original prefix. Bounded publication is limited to the original CI workflow attempt and is not retried under the same allowance.

Before any upload, the publisher validates all outgoing bundles: at most 64 MiB and 256 files for the immutable campaign including its manifest, 16 MiB and eight files for mutable history, and 64 MiB for the Pages bundle. History JSON reads are limited to 4 MiB. The active index keeps at most 64 campaigns from the last 28 days and rebuilds latest/green pointers from those records. If publication stops, the static index can retain links that later expire.

The bucket must have versioning enabled and exact prefix lifecycle rules for 30-day current-version expiration, one-day noncurrent-version expiration without a retained-version floor, expired delete-marker removal and one-day incomplete multipart cleanup. The publisher rejects overlapping transitions, different expiration policies, Object Lock and replication. Access denied is a failure, not proof that a feature is absent. Its role needs s3:GetBucketVersioning, s3:GetLifecycleConfiguration, s3:GetBucketObjectLockConfiguration and s3:GetReplicationConfiguration on that bucket. Configuration is verified at each publication; it cannot guarantee that an operator will preserve it afterward.

The publisher admits at most 280 S3 CLI commands over ten minutes, with a 30-second command timeout and automatic AWS retries disabled. Uploads use single PutObject operations in STANDARD storage; immutable objects use If-None-Match. A private receipt is synced before each command and retained locally for the operator to retain privately. Unknown responses remain unknown. Source bytes, S3 commands and expiration eligibility are not invoices or exact deletion times. Credential lookup, public downloads/CDN egress, Pages hosting and shared service charges require separate accounting; these limits do not cap public traffic.

The launcher owns one loopback HTTP gateway per attempt. The real provider key stays in the parent launcher. Playwright receives an ephemeral token, and fixture setup saves that token through a shared AI connection with the existing custom Responses/Messages routing contract. The agent does not receive ambient provider keys. The server environment strips the gateway token and upstream credentials.

Every upstream attempt reserves its maximum estimated charge in an fsynced journal before dispatch. Requests, retries, and cancelled calls retain distinct IDs. Unknown usage keeps the reservation held. One shared writer serializes admission across cells, so concurrent requests cannot each spend the same remaining allowance. Campaign deadline, request count, and in-flight count are also enforced. Private request receipts retain transport IDs, response IDs, usage and cache TTLs, route/cache/failover evidence, and final accounting status without request bodies or credentials. Raw provider responses remain only in the normal private evidence.

Manifest

A manifest contains no credentials. This structural example uses fixture prices; replace them with independently verified model prices and context limits before running a real campaign. The launcher does not discover current prices for you.

{
  "version": 1,
  "limits": {
    "campaignId": "qualification-example",
    "maxUsd": "100",
    "deadlineAt": "2026-10-20T23:00:00Z",
    "maxRequests": 200,
    "maxConcurrent": 1
  },
  "ledgerPath": "/private/path/qualification/ledger.jsonl",
  "executionIds": ["EXACT_SELECTED_EXECUTION_ID"],
  "maxAttemptsPerExecution": 2,
  "route": "direct",
  "models": [{
    "provider": "anthropic",
    "maximumInputTokens": 1000000,
    "maximumOutputTokens": 16384,
    "timeoutMs": 180000,
    "price": {
      "provider": "anthropic",
      "model": "EXACT_SELECTED_MODEL",
      "version": "verified-pricecard-date",
      "sourceUrl": "https://platform.claude.com/docs/en/about-claude/pricing",
      "inputUsdPerMillion": "2",
      "cachedReadUsdPerMillion": "0.2",
      "cacheWrite5mUsdPerMillion": "2.5",
      "cacheWrite1hUsdPerMillion": "4",
      "outputUsdPerMillion": "10"
    }
  }]
}

executionIds must equal the launcher's selected set, and the retry option must fit maxAttemptsPerExecution. Use --list to resolve the selection first. Operations records this attempt cap in the journal's immutable limits and fsyncs each attempt admission before starting its gateway. The count is per execution across the whole shared journal, including restarts, renamed output campaigns, and attempts that fail before making any provider request. Changing the cap or reusing an admitted attempt ID is refused. The maximum input value is the verified provider context ceiling, not a requested input cap. The gateway clamps output to maximumOutputTokens; it does not tokenize or truncate prompts. The conservative reservation prices the whole input ceiling at the highest cache-write/long-context rate, plus capped output. A small daily budget may therefore admit Claude but refuse Codex even when the expected charge is small. Do not reduce the context ceiling to bypass this guard.

For modern OpenAI models, specify both cacheWrite30mUsdPerMillion and cacheWrite12hUsdPerMillion. Optional longContext contains aboveInputTokens and a complete rates object with the same token-rate fields. Above that total input threshold (including cache), the estimate uses those rates for the entire request. Reservations use the maximum rate across both tiers. Missing cache-write counters, ambiguous TTL splits for nonzero writes, and model mismatches remain unknown. Older OpenAI models without these usage fields need separate qualification. Write prices are total token prices, not surcharges. Sources: OpenAI caching, OpenAI prices, and Claude prices.

For gpt-5.6-sol, a response that omits the entire write-TTL breakdown can use the actual request's documented implicit cache policy: prompt_cache_options.ttl or its 30-minute default. This fallback requires a visible input, no explicit breakpoints, no stored prompt/conversation/response references, and no compaction. The private receipt records cacheWriteAttribution with the policy version and lifetime. These amounts remain estimates. Provider splits take precedence; partial or malformed splits are never repaired. prompt_cache_retention: "24h" does not imply a 12-hour write lifetime. The gateway does not change cache policy, and this fallback does not reprice historical requests or release old holds.

For Braintrust, use route: "braintrust", add the exact projectId UUID, and provide BRAINTRUST_API_KEY securely. Configure project providers named OPENAI_API_KEY and ANTHROPIC_API_KEY. Disable default provider fallback in the project. The request pins that endpoint and project, disables gateway response caching, and never automatically retries or requests fallback. A cache hit, failover header, or unexpected successful endpoint invalidates the receipt and holds its reservation. Provider prompt caching stays enabled and is metered. The response stream is passed through and cumulative usage is counted once. Gateway and native tracing are separate opt-ins; see the README tracing section.

When both a bounded operations manifest and explicit Braintrust tracing are configured, the launcher starts a separate loopback OTLP relay for each attempt. The worker and its native providers receive an expiring ingest token. The Braintrust service key stays in the controller, even when model requests use Braintrust without native tracing. The relay accepts only trace/log exports to the configured project; neither request headers nor payload parent metadata can select another project or an API endpoint. JSON is canonicalized before export; protobuf payloads retain their original fields after bounded validation, which rejects ambiguous singular fields and routing values.

Each relay allows at most 2,048 admitted exports, 8 MiB per request before and after decompression, 64 MiB of forwarded payloads, and four simultaneous exports. It also limits each attempt to 16,384 records and 256 MiB of expanded source JSON. This measurement includes the full decoded source plus each record in its own envelope with the shared resource and scope fields repeated. It counts span names, IDs, times, events, links and log bodies, not only routing attributes. Protobuf fields use the pinned OTLP v1.9 schema; unknown fields fail admission until the schema is updated. Existing valid payloads are forwarded unchanged. It closes at the campaign deadline or four hours after startup, whichever comes first. Admission consumes the allowance before upstream dispatch. Failures and client retries do not refund it; the relay itself never retries. A durable receipt must be written before dispatch and before acknowledging the receiver outcome.

Version 2 private <ledgerPath>.otlp/<attempt-hash>.jsonl receipts record policy, admission time, wire/payload bytes, payload digest, dispatch/outcome, partial rejection counts, record counts, expanded source bytes, and closure totals. Version 1 receipts remain readable with unknown expanded size and record count. They contain no payload bodies, tokens or upstream error text. Receipt files are exclusive and cannot be reset on restart. An admission without an outcome is unresolved. Accepted ingestion does not prove indexing, and attempted forwarding bytes are not billed Braintrust storage. These receipts do not replace the observability overhead reservation or invoice reconciliation. After the worker relay closes, bounded controller finalization uses a separate one-request, 64 KiB relay with a 30-second deadline. Its exclusive expanded-source allowance is 1 MiB with at most 256 records. Its receipt is <ledgerPath>.otlp/<attempt-hash>-finalization.jsonl. This preserves the original trace and grade while accounting for the final controller export. This path currently supports the admitted local cells; remote transport remains unqualified.

Scheduled experiment creation, grade insertion and summary retrieval share a 60-second window, three-request cap, 4 MiB total payload allowance and eight-score cap. They run serially without automatic retry. Admission is persisted before dispatch in <ledgerPath>.experiment-ingestion.jsonl; uncertain responses retain their allowance and unresolved status. Reusing that file fails instead of resetting the budget. These limits cover the selected daily/weekly campaigns; the standalone experiment-import command does not use this transport budget.

Private ingestion-usage.json binds all three receipt paths to the campaign's admitted attempts and ledger hash. Missing, malformed, unclosed or unresolved receipts leave the total unknown while preserving readable partial measurements. These are admitted transport payload bytes, not Braintrust processed-data units or an invoice. Vendor charges remain unreconciled. The current two-attempt daily maximum is 132.125 MiB and the four-attempt weekly maximum is 260.25 MiB, including finalization and the campaign's experiment payload allowance. These bounds do not include other projects, standalone imports or retained storage. The corresponding expanded-source ceilings are 514 MiB daily and 1,028 MiB weekly before the separate 4-MiB experiment allowance. These source measurements support a priced forecast; they are not Braintrust's processed-data meter or a vendor-enforced dollar limit. Gateway auto-logging is a separate input to that forecast. Retain vendor usage reports for reconciliation rather than equating transport or expanded source bytes with an invoice.

Recurring campaigns

The existing runner-full-stack-e2e.yml owns both schedules: daily at 08:47 UTC and weekly on Sunday at 09:47 UTC. Scheduled runs require both RUNNER_FULL_STACK_E2E_NIGHTLY_ENABLED=true and RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED=true. Leave the operations gate unset until the workflow, billing coverage and infrastructure allowance are qualified. There is no separate scheduler. The protected runner-e2e-private-paid environment needs BRAINTRUST_API_KEY; repository variable RUNNER_E2E_BRAINTRUST_PROJECT_ID names the exact project UUID. The key is available only to the bounded execution and finalization step, after dependency installation and immutable builds.

Native Claude gateway cases also require the pinned Claude CLI for the managed account's setup hello probe. The agent still runs through the pinned ACP runtime. The workflow installs and checks the CLI before providing credentials for both native and legacy Claude selections, including the aggregate scheduled selection. For a local campaign, verify that same CLI prerequisite before starting a paid attempt. Installing the ACP package alone does not satisfy this setup path.

Bounded cases keep the gateway in the setup phase until the created agent's asynchronous lifecycle verification reaches ready. This wait uses the remaining attempt deadline, allows at most 45 seconds, and stops on an observed setup error or hold. The private budgeted-setup.json snapshot retains state transitions. Only then does the case begin agent-phase accounting and create its task. Older evidence that advanced after fixture creation may label hello probes as agent cost. Its campaign totals still include those requests; retain that phase limitation when comparing setup and task spending.

Attempt wall time includes setup and teardown. Agent time is the sum of captured heartbeat-run windows, which can overlap. Missing runtime evidence, no captured runs or an invalid run timestamp leaves agent time unknown; the report never substitutes attempt wall time. A campaign total stays unknown if one of its retained attempts lacks that measurement. A measured zero remains zero. Missing or invalid Daytona lease windows likewise leave lease time and its runtime estimate unknown. Local attempts do not contribute Daytona lease time. Existing published numeric measurements are retained as historical evidence; this change does not reconstruct their provenance.

Bounded daily/weekly campaigns require a public repository and use the existing standard ubuntu-latest runner for both compilation and execution. They do not follow the manual campaign's AWS runner switch. GitHub charges no standard hosted compute minutes for public repositories; retain runner/job evidence and recheck the billing policy when reviewing the allowance. This does not establish zero artifact, cache, external history, observability or subscription cost. Manual selections keep the existing AWS/GitHub runner choice.

Daily selects native Claude's marker and feedback-persistence cases with a shared $20 allowance. Weekly adds native Codex's two corresponding cases with a shared $100 allowance. Model requests and declared operating overhead share this limit. Each campaign runs sequentially in one job with one journal, one attempt per execution, zero automatic retries, a 30-minute request deadline and a finite request count. Codex's worst-case request reservation exceeds $20, so it is absent from daily selection. Its context ceiling is unchanged. The $20/$100 limits cover additional costs caused by each run. Show the existing Braintrust Pro subscription separately at $249/month (operator policy confirmed October 21, 2026); do not allocate it into these run limits. New processing, scoring, retention or capacity charges still count toward the run allowance. The monthly run allowance is at most $1,120 (31 daily plus five weekly runs), or $1,369 including the separately reported existing subscription. These are maximum allowances, not forecasts of actual spending. This is an admission limit with retained reservations, not a reconciled invoice total or a provider-enforced billing limit.

Before enabling recurrence, set RUNNER_E2E_OPERATING_OVERHEAD_JSON to an object with daily and weekly reviewed allowances. The catalog selects the allowance for the actual trigger, then passes only that validated allowance to preparation. A missing selected cadence or unknown outer field fails admission. A single allowance remains supported when the same bounds fit both schedules. Each allowance has version: 1, an evidence reference, the actual runner (github-hosted for bounded schedules), and verifiedAt/expiresAt timestamps no more than 31 days apart. Its components array must contain exactly ci_compute, artifact_storage, observability, and shared_subscription. Each component needs a USD-string maxUsd and its own evidence reference. References are stable audit IDs or paths, not credentials or arbitrary prose. No default amounts are supplied. The supporting audit must include rates, resource/time bounds, included allowances, allocation denominators, and the controls that keep actual resource use within those bounds. Zero requires evidence of an included or zero incremental charge; setting a zero allowance is not a receipt. Under the current policy, shared_subscription may reserve zero for the existing Braintrust subscription with a reference to that policy and current plan evidence. Any newly incurred subscription or capacity charge needs its own incremental allocation; the existing fee must remain visible separately.

The catalog checks that this allowance matches the selected runner, covers at least the next four hours, and leaves room for a full conservative model-request reservation. This happens before the build and test runners start. The workflow passes the exact validated allowance to preparation; it does not read a potentially changed repository variable again. Preparation rechecks it and retains it in the manifest and forecast. Setup, builds, failed jobs, publication and retained artifacts must all fit the declared components, not just the agent's runtime.

The journal initializes overhead:<component> reservations under campaign-overhead from its immutable limits before admitting model requests. These are allocation holds, not dispatched requests or completed resource lifecycles. They do not occupy model concurrency, but they count toward the finite charge-record limit. Resuming cannot change or drop them. Until matched cost observations arrive, the report shows them as unknown and keeps their allowance held; even a zero allowance does not claim a known zero charge. The existing finalizer reports incomplete accounting while these receipts are missing. Do not dismiss that finding to enable recurrence.

Reconcile a component through its existing request ID with provenance-bound infrastructure or invoice observations. Partial observations keep the remaining reservation held; a final nonconflicting observation can replace it. Never mark a receipt final before its resource and billing window are complete. An external bill above the allowance remains visible and stops further model admission. The journal cannot stop AWS, storage or subscription billing itself; provider resource limits and a verified receipt collector remain activation requirements. No operating-cost allowance has been configured or qualified by the presence of this code.

Collect retained GitHub usage after the workflow completes, including failed or cancelled workflows, without dispatching or retrying anything:

node --import tsx tests/runner-e2e/operating-receipts.ts github RUN_ID /private/path/github-usage.json

The collector uses the existing gh authentication and only reads paperclipai/paperclip. It verifies the Product E2E workflow, first attempt, revision and repository, reads every returned job/artifact page up to 1,000 records per inventory, and rejects changed or incomplete inventories. It retains job identities, summed job seconds, run-start delay, artifact sizes and expiration dates. It omits arbitrary job names, artifact names and upstream responses. Existing output files cannot be replaced. This is a current inventory, not an invoice: deleted artifacts are absent, compressed bytes are not billed byte-hours, job duration is not VM lifetime, and present repository visibility does not establish visibility throughout an earlier run. Caches and external S3 history remain separate. A visible public-hosted zero compute rate never automatically releases an overhead reservation.

Import a reviewed cost receipt and its exact source evidence into an existing campaign's declared overhead reservation:

node --import tsx tests/runner-e2e/operating-receipts.ts import /private/path/ledger.jsonl /private/path/receipt.json /private/path/source-evidence.json

The receipt contains version: 2, costBasis: "component_cumulative", campaignId, allowanceReference, component, the source file's evidenceSha256, windowStartedAt, windowEndedAt, coverage (partial or complete), source (infrastructure or invoice), provider, classification, and an explicit USD-string costUsd when known. Estimated and allocated charges also require a stable pricingVersion. reconciled requires an invoice source. An unknown amount has no costUsd and always remains partial. The observation window must have ended; completeness is the reviewer's assertion that this component's entire resource and billing window is represented. A checksum proves which file was used, not that the invoice or allocation is true. Shared allocations must record their numerator, denominator, window and excluded costs in that source file. Do not import a whole shared invoice as one campaign's charge without an explicit allocation method.

Each amount is the cumulative total for the entire component, including all its providers and separate charges through windowEndedAt. Later receipts must keep the same windowStartedAt and cannot shorten the window or make completed coverage partial again. Combine separate billing windows in the retained source evidence before importing their total; two separate $0.80 charges require a cumulative $1.60 receipt. Version 1 receipts lacked this declaration and are rejected on import. Existing journal records are preserved; the importer also refuses to extend a component whose prior receipts lack cumulative evidence. It also checks the ledger's precedence rules before import: a larger cumulative estimate cannot be hidden by an earlier, smaller invoice. Such imports fail without changing retained evidence or the journal; reconcile the entire expanded window with evidence the ledger can account for before retrying.

The importer validates campaign/allowance identity and source hash, retains both files privately under ledger.jsonl.overhead/, and syncs them before appending the observation. Files are written and synced under unique temporary names, then published atomically without replacing existing evidence. A partial write is removed without occupying the final path, so retry needs no manual repair. Retries sync matching existing files too. Reimporting an identical receipt adds no journal entry. A partial amount keeps the rest of the allowance held. Complete invoices can supersede estimates; conflicting complete invoices remain visible and block admission. This command does not open a new spending window, change limits, dispatch tests or rewrite previous records. Regenerate the campaign report from the same ledger afterward; an old report snapshot is not rewritten by an offline receipt import.

The checked pricecard expires on November 19, 2026. Admission stops when the pricecard expires, the catalog model changes, the project is missing, or the preflight already exists. Review current official prices and context limits before updating scheduled-prices.json. forecast.json records the selected IDs, provenance-linked pilot measurements, compatibility gaps, uncertainty, reservations and price expiry. The historical daily/weekly successful model subtotals are $0.427624800/$0.791539400; they are not campaign bills. Setup model calls and attempt setup/teardown are included, while CI setup/build/queue and non-model charges are excluded. Current compatible totals stay unknown if any selected cell lacks a matching measurement. Hosted timing is still unmeasured. Per-cell medians describe only the retained samples; P95 requires at least 20 matching samples and does not establish reliability. See measurement provenance and limitations. Daily coverage is limited to the two Claude journeys. The feedback definition has changed since its passing pilot. Broader rotation and Daytona remain unqualified and are not silently included by --all.

Dispatch cadence: daily or weekly from the default branch to qualify the same path manually. Bounded dispatch rejects custom target revisions and selectors. Bounded authorization pins the commit from the workflow event; a later merge does not replace that revision or invalidate a queued campaign. Workflow attempts after the first cannot start paid work with a fresh journal. Rerunning reporting can reuse the original attempt-one evidence. Recover the retained ledger for a deliberately resumed campaign; do not reset it or launch a new campaign merely to hide a failure. The existing manual mode remains an explicit, unbounded qualification path and does not inherit these allowances.

After execution, deterministic review dismisses only exact synthetic feedback bodies corroborated by all feedback-persistence graders. Novel feedback remains needs_investigation; no extra model call is made. Missing accounting, unresolved receipts, feedback collection/submission failures, novel feedback, export failure and incomplete experiment import or indexing fail the job with bounded diagnostic codes. Successful unchanged runs produce artifacts without posting messages. GitHub's workflow failure notifications remain the operator's notification channel. Missing/non-starting schedules require an external workflow monitor; this change does not claim that a workflow can detect its own failure to start.

The launcher starts the public Playwright JavaScript entry point directly with Node and sends SIGINT for cancellation and watchdog cleanup. The package-manager wrapper can exit before Playwright completes its shutdown, so it is excluded. Playwright handles that signal by stopping workers and closing its web server; SIGTERM can exit before web-server cleanup. The web-server supervisor still receives SIGTERM and owns its server's shutdown. Both paths retain the existing bounded grace period and forced cleanup for surviving owned processes. When a worker exits without a result, the launcher also recovers its original trace identity from the private fixture snapshots. The supervisor can finalize that trace with the failed outcome. A mismatched project, execution or company is rejected. Recovered identity does not prove native export or complete phases.

If a worker exits before writing its result, the server supervisor spends at most five seconds collecting feedback from that attempt's saved company before stopping the isolated server. Recovery snapshots remain private and are kept separate from the worker's original evidence. The failed result includes any recovered counts and review fingerprints; active runs or missing reads remain partial or unavailable. Each atomic recovery checkpoint carries its attempt identity, so a later write timeout cannot hide already saved feedback when the final receipt is absent. Completed results are never replaced. The launcher retains the private database whenever a worker result is missing, including when the supervisor was killed before it could capture feedback. This retention is reported as unconfirmed cleanup and requires inspecting the retained state. Authenticated public-MCP fixtures do not use the local operator recovery path.

Original grades are imported into Braintrust without rerunning agents. The access-controlled artifact includes the journal, forecast, feedback dispositions, experiment receipt, indexing receipt and findings. The trusted report job rebuilds the dashboard and retains its evidence-bound experiment link. Export acceptance and native indexing are separate. A bounded, read-only Braintrust query verifies the attempt root, finalization, native API activity and any expected feedback tools. It also compares indexed experiment rows with original grades, revisions and attempt IDs. The verifier tries at most three rounds within 45 seconds, retains missing/query failures, and never reruns an agent. Native record counts measure export volume, not distinct provider requests or charges. Codex exports can be high volume and Braintrust can double-count native/log observations; use the deduplicated gateway ledger as the estimate authority. Private indexing receipts contain only counts, identities and verification status, without trace contents or credentials.

Inspection and limits

Retained deterministic grades can be imported into a private Braintrust experiment without invoking agents or judges. Prepare a JSON array of absolute paths to the original result.json for every retained attempt in normalized-results.json. The original files supply each attempt's actual source and definition hash; the selected result's revision must not be assigned to older attempts.

PAPERCLIP_E2E_BRAINTRUST_PROJECT_ID=PROJECT_UUID \
node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/braintrust-experiment-import.ts \
  /private/campaign/normalized-results.json /private/campaign/original-result-paths.json \
  /private/campaign/experiment-plan.json --dry-run

Inspect the plan. To import it, provide BRAINTRUST_API_KEY securely, remove --dry-run, and use a separate experiment-receipt.json output path. The importer requires explicit accounting identities and source/definition hashes; older incomplete provenance fails closed. It retains original failed and successful grades, cleanup outcomes, feedback counts, source revisions and trace identities. Carried-forward CI attempts retain their original campaign IDs and must match the report's retained receipts. Request costs attach only when the attempt's campaign also matches the journal's campaign, even if attempt IDs coincide. It does not upload prompts, feedback bodies, matcher details or raw errors.

The evidence digest determines the private experiment name. Identical evidence reuses that experiment and stable row IDs after an interrupted insert. Changed evidence creates a different experiment and requires a separate receipt path. The importer validates an existing receipt before retrying and never lowers its confirmed accepted-row count. The receipt checkpoints accepted batches; HTTP acceptance still requires independent indexing/UI verification. Retain failed receipts. A retry must use the same original evidence, not newly run agents. Set PAPERCLIP_E2E_BRAINTRUST_EXPERIMENT to the receipt path when rebuilding the campaign report to retain the experiment link. Its evidence hash must match the report's selected source/definition identities, attempts, request-ledger checkpoint and feedback review. Refreshing only the report rendering preserves the link; changed evidence requires a new import.

Request-ledger amounts remain metadata with their journal hash and classifications. They are not additive cost metrics, and missing rows remain unknown rather than zero. The campaign ledger may include setup probes outside the imported tests. Do not add experiment metadata amounts to native or gateway trace costs. Native trace rollups are not invoice reconciliation and can omit setup or duplicate logs. API contracts: create experiment and insert retained events.

• operations-preflight.json preserves the exact manifest in the campaign folder.
• ledger.jsonl is the append-only authority for request admission and observations.
• ledger.jsonl.requests/ contains private per-request checkpoints.
• ledger.jsonl.summary.json preserves known, reserved/held, conflicting, and unknown amounts on normal finalization, including failed campaign workers.
• campaign.json and the dashboard preserve retained attempts and show missing selected results after worker failure. Their request-ledger section shows the shared ledger's known subtotal, classifications, additional held reservations, unresolved requests and per-attempt totals. It can include earlier campaigns sharing that budget. It is a separate view of model spending: do not add it to heartbeat totals. The journal hash and record count identify the checkpoint.
• If the journal cannot be read or validated, the report records accounting as unavailable with no numeric totals. Retained results and reports are still written, and the launcher/report command exits unsuccessfully. Recover the original journal before resuming paid work; reporting does not repair or reset it.
• To regenerate a merged report with the same ledger, pass PAPERCLIP_E2E_OPERATIONS_LEDGER=/absolute/ledger.jsonl to report.ts. Dashboard refresh and public publication preserve and validate its closed numeric projection. Raw request receipts and the journal remain private.

All model amounts from usage times a pricecard are estimated, not billed or reconciled. This path does not yet price CI compute, storage, subscriptions, Daytona, separate judges, or gateway overhead. Unknown observations and these coverage gaps must be included in operational reporting. Provider billing can lag and discounts can differ; a local list-price cap is not a provider-side cap.

Resuming the same journal requires exactly the original limits. A dead process can leave a lock and held reservations; inspect the owner and upstream receipts before recovering it. Never delete a live writer's lock or reset a ledger merely to regain budget. Older journals without an attempt cap remain readable for reporting and reconciliation. Operations refuses an implicit upgrade because request receipts cannot reconstruct zero-request attempts. Use the explicit audited migration below only after reviewing all retained campaign attempts. Do not replace or reset an active pilot ledger to evade this check.

No recurring schedule is enabled by this integration. Qualification of actual agent streams, tools, continuation, cancellation, native traces, and remote routes is required before treating those routes as operationally proven. Forecasts, provider reconciliation, and the campaign feedback disposition workflow are separate remaining integration work.

Setup and agent request attribution

Each admitted attempt creates a private phase checkpoint under <ledgerPath>.attempts/. The launcher passes its path only to the test worker; the Paperclip server does not inherit it. Before fixture setup finishes, requests are recorded as setup, including connection checks and hello probes. The worker atomically advances the checkpoint to agent immediately after fixture creation and before task execution. Cleanup does not reset this phase, so late agent requests retain their task attribution.

The gateway reads and validates the checkpoint for each request before reserving spend. That admission fixes the request's phase even if the phase changes while its response streams. Missing, malformed, or foreign-attempt checkpoints stop new dispatches rather than guessing a phase. Request records remain in the fsynced journal after the worker exits. This boundary applies to the admitted local core cells; it does not establish coverage for separate judges, paid tools, child-agent attribution, or remote routes. Historical journals retain their original attribution and are not retroactively rewritten.

Auditing a legacy journal before adding an attempt cap

A journal created before durable attempt admission can acquire one attempt policy through ledger-attempt-policy.ts. Stop its launcher and gateway first. Review all retained campaigns, including startup failures with no model request. Save the review and its source references as a private evidence file. Each prior attempt needs its original execution and accounting attempt ID; request receipts alone are insufficient. If that history is incomplete, do not resume paid work.

Create a private JSON audit with these fields:

• version: 1.
• ledgerPath and evidencePath: absolute private file paths.
• previousJournalHash: the current journal's final record hash.
• evidenceSha256: SHA-256 of the reviewed evidence file.
• maxAttemptsPerExecution: the finite cap to adopt, including past attempts.
• priorAttempts: an array of { "executionId": "...", "attemptId": "..." }.

Run node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/ledger-attempt-policy.ts /absolute/private-audit.json. The command checks evidence and journal hashes, takes the normal exclusive writer lock, and appends one fsynced policy record. It refuses unfinished requests, omitted recorded request identities, duplicate attempts, or an existing attempt policy. The imported count can already exceed the cap; then no new attempt is admitted for that execution. The command cannot prove the completeness of the operator's review of zero-request attempts. Retain the evidence for inspection.

All original journal bytes, request counts, charges, held reservations, campaign identity, dollar limit, concurrency limit and deadline remain intact. Subsequent launch manifests must use the adopted cap. The migration cannot extend an expired deadline, change an existing cap, or replenish an allowance. An interrupted append remains an accounting failure and must be recovered before more paid work.

The explicit local feedback-persistence native Claude/Codex cells also accept an operations manifest. They use the same metered connection, prices, campaign journal, setup boundary, and durable attempt allowance as core compatibility. Collection and the independent saved-evidence oracle require no additional model call. The synthetic items need review as test data, not product incidents.