From 3bd709df8b83aa855dbf300dcc57383b03ec1c88 Mon Sep 17 00:00:00 2001
From: Dotta <34892728+cryppadotta@users.noreply.github.com>
Date: Fri, 23 Oct 2026 07:18:37 -0700
Subject: [PATCH] Operationalize Product E2E accounting, tracing, and
scheduling (#17009)
## Thinking Path
> - Paperclip manages agents and the work they perform.
> - Product E2E tests exercise real agents, tools, setup and cleanup.
> - Repeated runs need retained costs and traces, including failed
attempts.
> - A passing retry must not hide earlier spending or feedback.
> - Recurring work also needs admission limits, private reports and
failure detection.
> - This change supplies those controls while keeping paid activation
behind existing gates.
## Linked Issues or Issue Description
Refs #16996 and #17158. This is the consolidated landing of the reviewed
eval operations stack: #17019, #17024, #17025, #17027, #17030, #17033,
#17035, #17038, #17043, #17044, #17048, #17050, #17051, #17092, #17096,
#17100, #17107, #17109, #17112, #17115, #17117, #17121, #17123, #17127,
#17130, #17133, #17134, #17137, #17140, #17143, #17154. The repository
permits squash merges only. A local rehearsal showed that squashing each
slice into master causes repeated conflicts in dependent slices. This PR
preserves their commits and lands their combined implementation in one
reviewed integration. Superseded slices will close after this merge.
## What Changed
- Retain every attempt's cost receipt, including setup, failures,
interruptions and retries. Separate reported amounts, estimates, missing
costs and reserved capacity in reports and dashboards.
- Add bounded request admission, exact model pricecards, gateway error
accounting, cache policy and reconciliation evidence. Refuse unsupported
routes and requests that cannot fit the allowance.
- Integrate Braintrust gateway traces and OTLP correlation, with secret
filtering and experiment imports. Record phase timing, including setup,
feedback capture and cleanup.
- Collect complaint and suggestion evidence after each attempt,
including recovery after failures. Retain explicit review dispositions
independently from the product grade.
- Add gated daily and weekly selections, measured forecasts,
setup/readiness checks, wall-time limits and retained monitor
notifications. Include declared non-model overhead in admission while
reporting the existing subscription separately.
- Keep recurring history private, bound uploads and retention, and use
isolated archive environments. Keep public publication an explicit
operation. Report and publication jobs install only the committed
lockfile, with scripts disabled and isolated stores; stale pins fail
closed.
- Integrate current master, preserve its Product E2E journey billing and
privacy checks, and test that shared setup receipts do not double-count
estimated agent usage. Preserve upstream fixture repairs.
## Verification
- Complete integration baseline
`0649a5ce7544ab20825c2183d2f421157db49f24` includes master
`a3e1d915a7110a4e115b46396a3f79e6dfb4d05e` and every original stack head
as ancestors.
- On that integration baseline, `pnpm test:e2e:runner:unit`: 4,012
Vitest tests passed, two skipped, across 193 files; all 137 Node
source-contract tests passed.
- Focused billing and Product E2E journey contract checks: 24 tests
passed. They cover estimates, unknown receipts, single-owner setup
costs, browser-only execution and private publication guards.
- Monitor and workflow wiring: all 96 Node tests passed. `node
scripts/check-token-gates.mjs`: all gates clean.
- `pnpm test:e2e:runner:typecheck` passes after the standard Runner
TypeScript build refreshed local declarations. The first attempt
rejected the new master `runner_prepared_launch` union member because
this checkout retained an older generated declaration; the source union
already contained that member.
- Current head `492b0708a1c531a023d9ac10f0f3dc6383cc1754` removes
dependency re-resolution from archive-authorized reporting jobs after
the security scan flagged it. All 15 workflow-security tests pass after
updating the old two-install assertion.
- Hosted [CI run
39965460180](https://github.com/paperclipai/paperclip/actions/runs/39965460180)
passes on the current repaired head, including typecheck, build, general
tests, Runner checks, browser shards and canary dry run. Greptile is 5/5
on this exact head with no unresolved threads. The security scan passes
after the fixed lockfile-resolution thread was resolved.
- Paperclip Review returned a terminal neutral check but an incomplete
0-file assessment on the repaired head: its platform blocked both diff
responses as prompt-injection content. This is not a successful code
review. No bypass or fabricated pass was attempted; current-head
Greptile, security and hosted CI are independent gates.
- No paid test, reservation release or recurring-run activation
accompanies this landing. Live route qualification and invoice
reconciliation remain separate evidence.
Retained earlier validation failures
Earlier broad local runs had two 15-second test timeouts (CLI poll
cadence and Retry validation), one embedded-database setup timeout, and
later PostgreSQL semaphore exhaustion. The exact timed-out tests passed
separately without larger timeouts; the broad-run causes were not
established. Those local failures remain distinct from subsequent hosted
CI results. The independent #17158 fix records its Muse fixture
collision, its UUID correction and hosted verification. Local Muse
execution was also blocked by macOS embedded-PostgreSQL startup. No
historical failure is treated as a successful paid qualification.
## Risks
- Admission is a local control, not a provider invoice cap. Missing
provider receipts or resource charges retain unknown amounts and holds;
estimates do not become billed dollars.
- The bounded path admits only documented local suites and provider
routes. Ordinary legacy or separate Runner protocol campaigns have
different controls. Merging this code does not qualify additional
routes.
- Recurrence still needs the existing enable gates and current
qualification/allowance evidence. Archive permissions and resources must
match the documented private-environment contract.
- Legacy reports may lack complete attempt or phase evidence. Reports
preserve that limitation and do not infer zero usage from missing data.
## Model Used
OpenAI GPT-6 through Codex, with reasoning, repository tools and code
execution. The exact deployment variant and context-window size are not
exposed by this session.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (targeted checks;
full-suite host limitation disclosed above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip
---
.github/infra/runner-e2e-private-history.yml | 117 +++
.github/workflows/pr-trusted.yml | 2 +-
.github/workflows/runner-full-stack-e2e.yml | 316 +++++++-
doc/agent-observability.md | 14 +-
doc/evals.md | 5 +
.../src/agent-observability.test.ts | 16 +-
.../adapter-utils/src/agent-observability.ts | 14 +-
.../runner-core/src/acpx_sidecar_transport.rs | 2 +
.../src/bin/fake-codex-app-server.rs | 19 +
.../crates/runner-core/src/codex_provider.rs | 2 +
.../runner-core/tests/codex_provider.rs | 18 +-
.../scripts/build-inputs.test.mjs | 38 +-
.../src/drivers/agent-observability.test.ts | 19 +-
.../src/drivers/agent-observability.ts | 3 +-
.../codex/app-server-transport.test.ts | 40 +
.../src/drivers/codex/app-server-transport.ts | 6 +-
.../opencode/opencode-server-driver.test.ts | 21 +-
.../__tests__/eval-schedule-monitor.test.mjs | 196 +++++
.../release-verify-workflow.test.mjs | 18 +-
scripts/eval-schedule-monitor.mjs | 176 +++++
...eval-feedback-recovery.integration.test.ts | 87 +++
.../fixtures/otlp-observability.proto | 3 +-
.../native-session-executor.test.ts | 14 +-
tests/e2e/cases-read-only.spec.ts | 13 +-
tests/e2e/runner-e2e-dashboard.spec.ts | 29 +
tests/runner-e2e/FIXTURES.md | 27 +
tests/runner-e2e/OPERATIONS-COVERAGE.md | 232 ++++++
tests/runner-e2e/OPERATIONS.md | 707 ++++++++++++++++++
tests/runner-e2e/OTLP-LICENSE.txt | 201 +++++
tests/runner-e2e/OTLP-NOTICE.md | 11 +
tests/runner-e2e/README.md | 195 ++++-
tests/runner-e2e/SECURITY.md | 60 ++
tests/runner-e2e/api.ts | 28 +-
tests/runner-e2e/artifact-budget.mjs | 163 ++++
tests/runner-e2e/artifact-budget.test.mjs | 120 +++
tests/runner-e2e/attempt-accounting.test.ts | 87 +++
tests/runner-e2e/attempt-accounting.ts | 99 +++
tests/runner-e2e/attempt-phase.ts | 34 +
tests/runner-e2e/billing.test.ts | 76 ++
tests/runner-e2e/billing.ts | 141 ++--
.../braintrust-experiment-import.ts | 45 ++
.../runner-e2e/braintrust-experiment.test.ts | 328 ++++++++
tests/runner-e2e/braintrust-experiment.ts | 207 +++++
tests/runner-e2e/braintrust-indexing.test.ts | 113 +++
tests/runner-e2e/braintrust-indexing.ts | 118 +++
tests/runner-e2e/braintrust.test.ts | 213 ++++++
tests/runner-e2e/braintrust.ts | 218 ++++++
tests/runner-e2e/budgeted-setup.test.ts | 75 ++
tests/runner-e2e/budgeted-setup.ts | 44 ++
tests/runner-e2e/campaign-timing.test.ts | 42 ++
tests/runner-e2e/campaign-timing.ts | 83 ++
tests/runner-e2e/catalog.test.ts | 6 +-
tests/runner-e2e/catalog.ts | 12 +
tests/runner-e2e/cleanup-verification.ts | 2 +-
tests/runner-e2e/codex-ci-sandbox.test.ts | 4 +-
tests/runner-e2e/cost-ledger.test.ts | 197 +++++
tests/runner-e2e/cost-ledger.ts | 293 ++++++++
tests/runner-e2e/dashboard-regenerate.ts | 30 +-
tests/runner-e2e/dashboard.ts | 33 +-
tests/runner-e2e/daytona-image-content.ts | 4 +
.../eval-finalization-relay.test.ts | 40 +
tests/runner-e2e/eval-finalization-relay.ts | 24 +
tests/runner-e2e/eval-gateway.test.ts | 315 ++++++++
tests/runner-e2e/eval-gateway.ts | 256 +++++++
tests/runner-e2e/eval-otlp-payload.ts | 127 ++++
tests/runner-e2e/eval-otlp-relay.test.ts | 258 +++++++
tests/runner-e2e/eval-otlp-relay.ts | 139 ++++
tests/runner-e2e/eval-otlp-schema.ts | 170 +++++
tests/runner-e2e/experiment-ingestion.test.ts | 108 +++
tests/runner-e2e/experiment-ingestion.ts | 91 +++
.../runner-e2e/feedback-dispositions.test.ts | 129 ++++
tests/runner-e2e/feedback-dispositions.ts | 95 +++
tests/runner-e2e/feedback-persistence.test.ts | 68 ++
tests/runner-e2e/feedback-persistence.ts | 78 ++
tests/runner-e2e/feedback-recovery.test.ts | 111 +++
tests/runner-e2e/feedback-recovery.ts | 101 +++
.../feedback-shutdown-process.test.ts | 113 +++
tests/runner-e2e/feedback.test.ts | 319 ++++++++
tests/runner-e2e/feedback.ts | 268 +++++++
tests/runner-e2e/first-task-flow.ts | 2 +
.../runner-e2e/github-operating-usage.test.ts | 64 ++
tests/runner-e2e/github-operating-usage.ts | 100 +++
tests/runner-e2e/harness-env.ts | 25 +-
tests/runner-e2e/history-budget.test.ts | 170 +++++
tests/runner-e2e/history-budget.ts | 177 +++++
tests/runner-e2e/history-index.ts | 3 +-
tests/runner-e2e/history-publish.ts | 96 ++-
tests/runner-e2e/history.test.ts | 35 +
tests/runner-e2e/history.ts | 61 +-
tests/runner-e2e/ingestion-usage.test.ts | 108 +++
tests/runner-e2e/ingestion-usage.ts | 98 +++
tests/runner-e2e/installed-cli.test.ts | 3 +-
tests/runner-e2e/journey-features.ts | 3 +
tests/runner-e2e/launch.ts | 110 ++-
.../runner-e2e/ledger-attempt-policy.test.ts | 120 +++
tests/runner-e2e/ledger-attempt-policy.ts | 40 +
tests/runner-e2e/live-fixtures.ts | 41 +-
tests/runner-e2e/operating-overhead.test.ts | 119 +++
tests/runner-e2e/operating-overhead.ts | 53 ++
tests/runner-e2e/operating-receipts.test.ts | 216 ++++++
tests/runner-e2e/operating-receipts.ts | 136 ++++
tests/runner-e2e/operations-report.test.ts | 93 +++
tests/runner-e2e/operations-report.ts | 114 +++
tests/runner-e2e/operations.test.ts | 173 +++++
tests/runner-e2e/operations.ts | 123 +++
tests/runner-e2e/paid-runner-routing.test.mjs | 51 ++
tests/runner-e2e/pi-controls-evidence.test.ts | 2 +-
tests/runner-e2e/private-history-archive.py | 148 ++++
tests/runner-e2e/private-history.test.ts | 162 ++++
tests/runner-e2e/private-history.ts | 142 ++++
tests/runner-e2e/process-tree-owner.ts | 16 +-
.../product-journeys/contracts.test.ts | 2 +
tests/runner-e2e/provider-usage.test.ts | 113 +++
tests/runner-e2e/provider-usage.ts | 154 ++++
tests/runner-e2e/report-format.ts | 11 +-
tests/runner-e2e/report-playwright.config.ts | 7 +
tests/runner-e2e/report.test.ts | 100 ++-
tests/runner-e2e/report.ts | 47 +-
tests/runner-e2e/result-validation.ts | 52 +-
tests/runner-e2e/run-unit.mjs | 2 +-
tests/runner-e2e/runner.spec.ts | 86 ++-
.../runner-e2e/scheduled-finalization.test.ts | 54 ++
tests/runner-e2e/scheduled-finalization.ts | 117 +++
tests/runner-e2e/scheduled-forecast.test.ts | 91 +++
tests/runner-e2e/scheduled-forecast.ts | 92 +++
tests/runner-e2e/scheduled-measurements.json | 84 +++
tests/runner-e2e/scheduled-operations.test.ts | 77 ++
tests/runner-e2e/scheduled-operations.ts | 121 +++
tests/runner-e2e/scheduled-prices.json | 51 ++
tests/runner-e2e/select-rerun-artifacts.ts | 7 +-
tests/runner-e2e/server.ts | 49 +-
tests/runner-e2e/trace-recovery.test.ts | 46 ++
tests/runner-e2e/trace-recovery.ts | 23 +
tests/runner-e2e/types.ts | 44 +-
tests/runner-e2e/vitest.config.ts | 2 +-
tests/runner-e2e/web-server-command.ts | 11 +-
tests/runner-e2e/worker-timing.test.ts | 127 ++++
tests/runner-e2e/worker-timing.ts | 148 ++++
tests/runner-e2e/workflow-security.test.ts | 125 +++-
tests/runner-e2e/workflow-target.test.ts | 49 ++
ui/src/pages/Cases.test.tsx | 1 +
141 files changed, 12644 insertions(+), 299 deletions(-)
create mode 100644 .github/infra/runner-e2e-private-history.yml
create mode 100644 scripts/__tests__/eval-schedule-monitor.test.mjs
create mode 100644 scripts/eval-schedule-monitor.mjs
create mode 100644 server/src/__tests__/eval-feedback-recovery.integration.test.ts
create mode 100644 tests/runner-e2e/OPERATIONS-COVERAGE.md
create mode 100644 tests/runner-e2e/OPERATIONS.md
create mode 100644 tests/runner-e2e/OTLP-LICENSE.txt
create mode 100644 tests/runner-e2e/OTLP-NOTICE.md
create mode 100644 tests/runner-e2e/artifact-budget.mjs
create mode 100644 tests/runner-e2e/artifact-budget.test.mjs
create mode 100644 tests/runner-e2e/attempt-accounting.test.ts
create mode 100644 tests/runner-e2e/attempt-accounting.ts
create mode 100644 tests/runner-e2e/attempt-phase.ts
create mode 100644 tests/runner-e2e/braintrust-experiment-import.ts
create mode 100644 tests/runner-e2e/braintrust-experiment.test.ts
create mode 100644 tests/runner-e2e/braintrust-experiment.ts
create mode 100644 tests/runner-e2e/braintrust-indexing.test.ts
create mode 100644 tests/runner-e2e/braintrust-indexing.ts
create mode 100644 tests/runner-e2e/braintrust.test.ts
create mode 100644 tests/runner-e2e/braintrust.ts
create mode 100644 tests/runner-e2e/budgeted-setup.test.ts
create mode 100644 tests/runner-e2e/budgeted-setup.ts
create mode 100644 tests/runner-e2e/campaign-timing.test.ts
create mode 100644 tests/runner-e2e/campaign-timing.ts
create mode 100644 tests/runner-e2e/cost-ledger.test.ts
create mode 100644 tests/runner-e2e/cost-ledger.ts
create mode 100644 tests/runner-e2e/eval-finalization-relay.test.ts
create mode 100644 tests/runner-e2e/eval-finalization-relay.ts
create mode 100644 tests/runner-e2e/eval-gateway.test.ts
create mode 100644 tests/runner-e2e/eval-gateway.ts
create mode 100644 tests/runner-e2e/eval-otlp-payload.ts
create mode 100644 tests/runner-e2e/eval-otlp-relay.test.ts
create mode 100644 tests/runner-e2e/eval-otlp-relay.ts
create mode 100644 tests/runner-e2e/eval-otlp-schema.ts
create mode 100644 tests/runner-e2e/experiment-ingestion.test.ts
create mode 100644 tests/runner-e2e/experiment-ingestion.ts
create mode 100644 tests/runner-e2e/feedback-dispositions.test.ts
create mode 100644 tests/runner-e2e/feedback-dispositions.ts
create mode 100644 tests/runner-e2e/feedback-persistence.test.ts
create mode 100644 tests/runner-e2e/feedback-persistence.ts
create mode 100644 tests/runner-e2e/feedback-recovery.test.ts
create mode 100644 tests/runner-e2e/feedback-recovery.ts
create mode 100644 tests/runner-e2e/feedback-shutdown-process.test.ts
create mode 100644 tests/runner-e2e/feedback.test.ts
create mode 100644 tests/runner-e2e/feedback.ts
create mode 100644 tests/runner-e2e/github-operating-usage.test.ts
create mode 100644 tests/runner-e2e/github-operating-usage.ts
create mode 100644 tests/runner-e2e/history-budget.test.ts
create mode 100644 tests/runner-e2e/history-budget.ts
create mode 100644 tests/runner-e2e/ingestion-usage.test.ts
create mode 100644 tests/runner-e2e/ingestion-usage.ts
create mode 100644 tests/runner-e2e/ledger-attempt-policy.test.ts
create mode 100644 tests/runner-e2e/ledger-attempt-policy.ts
create mode 100644 tests/runner-e2e/operating-overhead.test.ts
create mode 100644 tests/runner-e2e/operating-overhead.ts
create mode 100644 tests/runner-e2e/operating-receipts.test.ts
create mode 100644 tests/runner-e2e/operating-receipts.ts
create mode 100644 tests/runner-e2e/operations-report.test.ts
create mode 100644 tests/runner-e2e/operations-report.ts
create mode 100644 tests/runner-e2e/operations.test.ts
create mode 100644 tests/runner-e2e/operations.ts
create mode 100644 tests/runner-e2e/paid-runner-routing.test.mjs
create mode 100644 tests/runner-e2e/private-history-archive.py
create mode 100644 tests/runner-e2e/private-history.test.ts
create mode 100644 tests/runner-e2e/private-history.ts
create mode 100644 tests/runner-e2e/provider-usage.test.ts
create mode 100644 tests/runner-e2e/provider-usage.ts
create mode 100644 tests/runner-e2e/report-playwright.config.ts
create mode 100644 tests/runner-e2e/scheduled-finalization.test.ts
create mode 100644 tests/runner-e2e/scheduled-finalization.ts
create mode 100644 tests/runner-e2e/scheduled-forecast.test.ts
create mode 100644 tests/runner-e2e/scheduled-forecast.ts
create mode 100644 tests/runner-e2e/scheduled-measurements.json
create mode 100644 tests/runner-e2e/scheduled-operations.test.ts
create mode 100644 tests/runner-e2e/scheduled-operations.ts
create mode 100644 tests/runner-e2e/scheduled-prices.json
create mode 100644 tests/runner-e2e/trace-recovery.test.ts
create mode 100644 tests/runner-e2e/trace-recovery.ts
create mode 100644 tests/runner-e2e/worker-timing.test.ts
create mode 100644 tests/runner-e2e/worker-timing.ts
create mode 100644 tests/runner-e2e/workflow-target.test.ts
diff --git a/.github/infra/runner-e2e-private-history.yml b/.github/infra/runner-e2e-private-history.yml
new file mode 100644
index 00000000000..0d837d24793
--- /dev/null
+++ b/.github/infra/runner-e2e-private-history.yml
@@ -0,0 +1,117 @@
+AWSTemplateFormatVersion: '2010-09-09'
+Description: Private bounded Paperclip eval evidence and reports; no CDN reader
+Parameters:
+ BucketName:
+ Type: String
+ GitHubRepository:
+ Type: String
+ Default: paperclipai/paperclip
+ GitHubOidcProviderArn:
+ Type: String
+Resources:
+ PrivateHistoryBucket:
+ Type: AWS::S3::Bucket
+ DeletionPolicy: Retain
+ UpdateReplacePolicy: Retain
+ Properties:
+ BucketName: !Ref BucketName
+ PublicAccessBlockConfiguration:
+ BlockPublicAcls: true
+ IgnorePublicAcls: true
+ BlockPublicPolicy: true
+ RestrictPublicBuckets: true
+ OwnershipControls:
+ Rules:
+ - ObjectOwnership: BucketOwnerEnforced
+ VersioningConfiguration:
+ Status: Enabled
+ BucketEncryption:
+ ServerSideEncryptionConfiguration:
+ - ServerSideEncryptionByDefault:
+ SSEAlgorithm: AES256
+ LifecycleConfiguration:
+ Rules:
+ - Id: BoundedEvalRetentionV1
+ Status: Enabled
+ Prefix: runner-e2e-private/bounded-v1/
+ ExpirationInDays: 30
+ NoncurrentVersionExpiration:
+ NoncurrentDays: 1
+ AbortIncompleteMultipartUpload:
+ DaysAfterInitiation: 1
+ - Id: BoundedEvalDeleteMarkersV1
+ Status: Enabled
+ Prefix: runner-e2e-private/bounded-v1/
+ ExpiredObjectDeleteMarker: true
+ PrivateHistoryBucketPolicy:
+ Type: AWS::S3::BucketPolicy
+ Properties:
+ Bucket: !Ref PrivateHistoryBucket
+ PolicyDocument:
+ Version: '2012-10-17'
+ Statement:
+ - Sid: DenyInsecureTransport
+ Effect: Deny
+ Principal: '*'
+ Action: s3:*
+ Resource:
+ - !GetAtt PrivateHistoryBucket.Arn
+ - !Sub '${PrivateHistoryBucket.Arn}/*'
+ Condition:
+ Bool:
+ aws:SecureTransport: 'false'
+ - Sid: DenyUnconditionalWrites
+ Effect: Deny
+ Principal: '*'
+ Action: s3:PutObject
+ Resource: !Sub '${PrivateHistoryBucket.Arn}/runner-e2e-private/bounded-v1/*'
+ Condition:
+ 'Null':
+ s3:if-none-match: 'true'
+ PrivateHistoryRole:
+ Type: AWS::IAM::Role
+ Properties:
+ RoleName: paperclip-runner-e2e-private-history-github
+ MaxSessionDuration: 3600
+ AssumeRolePolicyDocument:
+ Version: '2012-10-17'
+ Statement:
+ - Effect: Allow
+ Principal:
+ Federated: !Ref GitHubOidcProviderArn
+ Action: sts:AssumeRoleWithWebIdentity
+ Condition:
+ StringEquals:
+ token.actions.githubusercontent.com:aud: sts.amazonaws.com
+ token.actions.githubusercontent.com:sub:
+ - !Sub 'repo:${GitHubRepository}:environment:runner-e2e-private-paid'
+ - !Sub 'repo:${GitHubRepository}:environment:runner-e2e-private-history'
+ Policies:
+ - PolicyName: private-eval-artifacts
+ PolicyDocument:
+ Version: '2012-10-17'
+ Statement:
+ - Sid: VerifyPrivateRetention
+ Effect: Allow
+ Action:
+ - s3:GetBucketVersioning
+ - s3:GetLifecycleConfiguration
+ - s3:GetBucketObjectLockConfiguration
+ - s3:GetReplicationConfiguration
+ - s3:GetBucketPublicAccessBlock
+ - s3:GetBucketOwnershipControls
+ - s3:GetBucketPolicy
+ Resource: !GetAtt PrivateHistoryBucket.Arn
+ - Sid: RetainAndReadPrivateArtifacts
+ Effect: Allow
+ Action:
+ - s3:GetObject
+ - s3:PutObject
+ Resource: !Sub '${PrivateHistoryBucket.Arn}/runner-e2e-private/bounded-v1/*'
+Outputs:
+ Bucket:
+ Value: !Ref PrivateHistoryBucket
+ RoleArn:
+ Value: !GetAtt PrivateHistoryRole.Arn
+ Region:
+ Value: !Ref AWS::Region
diff --git a/.github/workflows/pr-trusted.yml b/.github/workflows/pr-trusted.yml
index c15a34085fa..a04db993e7c 100644
--- a/.github/workflows/pr-trusted.yml
+++ b/.github/workflows/pr-trusted.yml
@@ -119,7 +119,7 @@ jobs:
run: node "$CI_NODE_TEST" ./scripts/__tests__/e2e-shard.test.mjs
- name: Test release verify workflow wiring
- run: node "$CI_NODE_TEST" ./scripts/__tests__/release-verify-workflow.test.mjs ./scripts/__tests__/prepare-runner-release-image.test.mjs ./scripts/cloud-source-verification.test.mjs ./scripts/standard-image-contract.test.mjs
+ run: node "$CI_NODE_TEST" ./scripts/__tests__/release-verify-workflow.test.mjs ./scripts/__tests__/eval-schedule-monitor.test.mjs ./scripts/__tests__/prepare-runner-release-image.test.mjs ./scripts/cloud-source-verification.test.mjs ./scripts/standard-image-contract.test.mjs
- name: Test standalone package build concurrency
run: node "$CI_NODE_TEST" ./scripts/__tests__/build-standalone-concurrency.test.mjs
diff --git a/.github/workflows/runner-full-stack-e2e.yml b/.github/workflows/runner-full-stack-e2e.yml
index 65eebf32ed3..99cdaf52326 100644
--- a/.github/workflows/runner-full-stack-e2e.yml
+++ b/.github/workflows/runner-full-stack-e2e.yml
@@ -1,10 +1,17 @@
name: Runner Full-Stack E2E
+run-name: Runner Full-Stack E2E / ${{ github.event.schedule || inputs.cadence || 'manual' }}
on:
schedule:
- - cron: "47 8 * * 0"
+ - cron: "47 8 * * *"
+ - cron: "47 9 * * 0"
workflow_dispatch:
inputs:
+ cadence:
+ description: "Bounded daily ($20) or weekly ($100) selection; manual keeps explicit selectors"
+ type: choice
+ options: [manual, daily, weekly]
+ default: manual
target_branch:
description: "Branch in paperclipai/paperclip to test; the trusted workflow still runs from master"
type: string
@@ -55,7 +62,7 @@ concurrency:
jobs:
authorize:
name: Authorize paid campaign
- if: github.event_name != 'schedule' || vars.RUNNER_FULL_STACK_E2E_NIGHTLY_ENABLED == 'true'
+ if: github.event_name != 'schedule' || (vars.RUNNER_FULL_STACK_E2E_NIGHTLY_ENABLED == 'true' && vars.RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED == 'true')
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
@@ -115,14 +122,25 @@ jobs:
GH_TOKEN: ${{ github.token }}
REPOSITORY: ${{ github.repository }}
TARGET_BRANCH: ${{ inputs.target_branch || github.event.repository.default_branch }}
+ DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
+ EVENT_NAME: ${{ github.event_name }}
+ EVAL_CADENCE: ${{ inputs.cadence || 'manual' }}
run: |
set -euo pipefail
if [ -z "$TARGET_BRANCH" ] || [[ "$TARGET_BRANCH" == refs/* ]]; then
echo "target_branch must name a branch in this repository without a refs/ prefix." >&2
exit 1
fi
- encoded_branch="$(jq -rn --arg branch "$TARGET_BRANCH" '$branch | @uri')"
- target_sha="$(gh api -X GET "repos/$REPOSITORY/branches/$encoded_branch" --jq .commit.sha)"
+ if [ "$EVENT_NAME" = schedule ] || [ "$EVAL_CADENCE" != manual ]; then
+ if [ "$TARGET_BRANCH" != "$DEFAULT_BRANCH" ]; then
+ echo "Bounded campaigns require the workflow's default-branch commit." >&2
+ exit 1
+ fi
+ target_sha="$GITHUB_SHA"
+ else
+ encoded_branch="$(jq -rn --arg branch "$TARGET_BRANCH" '$branch | @uri')"
+ target_sha="$(gh api -X GET "repos/$REPOSITORY/branches/$encoded_branch" --jq .commit.sha)"
+ fi
if ! [[ "$target_sha" =~ ^[0-9a-f]{40}$ ]]; then
echo "The requested repository branch did not resolve to a commit." >&2
exit 1
@@ -135,12 +153,23 @@ jobs:
id: runner
env:
AWS_PAID_RUNNER_ENABLED: ${{ vars.RUNNER_E2E_AWS_ENABLED }}
+ EVENT_NAME: ${{ github.event_name }}
+ EVAL_CADENCE: ${{ inputs.cadence || 'manual' }}
+ REPOSITORY_PRIVATE: ${{ github.event.repository.private }}
run: |
set -euo pipefail
github_runner='ubuntu-latest'
aws_runner='runs-on/fleet=paperclip-public-pr-x64/env=public-ci'
+ bounded=false
+ if [ "$EVENT_NAME" = schedule ] || [ "$EVAL_CADENCE" != manual ]; then
+ if [ "$REPOSITORY_PRIVATE" != false ]; then
+ echo 'Bounded campaigns require a public repository and standard GitHub-hosted compute.' >&2
+ exit 1
+ fi
+ bounded=true
+ fi
- if [ "$AWS_PAID_RUNNER_ENABLED" = true ]; then
+ if [ "$AWS_PAID_RUNNER_ENABLED" = true ] && [ "$bounded" = false ]; then
{
echo "runner=$aws_runner"
echo "max_parallel_default=100"
@@ -155,7 +184,7 @@ jobs:
echo "max_parallel_limit=57"
echo "playwright_channel="
} >> "$GITHUB_OUTPUT"
- echo '::notice title=Paid runner routing::RUNNER_E2E_AWS_ENABLED is not true; using the proven GitHub-hosted runner'
+ echo '::notice title=Paid runner routing::Using a standard GitHub-hosted runner'
fi
target_lock:
@@ -200,6 +229,10 @@ jobs:
fi
echo "sha256=$(sha256sum pnpm-lock.yaml | cut -d ' ' -f 1)" >> "$GITHUB_OUTPUT"
+ - name: Bound scheduled lockfile upload
+ if: github.event_name == 'schedule' || (inputs.cadence != '' && inputs.cadence != 'manual')
+ run: node tests/runner-e2e/artifact-budget.mjs check lock pnpm-lock.yaml
+
- name: Upload resolved target lockfile
id: upload
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
@@ -212,13 +245,15 @@ jobs:
catalog:
name: Validate catalog and select cells
needs: [authorize, target_lock]
- if: github.event_name != 'schedule' || vars.RUNNER_FULL_STACK_E2E_NIGHTLY_ENABLED == 'true'
+ if: github.event_name != 'schedule' || (vars.RUNNER_FULL_STACK_E2E_NIGHTLY_ENABLED == 'true' && vars.RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED == 'true')
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
outputs:
matrix: ${{ steps.catalog.outputs.matrix }}
+ operations_cadence: ${{ steps.catalog.outputs.operations_cadence }}
+ operating_overhead: ${{ steps.catalog.outputs.operating_overhead }}
needs_daytona: ${{ steps.catalog.outputs.needs_daytona }}
needs_runner_typescript: ${{ steps.catalog.outputs.needs_runner_typescript }}
needs_native_binaries: ${{ steps.catalog.outputs.needs_native_binaries }}
@@ -281,6 +316,13 @@ jobs:
id: catalog
env:
EVENT_NAME: ${{ github.event_name }}
+ EVAL_EVENT: ${{ github.event_name }}
+ EVAL_SCHEDULE: ${{ github.event.schedule }}
+ EVAL_CADENCE: ${{ inputs.cadence || 'manual' }}
+ EVAL_OPERATIONS_ENABLED: ${{ vars.RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED }}
+ EVAL_OPERATING_OVERHEAD_JSON: ${{ vars.RUNNER_E2E_OPERATING_OVERHEAD_JSON }}
+ EVAL_TEST_RUNNER: ${{ needs.authorize.outputs.test_runner }}
+ TARGET_SHA: ${{ needs.authorize.outputs.target_sha }}
SELECT_ALL: ${{ inputs.all }}
SELECT_SUITE: ${{ inputs.suite }}
SELECT_GROUP: ${{ inputs.group }}
@@ -293,6 +335,31 @@ jobs:
REQUESTED_MAX_PARALLEL: ${{ inputs.max_parallel }}
run: |
set -euo pipefail
+ if [ "$EVAL_EVENT" = schedule ] || [ "$EVAL_CADENCE" != manual ]; then
+ test "$TARGET_SHA" = "$GITHUB_SHA"
+ test -z "${SELECT_SUITE:-}${SELECT_GROUP:-}${SELECT_PROFILE:-}${SELECT_ENVIRONMENT:-}${SELECT_CASE:-}${SELECT_ID:-}"
+ catalog_json="$(node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/scheduled-operations.ts matrix)"
+ cadence="$EVAL_CADENCE"
+ if [ "$EVAL_EVENT" = schedule ]; then
+ case "$EVAL_SCHEDULE" in
+ '47 8 * * *') cadence=daily ;;
+ '47 9 * * 0') cadence=weekly ;;
+ *) exit 1 ;;
+ esac
+ fi
+ {
+ echo "operations_cadence=$cadence"
+ echo "operating_overhead=$(jq -c '.overhead' <<< "$catalog_json")"
+ echo "matrix=$(jq -c '{include: .include}' <<< "$catalog_json")"
+ echo "execution_ids=$(jq -c '.executionIds' <<< "$catalog_json")"
+ echo "needs_daytona=false"
+ echo "needs_runner_typescript=true"
+ echo "needs_native_binaries=true"
+ echo "needs_remote_provider_pack=false"
+ echo "max_parallel=1"
+ } >> "$GITHUB_OUTPUT"
+ exit 0
+ fi
args=(--matrix-json)
add_values() {
local flag="$1"
@@ -509,7 +576,7 @@ jobs:
build_runner_artifacts:
name: Build reusable runner campaign artifacts
needs: [authorize, target_lock, catalog]
- if: github.event_name != 'schedule' || vars.RUNNER_FULL_STACK_E2E_NIGHTLY_ENABLED == 'true'
+ if: github.event_name != 'schedule' || (vars.RUNNER_FULL_STACK_E2E_NIGHTLY_ENABLED == 'true' && vars.RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED == 'true')
# Compile native binaries on the same reviewed image used to execute them,
# avoiding libc/architecture drift between GitHub-hosted and AWS lanes.
runs-on: ${{ needs.authorize.outputs.test_runner }}
@@ -618,6 +685,10 @@ jobs:
TARGET_SHA: ${{ needs.authorize.outputs.target_sha }}
run: echo "name=runner-e2e-build-${TARGET_SHA}-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}" >> "$GITHUB_OUTPUT"
+ - name: Bound scheduled build upload
+ if: needs.catalog.outputs.operations_cadence != ''
+ run: node tests/runner-e2e/artifact-budget.mjs check build runner-e2e-build-bundle.tar.gz runner-e2e-build-bundle.tar.gz.sha256
+
- name: Upload immutable shared campaign outputs
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
@@ -633,7 +704,7 @@ jobs:
name: Build reusable remote provider pack
needs:
[authorize, target_lock, catalog, daytona_image, build_runner_artifacts]
- if: github.event_name != 'schedule' || vars.RUNNER_FULL_STACK_E2E_NIGHTLY_ENABLED == 'true'
+ if: github.event_name != 'schedule' || (vars.RUNNER_FULL_STACK_E2E_NIGHTLY_ENABLED == 'true' && vars.RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED == 'true')
runs-on: ubuntu-latest
timeout-minutes: 15
permissions:
@@ -769,6 +840,7 @@ jobs:
test:
name: ${{ matrix.executionId }}
+ if: needs.catalog.outputs.operations_cadence == ''
needs:
[
authorize,
@@ -790,7 +862,7 @@ jobs:
fail-fast: false
max-parallel: ${{ fromJSON(needs.catalog.outputs.max_parallel) }}
matrix: ${{ fromJSON(needs.catalog.outputs.matrix) }}
- steps:
+ steps: &paid_test_steps
- name: Reauthorize paid execution before provider access
env:
GH_TOKEN: ${{ github.token }}
@@ -962,8 +1034,8 @@ jobs:
}
NODE
- - name: Install pinned legacy Claude CLI
- if: matrix.profileId == 'legacy-claude'
+ - name: Install pinned Claude CLI for setup verification
+ if: matrix.profileId == 'legacy-claude' || matrix.profileId == 'runner-acpx-claude'
run: |
npm install --global --omit=dev --ignore-scripts @anthropic-ai/claude-code@2.1.277
launcher="$(npm root --global)/@anthropic-ai/claude-code/cli-wrapper.cjs"
@@ -1012,7 +1084,7 @@ jobs:
# This definition executes only from the authorized default-branch workflow.
# Provision host policy before credentials reach target-controlled tests.
- name: Provision Codex sandbox on the disposable trusted runner
- if: matrix.environmentId == 'local' && (matrix.profileId == 'runner-codex' || matrix.profileId == 'runner-codex-mini' || matrix.profileId == 'runner-acpx-codex' || matrix.profileId == 'runner-codex-subscription')
+ if: matrix.environmentId == 'local' && (needs.catalog.outputs.operations_cadence == 'weekly' || matrix.profileId == 'runner-codex' || matrix.profileId == 'runner-codex-mini' || matrix.profileId == 'runner-acpx-codex' || matrix.profileId == 'runner-codex-subscription')
run: |
node --input-type=module <<'NODE'
import { execFileSync } from "node:child_process";
@@ -1052,7 +1124,60 @@ jobs:
timeout 120s docker pull "$oracle_image"
timeout 30s docker image inspect "$oracle_image" --format '{{.Id}}'
+ - name: Obtain scoped private history credentials
+ id: private_history_credentials
+ if: needs.catalog.outputs.operations_cadence != ''
+ uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6
+ with:
+ role-to-assume: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_ROLE_ARN }}
+ aws-region: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }}
+ output-env-credentials: false
+ output-credentials: true
+ disable-retry: true
+ action-timeout-s: 60
+
+ - name: Verify private evidence retention before paid execution
+ if: needs.catalog.outputs.operations_cadence != ''
+ env:
+ AWS_ACCESS_KEY_ID: ${{ steps.private_history_credentials.outputs.aws-access-key-id }}
+ AWS_SECRET_ACCESS_KEY: ${{ steps.private_history_credentials.outputs.aws-secret-access-key }}
+ AWS_SESSION_TOKEN: ${{ steps.private_history_credentials.outputs.aws-session-token }}
+ AWS_DEFAULT_REGION: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }}
+ RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET }}
+ run: node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/private-history.ts verify evidence "$GITHUB_RUN_ID" bounded-runner-e2e-evidence
+
+ - name: Prepare bounded campaign before credentials
+ if: needs.catalog.outputs.operations_cadence != ''
+ env:
+ EVAL_EVENT: ${{ github.event_name }}
+ EVAL_SCHEDULE: ${{ github.event.schedule }}
+ EVAL_CADENCE: ${{ needs.catalog.outputs.operations_cadence }}
+ EVAL_OPERATIONS_ENABLED: ${{ vars.RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED }}
+ EVAL_OPERATING_OVERHEAD_JSON: ${{ needs.catalog.outputs.operating_overhead }}
+ EVAL_TEST_RUNNER: ${{ needs.authorize.outputs.test_runner }}
+ PAPERCLIP_E2E_BRAINTRUST_PROJECT_ID: ${{ vars.RUNNER_E2E_BRAINTRUST_PROJECT_ID }}
+ run: node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/scheduled-operations.ts prepare
+
+ - name: Run bounded campaign and retain feedback review
+ if: needs.catalog.outputs.operations_cadence != ''
+ env:
+ BRAINTRUST_API_KEY: ${{ secrets.BRAINTRUST_API_KEY }}
+ PAPERCLIP_E2E_BRAINTRUST_PROJECT_ID: ${{ vars.RUNNER_E2E_BRAINTRUST_PROJECT_ID }}
+ PAPERCLIP_RUNNER_E2E_SOURCE_SHA: ${{ needs.authorize.outputs.target_sha }}
+ PAPERCLIP_RUNNER_E2E_SOURCE_REF: ${{ needs.authorize.outputs.target_ref }}
+ PAPERCLIP_PLAYWRIGHT_CHANNEL: ${{ needs.authorize.outputs.playwright_channel }}
+ run: |
+ set +e
+ args=(--max-parallel 1 --max-automatic-retries 0)
+ while IFS= read -r id; do args+=(--id "$id"); done < <(jq -r '.executionIds[]' "$PAPERCLIP_E2E_OPERATIONS_MANIFEST")
+ pnpm test:e2e:runner -- "${args[@]}"
+ run_status=$?
+ node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/scheduled-finalization.ts
+ finalization_status=$?
+ test "$run_status" = 0 && test "$finalization_status" = 0
+
- name: Run paid cell
+ if: needs.catalog.outputs.operations_cadence == ''
env:
OPENAI_API_KEY: ${{ matrix.credentialName == 'OPENAI_API_KEY' && secrets.OPENAI_API_KEY || '' }}
ANTHROPIC_API_KEY: ${{ matrix.credentialName == 'ANTHROPIC_API_KEY' && secrets.ANTHROPIC_API_KEY || '' }}
@@ -1063,13 +1188,19 @@ jobs:
PAPERCLIP_E2E_DAYTONA_IMAGE: ${{ needs.daytona_image.outputs.image }}
PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH: ${{ github.workspace }}/packages/paperclip-runner/provider-pack
PAPERCLIP_E2E_CAMPAIGN_ID: gha-${{ github.run_id }}-${{ github.run_attempt }}-${{ matrix.executionId }}
+ PAPERCLIP_E2E_ACCOUNTING_CAMPAIGN_ID: gha-${{ github.run_id }}-${{ github.run_attempt }}
PAPERCLIP_RUNNER_E2E_SOURCE_SHA: ${{ needs.authorize.outputs.target_sha }}
PAPERCLIP_RUNNER_E2E_SOURCE_REF: ${{ needs.authorize.outputs.target_ref }}
PAPERCLIP_PLAYWRIGHT_CHANNEL: ${{ needs.authorize.outputs.playwright_channel }}
run: pnpm test:e2e:runner -- --id "${{ matrix.executionId }}"
+ - name: Stage bounded campaign evidence
+ id: stage_evidence
+ if: always() && needs.catalog.outputs.operations_cadence != ''
+ run: node tests/runner-e2e/artifact-budget.mjs stage evidence tests/runner-e2e/results bounded-runner-e2e-evidence
+
- name: Upload access-controlled packaged cell evidence
- if: always()
+ if: always() && needs.catalog.outputs.operations_cadence == ''
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: runner-e2e-${{ github.run_id }}-${{ github.run_attempt }}-${{ matrix.executionId }}
@@ -1077,10 +1208,41 @@ jobs:
retention-days: 30
if-no-files-found: error
+ - name: Retain bounded campaign evidence in private AWS storage
+ if: always() && needs.catalog.outputs.operations_cadence != '' && steps.private_history_credentials.outcome == 'success'
+ env:
+ AWS_ACCESS_KEY_ID: ${{ steps.private_history_credentials.outputs.aws-access-key-id }}
+ AWS_SECRET_ACCESS_KEY: ${{ steps.private_history_credentials.outputs.aws-secret-access-key }}
+ AWS_SESSION_TOKEN: ${{ steps.private_history_credentials.outputs.aws-session-token }}
+ AWS_DEFAULT_REGION: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }}
+ RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET }}
+ run: node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/private-history.ts upload evidence "$GITHUB_RUN_ID" bounded-runner-e2e-evidence
+
+ - name: Enforce complete bounded evidence retention
+ if: always() && needs.catalog.outputs.operations_cadence != '' && steps.stage_evidence.outputs.complete != 'true'
+ run: exit 1
+
+ test_recurring:
+ name: ${{ matrix.executionId }}
+ if: needs.catalog.outputs.operations_cadence != '' && needs.authorize.outputs.target_sha == github.sha && github.ref == format('refs/heads/{0}', github.event.repository.default_branch)
+ needs: [authorize, target_lock, catalog, daytona_image, build_runner_artifacts, build_remote_provider_pack]
+ runs-on: ${{ needs.authorize.outputs.test_runner }}
+ timeout-minutes: ${{ matrix.timeoutMinutes }}
+ permissions:
+ contents: read
+ id-token: write
+ environment:
+ name: runner-e2e-private-paid
+ strategy:
+ fail-fast: false
+ max-parallel: ${{ fromJSON(needs.catalog.outputs.max_parallel) }}
+ matrix: ${{ fromJSON(needs.catalog.outputs.matrix) }}
+ steps: *paid_test_steps
+
report:
name: Merge and enforce campaign result
- if: always() && !cancelled() && needs.catalog.result == 'success'
- needs: [authorize, catalog, daytona_image, test]
+ if: always() && !cancelled() && needs.catalog.result == 'success' && github.ref == format('refs/heads/{0}', github.event.repository.default_branch)
+ needs: [authorize, catalog, daytona_image, test, test_recurring]
outputs:
history_source_ready: ${{ steps.history_source_ready.outputs.ready }}
runs-on: ubuntu-latest
@@ -1088,6 +1250,9 @@ jobs:
permissions:
actions: read
contents: read
+ id-token: write
+ environment:
+ name: runner-e2e-private-history
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
@@ -1107,15 +1272,8 @@ jobs:
with:
version: 9.15.4
- - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
- with:
- node-version: 24
- cache: pnpm
-
- # Resolve from this trusted checkout only. Never use the target branch's
- # lockfile or install scripts in the publication job.
- - run: pnpm install --lockfile-only --ignore-scripts --no-frozen-lockfile
- - run: pnpm install --frozen-lockfile
+ - name: Install pinned reporting dependencies
+ run: pnpm install --frozen-lockfile --ignore-scripts --store-dir "${{ runner.temp }}/eval-report-store"
- name: Resolve workflow job attempts
env:
@@ -1138,6 +1296,7 @@ jobs:
- name: Download cell evidence
id: download_evidence
+ if: needs.catalog.outputs.operations_cadence == ''
continue-on-error: true
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
@@ -1146,15 +1305,37 @@ jobs:
merge-multiple: false
- name: Retry cell evidence download after transport failure
- if: steps.download_evidence.outcome == 'failure'
+ if: needs.catalog.outputs.operations_cadence == '' && steps.download_evidence.outcome == 'failure'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
pattern: runner-e2e-${{ github.run_id }}-*-*
path: downloaded-runner-e2e
merge-multiple: false
+ - name: Obtain scoped private history credentials
+ id: private_history_credentials
+ if: needs.catalog.outputs.operations_cadence != ''
+ uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6
+ with:
+ role-to-assume: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_ROLE_ARN }}
+ aws-region: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }}
+ output-env-credentials: false
+ output-credentials: true
+ disable-retry: true
+ action-timeout-s: 60
+
+ - name: Download private bounded campaign evidence
+ if: needs.catalog.outputs.operations_cadence != ''
+ env:
+ AWS_ACCESS_KEY_ID: ${{ steps.private_history_credentials.outputs.aws-access-key-id }}
+ AWS_SECRET_ACCESS_KEY: ${{ steps.private_history_credentials.outputs.aws-secret-access-key }}
+ AWS_SESSION_TOKEN: ${{ steps.private_history_credentials.outputs.aws-session-token }}
+ AWS_DEFAULT_REGION: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }}
+ RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET }}
+ run: node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/private-history.ts download evidence "$GITHUB_RUN_ID" "downloaded-runner-e2e/runner-e2e-$GITHUB_RUN_ID-1-scheduled-campaign"
+
- name: Select latest workflow attempt per cell
- if: always()
+ if: always() && needs.catalog.outputs.operations_cadence == ''
env:
PAPERCLIP_RUNNER_E2E_ARTIFACT_ROOT: ${{ github.workspace }}/downloaded-runner-e2e
PAPERCLIP_RUNNER_E2E_SELECTED_ROOT: ${{ github.workspace }}/selected-runner-e2e
@@ -1164,6 +1345,25 @@ jobs:
PAPERCLIP_RUNNER_E2E_SOURCE_REF: ${{ needs.authorize.outputs.target_ref }}
run: node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/select-rerun-artifacts.ts
+ - name: Preserve bounded campaign evidence and journal
+ if: always() && needs.catalog.outputs.operations_cadence != ''
+ env:
+ RUN_ID: ${{ github.run_id }}
+ run: |
+ set -euo pipefail
+ artifact="downloaded-runner-e2e/runner-e2e-$RUN_ID-1-scheduled-campaign"
+ directory="$GITHUB_WORKSPACE/selected-runner-e2e/runner-e2e-$RUN_ID-1-scheduled-campaign/scheduled"
+ echo "PAPERCLIP_E2E_OPERATIONS_LEDGER=$directory/ledger.jsonl" >> "$GITHUB_ENV"
+ test -d "$artifact"
+ mkdir -p selected-runner-e2e
+ cp -R "$artifact" selected-runner-e2e/
+ if [ -f "$directory/feedback-review.json" ]; then
+ echo "PAPERCLIP_E2E_FEEDBACK_REVIEW=$directory/feedback-review.json" >> "$GITHUB_ENV"
+ fi
+ if [ -f "$directory/experiment-receipt.json" ]; then
+ echo "PAPERCLIP_E2E_BRAINTRUST_EXPERIMENT=$directory/experiment-receipt.json" >> "$GITHUB_ENV"
+ fi
+
- name: Collect blob reports
run: |
set -euo pipefail
@@ -1188,21 +1388,34 @@ jobs:
PAPERCLIP_RUNNER_E2E_REPORT_ROOT: ${{ github.workspace }}/selected-runner-e2e
PAPERCLIP_RUNNER_E2E_REPORT_OUT: ${{ github.workspace }}/runner-e2e-merged-report/normalized
PAPERCLIP_RUNNER_E2E_EXPECTED_IDS: ${{ needs.catalog.outputs.execution_ids }}
- PAPERCLIP_E2E_CAMPAIGN_ID: gha-${{ github.run_id }}-${{ github.run_attempt }}
+ PAPERCLIP_E2E_CAMPAIGN_ID: ${{ needs.catalog.outputs.operations_cadence != '' && format('gha-{0}-1-scheduled-campaign', github.run_id) || format('gha-{0}-{1}', github.run_id, github.run_attempt) }}
PAPERCLIP_RUNNER_E2E_SOURCE_SHA: ${{ needs.authorize.outputs.target_sha }}
PAPERCLIP_RUNNER_E2E_SOURCE_REF: ${{ needs.authorize.outputs.target_ref }}
- PAPERCLIP_RUNNER_E2E_HISTORY_PUBLIC_BASE_URL: ${{ vars.RUNNER_E2E_HISTORY_PUBLIC_BASE_URL }}
+ PAPERCLIP_RUNNER_E2E_HISTORY_PUBLIC_BASE_URL: ${{ needs.catalog.outputs.operations_cadence == '' && vars.RUNNER_E2E_HISTORY_PUBLIC_BASE_URL || '' }}
PAPERCLIP_RUNNER_E2E_HISTORY_PREFIX: ${{ vars.RUNNER_E2E_HISTORY_PREFIX || 'runner-e2e' }}
+ EVAL_OPERATIONS_CADENCE: ${{ needs.catalog.outputs.operations_cadence }}
run: |
set +e
- pnpm test:e2e:runner:report
+ if [ -n "$EVAL_OPERATIONS_CADENCE" ]; then
+ mkdir -p runner-e2e-merged-report
+ pnpm test:e2e:runner:report > runner-e2e-merged-report/report.log 2>&1
+ else
+ pnpm test:e2e:runner:report
+ fi
report_status=$?
set -e
- cat runner-e2e-merged-report/normalized/summary.md >> "$GITHUB_STEP_SUMMARY"
+ if [ -z "$EVAL_OPERATIONS_CADENCE" ]; then
+ cat runner-e2e-merged-report/normalized/summary.md >> "$GITHUB_STEP_SUMMARY"
+ fi
exit "$report_status"
+ - name: Stage bounded merged report
+ id: stage_report
+ if: always() && needs.catalog.outputs.operations_cadence != ''
+ run: node tests/runner-e2e/artifact-budget.mjs stage report runner-e2e-merged-report bounded-runner-e2e-report
+
- name: Upload access-controlled merged report
- if: always()
+ if: always() && needs.catalog.outputs.operations_cadence == ''
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: runner-e2e-report-${{ github.run_id }}-${{ github.run_attempt }}
@@ -1210,6 +1423,33 @@ jobs:
retention-days: 30
if-no-files-found: error
+ - name: Retain bounded report in private AWS storage
+ id: private_report
+ if: always() && needs.catalog.outputs.operations_cadence != '' && steps.private_history_credentials.outcome == 'success'
+ env:
+ AWS_ACCESS_KEY_ID: ${{ steps.private_history_credentials.outputs.aws-access-key-id }}
+ AWS_SECRET_ACCESS_KEY: ${{ steps.private_history_credentials.outputs.aws-secret-access-key }}
+ AWS_SESSION_TOKEN: ${{ steps.private_history_credentials.outputs.aws-session-token }}
+ AWS_DEFAULT_REGION: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION }}
+ RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET: ${{ vars.RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET }}
+ run: |
+ node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/private-history.ts upload report "$GITHUB_RUN_ID" bounded-runner-e2e-report
+ echo '{"version":1,"privateReportRetained":true}' > private-report-status.json
+ echo 'The recurring campaign report is retained in private AWS storage. See the eval operations runbook for authenticated retrieval.' >> "$GITHUB_STEP_SUMMARY"
+
+ - name: Retain report availability for the schedule monitor
+ if: always() && needs.catalog.outputs.operations_cadence != '' && steps.private_report.outcome == 'success'
+ uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
+ with:
+ name: runner-e2e-report-${{ github.run_id }}-${{ github.run_attempt }}
+ path: private-report-status.json
+ retention-days: 30
+ if-no-files-found: error
+
+ - name: Enforce complete bounded report retention
+ if: always() && needs.catalog.outputs.operations_cadence != '' && steps.stage_report.outputs.complete != 'true'
+ run: exit 1
+
- name: Verify normalized history source report
id: history_source_ready
if: always()
@@ -1225,7 +1465,7 @@ jobs:
publish_history:
name: Publish S3 history and Pages bundle with declared screenshots
needs: [authorize, catalog, report]
- if: always() && needs.catalog.result == 'success' && needs.report.outputs.history_source_ready == 'true'
+ if: always() && needs.catalog.result == 'success' && needs.catalog.outputs.operations_cadence == '' && needs.report.outputs.history_source_ready == 'true'
runs-on: ubuntu-latest
timeout-minutes: 15
outputs:
@@ -1257,12 +1497,10 @@ jobs:
with:
version: 9.15.4
- # Resolve the trusted publication checkout's lockfile without lifecycle
- # scripts, then install the exact result. This matches the report job's
- # frozen-install preparation while keeping target-controlled code out of
- # the AWS credentialed publisher.
- - run: pnpm install --lockfile-only --ignore-scripts --no-frozen-lockfile
- - run: pnpm install --frozen-lockfile
+ # Use a job-local store and disable lifecycle scripts throughout install.
+ # OIDC is available to every step, even before the AWS credential exchange.
+ - name: Install pinned publication dependencies
+ run: pnpm install --frozen-lockfile --ignore-scripts --store-dir "${{ runner.temp }}/eval-publisher-store"
- name: Install publisher-only Chromium
run: pnpm exec playwright install --with-deps --only-shell chromium
diff --git a/doc/agent-observability.md b/doc/agent-observability.md
index 2932330f2dc..2968ee6f507 100644
--- a/doc/agent-observability.md
+++ b/doc/agent-observability.md
@@ -28,9 +28,13 @@ requires the qualified executable version. Codex ACPX writes its isolated home.
The legacy ACPX adapter keeps export configuration in a private home per agent
and settings compatibility key, and shares conversation files with its stable managed home;
export changes do not change its logical conversation fingerprint.
-Codex's bundled Rust OTLP exporter percent-decodes header values, so launch
-material encodes them once to preserve literal percent sequences. Credentials
-remain literal values in the company secret; environment interpolation is not used.
+Codex receives authentication through `OTEL_EXPORTER_OTLP_LOGS_HEADERS` and
+`OTEL_EXPORTER_OTLP_TRACES_HEADERS` in its process environment. Its managed
+configuration and invocation profiles contain empty header maps, so exporting
+does not save the ingestion credential in those files. The bundled Rust OTLP
+exporter percent-decodes environment header values; launch material encodes
+them once to preserve literal percent sequences. Credentials remain literal
+values in the company secret; config-file environment interpolation is not used.
A caller-owned/preconfigured Codex home cannot be automatically configured in
this version: its files remain untouched and execution emits a diagnostic.
Other harnesses, personal Codex desktop sessions and hosted claude_managed are
@@ -69,8 +73,8 @@ entered secrets. Revisions prevent another administrator's edit being silently
overwritten. Credential values are excluded from task environment manifests,
run/config snapshots, argv, activity records, recovery configuration and query
cache data. Export credentials are necessarily accessible to the exporting
-harness itself. Claude Code supplies these headers through its process environment,
-which its tools can inherit or inspect. Codex can read its managed configuration.
+harness itself. Both Codex and Claude Code receive these headers through their
+process environment, which their tools can inherit or inspect.
This feature does not isolate export credentials from agent code. Use a credential
limited to telemetry ingestion for the selected destination; do not supply a broad
organization administration key. A user-operated collector can hold upstream
diff --git a/doc/evals.md b/doc/evals.md
index 985d358469f..9d89aa297eb 100644
--- a/doc/evals.md
+++ b/doc/evals.md
@@ -511,3 +511,8 @@ covers projectless task files, deferred repository acquisition, controller resta
and nested Git/dirty-file persistence on legacy local, native local and native
Daytona OpenCode. Its three configured cells use independent filesystem, public
API and download oracles; no live qualification is implied by fixture registration.
+
+The explicit-only [feedback persistence case](../tests/runner-e2e/README.md#feedback-persistence-explicit-only)
+checks real native complaint/suggestion submissions and retained bodies with
+run-attributed tool receipts. It is manual, uses the normal Product E2E report,
+and keeps synthetic feedback distinct from actionable product findings.
diff --git a/packages/adapter-utils/src/agent-observability.test.ts b/packages/adapter-utils/src/agent-observability.test.ts
index cb85c54eef9..22fb580541e 100644
--- a/packages/adapter-utils/src/agent-observability.test.ts
+++ b/packages/adapter-utils/src/agent-observability.test.ts
@@ -23,13 +23,23 @@ it("encodes headers losslessly and clears ambient exporter conflicts without cha
expect(observabilitySecretValues(launch)).toContain(input.headers.authorization);
expect(JSON.stringify(buildInvocationEnvForLogs({ ...launch.environment }))).not.toContain("a%2Cb");
});
-it("preserves literal percent sequences through the qualified Codex exporter's header decoder", () => {
+it("keeps Codex headers in the launch environment and preserves literal percent sequences", () => {
const original = "Bearer a,b=%26 c%";
const launch = buildProviderObservabilityLaunch({ ...input, kind: "codex", headers: { authorization: original } });
const exporter = launch.otelConfig!.exporter as { "otlp-http": { headers: Record } };
- expect(decodeURIComponent(exporter["otlp-http"].headers.authorization)).toBe(original);
+ expect(exporter["otlp-http"].headers).toEqual({});
+ for (const key of ["OTEL_EXPORTER_OTLP_LOGS_HEADERS", "OTEL_EXPORTER_OTLP_TRACES_HEADERS"]) {
+ expect(launch.environment[key]).toBe(`authorization=${encodeURIComponent(original)}`);
+ }
+ expect(JSON.stringify(launch.otelConfig)).not.toContain(original);
+ expect(JSON.stringify(launch.otelConfig)).not.toContain(encodeURIComponent(original));
expect(observabilitySecretValues(launch)).toContain(original);
+ expect(observabilitySecretValues(launch)).toContain(encodeURIComponent(original));
expect(observabilitySecretValues(launch)).toContain(original.slice(7));
+ expect(JSON.stringify(buildInvocationEnvForLogs({ ...launch.environment }))).not.toContain("a%2Cb");
+ const disabled = buildProviderObservabilityLaunch({ ...input, kind: "codex", enabled: false });
+ expect(disabled.environment).not.toHaveProperty("OTEL_EXPORTER_OTLP_LOGS_HEADERS");
+ expect(disabled.environment).not.toHaveProperty("OTEL_EXPORTER_OTLP_TRACES_HEADERS");
});
it("replaces a compact explicit profile without leaving competing profile flags", () => {
const args = ["exec", "-preview", "-"];
@@ -52,6 +62,8 @@ it("creates invocation-owned profiles, preserves selected profile settings, and
const a = await prepareCodexObservabilityProfile({ home: root, runId: "run-a", sourceProfile: "review", launch });
const b = await prepareCodexObservabilityProfile({ home: root, runId: "run-b", launch });
const parsed = parse(await readFile(join(root, a.filename), "utf8"));
+ expect(JSON.stringify(parsed)).not.toContain(input.headers.authorization);
+ expect(JSON.stringify(parsed)).not.toContain(encodeURIComponent(input.headers.authorization));
expect(parsed.model).toBe("chosen-model");
expect(parsed.otel).toMatchObject({ log_agent_responses: true, tool_result: { max_bytes: 65536 } });
expect((await stat(join(root, a.filename))).mode & 0o777).toBe(0o600);
diff --git a/packages/adapter-utils/src/agent-observability.ts b/packages/adapter-utils/src/agent-observability.ts
index aab46b24457..051e7cb2c78 100644
--- a/packages/adapter-utils/src/agent-observability.ts
+++ b/packages/adapter-utils/src/agent-observability.ts
@@ -35,7 +35,10 @@ export function observabilitySecretValues(launch: ProviderObservabilityLaunch |
const headers = ((launch.otelConfig?.exporter as Record | undefined)?.["otlp-http"] as { headers?: Record } | undefined)?.headers;
const encoded = launch.environment.OTEL_EXPORTER_OTLP_LOGS_HEADERS;
const decoded = (value: string) => { try { return decodeURIComponent(value); } catch { return value; } };
- const values = [...Object.values(headers ?? {}).flatMap(value => [value, decoded(value)]), ...(encoded ? [encoded, ...encoded.split(",").map(item => decoded(item.slice(item.indexOf("=") + 1)))] : [])];
+ const values = [...Object.values(headers ?? {}).flatMap(value => [value, decoded(value)]), ...(encoded ? [encoded, ...encoded.split(",").flatMap(item => {
+ const value = item.slice(item.indexOf("=") + 1);
+ return [value, decoded(value)];
+ })] : [])];
return [...new Set(values.flatMap(value => [value, value.replace(/^Bearer /i, ""), JSON.stringify(value).slice(1, -1)]).filter(Boolean))];
}
@@ -50,10 +53,11 @@ export function buildProviderObservabilityLaunch(input: {
OTEL_EXPORTER_OTLP_LOGS_TIMEOUT: "3000", OTEL_EXPORTER_OTLP_TRACES_TIMEOUT: "3000",
};
if (kind === "codex") {
- // Codex passes these through opentelemetry-otlp 0.31's with_headers, which
- // percent-decodes values. Encode once so literal percent sequences survive.
- const headers = Object.fromEntries(Object.entries(input.headers ?? {}).map(([name, value]) => [name, encodeURIComponent(value)]));
- const exporter = (endpoint: string | undefined) => enabled ? { "otlp-http": { endpoint, protocol: "binary", headers } } : "none";
+ if (enabled) Object.assign(environment, {
+ OTEL_EXPORTER_OTLP_LOGS_HEADERS: encodeObservabilityHeaders(input.headers ?? {}),
+ OTEL_EXPORTER_OTLP_TRACES_HEADERS: encodeObservabilityHeaders(input.headers ?? {}),
+ });
+ const exporter = (endpoint: string | undefined) => enabled ? { "otlp-http": { endpoint, protocol: "binary", headers: {} } } : "none";
return { kind, compatibilityKey, environment, otelConfig: {
exporter: exporter(input.logsEndpoint), trace_exporter: exporter(input.tracesEndpoint), metrics_exporter: "none",
log_user_prompt: enabled && capture.userPrompts, log_agent_responses: enabled && capture.assistantResponses,
diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_sidecar_transport.rs b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_sidecar_transport.rs
index 8f4e4dc9c18..43edd3e0196 100644
--- a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_sidecar_transport.rs
+++ b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_sidecar_transport.rs
@@ -302,6 +302,8 @@ impl AcpxSidecarTransport {
"PAPERCLIP_OBSERVABILITY_CODEX_CONFIG",
"OTEL_EXPORTER_OTLP_LOGS_TIMEOUT",
"OTEL_EXPORTER_OTLP_TRACES_TIMEOUT",
+ "OTEL_EXPORTER_OTLP_LOGS_HEADERS",
+ "OTEL_EXPORTER_OTLP_TRACES_HEADERS",
]);
}
let pi_keys = if agent == "pi" {
diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-codex-app-server.rs b/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-codex-app-server.rs
index 07e5cf81451..e5c77b954be 100644
--- a/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-codex-app-server.rs
+++ b/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-codex-app-server.rs
@@ -651,6 +651,25 @@ fn run() -> Result<(), Box> {
let require_codex_home_auth = args
.iter()
.any(|value| value == "--require-codex-home-auth");
+ if args
+ .iter()
+ .any(|value| value == "--require-otel-environment")
+ {
+ for (name, expected) in [
+ (
+ "OTEL_EXPORTER_OTLP_LOGS_HEADERS",
+ "authorization=Bearer%20logs-fixture",
+ ),
+ (
+ "OTEL_EXPORTER_OTLP_TRACES_HEADERS",
+ "authorization=Bearer%20traces-fixture",
+ ),
+ ] {
+ assert_eq!(std::env::var(name).as_deref(), Ok(expected));
+ }
+ assert!(std::env::var_os("OTEL_EXPORTER_OTLP_HEADERS").is_none());
+ assert!(std::env::var_os("OTEL_EXPORTER_OTLP_ENDPOINT").is_none());
+ }
let durable_turn_ids = args.iter().any(|value| value == "--durable-turn-ids");
let hold_first_durable_turn = args
.iter()
diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/codex_provider.rs b/packages/paperclip-runner/runner/crates/runner-core/src/codex_provider.rs
index b6f7675c5e0..70aa6735095 100644
--- a/packages/paperclip-runner/runner/crates/runner-core/src/codex_provider.rs
+++ b/packages/paperclip-runner/runner/crates/runner-core/src/codex_provider.rs
@@ -823,6 +823,8 @@ const GITHUB_CREDENTIAL_ENVIRONMENT_KEYS: &[&str] = &[
const CODEX_PROVIDER_ENVIRONMENT_KEYS: &[&str] = &[
"OTEL_EXPORTER_OTLP_LOGS_TIMEOUT",
"OTEL_EXPORTER_OTLP_TRACES_TIMEOUT",
+ "OTEL_EXPORTER_OTLP_LOGS_HEADERS",
+ "OTEL_EXPORTER_OTLP_TRACES_HEADERS",
"PAPERCLIP_AI_PROVIDER_KEY",
"PAPERCLIP_AGENT_KEY_ID",
"PAPERCLIP_AGENT_PUBLIC_KEY",
diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/codex_provider.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/codex_provider.rs
index 07c7912635c..5d8ccc10ab7 100644
--- a/packages/paperclip-runner/runner/crates/runner-core/tests/codex_provider.rs
+++ b/packages/paperclip-runner/runner/crates/runner-core/tests/codex_provider.rs
@@ -106,6 +106,19 @@ fn provider_receives_the_isolated_codex_auth_home() {
.env("PAPERCLIP_CODEX_AUTH_TEST_HOME", &directory)
.env("HOME", &directory)
.env("CODEX_HOME", &directory)
+ .env(
+ "OTEL_EXPORTER_OTLP_LOGS_HEADERS",
+ "authorization=Bearer%20logs-fixture",
+ )
+ .env(
+ "OTEL_EXPORTER_OTLP_TRACES_HEADERS",
+ "authorization=Bearer%20traces-fixture",
+ )
+ .env(
+ "OTEL_EXPORTER_OTLP_HEADERS",
+ "authorization=unwanted-ambient",
+ )
+ .env("OTEL_EXPORTER_OTLP_ENDPOINT", "https://unwanted.example")
.status()
.expect("run isolated auth-home assertion");
assert!(status.success());
@@ -119,7 +132,10 @@ fn provider_receives_isolated_codex_auth_home_subprocess() {
else {
return;
};
- let config = provider_config(&directory, &["--require-codex-home-auth"]);
+ let config = provider_config(
+ &directory,
+ &["--require-codex-home-auth", "--require-otel-environment"],
+ );
let mut provider = CodexProvider::start(&config, None)
.expect("Codex provider should read auth from the isolated CODEX_HOME");
provider.shutdown().expect("stop fake Codex provider");
diff --git a/packages/paperclip-runner/scripts/build-inputs.test.mjs b/packages/paperclip-runner/scripts/build-inputs.test.mjs
index ce650a871df..7822de2cbb3 100644
--- a/packages/paperclip-runner/scripts/build-inputs.test.mjs
+++ b/packages/paperclip-runner/scripts/build-inputs.test.mjs
@@ -491,26 +491,34 @@ test("two chains run by separate shells at the same time both record", { skip: p
await withRepo(async ({ repoRoot, packageRoot }) => {
const pendingDirectory = buildInputsPendingDirectory(packageRoot);
const marker = path.join(repoRoot, "release-the-chains");
- const records = () => (existsSync(pendingDirectory) ? readdirSync(pendingDirectory).sort() : []);
+ const records = () => (existsSync(pendingDirectory) ? readdirSync(pendingDirectory).filter(name => name.endsWith(".json")).sort() : []);
- const first = runChainInShell("sh", packageRoot, marker);
- await waitFor(() => records().length === 1, "the first chain's record");
- const second = runChainInShell("sh", packageRoot, marker);
- await waitFor(() => records().length === 2, "the second chain's record");
- const [firstId, secondId] = records().map((name) => name.replace(/\.json$/, ""));
- assert.notEqual(firstId, secondId);
- for (const id of [firstId, secondId]) {
- assert.match(id, /^\d+$/, "a chain's record is named after its shell's pid");
- assert.equal(readBuildInputsStamp(buildInputsPendingPath(packageRoot, id)).buildId, id);
+ const staged = path.join(pendingDirectory, "999999.json.999999.tmp");
+ const chains = [runChainInShell("sh", packageRoot, marker)];
+ let results;
+ try {
+ await waitFor(() => records().length === 1, "the first chain's record");
+ await writeFile(staged, "{");
+ chains.push(runChainInShell("sh", packageRoot, marker));
+ await waitFor(() => records().length === 2, "the second chain's record");
+ const [firstId, secondId] = records().map((name) => name.replace(/\.json$/, ""));
+ assert.notEqual(firstId, secondId);
+ for (const id of [firstId, secondId]) {
+ assert.match(id, /^\d+$/, "a chain's record is named after its shell's pid");
+ assert.equal(readBuildInputsStamp(buildInputsPendingPath(packageRoot, id)).buildId, id);
+ }
+ } finally {
+ await rm(staged, { force: true });
+ await writeFile(marker, "go\n");
+ results = await Promise.allSettled(chains);
}
- await writeFile(marker, "go\n");
- const results = await Promise.all([first, second]);
for (const result of results) {
- assert.match(result.stdout, new RegExp(`${BUILD_INPUTS_SCHEMA} begin: `));
- assert.match(result.stdout, new RegExp(`${BUILD_INPUTS_SCHEMA} recorded: `), "a chain finished without recording");
+ assert.equal(result.status, "fulfilled", result.reason?.message);
+ assert.match(result.value.stdout, new RegExp(`${BUILD_INPUTS_SCHEMA} begin: `));
+ assert.match(result.value.stdout, new RegExp(`${BUILD_INPUTS_SCHEMA} recorded: `), "a chain finished without recording");
}
- assert.deepEqual(records(), []);
+ assert.deepEqual(await readdir(pendingDirectory), []);
assert.equal(assessRunnerBuild({ repoRoot, packageRoot }).current, true);
// The same chain under bash, which may exec its last command: `--finish`
diff --git a/packages/paperclip-runner/src/drivers/agent-observability.test.ts b/packages/paperclip-runner/src/drivers/agent-observability.test.ts
index 4fdc2f93348..9e932db0adc 100644
--- a/packages/paperclip-runner/src/drivers/agent-observability.test.ts
+++ b/packages/paperclip-runner/src/drivers/agent-observability.test.ts
@@ -9,8 +9,12 @@ import { createSanitizedAcpxSpawnInput } from "./acpx/environment.js";
import { configuredEnvironmentProjection } from "../configured-environment.js";
const roots: string[] = [];
afterEach(async () => { await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true }))); });
-const launch: ProviderObservabilityLaunch = { kind: "codex", compatibilityKey: "revision", environment: { OTEL_EXPORTER_OTLP_LOGS_TIMEOUT: "3000" }, otelConfig: {
- exporter: { "otlp-http": { endpoint: "https://collector.example/v1/logs", protocol: "binary", headers: { authorization: "Bearer secret-canary" } } }, metrics_exporter: "none", log_agent_responses: true,
+const launch: ProviderObservabilityLaunch = { kind: "codex", compatibilityKey: "revision", environment: {
+ OTEL_EXPORTER_OTLP_LOGS_TIMEOUT: "3000",
+ OTEL_EXPORTER_OTLP_LOGS_HEADERS: "authorization=Bearer%20secret-canary",
+ OTEL_EXPORTER_OTLP_TRACES_HEADERS: "authorization=Bearer%20secret-canary",
+}, otelConfig: {
+ exporter: { "otlp-http": { endpoint: "https://collector.example/v1/logs", protocol: "binary", headers: {} } }, metrics_exporter: "none", log_agent_responses: true,
} };
it("transports a valid encoded header set in both Codex exporters without rejecting the launch", () => {
const headers = Object.fromEntries([0, 1, 2, 3].map(index => [`x-custom-${index}`, encodeURIComponent("%".repeat(4000))]));
@@ -28,8 +32,19 @@ it("materializes the real native home with both governed MCP and protected expor
nativeMcp: { name: "paperclip-assigned", url: "https://paperclip.example/mcp", token: "mcp-secret" } });
const config = parse(await readFile(join(home, "config.toml"), "utf8"));
expect(config.otel).toEqual(launch.otelConfig); expect(config.mcp_servers).toHaveProperty("paperclip-assigned");
+ expect(JSON.stringify(config)).not.toContain("secret-canary");
expect((await stat(join(home, "config.toml"))).mode & 0o777).toBe(0o600);
});
+it("delivers Codex headers only through its provider environment and redacts diagnostics", () => {
+ const source = applyProviderObservabilityEnvironment({ KEEP: "task" }, launch, true);
+ const sanitized = createSanitizedAcpxSpawnInput(source, "codex").env;
+ expect(sanitized).toMatchObject(launch.environment);
+ expect(sanitized.PAPERCLIP_OBSERVABILITY_CODEX_CONFIG).not.toContain("secret-canary");
+ expect(configuredEnvironmentProjection(source).PAPERCLIP_CONFIGURED_ENV_KEYS).toBe('["KEEP"]');
+ expect(createSanitizedAcpxSpawnInput(source, "pi").env).not.toHaveProperty("OTEL_EXPORTER_OTLP_LOGS_HEADERS");
+ const diagnostic = redactObservabilityDiagnostic("Bearer secret-canary Bearer%20secret-canary secret-canary", sanitized);
+ expect(diagnostic).toBe("[REDACTED] [REDACTED] [REDACTED]");
+});
it("passes only explicit Claude observability through its sanitizer and excludes task projection", () => {
const source = { OTEL_LOGS_EXPORTER: "otlp", OTEL_EXPORTER_OTLP_LOGS_HEADERS: "authorization=Bearer%20secret-canary", KEEP: "task" };
expect(createSanitizedAcpxSpawnInput(source, "claude").env).toMatchObject({ OTEL_LOGS_EXPORTER: "otlp", OTEL_EXPORTER_OTLP_LOGS_HEADERS: source.OTEL_EXPORTER_OTLP_LOGS_HEADERS });
diff --git a/packages/paperclip-runner/src/drivers/agent-observability.ts b/packages/paperclip-runner/src/drivers/agent-observability.ts
index 37b11a13032..7ee8159df14 100644
--- a/packages/paperclip-runner/src/drivers/agent-observability.ts
+++ b/packages/paperclip-runner/src/drivers/agent-observability.ts
@@ -27,7 +27,7 @@ export function providerObservabilityEnvironment(source: NodeJS.ProcessEnv | und
if (!source || !["codex", "claude"].includes(agent)) return {};
const result: NodeJS.ProcessEnv = {};
const names: readonly string[] = agent === "claude" ? AGENT_OBSERVABILITY_ENV_KEYS.filter(key => key !== "PAPERCLIP_OBSERVABILITY_CODEX_CONFIG")
- : ["OTEL_EXPORTER_OTLP_LOGS_TIMEOUT", "OTEL_EXPORTER_OTLP_TRACES_TIMEOUT", "PAPERCLIP_OBSERVABILITY_CODEX_CONFIG"];
+ : ["OTEL_EXPORTER_OTLP_LOGS_TIMEOUT", "OTEL_EXPORTER_OTLP_TRACES_TIMEOUT", "OTEL_EXPORTER_OTLP_LOGS_HEADERS", "OTEL_EXPORTER_OTLP_TRACES_HEADERS", "PAPERCLIP_OBSERVABILITY_CODEX_CONFIG"];
for (const key of names) if (source[key] !== undefined) {
// Codex carries two percent-encoded header maps. A valid 16-KiB company
// header set can exceed 64 KiB after encoding and duplication.
@@ -74,6 +74,7 @@ export function redactObservabilityDiagnostic(message: string, environment: Node
values.add(value);
for (const item of value.split(",")) {
const encoded = item.slice(item.indexOf("=") + 1);
+ values.add(encoded);
try { values.add(decodeURIComponent(encoded)); } catch { values.add(encoded); }
}
}
diff --git a/packages/paperclip-runner/src/drivers/codex/app-server-transport.test.ts b/packages/paperclip-runner/src/drivers/codex/app-server-transport.test.ts
index 4184701c56d..ddeef5fc806 100644
--- a/packages/paperclip-runner/src/drivers/codex/app-server-transport.test.ts
+++ b/packages/paperclip-runner/src/drivers/codex/app-server-transport.test.ts
@@ -25,6 +25,46 @@ describe("Codex app-server transport limits", () => {
})).toEqual({ PAPERCLIP_AI_PROVIDER_KEY: "selected-provider-key", CODEX_HOME: "/isolated/connection" });
});
+ it.each([true, false])("passes configured OTLP authentication to the child only when present (%s)", async (enabled) => {
+ const headers = enabled ? {
+ OTEL_EXPORTER_OTLP_LOGS_HEADERS: "authorization=Bearer%20fixture-logs",
+ OTEL_EXPORTER_OTLP_TRACES_HEADERS: "authorization=Bearer%20fixture-traces",
+ } : {};
+ const transport = new ProcessCodexAppServerTransport({
+ command: process.execPath,
+ args: ["-e", `
+ process.stdin.resume();
+ process.stdout.write(JSON.stringify({ method: "environment", params: {
+ OTEL_EXPORTER_OTLP_LOGS_HEADERS: process.env.OTEL_EXPORTER_OTLP_LOGS_HEADERS,
+ OTEL_EXPORTER_OTLP_TRACES_HEADERS: process.env.OTEL_EXPORTER_OTLP_TRACES_HEADERS,
+ DATABASE_URL: process.env.DATABASE_URL,
+ OPENAI_API_KEY: process.env.OPENAI_API_KEY,
+ } }) + "\\n");
+ `],
+ environment: createSanitizedCodexEnvironment({
+ ...headers, DATABASE_URL: "excluded-database", OPENAI_API_KEY: "excluded-provider-key",
+ }),
+ });
+ try {
+ const notification = await transport.notifications()[Symbol.asyncIterator]().next();
+ expect(notification.value).toEqual({ method: "environment", params: headers });
+ } finally {
+ await transport.close();
+ }
+ });
+
+ it("does not inherit ambient OTLP credentials without an explicit launch environment", () => {
+ vi.stubEnv("OTEL_EXPORTER_OTLP_LOGS_HEADERS", "ambient-log-secret");
+ vi.stubEnv("OTEL_EXPORTER_OTLP_TRACES_HEADERS", "ambient-trace-secret");
+ try {
+ const environment = createSanitizedCodexEnvironment();
+ expect(environment.OTEL_EXPORTER_OTLP_LOGS_HEADERS).toBeUndefined();
+ expect(environment.OTEL_EXPORTER_OTLP_TRACES_HEADERS).toBeUndefined();
+ } finally {
+ vi.unstubAllEnvs();
+ }
+ });
+
it("passes only bounded controller-projected GitHub credentials", () => {
expect(
createSanitizedCodexEnvironment({
diff --git a/packages/paperclip-runner/src/drivers/codex/app-server-transport.ts b/packages/paperclip-runner/src/drivers/codex/app-server-transport.ts
index 886e86c7a4e..c54663b1b28 100644
--- a/packages/paperclip-runner/src/drivers/codex/app-server-transport.ts
+++ b/packages/paperclip-runner/src/drivers/codex/app-server-transport.ts
@@ -251,7 +251,11 @@ export function createSanitizedCodexEnvironment(
if (key.includes("PROXY") && proxyContainsCredentials(value)) continue;
environment[key] = value;
}
- for (const key of ["PAPERCLIP_AGENT_KEY_ID", "PAPERCLIP_AGENT_PUBLIC_KEY", "PAPERCLIP_AGENT_PRIVATE_KEY"]) {
+ // Secret-bearing launch material must come from the controller's explicit environment.
+ for (const key of [
+ "PAPERCLIP_AGENT_KEY_ID", "PAPERCLIP_AGENT_PUBLIC_KEY", "PAPERCLIP_AGENT_PRIVATE_KEY",
+ "OTEL_EXPORTER_OTLP_LOGS_HEADERS", "OTEL_EXPORTER_OTLP_TRACES_HEADERS",
+ ]) {
if (explicit?.[key] !== undefined) environment[key] = explicit[key];
}
Object.assign(environment, githubCredentialEnvironment(source), configuredEnvironment(explicit));
diff --git a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts
index 30ed4b970a6..a63abc777f8 100644
--- a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts
+++ b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts
@@ -18,7 +18,7 @@ import { execFile, type ChildProcess } from "node:child_process";
import { EventEmitter } from "node:events";
import { promisify } from "node:util";
import { getCACertificates } from "node:tls";
-import { join, resolve } from "node:path";
+import { basename, dirname, join, resolve } from "node:path";
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
import {
@@ -1441,7 +1441,7 @@ describe("OpenCodeServerDriver", () => {
await session.close({ reason: "test" });
});
- it("uses the native system channel, selected skill tree, instruction root, and assigned MCP gateway", async () => {
+ it.each([false, true])("uses the native system channel, selected skill tree, instruction root, and assigned MCP gateway after startup retry=%s", async (retryStartup) => {
await chmod(fixture, 0o755);
const root = await mkdtemp(join(tmpdir(), "paperclip-opencode-context-"));
const workspace = await mkdtemp(
@@ -1465,6 +1465,7 @@ describe("OpenCodeServerDriver", () => {
]);
const systemInstructions = `${PAPERCLIP_EXECUTION_PROMPT}\n\nRead sibling.md\n\nRead-only instruction sibling root: ${instructionRoot}`;
let submittedPrompt: Record | null = null;
+ let startupFailurePending = retryStartup;
const driver = new OpenCodeServerDriver({
model: "openrouter/deepseek/deepseek-v4-flash-0731",
runtimeDirectory: root,
@@ -1479,6 +1480,10 @@ describe("OpenCodeServerDriver", () => {
PAPERCLIP_NATIVE_MCP_TOKEN: "x".repeat(40),
},
fetch: async (input, init) => {
+ if (startupFailurePending && String(input).endsWith("/session") && init?.method === "POST") {
+ startupFailurePending = false;
+ return new Response("Fixture startup failure", { status: 503 });
+ }
if (String(input).endsWith("/prompt_async"))
submittedPrompt = JSON.parse(String(init?.body));
return fetch(input, init);
@@ -1503,15 +1508,17 @@ describe("OpenCodeServerDriver", () => {
PAPERCLIP_EXECUTION_PROMPT,
);
const sessionRoot = join(root, "context");
- const isolatedHomes = (await readdir(sessionRoot)).filter((entry) =>
- entry.startsWith("home-"),
+ const runtimeEnvironment = JSON.parse(
+ await readFile(join(sessionRoot, "data", "fake-environment.json"), "utf8"),
);
- expect(isolatedHomes).toHaveLength(1);
+ const isolatedHome = runtimeEnvironment.home;
+ expect(startupFailurePending).toBe(false);
+ expect(dirname(isolatedHome)).toBe(sessionRoot);
+ expect(basename(isolatedHome)).toMatch(/^home-/);
await expect(
readFile(
join(
- sessionRoot,
- isolatedHomes[0]!,
+ isolatedHome,
".claude",
"skills",
"assigned",
diff --git a/scripts/__tests__/eval-schedule-monitor.test.mjs b/scripts/__tests__/eval-schedule-monitor.test.mjs
new file mode 100644
index 00000000000..ddf6db789e6
--- /dev/null
+++ b/scripts/__tests__/eval-schedule-monitor.test.mjs
@@ -0,0 +1,196 @@
+import assert from "node:assert/strict";
+import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import test from "node:test";
+import { acknowledgeEvalNotifications, expectedCampaign, inspectEvalSchedule, monitorTransition, runEvalMonitor, MONITORED_SCHEDULES } from "../eval-schedule-monitor.mjs";
+
+const now = Date.parse("2026-10-25T13:00:00Z");
+const enabledAt = "2026-10-24T00:00:00Z";
+const variables = { total_count: 3, variables: [
+ { name: "RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED", value: "true" },
+ { name: "RUNNER_FULL_STACK_E2E_NIGHTLY_ENABLED", value: "true" },
+ { name: "RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED_AT", value: enabledAt },
+] };
+const workflow = { id: 17, path: ".github/workflows/runner-full-stack-e2e.yml", state: "active" };
+const run = (cadence, id, extra = {}) => ({ id, workflow_id: 17, run_attempt: 1, event: "schedule", status: "completed", conclusion: "success",
+ display_title: `Runner Full-Stack E2E / ${MONITORED_SCHEDULES[cadence]}`,
+ created_at: cadence === "daily" ? "2026-10-25T08:50:00Z" : "2026-10-25T09:50:00Z", ...extra });
+function fixture(overrides = {}) {
+ const calls = [];
+ const get = async endpoint => {
+ calls.push(endpoint);
+ if (endpoint.includes("/variables?")) return overrides.variables ?? structuredClone(variables);
+ if (endpoint.endsWith("runner-full-stack-e2e.yml")) return overrides.workflow ?? workflow;
+ if (endpoint.includes("/runs?")) {
+ const workflow_runs = overrides.runs ?? [run("daily", 101), run("weekly", 102)];
+ return { total_count: overrides.total_count ?? workflow_runs.length, workflow_runs };
+ }
+ if (endpoint.includes("/artifacts?")) {
+ const id = Number(endpoint.match(/\/runs\/(\d+)\//)[1]);
+ const artifacts = overrides.artifacts ?? [{ name: `runner-e2e-report-${id}-1`, expired: false, size_in_bytes: 100 }];
+ return { artifacts, total_count: artifacts.length };
+ }
+ throw new Error("Unexpected read");
+ };
+ return { get, calls };
+}
+
+test("UTC due windows honor grace, activation and the weekly boundary", () => {
+ assert.equal(expectedCampaign("daily", now, Date.parse(enabledAt)), "2026-10-25T08:47:00.000Z");
+ assert.equal(expectedCampaign("weekly", now, Date.parse(enabledAt)), "2026-10-25T09:47:00.000Z");
+ assert.equal(expectedCampaign("weekly", Date.parse("2026-10-25T10:00:00Z"), Date.parse(enabledAt)), undefined);
+ assert.equal(expectedCampaign("daily", now, now), undefined);
+ assert.equal(expectedCampaign("daily", Date.parse("2026-11-01T10:17:00Z"), 0), "2026-11-01T08:47:00.000Z");
+});
+
+test("disabled gates do not query or diagnose old paid campaigns", async () => {
+ const f = fixture({ variables: { total_count: 0, variables: [] } });
+ assert.equal((await inspectEvalSchedule({ ...f, now })).status, "disabled");
+ assert.equal(f.calls.length, 1);
+});
+
+for (const budgetGate of [undefined, "false"]) test(`an enabled legacy schedule with ${budgetGate ?? "missing"} budget gate alerts once`, async () => {
+ const rows = [{ name: "RUNNER_FULL_STACK_E2E_NIGHTLY_ENABLED", value: "true" }];
+ if (budgetGate !== undefined) rows.push({ name: "RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED", value: budgetGate });
+ const f = fixture({ variables: { total_count: rows.length, variables: rows } });
+ const inspection = await inspectEvalSchedule({ ...f, now });
+ assert.equal(inspection.status, "configuration_incomplete");
+ assert.equal(inspection.reason, "legacy_schedule_enabled_without_budget_gate");
+ assert.equal(f.calls.length, 1);
+ const first = monitorTransition(undefined, inspection);
+ assert.equal(first.notifications.length, 1);
+ assert.equal(first.notifications[0].kind, "problem");
+ assert.deepEqual(monitorTransition(first, inspection).notifications, first.notifications);
+});
+
+test("an explicitly disabled legacy gate stays quiet even if the new budget gate is enabled", async () => {
+ const changed = structuredClone(variables); changed.variables[1].value = "false";
+ const f = fixture({ variables: changed });
+ const inspection = await inspectEvalSchedule({ ...f, now });
+ assert.equal(inspection.status, "disabled"); assert.equal(f.calls.length, 1);
+ assert.deepEqual(monitorTransition(undefined, inspection).notifications, []);
+});
+
+test("missing activation time and a disabled workflow stay distinct from missing runs", async () => {
+ const invalid = structuredClone(variables); invalid.variables[2].value = "not a date";
+ assert.equal((await inspectEvalSchedule({ ...fixture({ variables: invalid }), now })).status, "configuration_incomplete");
+ assert.equal((await inspectEvalSchedule({ ...fixture({ workflow: { ...workflow, state: "disabled_inactivity" } }), now })).status, "workflow_disabled");
+});
+
+test("requires one completed scheduled campaign and its unexpired report for each cadence", async () => {
+ const result = await inspectEvalSchedule({ ...fixture(), now });
+ assert.deepEqual(result.observations.map(row => row.status), ["healthy", "healthy"]);
+ assert.deepEqual(result.observations.map(row => row.runId), [101, 102]);
+ assert.deepEqual(monitorTransition(undefined, result).notifications, []);
+});
+
+for (const [name, overrides, status] of [
+ ["no campaign", { runs: [] }, "missing"],
+ ["manual run cannot substitute for schedule", { runs: [run("daily", 101, { event: "workflow_dispatch" })] }, "missing"],
+ ["failed campaign", { runs: [run("daily", 101, { conclusion: "failure" })] }, "failed"],
+ ["canceled campaign", { runs: [run("daily", 101, { conclusion: "cancelled" })] }, "failed"],
+ ["overdue queued campaign", { runs: [run("daily", 101, { status: "queued" })] }, "overdue"],
+ ["unexpected paid rerun", { runs: [run("daily", 101, { run_attempt: 2 })] }, "unexpected_retry"],
+ ["multiple matching runs", { runs: [run("daily", 101), run("daily", 103)] }, "ambiguous_runs"],
+ ["missing report", { artifacts: [] }, "missing_report"],
+ ["expired report", { artifacts: [{ name: "runner-e2e-report-101-1", expired: true, size_in_bytes: 100 }] }, "missing_report"],
+ ["empty report", { artifacts: [{ name: "runner-e2e-report-101-1", expired: false, size_in_bytes: 0 }] }, "missing_report"],
+]) test(name, async () => {
+ const result = await inspectEvalSchedule({ ...fixture(overrides), now });
+ assert.equal(result.observations[0].status, status);
+});
+
+test("new-day early run does not collide with the prior day during grace", async () => {
+ const runs = [run("daily", 100, { created_at: "2026-10-24T08:50:00Z" }), run("daily", 101)];
+ const result = await inspectEvalSchedule({ ...fixture({ runs }), now: Date.parse("2026-10-25T09:00:00Z") });
+ assert.equal(result.observations[0].runId, 100);
+ assert.equal(result.observations[0].status, "healthy");
+});
+
+test("running is quiet inside the runtime limit", async () => {
+ const result = await inspectEvalSchedule({ ...fixture({ runs: [run("daily", 101, { status: "in_progress", created_at: "2026-10-25T10:00:00Z" })] }), now });
+ assert.equal(result.observations[0].status, "running");
+});
+
+test("API errors and incomplete lists cannot imply a missing or successful run", async () => {
+ assert.equal((await inspectEvalSchedule({ ...fixture({ total_count: 101 }), now })).status, "api_unavailable");
+ assert.equal((await inspectEvalSchedule({ get: async () => { throw new Error("private upstream body"); }, now })).status, "api_unavailable");
+});
+
+test("alerts once per incident and recovery while keeping last observed run IDs through API failure", async () => {
+ const failed = await inspectEvalSchedule({ ...fixture({ runs: [run("daily", 101, { conclusion: "failure" }), run("weekly", 102)] }), now });
+ const first = monitorTransition(undefined, failed);
+ assert.equal(first.notifications.length, 1);
+ assert.deepEqual(monitorTransition(first, failed).notifications, first.notifications);
+ const outage = { observedAt: new Date(now + 1000).toISOString(), status: "api_unavailable", observations: [] };
+ const unavailable = monitorTransition({ ...first, notifications: [] }, outage);
+ assert.equal(unavailable.lastSuccessfulInspectionAt, first.lastSuccessfulInspectionAt);
+ assert.deepEqual(unavailable.lastInspectedRunIds, { daily: 101, weekly: 102 });
+ assert.equal(unavailable.notifications[0].status, "api_unavailable");
+ assert.deepEqual(monitorTransition(unavailable, outage).notifications, unavailable.notifications);
+ const resumed = monitorTransition({ ...unavailable, notifications: [] }, failed);
+ assert.deepEqual(resumed.notifications.map(row => [row.kind, row.cadence]), [["recovery", "monitor"]]);
+ const healthy = monitorTransition({ ...resumed, notifications: [] }, await inspectEvalSchedule({ ...fixture(), now }));
+ assert.deepEqual(healthy.notifications.map(row => [row.kind, row.cadence]), [["recovery", "daily"]]);
+});
+
+test("persists private state across invocations and refuses corrupt history", async () => {
+ const directory = await mkdtemp(path.join(os.tmpdir(), "eval-monitor-"));
+ try {
+ const file = path.join(directory, "state.json");
+ const first = await runEvalMonitor(file, fixture({ runs: [] }).get, now);
+ assert.equal(first.notifications.length, 2);
+ const repeated = await runEvalMonitor(file, fixture({ runs: [] }).get, now);
+ assert.deepEqual(repeated.notifications, first.notifications);
+ assert.deepEqual(await acknowledgeEvalNotifications(file, [first.notifications[0].id]), { acknowledged: 1, pending: 1 });
+ const afterPartialDelivery = await runEvalMonitor(file, fixture({ runs: [] }).get, now);
+ assert.deepEqual(afterPartialDelivery.notifications, first.notifications.slice(1));
+ assert.deepEqual(await acknowledgeEvalNotifications(file, first.notifications.map(row => row.id)), { acknowledged: 1, pending: 0 });
+ assert.deepEqual((await runEvalMonitor(file, fixture({ runs: [] }).get, now)).notifications, []);
+ assert.equal(JSON.parse(await readFile(file, "utf8")).repository, "paperclipai/paperclip");
+ await writeFile(file, "bad history");
+ await assert.rejects(runEvalMonitor(file, fixture().get, now));
+ assert.equal(await readFile(file, "utf8"), "bad history");
+ } finally { await rm(directory, { recursive: true, force: true }); }
+});
+
+test("pending alerts survive later recovery and are acknowledged only after delivery", async () => {
+ const directory = await mkdtemp(path.join(os.tmpdir(), "eval-monitor-outbox-"));
+ try {
+ const file = path.join(directory, "state.json");
+ const lostOutput = await runEvalMonitor(file, fixture({ runs: [] }).get, now);
+ const recovered = await runEvalMonitor(file, fixture().get, now + 1000);
+ assert.deepEqual(recovered.notifications.slice(0, 2), lostOutput.notifications);
+ assert.deepEqual(recovered.notifications.map(row => row.kind), ["problem", "problem", "recovery", "recovery"]);
+ const unchanged = await runEvalMonitor(file, fixture().get, now + 2000);
+ assert.deepEqual(unchanged.notifications, recovered.notifications);
+ await assert.rejects(acknowledgeEvalNotifications(file, ["invalid"]));
+ assert.deepEqual(await acknowledgeEvalNotifications(file, recovered.notifications.map(row => row.id)), { acknowledged: 4, pending: 0 });
+ assert.deepEqual((await runEvalMonitor(file, fixture().get, now + 3000)).notifications, []);
+ } finally { await rm(directory, { recursive: true, force: true }); }
+});
+
+test("rejects valid JSON with malformed history without querying or overwriting it", async () => {
+ const directory = await mkdtemp(path.join(os.tmpdir(), "eval-monitor-corrupt-"));
+ try {
+ const file = path.join(directory, "state.json");
+ await runEvalMonitor(file, fixture().get, now);
+ const valid = JSON.parse(await readFile(file, "utf8"));
+ for (const malformed of [null, false, 0, [], {}, { ...valid, signals: [] }, { ...valid, signals: { daily: null } },
+ { ...valid, notifications: null }, { ...valid, notifications: [{ id: "invalid" }] }]) {
+ const original = JSON.stringify(malformed);
+ await writeFile(file, original);
+ const f = fixture();
+ await assert.rejects(runEvalMonitor(file, f.get, now), /Invalid retained monitor state/);
+ assert.equal(f.calls.length, 0);
+ assert.equal(await readFile(file, "utf8"), original);
+ }
+ } finally { await rm(directory, { recursive: true, force: true }); }
+});
+
+test("workflow titles carry the literal schedule used to distinguish daily and weekly runs", async () => {
+ const text = await readFile(new URL("../../.github/workflows/runner-full-stack-e2e.yml", import.meta.url), "utf8");
+ assert.ok(text.includes("run-name: Runner Full-Stack E2E / ${{ github.event.schedule || inputs.cadence || 'manual' }}"));
+ for (const cron of Object.values(MONITORED_SCHEDULES)) assert.ok(text.includes(`cron: "${cron}"`));
+});
diff --git a/scripts/__tests__/release-verify-workflow.test.mjs b/scripts/__tests__/release-verify-workflow.test.mjs
index 8ea57dadc77..69b7df521dd 100644
--- a/scripts/__tests__/release-verify-workflow.test.mjs
+++ b/scripts/__tests__/release-verify-workflow.test.mjs
@@ -698,8 +698,22 @@ test("Runner eval workflows pin actions and gate paid live execution", () => {
const crons = [...workflow.matchAll(/cron:\s*"([^"]+)"/g)].map(
(match) => match[1],
);
- assert.equal(crons.length, 1, `${name} must have one schedule`);
- assert.match(crons[0], /^\d{1,2} \d{1,2} \* \* 0$/);
+ if (name === "runner-full-stack-e2e.yml") {
+ assert.deepEqual(crons, ["47 8 * * *", "47 9 * * 0"]);
+ assert.ok(workflow.includes("vars.RUNNER_FULL_STACK_E2E_NIGHTLY_ENABLED == 'true' && vars.RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED == 'true'"));
+ assert.ok(workflow.includes("Prepare bounded campaign before credentials"));
+ assert.ok(workflow.includes("--max-parallel 1 --max-automatic-retries 0"));
+ const catalog = workflow.slice(workflow.indexOf(" catalog:"), workflow.indexOf(" daytona_image:"));
+ assert.ok(catalog.includes("EVAL_OPERATING_OVERHEAD_JSON: ${{ vars.RUNNER_E2E_OPERATING_OVERHEAD_JSON }}"));
+ assert.ok(catalog.includes("operating_overhead: ${{ steps.catalog.outputs.operating_overhead }}"));
+ const prepare = workflow.slice(workflow.indexOf(" - name: Prepare bounded campaign before credentials"), workflow.indexOf(" - name: Run bounded campaign and retain feedback review"));
+ assert.ok(prepare.includes("EVAL_OPERATING_OVERHEAD_JSON: ${{ needs.catalog.outputs.operating_overhead }}"));
+ assert.ok(prepare.includes("EVAL_TEST_RUNNER: ${{ needs.authorize.outputs.test_runner }}"));
+ assert.ok(!prepare.includes("vars.RUNNER_E2E_OPERATING_OVERHEAD_JSON"));
+ } else {
+ assert.equal(crons.length, 1, `${name} must have one schedule`);
+ assert.match(crons[0], /^\d{1,2} \d{1,2} \* \* 0$/);
+ }
}
const chaosWorkflow = actionPinWorkflows[2];
diff --git a/scripts/eval-schedule-monitor.mjs b/scripts/eval-schedule-monitor.mjs
new file mode 100644
index 00000000000..6bee0e19566
--- /dev/null
+++ b/scripts/eval-schedule-monitor.mjs
@@ -0,0 +1,176 @@
+import { execFile } from "node:child_process";
+import { randomUUID } from "node:crypto";
+import { mkdir, readFile, rename, writeFile } from "node:fs/promises";
+import path from "node:path";
+import { promisify } from "node:util";
+import { pathToFileURL } from "node:url";
+
+export const MONITORED_REPOSITORY = "paperclipai/paperclip";
+export const MONITORED_WORKFLOW = "runner-full-stack-e2e.yml";
+export const MONITORED_SCHEDULES = { daily: "47 8 * * *", weekly: "47 9 * * 0" };
+const hour = 3_600_000;
+const prefix = `repos/${MONITORED_REPOSITORY}/actions`;
+const numericId = value => Number.isSafeInteger(value) && value > 0;
+
+export function expectedCampaign(cadence, now, enabledAt) {
+ if (!Object.hasOwn(MONITORED_SCHEDULES, cadence) || !Number.isFinite(now) || !Number.isFinite(enabledAt)) {
+ throw new Error("Invalid monitor schedule input");
+ }
+ const due = new Date(now - 90 * 60_000);
+ due.setUTCHours(cadence === "daily" ? 8 : 9, 47, 0, 0);
+ if (due.getTime() > now - 90 * 60_000) due.setUTCDate(due.getUTCDate() - 1);
+ if (cadence === "weekly") due.setUTCDate(due.getUTCDate() - due.getUTCDay());
+ return due.getTime() >= enabledAt ? due.toISOString() : undefined;
+}
+
+function signal(cadence, status, dueAt, runId) {
+ return { cadence, status, ...(dueAt ? { dueAt } : {}), ...(runId ? {
+ runId, url: `https://github.com/${MONITORED_REPOSITORY}/actions/runs/${runId}`,
+ } : {}) };
+}
+
+export async function inspectEvalSchedule({ get, now = Date.now() }) {
+ const observations = [];
+ try {
+ const variables = await get(`${prefix}/variables?per_page=100`);
+ if (!Array.isArray(variables.variables) || !Number.isSafeInteger(variables.total_count)
+ || variables.total_count !== variables.variables.length) throw new Error("Incomplete variables");
+ const values = Object.fromEntries(variables.variables.map(row => [row.name, row.value]));
+ if (values.RUNNER_FULL_STACK_E2E_NIGHTLY_ENABLED !== "true") {
+ return { observedAt: new Date(now).toISOString(), status: "disabled", observations: [] };
+ }
+ if (values.RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED !== "true") {
+ return { observedAt: new Date(now).toISOString(), status: "configuration_incomplete",
+ reason: "legacy_schedule_enabled_without_budget_gate", observations: [] };
+ }
+ const enabledAt = Date.parse(values.RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED_AT ?? "");
+ if (!Number.isFinite(enabledAt) || enabledAt > now) {
+ return { observedAt: new Date(now).toISOString(), status: "configuration_incomplete", observations: [] };
+ }
+ const workflow = await get(`${prefix}/workflows/${MONITORED_WORKFLOW}`);
+ if (workflow.path !== `.github/workflows/${MONITORED_WORKFLOW}` || !numericId(workflow.id)) throw new Error("Wrong workflow");
+ if (workflow.state !== "active") return { observedAt: new Date(now).toISOString(), status: "workflow_disabled", observations: [] };
+ const expected = Object.keys(MONITORED_SCHEDULES).map(cadence => ({ cadence, dueAt: expectedCampaign(cadence, now, enabledAt) }));
+ const due = expected.filter(row => row.dueAt);
+ if (!due.length) return { observedAt: new Date(now).toISOString(), status: "waiting", observations: [] };
+ const since = due.map(row => row.dueAt).sort()[0];
+ const runs = await get(`${prefix}/workflows/${MONITORED_WORKFLOW}/runs?event=schedule&per_page=100&created=${encodeURIComponent(`>=${since}`)}`);
+ if (!Array.isArray(runs.workflow_runs) || !Number.isSafeInteger(runs.total_count)
+ || runs.total_count !== runs.workflow_runs.length) throw new Error("Incomplete runs");
+ for (const { cadence, dueAt } of expected) {
+ if (!dueAt) { observations.push(signal(cadence, "waiting")); continue; }
+ const matches = runs.workflow_runs.filter(run => run.event === "schedule" && run.workflow_id === workflow.id
+ && run.display_title === `Runner Full-Stack E2E / ${MONITORED_SCHEDULES[cadence]}`
+ && Date.parse(run.created_at) >= Date.parse(dueAt) && Date.parse(run.created_at) <= now
+ && Date.parse(run.created_at) < Date.parse(dueAt) + (cadence === "daily" ? 24 : 168) * hour);
+ if (!matches.length) { observations.push(signal(cadence, "missing", dueAt)); continue; }
+ if (matches.length !== 1) { observations.push(signal(cadence, "ambiguous_runs", dueAt)); continue; }
+ const run = matches[0];
+ if (!numericId(run.id) || !numericId(run.run_attempt) || typeof run.status !== "string") throw new Error("Invalid run identity");
+ if (run.run_attempt !== 1) { observations.push(signal(cadence, "unexpected_retry", dueAt, run.id)); continue; }
+ if (run.status !== "completed") {
+ observations.push(signal(cadence, now - Date.parse(run.created_at) > 4 * hour ? "overdue" : "running", dueAt, run.id));
+ continue;
+ }
+ if (run.conclusion !== "success") { observations.push(signal(cadence, "failed", dueAt, run.id)); continue; }
+ const artifacts = await get(`${prefix}/runs/${run.id}/artifacts?per_page=100`);
+ if (!Array.isArray(artifacts.artifacts) || !Number.isSafeInteger(artifacts.total_count)
+ || artifacts.total_count !== artifacts.artifacts.length) throw new Error("Incomplete artifacts");
+ const reports = artifacts.artifacts.filter(row => row.name === `runner-e2e-report-${run.id}-1`
+ && row.expired === false && Number.isSafeInteger(row.size_in_bytes) && row.size_in_bytes > 0);
+ observations.push(signal(cadence, reports.length === 1 ? "healthy" : "missing_report", dueAt, run.id));
+ }
+ return { observedAt: new Date(now).toISOString(), status: "observed", observations };
+ } catch {
+ return { observedAt: new Date(now).toISOString(), status: "api_unavailable", observations };
+ }
+}
+
+const quiet = new Set(["disabled", "waiting", "running", "healthy"]);
+const statuses = new Set([...quiet, "observed", "configuration_incomplete", "workflow_disabled", "api_unavailable",
+ "missing", "ambiguous_runs", "unexpected_retry", "overdue", "failed", "missing_report"]);
+const object = value => value !== null && typeof value === "object" && !Array.isArray(value);
+const timestamp = value => typeof value === "string" && Number.isFinite(Date.parse(value));
+const notificationId = value => typeof value === "string" && /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/.test(value);
+const signalKey = row => !quiet.has(row.status) ? JSON.stringify([row.status, row.dueAt ?? null, row.runId ?? null]) : row.status;
+const validSignal = row => object(row) && ["monitor", "daily", "weekly"].includes(row.cadence) && statuses.has(row.status)
+ && (row.dueAt === undefined || timestamp(row.dueAt)) && (row.runId === undefined || numericId(row.runId))
+ && (row.runId === undefined ? row.url === undefined : row.url === `https://github.com/${MONITORED_REPOSITORY}/actions/runs/${row.runId}`);
+
+function validateState(previous) {
+ if (!object(previous) || previous.version !== 1 || previous.repository !== MONITORED_REPOSITORY
+ || previous.workflow !== MONITORED_WORKFLOW || !object(previous.signals)
+ || !timestamp(previous.checkedAt) || !(previous.lastSuccessfulInspectionAt === null || timestamp(previous.lastSuccessfulInspectionAt))
+ || !object(previous.lastInspectedRunIds) || Object.entries(previous.lastInspectedRunIds).some(([cadence, id]) => !["daily", "weekly"].includes(cadence) || !numericId(id))
+ || Object.entries(previous.signals).some(([cadence, row]) => !validSignal(row) || row.cadence !== cadence
+ || row.issue !== !quiet.has(row.status) || row.key !== signalKey(row))
+ || !Array.isArray(previous.notifications) || previous.notifications.some(row => !validSignal(row)
+ || !notificationId(row.id) || !["problem", "recovery"].includes(row.kind))
+ || new Set(previous.notifications.map(row => row.id)).size !== previous.notifications.length) {
+ throw new Error("Invalid retained monitor state; refusing to reset notification history");
+ }
+}
+
+export function monitorTransition(previous, inspection) {
+ if (previous !== undefined) validateState(previous);
+ const current = inspection.status === "observed" ? inspection.observations : [signal("monitor", inspection.status)];
+ const signals = {}, notifications = [];
+ const prior = previous?.signals ?? {};
+ for (const row of current) {
+ const issue = !quiet.has(row.status);
+ const key = signalKey(row);
+ signals[row.cadence] = { ...row, issue, key };
+ if (issue && prior[row.cadence]?.key !== key) notifications.push({ kind: "problem", ...row });
+ else if (!issue && prior[row.cadence]?.issue && row.status !== "disabled") notifications.push({ kind: "recovery", ...row });
+ }
+ if (inspection.status === "observed" && prior.monitor?.issue) notifications.push({ kind: "recovery", cadence: "monitor", status: "observed" });
+ const readable = !["api_unavailable", "configuration_incomplete"].includes(inspection.status);
+ return { version: 1, repository: MONITORED_REPOSITORY, workflow: MONITORED_WORKFLOW,
+ checkedAt: inspection.observedAt,
+ lastSuccessfulInspectionAt: readable ? inspection.observedAt : previous?.lastSuccessfulInspectionAt ?? null,
+ lastInspectedRunIds: { ...previous?.lastInspectedRunIds,
+ ...Object.fromEntries(inspection.observations.filter(row => row.runId).map(row => [row.cadence, row.runId])) },
+ signals: inspection.status === "observed" || quiet.has(inspection.status) ? signals : { ...prior, ...signals },
+ notifications: [...previous?.notifications ?? [], ...notifications.map(row => ({ id: randomUUID(), ...row }))] };
+}
+
+async function saveState(stateFile, next) {
+ await mkdir(path.dirname(stateFile), { recursive: true, mode: 0o700 });
+ await writeFile(`${stateFile}.pending`, JSON.stringify(next, null, 2) + "\n", { mode: 0o600 });
+ await rename(`${stateFile}.pending`, stateFile);
+}
+
+export async function acknowledgeEvalNotifications(stateFile, ids) {
+ if (!Array.isArray(ids) || !ids.length || ids.some(id => !notificationId(id))) throw new Error("Invalid notification acknowledgments");
+ const previous = JSON.parse(await readFile(stateFile, "utf8"));
+ validateState(previous);
+ const next = { ...previous, notifications: previous.notifications.filter(row => !ids.includes(row.id)) };
+ await saveState(stateFile, next);
+ return { acknowledged: previous.notifications.length - next.notifications.length, pending: next.notifications.length };
+}
+
+export async function runEvalMonitor(stateFile, get, now = Date.now()) {
+ let previous;
+ try { previous = JSON.parse(await readFile(stateFile, "utf8")); validateState(previous); }
+ catch (error) { if (error.code !== "ENOENT") throw error; }
+ const inspection = await inspectEvalSchedule({ get, now });
+ const next = monitorTransition(previous, inspection);
+ await saveState(stateFile, next);
+ return { ...inspection, lastSuccessfulInspectionAt: next.lastSuccessfulInspectionAt,
+ lastInspectedRunIds: next.lastInspectedRunIds, notifications: next.notifications };
+}
+
+if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
+ if (process.argv.length !== 3 && !(process.argv[3] === "--ack" && process.argv.length >= 5)) {
+ throw new Error("Usage: node scripts/eval-schedule-monitor.mjs [--ack ...]");
+ }
+ const execute = promisify(execFile);
+ const get = async endpoint => {
+ if (!endpoint.startsWith(`${prefix}/`)) throw new Error("Unapproved monitor endpoint");
+ const { stdout } = await execute("gh", ["api", "--method", "GET", endpoint], { timeout: 15_000, maxBuffer: 4 * 1024 * 1024 });
+ return JSON.parse(stdout);
+ };
+ console.log(JSON.stringify(process.argv[3] === "--ack"
+ ? await acknowledgeEvalNotifications(path.resolve(process.argv[2]), process.argv.slice(4))
+ : await runEvalMonitor(path.resolve(process.argv[2]), get), null, 2));
+}
diff --git a/server/src/__tests__/eval-feedback-recovery.integration.test.ts b/server/src/__tests__/eval-feedback-recovery.integration.test.ts
new file mode 100644
index 00000000000..873dc701afc
--- /dev/null
+++ b/server/src/__tests__/eval-feedback-recovery.integration.test.ts
@@ -0,0 +1,87 @@
+import { spawn } from "node:child_process";
+import { randomUUID } from "node:crypto";
+import { once } from "node:events";
+import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import path from "node:path";
+import { eq } from "drizzle-orm";
+import { agents, heartbeatRuns } from "@paperclipai/db";
+import { expect, it } from "vitest";
+import { createLocalAgentJwt } from "../agent-auth-jwt.js";
+import { startRunnerApiTestServer } from "./helpers/runner-api-server.js";
+import { recoverInterruptedFeedback, readRecoveredFeedback } from "../../../tests/runner-e2e/feedback-recovery.js";
+
+it("recovers real saved complaints and suggestions after their producer is killed", async () => {
+ const root = await mkdtemp(path.join(tmpdir(), "eval-feedback-interruption-"));
+ const previousHome = process.env.PAPERCLIP_HOME, previousSecret = process.env.PAPERCLIP_AGENT_JWT_SECRET;
+ process.env.PAPERCLIP_HOME = path.join(root, "home");
+ process.env.PAPERCLIP_AGENT_JWT_SECRET = randomUUID();
+ let server: Awaited> | undefined;
+ let worker: ReturnType | undefined;
+ try {
+ server = await startRunnerApiTestServer({ deploymentMode: "local_trusted" });
+ const fixture = await server.fixture({ disableWakeOnDemand: true });
+ await server.db.update(heartbeatRuns).set({ runtimeMode: "legacy", nativeIssueId: null }).where(eq(heartbeatRuns.id, fixture.runId));
+ await server.db.update(agents).set({ adapterType: "codex_local" }).where(eq(agents.id, fixture.agentId));
+ const token = createLocalAgentJwt(fixture.agentId, fixture.companyId, "codex_local", fixture.runId)!;
+ const caseDirectory = path.join(root, "case");
+ const snapshots = path.join(caseDirectory, "snapshots");
+ const identity = { campaignId: randomUUID(), attemptId: randomUUID(), executionId: "synthetic.interruption.local" };
+ await mkdir(snapshots, { recursive: true });
+ await writeFile(path.join(snapshots, "fixtures.json"), JSON.stringify({ executionId: identity.executionId, companyId: fixture.companyId }));
+ worker = spawn(process.execPath, ["--input-type=module", "-e", `
+ for (const kind of ['complaint', 'suggestion']) {
+ const response = await fetch(process.env.FEEDBACK_URL, {
+ method: 'POST', headers: { 'content-type': 'application/json',
+ authorization: 'Bearer ' + process.env.FEEDBACK_TOKEN, 'X-Paperclip-Run-Id': process.env.FEEDBACK_RUN },
+ body: JSON.stringify({ kind, body: 'Synthetic interrupted producer ' + kind, idempotencyKey: kind }),
+ });
+ if (response.status !== 201) throw new Error('Synthetic feedback submission failed: ' + response.status);
+ await response.json();
+ }
+ process.send({ saved: 2 });
+ setInterval(() => {}, 1000);
+ `], { stdio: ["ignore", "pipe", "pipe", "ipc"], env: {
+ PATH: process.env.PATH, FEEDBACK_URL: `${server.apiUrl}/api/companies/${fixture.companyId}/agent-commentary`,
+ FEEDBACK_TOKEN: token, FEEDBACK_RUN: fixture.runId,
+ } });
+ const saved = await Promise.race([
+ once(worker, "message", { signal: AbortSignal.timeout(10_000) }),
+ once(worker, "exit").then(([code]) => { throw new Error(`Producer exited before saving feedback: ${code}`); }),
+ ]);
+ expect(saved[0]).toEqual({ saved: 2 });
+ const exited = once(worker, "exit");
+ worker.kill("SIGKILL");
+ expect((await exited)[1]).toBe("SIGKILL");
+ const apiUrl = server.apiUrl;
+ await recoverInterruptedFeedback({ identity, caseDirectory, secrets: [token], deadlineAt: Date.now() + 5_000,
+ api: { async get(endpoint: string, options?: { timeout: number }): Promise {
+ const response = await fetch(apiUrl + endpoint, { signal: AbortSignal.timeout(options?.timeout ?? 1_000) });
+ if (!response.ok) throw new Error(`Read failed: ${response.status}`);
+ return response.json() as Promise;
+ } },
+ });
+ expect(await readRecoveredFeedback(caseDirectory, identity)).toMatchObject({
+ complaints: 1, suggestions: 1, bodiesCollected: 2, collection: "partial", invocationCollection: "partial",
+ invocations: { completed: 0, failed: 0, unknown: 0 },
+ });
+ await server.close(); server = undefined;
+ const evidenceText = await readFile(path.join(snapshots, "agent-feedback-recovered.json"), "utf8");
+ const evidence = JSON.parse(evidenceText);
+ expect(evidence.entries.map((entry: { body: string }) => entry.body).sort()).toEqual([
+ "Synthetic interrupted producer complaint", "Synthetic interrupted producer suggestion",
+ ]);
+ expect(evidence.entries.every((entry: { runId: string; agent: { id: string }; task: { id: string } }) =>
+ entry.runId === fixture.runId && entry.agent.id === fixture.agentId && entry.task.id === fixture.issueId)).toBe(true);
+ expect(evidenceText).not.toContain(token);
+ expect((await readRecoveredFeedback(caseDirectory, identity))?.review).toHaveLength(2);
+ } finally {
+ if (worker && worker.exitCode === null && worker.signalCode === null) {
+ const exited = once(worker, "exit"); worker.kill("SIGKILL"); await exited;
+ }
+ await server?.close();
+ if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = previousHome;
+ if (previousSecret === undefined) delete process.env.PAPERCLIP_AGENT_JWT_SECRET; else process.env.PAPERCLIP_AGENT_JWT_SECRET = previousSecret;
+ await rm(root, { recursive: true, force: true });
+ }
+}, 90_000);
diff --git a/server/src/__tests__/fixtures/otlp-observability.proto b/server/src/__tests__/fixtures/otlp-observability.proto
index ed258755a2d..98feb312054 100644
--- a/server/src/__tests__/fixtures/otlp-observability.proto
+++ b/server/src/__tests__/fixtures/otlp-observability.proto
@@ -13,7 +13,8 @@ message LogRecord { fixed64 time_unix_nano = 1; int32 severity_number = 2; strin
message ScopeLogs { Scope scope = 1; repeated LogRecord log_records = 2; }
message ResourceLogs { Resource resource = 1; repeated ScopeLogs scope_logs = 2; }
message ExportLogsServiceRequest { repeated ResourceLogs resource_logs = 1; }
-message Span { bytes trace_id = 1; bytes span_id = 2; bytes parent_span_id = 4; string name = 5; int32 kind = 6; fixed64 start_time_unix_nano = 7; fixed64 end_time_unix_nano = 8; repeated KeyValue attributes = 9; }
+message Status { string message = 2; int32 code = 3; }
+message Span { bytes trace_id = 1; bytes span_id = 2; bytes parent_span_id = 4; string name = 5; int32 kind = 6; fixed64 start_time_unix_nano = 7; fixed64 end_time_unix_nano = 8; repeated KeyValue attributes = 9; Status status = 15; }
message ScopeSpans { Scope scope = 1; repeated Span spans = 2; }
message ResourceSpans { Resource resource = 1; repeated ScopeSpans scope_spans = 2; }
message ExportTraceServiceRequest { repeated ResourceSpans resource_spans = 1; }
diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts
index fb08b142c6f..f33983f14c0 100644
--- a/server/src/services/native-runtime/native-session-executor.test.ts
+++ b/server/src/services/native-runtime/native-session-executor.test.ts
@@ -802,6 +802,9 @@ describe("remote runner process supervision", () => {
await writeFile(fixture, 'require("node:net").createServer().listen(0,"127.0.0.1",()=>process.stdout.write("ready\\n"));');
let child: ReturnType | undefined;
let probes = 0;
+ let healthyProbes = 0;
+ let releaseExitProbe!: () => void;
+ const exitProbeReady = new Promise(resolve => { releaseExitProbe = resolve; });
let settled = false;
const ok = { exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "" };
const execute = vi.fn(async (input: { command: string; args?: string[] }) => {
@@ -817,12 +820,15 @@ describe("remote runner process supervision", () => {
return ok;
}
if (label === "paperclip-runner-monitor" && ++probes === 1) return { ...ok, exitCode: null, timedOut: true };
- return await new Promise<{ exitCode: number | null; signal: null; timedOut: boolean; stdout: string; stderr: string }>(resolve => {
+ if (label === "paperclip-runner-monitor" && probes > 2) await exitProbeReady;
+ const observed = await new Promise<{ exitCode: number | null; signal: null; timedOut: boolean; stdout: string; stderr: string }>(resolve => {
execFile(input.command, input.args ?? [], { timeout: 2_000 }, (error, stdout, stderr) => resolve({
exitCode: error ? typeof error.code === "number" ? error.code : null : 0,
signal: null, timedOut: !!error?.killed, stdout, stderr,
}));
});
+ if (label === "paperclip-runner-monitor" && observed.exitCode === 0 && !observed.timedOut) healthyProbes++;
+ return observed;
});
const handle = createRemoteRunnerProcessLauncher({
target: { kind: "remote", transport: "sandbox", environmentId: "environment", leaseId: "lease", remoteCwd: root },
@@ -831,17 +837,21 @@ describe("remote runner process supervision", () => {
})({ command: fixture, args: [], cwd: root, environment: {} });
void handle.completion.then(() => { settled = true; }, () => { settled = true; });
try {
- await vi.waitFor(() => expect(probes).toBeGreaterThanOrEqual(2), { timeout: 5_000 });
+ await vi.waitFor(() => expect(healthyProbes).toBe(1), { timeout: 5_000 });
expect(settled).toBe(false);
expect(handle.isExitConfirmed?.()).toBe(false);
expect(() => process.kill(child!.pid!, 0)).not.toThrow();
expect((await readFile(marker, "utf8")).split("\n")[1]).toBe(String(child!.pid));
expect(execute.mock.calls.filter(([x]) => x.args?.[2] === "paperclip-runner-launch")).toHaveLength(1);
+ const childStopped = new Promise(resolve => { child!.once("exit", () => resolve()); });
child!.kill("SIGTERM");
+ await childStopped;
+ releaseExitProbe();
await handle.completion;
expect(handle.isExitConfirmed?.()).toBe(true);
} finally {
child?.kill("SIGKILL");
+ releaseExitProbe();
await handle.completion.catch(() => undefined);
await rm(root, { recursive: true, force: true });
}
diff --git a/tests/e2e/cases-read-only.spec.ts b/tests/e2e/cases-read-only.spec.ts
index 83f272b46b4..fbd98223df4 100644
--- a/tests/e2e/cases-read-only.spec.ts
+++ b/tests/e2e/cases-read-only.spec.ts
@@ -28,12 +28,9 @@ test.afterEach(async ({ request }, testInfo) => {
test("browses passive cases and rolled-up task documents without starting work or editing", async ({ page }, testInfo) => {
page.setDefaultTimeout(10_000);
- // Announcements can arrive after navigation and cover otherwise valid links.
- // Dismiss through the ordinary Board control, including after reloads.
- const dismissAnnouncement = page.getByRole("complementary", { name: "Paperclip announcements", exact: true })
- .getByRole("button", { name: "Dismiss announcement", exact: true });
- const dismissAnnouncements = async () => { await dismissAnnouncement.click(); };
- await page.addLocatorHandler(dismissAnnouncement, dismissAnnouncements);
+ await page.route("**/api/announcements/current", async (route) => {
+ await route.fulfill({ json: null });
+ });
const api = page.request;
const original = await json(await api.get("/api/instance/settings/experimental"));
const company = await json(await api.post("/api/companies", { data: { name: `Read-only cases ${randomUUID()}` } }));
@@ -189,13 +186,9 @@ test("browses passive cases and rolled-up task documents without starting work o
await expect(dialog.getByRole("button", { name: /^(edit|save|restore|delete)\b/i })).toHaveCount(0);
await expect(dialog.locator('textarea, [contenteditable="true"]')).toHaveCount(0);
await testInfo.attach("read-only-document-revisions", { body: await page.screenshot(), contentType: "image/png" });
- // The handler's own click moves focus. Keep it out of this focus-restoration
- // assertion, then resume ordinary dismissal before the next navigation.
- await page.removeLocatorHandler(dismissAnnouncement);
await page.keyboard.press("Escape");
await expect(dialog).toHaveCount(0);
await expect(openReceipt).toBeFocused();
- await page.addLocatorHandler(dismissAnnouncement, dismissAnnouncements);
const auditLink = casePage.getByRole("link", { name: "View in Audit", exact: true });
await expect(auditLink).toHaveAttribute("href", `/${company.issuePrefix}/activity?entityType=case&entityId=${receipt.id}`);
await auditLink.click();
diff --git a/tests/e2e/runner-e2e-dashboard.spec.ts b/tests/e2e/runner-e2e-dashboard.spec.ts
index e9e1d8e550e..e9cc5c5237b 100644
--- a/tests/e2e/runner-e2e-dashboard.spec.ts
+++ b/tests/e2e/runner-e2e-dashboard.spec.ts
@@ -363,3 +363,32 @@ test("case status dots summarize matching results and open failures", async ({ p
await page.getByRole("button", { name: "Legacy Codex · Daytona sandbox", exact: true }).click();
await expect(topLink).toHaveAttribute("data-case-status", "failed");
});
+
+test("keeps long feedback fingerprints inside a scrollable review table on phones", async ({ page }) => {
+ const { emptyFeedbackSummary } = await import("../runner-e2e/feedback.js");
+ const { feedbackReviewTemplate } = await import("../runner-e2e/feedback-dispositions.js");
+ const execution = runnerExecutionById(executionIds[0]);
+ const entry = resultFor(execution, "passed");
+ entry.result.feedback = { ...emptyFeedbackSummary(), collection: "complete", invocationCollection: "complete",
+ complaints: 2, bodiesCollected: 2,
+ review: ["a", "b"].map(value => ({ fingerprint: value.repeat(64), kind: "complaint", occurrences: 1, disposition: "needs_investigation" })),
+ };
+ const campaign = buildRunnerCampaign({ campaignId: "feedback-mobile", generatedAt: entry.result.finishedAt, expected: [execution.id], results: [entry.result] });
+ campaign.feedbackReview = { ...feedbackReviewTemplate(campaign), version: 1, reviewedAt: "2026-10-20T01:00:00Z", reviewedBy: "agent",
+ decisions: [{ fingerprint: "a".repeat(64), disposition: "actionable", issueNumber: 123 },
+ { fingerprint: "b".repeat(64), disposition: "duplicate", duplicateOf: "a".repeat(64) }],
+ };
+ await page.setViewportSize({ width: 390, height: 844 });
+ await openReport(page, renderRunnerE2EDashboard({ title: "Feedback review", generatedAt: campaign.generatedAt,
+ expected: campaign.expected, catalog: [execution], entries: [entry], campaign }));
+ await page.getByText("Campaign usage and spend", { exact: true }).click();
+ const decisions = page.getByRole("region", { name: "Feedback review decisions" });
+ await expect(decisions).toBeVisible();
+ expect(await decisions.evaluate(element => element.scrollWidth > element.clientWidth)).toBe(true);
+ expect(await page.evaluate(() => document.documentElement.scrollWidth)).toBeLessThanOrEqual(390);
+ await decisions.focus();
+ await page.keyboard.press("Tab");
+ const followup = decisions.getByRole("link", { name: "#123" });
+ await expect(followup).toBeFocused();
+ await expect(followup).toBeInViewport();
+});
diff --git a/tests/runner-e2e/FIXTURES.md b/tests/runner-e2e/FIXTURES.md
index e1a34eee982..e499517506b 100644
--- a/tests/runner-e2e/FIXTURES.md
+++ b/tests/runner-e2e/FIXTURES.md
@@ -1011,3 +1011,30 @@ The arithmetic journey records four delivered agent handoffs and one completed
terminal intent. The final case is `done` at handoff version 5, while the same
four provider work runs and unchanged workflow document revision finish the task.
The oracle rejects a missing or pending terminal intent.
+
+## Feedback persistence (explicit only)
+
+`--suite feedback-persistence` selects two local native Claude/Codex cells.
+Each creates one standard task through the browser, asks for two distinct
+synthetic feedback items, and requires the real `submit_complaint` and
+`submit_suggestion` tools. The five-minute case expects one task work run and
+disables automatic retries. Company and agent accounted-work budgets are $10.
+Setup probes and any background work must also pass through the separately
+budgeted gateway when using an operations manifest. Local compute is unmetered.
+Cost and duration forecasts remain unmeasured until a retained live attempt.
+The runner E2E maintainers own this manual case; `--all` excludes it.
+
+After collection and fixture teardown, the oracle reads the retained evidence
+file and requires exact nonce-bearing bodies, the original company/agent/task
+and observed run, plus one completed native receipt for each tool. Missing or
+partial collection, prose-only claims, foreign attribution, duplicate records,
+and failed submissions cannot pass. Full bodies stay in private evidence;
+public reports retain counts, fingerprints, and closed oracle checks. The usual
+result screenshot, trace, secret scanning and supervisor cleanup still apply.
+This case proves explicit submission and persistence, not spontaneous feedback
+quality or survival after an abrupt worker kill.
+
+```sh
+pnpm test:e2e:runner -- --list --suite feedback-persistence
+pnpm test:e2e:runner -- --id feedback-persistence.runner-acpx-claude.local.complaint-and-suggestion --max-automatic-retries 0 --max-parallel 1
+```
diff --git a/tests/runner-e2e/OPERATIONS-COVERAGE.md b/tests/runner-e2e/OPERATIONS-COVERAGE.md
new file mode 100644
index 00000000000..da4e2efa6ec
--- /dev/null
+++ b/tests/runner-e2e/OPERATIONS-COVERAGE.md
@@ -0,0 +1,232 @@
+# Eval coverage and operating costs
+
+Use this with [OPERATIONS.md](OPERATIONS.md). The bounded daily and weekly
+selections cover Product E2E only. They do not authorize or cap the separate
+Runner protocol workflow, manual matrix runs, or other live probes.
+
+## Initial cadence
+
+The runner E2E maintainers own these selections and their failure triage.
+Both profiles use the exact model and limits in the reviewed pricecard.
+
+| Journey | Failure detected | Feature-map association | Cadence |
+| --- | --- | --- | --- |
+| Native Claude message marker | A real task cannot complete and persist its expected answer through the metered route | `runs-adapters/runs` | Daily and weekly |
+| Native Claude complaint and suggestion | Completed tools fail to save attributed feedback or collection loses it | `activity/agent-commentary` | Daily and weekly |
+| Native Codex message marker | The corresponding Codex task/connection path fails | `runs-adapters/runs` | Weekly |
+| Native Codex complaint and suggestion | The corresponding Codex feedback/tool path fails | `activity/agent-commentary` | Weekly |
+
+These associations identify partial coverage. They do not qualify every run
+lifecycle, the Commentary browser page, continuation, or remote execution.
+The October 21 local pilot measured one successful sample per selected cell:
+$0.427624800 and 72.757 summed attempt seconds for the daily selection,
+$0.791539400 and 127.069 seconds for the weekly selection. These are historical
+gateway estimates from the Linux ARM64 container. They include model setup
+requests and attempt setup/teardown, but omit queue/build time and non-model
+charges. Earlier failed and interrupted attempts remain in campaign spending.
+
+`scheduled-measurements.json` retains the exact attempt, source revision,
+definition, model, price version, result hash and journal hashes for each sample.
+`forecast.json` includes this evidence and a snapshot hash. It separates the
+historical successful selection from samples matching the current definition,
+model, environment, runtime, route and pricing. A missing matching cell makes
+the compatible selection total unknown. The feedback definition changed after
+these four samples; its earlier results are not current qualification.
+
+Per-cell mean and median are descriptive. P95 stays unknown below 20 samples
+and then uses nearest rank; this is not a statistical confidence guarantee.
+Do not sum per-cell percentiles into a campaign percentile. Hosted cost/time
+and the selection's P95 remain unknown until separately measured. A passing
+sample does not predict failure frequency or account for infrastructure,
+storage, observability or subscriptions. Forecasts never reduce reservations
+or the approved budget limits.
+
+At the October 21, 2026 catalog audit, 47 suites declared 738 cells. `--all`
+included 179; the other 559 required explicit selection. The bounded weekly
+selection included four cells, and daily two. These are dated inventory counts,
+not live acceptance results or an enduring catalog-size contract. Inspect the
+current `pnpm test:e2e:runner -- --list` before choosing work. In particular,
+Everyday Workflows, provider connections, feedback persistence, interruption,
+and case handoffs must not be inferred from a passing `--all` campaign.
+
+Keep the broader rotating lane disabled until its exact cells have a metered
+route, complete retained attempts, a measured cost/time distribution, and a
+budget that includes failures. Prefer targeted continuation, approval, and
+cleanup cases that detect different failures over another model running the
+same marker. A release lane may broaden coverage only with its own reviewed
+manifest; the daily/weekly allowance does not authorize it.
+
+## Adding an eval or changing its cadence
+
+The change's fixture documentation and PR must declare:
+
+- A responsible maintainer/team and the concrete failure the case detects.
+- Feature-map capability IDs, or an explicit unmapped gap. State what the
+ journey does not exercise within a broader capability.
+- Exact suite, case, profiles, models, environment, and intended cadence.
+ State whether it is explicit-only and how it differs from existing coverage.
+- A deterministic alternative considered, and why paid behavior is necessary.
+- Attempt, request, output-token, concurrency, retry, and wall-time limits.
+ Include setup, judges, child/background work, and cleanup in the route audit.
+- Estimated cost/time before the first run, then all-attempt measurements with
+ sample count, retained failures, source/definition IDs, and uncertainty.
+ An unmeasured field is unknown, never zero.
+- Feedback collection/review, trace verification, missing-receipt handling,
+ and a failure/cleanup owner. Novel complaints are reviewed independently
+ of the product grade.
+
+Use the existing [Product E2E authoring skill](../../.agents/skills/add-product-e2e-eval/SKILL.md)
+or the separate [Runner authoring skill](../../.agents/skills/add-runner-eval/SKILL.md).
+Update the existing scheduled selection and forecast only after qualification;
+adding a catalog entry alone must not enable recurring spending.
+
+## Reading campaign time
+
+The report's summed attempt time uses every retained accounting receipt,
+including failed retries. A missing attempt makes the total unknown; the report
+keeps the observed subtotal and missing count. The observed attempt window runs
+from the earliest retained start to the latest retained finish. Parallel attempts
+overlap in that window, while their individual durations add to the sum. Gaps
+between attempts remain in the window. Carried-forward evidence can span older
+runs, so this window is not the current workflow's duration.
+
+Agent execution and Daytona lease time remain separate measurements. Attempt
+durations include the setup and cleanup inside their recorded boundaries.
+New browser-worker results retain a separate phase receipt for fixture setup,
+execution, feedback capture, cleanup and worker finalization. Repeated cleanup
+segments add together. Atomic checkpoints preserve completed phases if the
+worker is interrupted; the active phase has no inferred end time. Parent
+recovery checks identity, interval bounds and file size before retaining that
+partial receipt. Missing, malformed or unavailable receipts keep phase totals
+unknown, while closed segments remain visible as observed subtotals.
+
+These phases start inside the test callback, after browser/server startup, and
+end before result publication. Parent teardown/finalization, CI queue/build time
+and provider-billed runtime need their own receipts. Historical results and
+non-browser launchers without a worker receipt remain explicitly unmeasured.
+Test details show each selected attempt's receipt status and raw timing metadata.
+Do not infer missing measurements by subtracting summed agent time from the
+observed window. Refreshing a report computes these views from retained
+results without making provider calls or changing the original measurements.
+
+## Costs outside the model journal
+
+Keep a separate cost component for each resource, with its currency, service,
+resource/request identity, source window, allocation basis, and status. Join
+components to campaign/attempt IDs where evidence permits. Do not sum a model
+estimate with the invoice that later reconciles that same usage.
+
+| Component | Evidence and allocation | When evidence is missing |
+| --- | --- | --- |
+| Provider models and paid tools | Dedicated project/key usage and billing exports; match provider request IDs or the narrowest available billing window | Preserve estimates, unknown requests, held reservations and the reconciliation difference |
+| Daytona | Owned sandbox ID, actual resources, billable lifecycle and provider receipt; include failed creates, retries, stopped storage and delayed deletion | Retain the reservation or an explicitly partial estimate; a requested TTL does not prove deletion |
+| CI compute | Workflow/job IDs, runner type, actual billed runtime and invoice rates; include setup, builds and failed jobs | Mark CI cost unknown; a short agent turn does not imply a cheap CI job |
+| Storage and transfer | Artifact size, retention window and billed storage/egress | Report the unpriced component, including retained failure evidence |
+| Braintrust/gateway usage | Project usage export and its window; distinguish included allowance from incremental overage | Do not treat free-looking trace metrics as proof of zero provider charges |
+| Shared subscriptions | Show the existing Braintrust Pro $249/month separately under the October 21 operator policy; allocate newly incurred subscription/capacity charges from invoices with an explicit denominator | Keep new shared charges unallocated rather than assign an invented exact per-test cost |
+
+Retain both the service total and the allocation denominator so allocated
+amounts can be reproduced. A zero incremental usage charge can coexist with a
+nonzero shared subscription. Never add the entire subscription to every test.
+The approved daily/weekly budgets cover additional run costs, so they exclude
+the existing $249/month fee while including new usage and capacity charges.
+Scheduled admission now reserves the declared CI, storage, observability and
+subscription allowances inside the same $20/$100 journal limit. Missing or stale
+allowances stop scheduling before the AWS test runner starts. Missing receipts
+keep their reservations held and accounting incomplete. This is a local admission
+control, not a provider billing cap. Actual resource bounds, cost collection and
+reviewed allowances still need qualification before recurrence is enabled; see
+[the allowance contract](OPERATIONS.md#recurring-campaigns).
+
+Recurring evidence and reports use private S3 archives with file/byte/request
+boundaries, retained transfer receipts and verified access/expiration rules;
+see [private recurring history](OPERATIONS.md#private-recurring-history). The
+public workflow retains only a content-free availability marker for monitoring.
+Public history is a separate explicit publication step. Private retention still
+needs hosted OIDC qualification and actual storage/transfer billing evidence;
+local archive tests and an infrastructure template do not establish those facts.
+
+## Separate Runner Eval audit
+
+The October 21 audit inspected Paperclip source on master `8b873e0112` plus the
+eval operations stack, and the available `paperclip-evals` checkout at
+`0bae4e0947dc7959cae25b73d27533c096f23a14`. This is a source audit, not a
+new live run or proof of the workflow's configured private-repository revision.
+
+Runner reports already retain immutable attempts, retry-inclusive estimated
+cost and provider list cost as separate views, compatible-cohort comparisons,
+and missing-usage coverage. Its mock control plane is a different authority
+from the Product E2E server/database; Product feedback persistence cannot be
+claimed from a Runner tool assertion.
+
+The private `run_live_campaign.py` requires an explicit authorization envelope.
+The hosted `runner-protocol-live-evals.yml` instead fans out individual
+`run_live_roster.py` cells, supplies their direct provider credentials, and uses
+configured per-attempt limits. It does not use the Product E2E shared gateway
+journal, Braintrust attempt parent, indexing verifier, or feedback finalizer.
+The private campaign wrapper's aggregate authorization must not be assumed to
+protect that different hosted path.
+
+Before adding Runner recurrence, its owners must establish an aggregate
+allowance across fanout and retries, meter setup/provider bypasses, and qualify
+native tracing through its actual packaged runner. Preserve direct-provider
+qualification as its own route. Implement definition/roster changes in the
+maintained private repository, pin that reviewed revision in the existing
+workflow, and retain Runner's existing Evalbook renderer and scores. These
+gaps remain open; Product E2E qualification does not close them.
+
+## Missing campaign detection
+
+Workflow failure notifications cover runs that start. The independent observer
+is `scripts/eval-schedule-monitor.mjs`. Run it through a separate host with an
+authenticated GitHub CLI and one private state file:
+
+```sh
+node scripts/eval-schedule-monitor.mjs "$HOME/.config/paperclip-evals/schedule-monitor.json"
+```
+
+The observer uses only GitHub GET requests. It never starts or retries paid
+work and does not need provider credentials. It reads the two existing enable
+gates plus `RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED_AT`, an ISO timestamp set
+when recurrence is enabled. Missing activation time fails visibly; occurrences
+before activation are not treated as missed. Workflow run titles carry the
+literal cron expression to distinguish daily and weekly runs, including Sunday.
+
+Only the legacy `RUNNER_FULL_STACK_E2E_NIGHTLY_ENABLED` gate being off establishes
+disabled recurrence. If that gate is on while the operations gate is absent or
+off, the observer reports `configuration_incomplete` and retains one incident
+notification. Older deployed workflows use only the legacy gate, so a missing
+new gate must not imply that those paid schedules are disabled. Verify the
+deployed default-branch workflow and both variables before enabling recurrence.
+
+The observer allows 90 minutes for schedule delay. It checks the most recent due
+occurrence per cadence, marks a run overdue after four hours, and requires a
+successful terminal run plus a nonempty, unexpired merged-report artifact.
+The workflow's finalizer remains responsible for checking report content and
+accounting. Metadata inspection does not repeat that validation. Unexpected
+reruns, ambiguous matching runs, incomplete API lists and API failures remain
+visible. The observer reads at most 100 rows per collection and fails closed
+if a response is truncated.
+
+Retained state includes the last successful inspection and inspected run IDs.
+`notifications` is a durable outbox of problems and recovery transitions. Each
+entry has an ID. After confirmed delivery, the host acknowledges those IDs:
+
+```sh
+node scripts/eval-schedule-monitor.mjs "$HOME/.config/paperclip-evals/schedule-monitor.json" --ack ...
+```
+
+Until acknowledgment, later inspections return the same pending entries even
+if the problem has recovered. The host uses IDs to deduplicate delivery. This
+survives a process exit between saving state and printing output; it does not
+claim exactly-once delivery across an external notification service. After
+acknowledgment, an unchanged problem stays quiet. A new missed occurrence is a
+new incident. Disabled recurrence and unchanged healthy runs create no new
+alerts. Use one writer for both checks and acknowledgments; do not reset the
+state file to clear an incident. Malformed history fails before a query or
+write. The external host owns notification delivery and its own liveness.
+
+Schedule this observer outside the paid workflow after source and recurrence
+are ready. It should inspect at least daily, after both scheduled start times
+and their grace windows. Until that external trigger and notification delivery
+are configured and exercised, missed-run detection remains an activation gap.
diff --git a/tests/runner-e2e/OPERATIONS.md b/tests/runner-e2e/OPERATIONS.md
new file mode 100644
index 00000000000..31b36ed856f
--- /dev/null
+++ b/tests/runner-e2e/OPERATIONS.md
@@ -0,0 +1,707 @@
+# Bounded eval operations
+
+The request ledger and gateway are opt-in. The ordinary launcher still supports
+unmetered legacy campaigns. Do not infer a spending limit from a campaign ID.
+Enable the bounded path with `PAPERCLIP_E2E_OPERATIONS_MANIFEST=/absolute/file.json`.
+Keep this manifest, its journal, and request receipts private.
+
+The [coverage and operating-cost guide](OPERATIONS-COVERAGE.md) records cadence
+ownership, suite-growth requirements, non-model allocation, the separate Runner
+Eval audit, and remaining activation gaps.
+
+## Supported admission
+
+The initial path admits local `core-compatibility` and `feedback-persistence` cells for `runner-codex`,
+`runner-acpx-claude`, `legacy-codex`, and `legacy-claude`. Each cell must have one
+provider credential and an exact priced model. Other suites, remote environments,
+subscription profiles, hosted tools, multimodal inputs, premium service modes,
+and separate judges fail admission. This whitelist is implementation scope, not
+evidence that every admitted harness has passed live qualification.
+
+Before creating the journal or requesting credentials, manifest validation checks
+that every selected model's worst-case single-request reservation fits the initial
+campaign allowance after declared operating holds. Prices for unused models do
+not affect this selection check. Passing it does not promise the whole campaign
+will fit: later requests still use the journal's current commitments and limits.
+
+Bounded daily and weekly workflows check source-byte limits before upload: 16 MiB
+for the resolved lockfile, 1 GiB for the build bundle, and 256 MiB each for cell
+evidence and the merged report. Build artifacts expire after one day; the others
+expire after 30 days. These are source-byte and retention controls, not measured
+compressed storage charges. ZIP framing and platform billing need separate receipts.
+
+Evidence and reports are copied into fresh private staging directories. Up to
+4,096 directory entries are inspected. Discovery visits accounting and snapshot
+directories across attempts before archive and Playwright diagnostic directories.
+Accounting under `scheduled/` and JSON records have copy priority over logs,
+images and archives. Each retained file has a
+SHA-256 digest. The staging receipt reserves 8 MiB inside the total and identifies
+missing, changed, unsafe or oversized files and incomplete traversal. Originals
+are never deleted or rewritten. Hidden paths remain excluded. The expected regular
+`playwright-output/.last-run.json` bookkeeping file is outside the retained evidence
+scope and does not fail retention. Other hidden paths make the scope incomplete.
+Skipped paths and unvisited subtrees are named; overlong names use a bounded preview
+and full-path digest. Partial staging is uploaded before the workflow
+fails, so a storage limit cannot silently turn missing evidence into success.
+These upload limits do not cap runtime disk writes or Actions caches. Each service
+still needs billing evidence.
+
+### Private recurring history
+
+Daily and weekly evidence and merged reports are retained in a dedicated private
+S3 bucket. They are not uploaded as plaintext GitHub artifacts, sent to the public
+history publisher, or deployed to Pages. The report command's output also stays
+inside the private report. The public workflow retains only a fixed availability
+marker for the existing schedule monitor, with no report contents, storage address
+or object identity. Public-repository GitHub artifacts are readable by signed-in
+repository readers; authentication alone does not make those reports private.
+
+Deploy `.github/infra/runner-e2e-private-history.yml` with the existing GitHub OIDC
+provider and set `RUNNER_E2E_PRIVATE_HISTORY_S3_BUCKET`,
+`RUNNER_E2E_PRIVATE_HISTORY_AWS_ROLE_ARN` and
+`RUNNER_E2E_PRIVATE_HISTORY_AWS_REGION`. The role trusts only this repository's
+`runner-e2e-private-paid` and `runner-e2e-private-history` environments. Create both
+with a deployment branch policy allowing only the default branch. Reserve them
+for this workflow: the shared `runner-e2e-paid` and `runner-e2e-history` environments
+also serve protocol evaluations that execute target-branch code and must not
+have access to the private archive role. Set the three archive variables in both
+dedicated environments. This workflow separately requires the default branch before
+paid access and reporting. Manual target-branch tests have no OIDC permission. A separate recurring
+job receives that permission only when the target SHA equals the default-branch
+workflow SHA. Both jobs reuse the same anchored setup and test steps. The role has configuration reads
+and GetObject/PutObject under `runner-e2e-private/bounded-v1/`; no deletion,
+policy changes or access to the public history bucket. AWS credentials are step
+outputs supplied only to private transfer commands, never exported to paid agents.
+
+Admission verifies retention before paid execution. Every transfer verifies all
+four public-access blocks, disabled object ACLs, no bucket-policy reader grants,
+versioning, the 30-day/current and one-day/noncurrent expiration rules, and absence
+of Object Lock and replication. Inspection failure stops the transfer. The private
+bucket is not a CDN origin and has no external reader grant.
+
+Evidence and report archives each contain at most 256 MiB of staged files and
+4,097 entries including the staging receipt, with an additional 8 MiB ceiling for
+ZIP framing. Archives use stored entries, without decompression. Extraction checks
+paths, entry types, duplicate names, directory size, payload totals and ZIP checksums
+before committing a fresh destination. Transfers use one conditional PutObject or
+one ranged GetObject. Automatic AWS retries and workflow reruns are refused. The
+bucket also rejects PutObject requests without an If-None-Match condition. A local
+transfer receipt cannot be overwritten by rerunning the command in the same directory.
+Each command retains admission and outcome metadata. After private storage passes
+inspection, completed and failed transfers each get one independent receipt-upload
+attempt without repeating the original transfer. Successful receipts include the
+archive hash and object version; lost responses remain failed or unknown. A failed
+privacy inspection prevents all uploads, and an unsuccessful receipt upload can
+leave only the local receipt. Its own upload remains pending in the snapshot,
+so the snapshot is not proof of complete billing. Operator downloads and STS
+credential exchanges also need separate cost attribution.
+
+For authenticated retrieval, set the bucket variable and AWS profile, then run:
+
+```sh
+node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/private-history.ts download report RUN_ID /tmp/eval-RUN_ID
+```
+
+Use `evidence` instead of `report` for the original staged accounting and diagnostics.
+The requested destination must not already exist. Keep the downloaded report private.
+To publish a selected campaign, review its retained normalized report and invoke the
+existing trusted history publisher explicitly with the public destination; no agent
+rerun is needed. Recurring execution never invokes that publication path.
+
+Reporting and publication install only the committed lockfile from the trusted
+workflow revision, with lifecycle scripts disabled and a job-local package store.
+They never resolve new dependency versions while the job has OIDC access. If
+that revision has an outdated lockfile, reporting fails before accessing the
+archive. Use the normal master lockfile maintenance process; do not bypass the
+frozen install in the credential-bearing job.
+
+### Public history retention
+
+Public history remains an explicit manual operation. The optional bounded publisher
+uses `/bounded-v1/` when `PAPERCLIP_E2E_BOUNDED_HISTORY=true`; existing
+manual history otherwise retains its original prefix. Bounded publication is limited
+to the original CI workflow attempt and is not retried under the same allowance.
+
+Before any upload, the publisher validates all outgoing bundles: at most 64 MiB
+and 256 files for the immutable campaign including its manifest, 16 MiB and eight
+files for mutable history, and 64 MiB for the Pages bundle. History JSON reads
+are limited to 4 MiB. The active index keeps at most 64 campaigns from the last
+28 days and rebuilds latest/green pointers from those records. If publication
+stops, the static index can retain links that later expire.
+
+The bucket must have versioning enabled and exact prefix lifecycle rules for
+30-day current-version expiration, one-day noncurrent-version expiration without
+a retained-version floor, expired delete-marker removal and one-day incomplete
+multipart cleanup. The publisher rejects overlapping transitions, different
+expiration policies, Object Lock and replication. Access denied is a failure,
+not proof that a feature is absent. Its role needs `s3:GetBucketVersioning`,
+`s3:GetLifecycleConfiguration`, `s3:GetBucketObjectLockConfiguration` and
+`s3:GetReplicationConfiguration` on that bucket. Configuration is verified at
+each publication; it cannot guarantee that an operator will preserve it afterward.
+
+The publisher admits at most 280 S3 CLI commands over ten minutes, with a
+30-second command timeout and automatic AWS retries disabled. Uploads use
+single `PutObject` operations in STANDARD storage; immutable objects use
+`If-None-Match`. A private receipt is synced before each command and retained
+locally for the operator to retain privately. Unknown responses remain unknown. Source bytes, S3
+commands and expiration eligibility are not invoices or exact deletion times.
+Credential lookup, public downloads/CDN egress, Pages hosting and shared service
+charges require separate accounting; these limits do not cap public traffic.
+
+The launcher owns one loopback HTTP gateway per attempt. The real provider key
+stays in the parent launcher. Playwright receives an ephemeral token, and fixture
+setup saves that token through a shared AI connection with the existing custom
+Responses/Messages routing contract. The agent does not receive ambient provider
+keys. The server environment strips the gateway token and upstream credentials.
+
+Every upstream attempt reserves its maximum estimated charge in an fsynced
+journal before dispatch. Requests, retries, and cancelled calls retain distinct
+IDs. Unknown usage keeps the reservation held. One shared writer serializes
+admission across cells, so concurrent requests cannot each spend the same remaining
+allowance. Campaign deadline, request count, and in-flight count are also enforced.
+Private request receipts retain transport IDs, response IDs, usage and cache TTLs,
+route/cache/failover evidence, and final accounting status without request bodies
+or credentials. Raw provider responses remain only in the normal private evidence.
+
+## Manifest
+
+A manifest contains no credentials. This structural example uses fixture prices;
+replace them with independently verified model prices and context limits before
+running a real campaign. The launcher does not discover current prices for you.
+
+```json
+{
+ "version": 1,
+ "limits": {
+ "campaignId": "qualification-example",
+ "maxUsd": "100",
+ "deadlineAt": "2026-10-20T23:00:00Z",
+ "maxRequests": 200,
+ "maxConcurrent": 1
+ },
+ "ledgerPath": "/private/path/qualification/ledger.jsonl",
+ "executionIds": ["EXACT_SELECTED_EXECUTION_ID"],
+ "maxAttemptsPerExecution": 2,
+ "route": "direct",
+ "models": [{
+ "provider": "anthropic",
+ "maximumInputTokens": 1000000,
+ "maximumOutputTokens": 16384,
+ "timeoutMs": 180000,
+ "price": {
+ "provider": "anthropic",
+ "model": "EXACT_SELECTED_MODEL",
+ "version": "verified-pricecard-date",
+ "sourceUrl": "https://platform.claude.com/docs/en/about-claude/pricing",
+ "inputUsdPerMillion": "2",
+ "cachedReadUsdPerMillion": "0.2",
+ "cacheWrite5mUsdPerMillion": "2.5",
+ "cacheWrite1hUsdPerMillion": "4",
+ "outputUsdPerMillion": "10"
+ }
+ }]
+}
+```
+
+`executionIds` must equal the launcher's selected set, and the retry option must
+fit `maxAttemptsPerExecution`. Use `--list` to resolve the selection first.
+Operations records this attempt cap in the journal's immutable limits and fsyncs
+each attempt admission before starting its gateway. The count is per execution
+across the whole shared journal, including restarts, renamed output campaigns,
+and attempts that fail before making any provider request. Changing the cap or
+reusing an admitted attempt ID is refused.
+The maximum input value is the verified provider context ceiling, **not** a
+requested input cap. The gateway clamps output to `maximumOutputTokens`; it does
+not tokenize or truncate prompts. The conservative reservation prices the whole
+input ceiling at the highest cache-write/long-context rate, plus capped output.
+A small daily budget may therefore admit Claude but refuse Codex even when the
+expected charge is small. Do not reduce the context ceiling to bypass this guard.
+
+For modern OpenAI models, specify both `cacheWrite30mUsdPerMillion` and
+`cacheWrite12hUsdPerMillion`. Optional `longContext` contains `aboveInputTokens`
+and a complete `rates` object with the same token-rate fields. Above that total
+input threshold (including cache), the estimate uses those rates for the entire
+request. Reservations use the maximum rate across both tiers. Missing cache-write
+counters, ambiguous TTL splits for nonzero writes, and model mismatches remain
+unknown. Older OpenAI models without these usage fields need separate qualification.
+Write prices are total token prices, not surcharges. Sources:
+[OpenAI caching](https://developers.openai.com/api/docs/guides/prompt-caching),
+[OpenAI prices](https://developers.openai.com/api/docs/pricing), and
+[Claude prices](https://platform.claude.com/docs/en/about-claude/pricing).
+
+For `gpt-5.6-sol`, a response that omits the entire write-TTL breakdown can use
+the actual request's documented implicit cache policy: `prompt_cache_options.ttl`
+or its 30-minute default. This fallback requires a visible input, no explicit
+breakpoints, no stored prompt/conversation/response references, and no compaction.
+The private receipt records `cacheWriteAttribution` with the policy version and
+lifetime. These amounts remain estimates. Provider splits take precedence;
+partial or malformed splits are never repaired. `prompt_cache_retention: "24h"`
+does not imply a 12-hour write lifetime. The gateway does not change cache policy,
+and this fallback does not reprice historical requests or release old holds.
+
+For Braintrust, use `route: "braintrust"`, add the exact `projectId` UUID, and
+provide `BRAINTRUST_API_KEY` securely. Configure project providers named
+`OPENAI_API_KEY` and `ANTHROPIC_API_KEY`. Disable default provider fallback in the
+project. The request pins that endpoint and project, disables gateway response
+caching, and never automatically retries or requests fallback. A cache hit,
+failover header, or unexpected successful endpoint invalidates the receipt and
+holds its reservation. Provider prompt caching stays enabled and is metered.
+The response stream is passed through and cumulative usage is counted once.
+Gateway and native tracing are separate opt-ins; see the README tracing section.
+
+When both a bounded operations manifest and explicit Braintrust tracing are
+configured, the launcher starts a separate loopback OTLP relay for each attempt.
+The worker and its native providers receive an expiring ingest token. The
+Braintrust service key stays in the controller, even when model requests use
+Braintrust without native tracing. The relay accepts only trace/log exports to
+the configured project; neither request headers nor payload parent metadata can
+select another project or an API endpoint. JSON is canonicalized before export;
+protobuf payloads retain their original fields after bounded validation, which
+rejects ambiguous singular fields and routing values.
+
+Each relay allows at most 2,048 admitted exports, 8 MiB per request before and
+after decompression, 64 MiB of forwarded payloads, and four simultaneous exports.
+It also limits each attempt to 16,384 records and 256 MiB of expanded source
+JSON. This measurement includes the full decoded source plus each record in
+its own envelope with the shared resource and scope fields repeated. It counts
+span names, IDs, times, events, links and log bodies, not only routing attributes.
+Protobuf fields use the pinned OTLP v1.9 schema; unknown fields fail admission
+until the schema is updated. Existing valid payloads are forwarded unchanged.
+It closes at the campaign deadline or four hours after startup, whichever comes
+first. Admission consumes the allowance before upstream dispatch. Failures and
+client retries do not refund it; the relay itself never retries. A durable receipt
+must be written before dispatch and before acknowledging the receiver outcome.
+
+Version 2 private `.otlp/.jsonl` receipts record policy,
+admission time, wire/payload bytes, payload digest, dispatch/outcome, partial
+rejection counts, record counts, expanded source bytes, and closure totals.
+Version 1 receipts remain readable with unknown expanded size and record count.
+They contain no payload bodies, tokens or
+upstream error text. Receipt files are exclusive and cannot be reset on restart.
+An admission without an outcome is unresolved. Accepted ingestion does not prove
+indexing, and attempted forwarding bytes are not billed Braintrust storage.
+These receipts do not replace the observability overhead reservation or invoice
+reconciliation. After the worker relay closes, bounded controller finalization
+uses a separate one-request, 64 KiB relay with a 30-second deadline. Its exclusive
+expanded-source allowance is 1 MiB with at most 256 records. Its
+receipt is `.otlp/-finalization.jsonl`. This preserves
+the original trace and grade while accounting for the final controller export.
+This path currently supports the admitted local cells; remote transport remains
+unqualified.
+
+Scheduled experiment creation, grade insertion and summary retrieval share a
+60-second window, three-request cap, 4 MiB total payload allowance and eight-score
+cap. They run serially without automatic retry. Admission is persisted before
+dispatch in `.experiment-ingestion.jsonl`; uncertain responses retain
+their allowance and unresolved status. Reusing that file fails instead of
+resetting the budget. These limits cover the selected daily/weekly campaigns;
+the standalone experiment-import command does not use this transport budget.
+
+Private `ingestion-usage.json` binds all three receipt paths to the campaign's
+admitted attempts and ledger hash. Missing, malformed, unclosed or unresolved
+receipts leave the total unknown while preserving readable partial measurements.
+These are admitted transport payload bytes, not Braintrust processed-data units
+or an invoice. Vendor charges remain unreconciled. The current two-attempt daily
+maximum is 132.125 MiB and the four-attempt weekly maximum is 260.25 MiB,
+including finalization and the campaign's experiment payload allowance. These
+bounds do not include other projects, standalone imports or retained storage.
+The corresponding expanded-source ceilings are 514 MiB daily and 1,028 MiB
+weekly before the separate 4-MiB experiment allowance. These source measurements
+support a priced forecast; they are not Braintrust's processed-data meter or a
+vendor-enforced dollar limit. Gateway auto-logging is a separate input to that
+forecast. Retain vendor usage reports for reconciliation rather than equating
+transport or expanded source bytes with an invoice.
+
+## Recurring campaigns
+
+The existing `runner-full-stack-e2e.yml` owns both schedules: daily at 08:47 UTC
+and weekly on Sunday at 09:47 UTC. Scheduled runs require both
+`RUNNER_FULL_STACK_E2E_NIGHTLY_ENABLED=true` and
+`RUNNER_FULL_STACK_E2E_OPERATIONS_ENABLED=true`. Leave the operations gate unset
+until the workflow, billing coverage and infrastructure allowance are qualified.
+There is no separate scheduler. The protected `runner-e2e-private-paid` environment needs
+`BRAINTRUST_API_KEY`; repository variable `RUNNER_E2E_BRAINTRUST_PROJECT_ID` names
+the exact project UUID. The key is available only to the bounded execution and
+finalization step, after dependency installation and immutable builds.
+
+Native Claude gateway cases also require the pinned Claude CLI for the managed
+account's setup hello probe. The agent still runs through the pinned ACP runtime.
+The workflow installs and checks the CLI before providing credentials for both
+native and legacy Claude selections, including the aggregate scheduled selection.
+For a local campaign, verify that same CLI prerequisite before starting a paid
+attempt. Installing the ACP package alone does not satisfy this setup path.
+
+Bounded cases keep the gateway in the setup phase until the created agent's
+asynchronous lifecycle verification reaches ready. This wait uses the remaining
+attempt deadline, allows at most 45 seconds, and stops on an observed setup error
+or hold. The private `budgeted-setup.json` snapshot retains state transitions.
+Only then does the case begin agent-phase accounting and create its task.
+Older evidence that advanced after fixture creation may label hello probes as
+agent cost. Its campaign totals still include those requests; retain that phase
+limitation when comparing setup and task spending.
+
+Attempt wall time includes setup and teardown. Agent time is the sum of captured
+heartbeat-run windows, which can overlap. Missing runtime evidence, no captured
+runs or an invalid run timestamp leaves agent time unknown; the report never substitutes attempt
+wall time. A campaign total stays unknown if one of its retained attempts lacks
+that measurement. A measured zero remains zero. Missing or invalid Daytona lease
+windows likewise leave lease time and its runtime estimate unknown. Local attempts
+do not contribute Daytona lease time. Existing published numeric measurements are
+retained as historical evidence; this change does not reconstruct their provenance.
+
+Bounded daily/weekly campaigns require a public repository and use the existing
+standard `ubuntu-latest` runner for both compilation and execution. They do not
+follow the manual campaign's AWS runner switch. GitHub charges no standard
+hosted compute minutes for public repositories; retain runner/job evidence and
+recheck the [billing policy](https://docs.github.com/en/billing/concepts/product-billing/github-actions)
+when reviewing the allowance. This does not establish zero artifact, cache,
+external history, observability or subscription cost. Manual selections keep
+the existing AWS/GitHub runner choice.
+
+Daily selects native Claude's marker and feedback-persistence cases with a shared
+$20 allowance. Weekly adds native Codex's two corresponding cases with a
+shared $100 allowance. Model requests and declared operating overhead share
+this limit. Each campaign runs sequentially in one job with one
+journal, one attempt per execution, zero automatic retries, a 30-minute request
+deadline and a finite request count. Codex's worst-case request reservation exceeds
+$20, so it is absent from daily selection. Its context ceiling is unchanged.
+The $20/$100 limits cover additional costs caused by each run. Show the existing
+Braintrust Pro subscription separately at $249/month (operator policy confirmed
+October 21, 2026); do not allocate it into these run limits. New processing,
+scoring, retention or capacity charges still count toward the run allowance.
+The monthly run allowance is at most $1,120 (31 daily plus five weekly runs),
+or $1,369 including the separately reported existing subscription. These are
+maximum allowances, not forecasts of actual spending.
+This is an admission limit with retained reservations, not a reconciled invoice
+total or a provider-enforced billing limit.
+
+Before enabling recurrence, set `RUNNER_E2E_OPERATING_OVERHEAD_JSON` to a reviewed
+allowance with `version: 1`, an evidence `reference`, the actual `runner`
+(`github-hosted` for bounded schedules), and `verifiedAt`/`expiresAt` timestamps no more
+than 31 days apart. Its `components` array must contain exactly `ci_compute`,
+`artifact_storage`, `observability`, and `shared_subscription`. Each component
+needs a USD-string `maxUsd` and its own evidence `reference`. References are
+stable audit IDs or paths, not credentials or arbitrary prose. No default
+amounts are supplied. The supporting audit must include rates, resource/time
+bounds, included allowances, allocation denominators, and the controls that
+keep actual resource use within those bounds. Zero requires evidence of an
+included or zero incremental charge; setting a zero allowance is not a receipt.
+Under the current policy, `shared_subscription` may reserve zero for the
+existing Braintrust subscription with a reference to that policy and current
+plan evidence. Any newly incurred subscription or capacity charge needs its own
+incremental allocation; the existing fee must remain visible separately.
+
+The catalog checks that this allowance matches the selected runner, covers at
+least the next four hours, and leaves room for a full conservative model-request
+reservation. This happens before the build and test runners start. The workflow passes
+the exact validated allowance to preparation; it does not read a potentially
+changed repository variable again. Preparation rechecks it and retains it in the
+manifest and forecast. Setup, builds, failed jobs, publication and retained
+artifacts must all fit the declared components, not just the agent's runtime.
+
+The journal initializes `overhead:` reservations under
+`campaign-overhead` from its immutable limits before admitting model requests.
+These are allocation holds, not dispatched requests or completed resource
+lifecycles. They do not occupy model concurrency, but they count toward the
+finite charge-record limit. Resuming cannot change or drop them. Until matched
+cost observations arrive, the report shows them as unknown and keeps their
+allowance held; even a zero allowance does not claim a known zero charge.
+The existing finalizer reports incomplete accounting while these receipts are
+missing. Do not dismiss that finding to enable recurrence.
+
+Reconcile a component through its existing request ID with provenance-bound
+`infrastructure` or `invoice` observations. Partial observations keep the
+remaining reservation held; a final nonconflicting observation can replace it.
+Never mark a receipt final before its resource and billing window are complete.
+An external bill above the allowance remains visible and stops further model
+admission. The journal cannot stop AWS, storage or subscription billing itself;
+provider resource limits and a verified receipt collector remain activation
+requirements. No operating-cost allowance has been configured or qualified by
+the presence of this code.
+
+Collect retained GitHub usage after the workflow completes, including failed or
+cancelled workflows, without dispatching or retrying anything:
+
+```sh
+node --import tsx tests/runner-e2e/operating-receipts.ts github RUN_ID /private/path/github-usage.json
+```
+
+The collector uses the existing `gh` authentication and only reads
+`paperclipai/paperclip`. It verifies the Product E2E workflow, first attempt,
+revision and repository, reads every returned job/artifact page up to 1,000
+records per inventory, and rejects changed or incomplete inventories. It retains
+job identities, summed job seconds, run-start delay, artifact sizes and expiration
+dates. It omits arbitrary job names, artifact names and upstream responses.
+Existing output files cannot be replaced. This is a current inventory, not an
+invoice: deleted artifacts are absent, compressed bytes are not billed
+byte-hours, job duration is not VM lifetime, and present repository visibility
+does not establish visibility throughout an earlier run. Caches and external
+S3 history remain separate. A visible public-hosted zero compute rate never
+automatically releases an overhead reservation.
+
+Import a reviewed cost receipt and its exact source evidence into an existing
+campaign's declared overhead reservation:
+
+```sh
+node --import tsx tests/runner-e2e/operating-receipts.ts import /private/path/ledger.jsonl /private/path/receipt.json /private/path/source-evidence.json
+```
+
+The receipt contains `version: 2`, `costBasis: "component_cumulative"`,
+`campaignId`, `allowanceReference`, `component`,
+the source file's `evidenceSha256`, `windowStartedAt`, `windowEndedAt`, `coverage`
+(`partial` or `complete`), `source` (`infrastructure` or `invoice`), `provider`,
+`classification`, and an explicit USD-string `costUsd` when known. Estimated and
+allocated charges also require a stable `pricingVersion`. `reconciled` requires
+an invoice source. An unknown amount has no `costUsd` and always remains partial.
+The observation window must have ended; completeness is the reviewer's assertion
+that this component's entire resource and billing window is represented. A
+checksum proves which file was used, not that the invoice or allocation is true.
+Shared allocations must record their numerator, denominator, window and excluded
+costs in that source file. Do not import a whole shared invoice as one campaign's
+charge without an explicit allocation method.
+
+Each amount is the cumulative total for the entire component, including all its
+providers and separate charges through `windowEndedAt`. Later receipts must keep
+the same `windowStartedAt` and cannot shorten the window or make completed
+coverage partial again. Combine separate billing windows in the retained source
+evidence before importing their total; two separate $0.80 charges require a
+cumulative $1.60 receipt. Version 1 receipts lacked this declaration and are
+rejected on import. Existing journal records are preserved; the importer also
+refuses to extend a component whose prior receipts lack cumulative evidence.
+It also checks the ledger's precedence rules before import: a larger cumulative
+estimate cannot be hidden by an earlier, smaller invoice. Such imports fail
+without changing retained evidence or the journal; reconcile the entire expanded
+window with evidence the ledger can account for before retrying.
+
+The importer validates campaign/allowance identity and source hash, retains both
+files privately under `ledger.jsonl.overhead/`, and syncs them before appending
+the observation. Files are written and synced under unique temporary names, then
+published atomically without replacing existing evidence. A partial write is
+removed without occupying the final path, so retry needs no manual repair.
+Retries sync matching existing files too. Reimporting an identical
+receipt adds no journal entry. A partial amount
+keeps the rest of the allowance held. Complete invoices can supersede estimates;
+conflicting complete invoices remain visible and block admission. This command
+does not open a new spending window, change limits, dispatch tests or rewrite
+previous records. Regenerate the campaign report from the same ledger afterward;
+an old report snapshot is not rewritten by an offline receipt import.
+
+The checked pricecard expires on November 19, 2026. Admission stops when the
+pricecard expires, the catalog model changes, the project is missing, or the
+preflight already exists. Review current official prices and context limits
+before updating `scheduled-prices.json`. `forecast.json` records the selected
+IDs, provenance-linked pilot measurements, compatibility gaps, uncertainty,
+reservations and price expiry. The historical daily/weekly successful model
+subtotals are $0.427624800/$0.791539400; they are not campaign bills. Setup model
+calls and attempt setup/teardown are included, while CI setup/build/queue and
+non-model charges are excluded. Current compatible totals stay unknown if any
+selected cell lacks a matching measurement. Hosted timing is still unmeasured.
+Per-cell medians describe only the retained samples; P95 requires at least 20
+matching samples and does not establish reliability. See
+[measurement provenance and limitations](OPERATIONS-COVERAGE.md#initial-cadence).
+Daily coverage is limited to the two Claude journeys. The feedback definition
+has changed since its passing pilot. Broader rotation and Daytona remain
+unqualified and are not silently included by `--all`.
+
+Dispatch `cadence: daily` or `weekly` from the default branch to qualify the same
+path manually. Bounded dispatch rejects custom target revisions and selectors.
+Bounded authorization pins the commit from the workflow event; a later merge
+does not replace that revision or invalidate a queued campaign.
+Workflow attempts after the first cannot start paid work with a fresh journal.
+Rerunning reporting can reuse the original attempt-one evidence. Recover the
+retained ledger for a deliberately resumed campaign; do not reset it or launch a
+new campaign merely to hide a failure. The existing `manual` mode remains an
+explicit, unbounded qualification path and does not inherit these allowances.
+
+After execution, deterministic review dismisses only exact synthetic feedback
+bodies corroborated by all feedback-persistence graders. Novel feedback remains
+`needs_investigation`; no extra model call is made. Missing accounting, unresolved
+receipts, feedback collection/submission failures, novel feedback, export failure
+and incomplete experiment import or indexing fail the job with bounded diagnostic codes.
+Successful unchanged runs produce artifacts without posting messages. GitHub's
+workflow failure notifications remain the operator's notification channel.
+Missing/non-starting schedules require an external workflow monitor; this change
+does not claim that a workflow can detect its own failure to start.
+
+The launcher starts the public Playwright JavaScript entry point directly with
+Node and sends SIGINT for cancellation and watchdog cleanup. The package-manager
+wrapper can exit before Playwright completes its shutdown, so it is excluded.
+Playwright handles that signal by stopping workers and closing its web server;
+SIGTERM can exit before web-server cleanup. The web-server supervisor still
+receives SIGTERM and owns its server's shutdown. Both paths retain the existing
+bounded grace period and forced cleanup for surviving owned processes.
+When a worker exits without a result, the launcher also recovers its original
+trace identity from the private fixture snapshots. The supervisor can finalize
+that trace with the failed outcome. A mismatched project, execution or company
+is rejected. Recovered identity does not prove native export or complete phases.
+
+If a worker exits before writing its result, the server supervisor spends at
+most five seconds collecting feedback from that attempt's saved company before
+stopping the isolated server. Recovery snapshots remain private and are kept
+separate from the worker's original evidence. The failed result includes any
+recovered counts and review fingerprints; active runs or missing reads remain
+partial or unavailable. Each atomic recovery checkpoint carries its attempt
+identity, so a later write timeout cannot hide already saved feedback when the
+final receipt is absent. Completed results are never replaced. The launcher
+retains the private database whenever a worker result is missing, including when
+the supervisor was killed before it could capture feedback. This retention is
+reported as unconfirmed cleanup and requires inspecting the retained state.
+Authenticated public-MCP fixtures do not use the local operator recovery path.
+
+Original grades are imported into Braintrust without rerunning agents. The
+access-controlled artifact includes the journal, forecast, feedback dispositions,
+experiment receipt, indexing receipt and findings. The trusted report job rebuilds the dashboard
+and retains its evidence-bound experiment link. Export acceptance and native
+indexing are separate. A bounded, read-only Braintrust query verifies the attempt
+root, finalization, native API activity and any expected feedback tools. It also
+compares indexed experiment rows with original grades, revisions and attempt IDs.
+The verifier tries at most three rounds within 45 seconds, retains missing/query
+failures, and never reruns an agent. Native record counts measure export volume,
+not distinct provider requests or charges. Codex exports can be high volume and
+Braintrust can double-count native/log observations; use the deduplicated gateway
+ledger as the estimate authority. Private indexing receipts contain only counts,
+identities and verification status, without trace contents or credentials.
+
+## Inspection and limits
+
+Retained deterministic grades can be imported into a private Braintrust experiment
+without invoking agents or judges. Prepare a JSON array of absolute paths to the
+original `result.json` for **every** retained attempt in `normalized-results.json`.
+The original files supply each attempt's actual source and definition hash;
+the selected result's revision must not be assigned to older attempts.
+
+```sh
+PAPERCLIP_E2E_BRAINTRUST_PROJECT_ID=PROJECT_UUID \
+node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/braintrust-experiment-import.ts \
+ /private/campaign/normalized-results.json /private/campaign/original-result-paths.json \
+ /private/campaign/experiment-plan.json --dry-run
+```
+
+Inspect the plan. To import it, provide `BRAINTRUST_API_KEY` securely, remove
+`--dry-run`, and use a separate `experiment-receipt.json` output path. The importer
+requires explicit accounting identities and source/definition hashes; older
+incomplete provenance fails closed. It retains original failed and successful
+grades, cleanup outcomes, feedback counts, source revisions and trace identities.
+Carried-forward CI attempts retain their original campaign IDs and must match
+the report's retained receipts. Request costs attach only when the attempt's
+campaign also matches the journal's campaign, even if attempt IDs coincide.
+It does not upload prompts, feedback bodies, matcher details or raw errors.
+
+The evidence digest determines the private experiment name. Identical evidence
+reuses that experiment and stable row IDs after an interrupted insert. Changed
+evidence creates a different experiment and requires a separate receipt path.
+The importer validates an existing receipt before retrying and never lowers its
+confirmed accepted-row count. The receipt checkpoints accepted batches;
+HTTP acceptance still requires independent indexing/UI verification. Retain failed
+receipts. A retry must use the same original evidence, not newly run agents.
+Set `PAPERCLIP_E2E_BRAINTRUST_EXPERIMENT` to the receipt path when rebuilding the
+campaign report to retain the experiment link. Its evidence hash must match the
+report's selected source/definition identities, attempts, request-ledger checkpoint
+and feedback review. Refreshing only
+the report rendering preserves the link; changed evidence requires a new import.
+
+Request-ledger amounts remain metadata with their journal hash and classifications.
+They are not additive cost metrics, and missing rows remain unknown rather than
+zero. The campaign ledger may include setup probes outside the imported tests.
+Do not add experiment metadata amounts to native or gateway trace costs. Native
+trace rollups are not invoice reconciliation and can omit setup or duplicate logs.
+API contracts: [create experiment](https://www.braintrust.dev/docs/api-reference/experiments/create-experiment)
+and [insert retained events](https://www.braintrust.dev/docs/api-reference/experiments/insert-experiment-events).
+
+- `operations-preflight.json` preserves the exact manifest in the campaign folder.
+- `ledger.jsonl` is the append-only authority for request admission and observations.
+- `ledger.jsonl.requests/` contains private per-request checkpoints.
+- `ledger.jsonl.summary.json` preserves known, reserved/held, conflicting, and
+ unknown amounts on normal finalization, including failed campaign workers.
+- `campaign.json` and the dashboard preserve retained attempts and show missing
+ selected results after worker failure. Their request-ledger section shows the
+ shared ledger's known subtotal, classifications, additional held reservations,
+ unresolved requests and per-attempt totals. It can include earlier campaigns
+ sharing that budget. It is a separate view of model spending: do not add it to
+ heartbeat totals. The journal hash and record count identify the checkpoint.
+- If the journal cannot be read or validated, the report records accounting as
+ unavailable with no numeric totals. Retained results and reports are still
+ written, and the launcher/report command exits unsuccessfully. Recover the
+ original journal before resuming paid work; reporting does not repair or reset it.
+- To regenerate a merged report with the same ledger, pass
+ `PAPERCLIP_E2E_OPERATIONS_LEDGER=/absolute/ledger.jsonl` to `report.ts`.
+ Dashboard refresh and public publication preserve and validate its closed
+ numeric projection. Raw request receipts and the journal remain private.
+
+All model amounts from usage times a pricecard are **estimated**, not billed or
+reconciled. This path does not yet price CI compute, storage, subscriptions,
+Daytona, separate judges, or gateway overhead. Unknown observations and these
+coverage gaps must be included in operational reporting. Provider billing can
+lag and discounts can differ; a local list-price cap is not a provider-side cap.
+
+Resuming the same journal requires exactly the original limits. A dead process
+can leave a lock and held reservations; inspect the owner and upstream receipts
+before recovering it. Never delete a live writer's lock or reset a ledger merely
+to regain budget. Older journals without an attempt cap remain readable for
+reporting and reconciliation. Operations refuses an implicit upgrade because request receipts cannot reconstruct
+zero-request attempts. Use the explicit audited migration below only after
+reviewing all retained campaign attempts. Do not replace or reset an active pilot
+ledger to evade this check.
+
+No recurring schedule is enabled by this integration. Qualification of actual
+agent streams, tools, continuation, cancellation, native traces, and remote routes
+is required before treating those routes as operationally proven. Forecasts,
+provider reconciliation, and the campaign feedback disposition workflow are
+separate remaining integration work.
+
+### Setup and agent request attribution
+
+Each admitted attempt creates a private phase checkpoint under
+`.attempts/`. The launcher passes its path only to the test worker;
+the Paperclip server does not inherit it. Before fixture setup finishes,
+requests are recorded as `setup`, including connection checks and hello probes.
+The worker atomically advances the checkpoint to `agent` immediately after
+fixture creation and before task execution. Cleanup does not reset this phase,
+so late agent requests retain their task attribution.
+
+The gateway reads and validates the checkpoint for each request before reserving
+spend. That admission fixes the request's phase even if the phase changes while
+its response streams. Missing, malformed, or foreign-attempt checkpoints stop
+new dispatches rather than guessing a phase. Request records remain in the
+fsynced journal after the worker exits. This boundary applies to the admitted
+local core cells; it does not establish coverage for separate judges, paid tools,
+child-agent attribution, or remote routes. Historical journals retain their
+original attribution and are not retroactively rewritten.
+
+### Auditing a legacy journal before adding an attempt cap
+
+A journal created before durable attempt admission can acquire one attempt policy
+through `ledger-attempt-policy.ts`. Stop its launcher and gateway first. Review
+all retained campaigns, including startup failures with no model request. Save
+the review and its source references as a private evidence file. Each prior
+attempt needs its original execution and accounting attempt ID; request receipts
+alone are insufficient. If that history is incomplete, do not resume paid work.
+
+Create a private JSON audit with these fields:
+
+- `version`: `1`.
+- `ledgerPath` and `evidencePath`: absolute private file paths.
+- `previousJournalHash`: the current journal's final record hash.
+- `evidenceSha256`: SHA-256 of the reviewed evidence file.
+- `maxAttemptsPerExecution`: the finite cap to adopt, including past attempts.
+- `priorAttempts`: an array of `{ "executionId": "...", "attemptId": "..." }`.
+
+Run `node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/ledger-attempt-policy.ts /absolute/private-audit.json`.
+The command checks evidence and journal hashes, takes the normal exclusive writer
+lock, and appends one fsynced policy record. It refuses unfinished requests,
+omitted recorded request identities, duplicate attempts, or an existing attempt
+policy. The imported count can already exceed the cap; then no new attempt is
+admitted for that execution. The command cannot prove the completeness of the
+operator's review of zero-request attempts. Retain the evidence for inspection.
+
+All original journal bytes, request counts, charges, held reservations, campaign
+identity, dollar limit, concurrency limit and deadline remain intact. Subsequent
+launch manifests must use the adopted cap. The migration cannot extend an expired
+deadline, change an existing cap, or replenish an allowance. An interrupted append
+remains an accounting failure and must be recovered before more paid work.
+
+The explicit local `feedback-persistence` native Claude/Codex cells also accept
+an operations manifest. They use the same metered connection, prices, campaign
+journal, setup boundary, and durable attempt allowance as core compatibility.
+Collection and the independent saved-evidence oracle require no additional
+model call. The synthetic items need review as test data, not product incidents.
diff --git a/tests/runner-e2e/OTLP-LICENSE.txt b/tests/runner-e2e/OTLP-LICENSE.txt
new file mode 100644
index 00000000000..261eeb9e9f8
--- /dev/null
+++ b/tests/runner-e2e/OTLP-LICENSE.txt
@@ -0,0 +1,201 @@
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright [yyyy] [name of copyright owner]
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
diff --git a/tests/runner-e2e/OTLP-NOTICE.md b/tests/runner-e2e/OTLP-NOTICE.md
new file mode 100644
index 00000000000..6ae059f624c
--- /dev/null
+++ b/tests/runner-e2e/OTLP-NOTICE.md
@@ -0,0 +1,11 @@
+# OTLP field definitions
+
+The field definitions in `eval-otlp-schema.ts` are derived from
+[OpenTelemetry Protocol v1.9.0](https://github.com/open-telemetry/opentelemetry-proto/tree/v1.9.0).
+
+Copyright The OpenTelemetry Authors.
+Licensed under the [Apache License, Version 2.0](./OTLP-LICENSE.txt).
+
+The source comments and package declarations were removed to embed these field
+definitions in the eval harness. The harness adds a combined response message.
+Accounting behavior and limits are documented in [OPERATIONS.md](./OPERATIONS.md).
diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md
index df516223c84..5f199a2fad4 100644
--- a/tests/runner-e2e/README.md
+++ b/tests/runner-e2e/README.md
@@ -1054,9 +1054,32 @@ timings, token and cost accounting, and evidence links. The campaign header
aggregates input, output, and cached tokens, provider-reported LLM spend,
Daytona list-price runtime estimates, and pricing coverage. Missing provider
usage is labeled `unavailable` or `unpriced`; it is never presented as zero
-cost. The CI report job stages the same portable site at
+cost. A known subtotal from a partially priced run stays in campaign spending,
+with a partial-amount label and incomplete billing coverage. The CI report job stages the same portable site at
`normalized/index.html` inside the access-controlled merged report artifact.
+Campaign billing retains a separate numeric receipt for each attempt, including
+failed retries. The selected result still controls the grade and result card.
+Copies of the same attempt do not add another charge. The supervisor stamps a
+campaign/attempt identity; CI cells share `PAPERCLIP_E2E_ACCOUNTING_CAMPAIGN_ID`
+while retaining their separate artifact directories. Report regeneration keeps
+these receipts and their totals. A later cleanup update replaces an earlier
+observation of the same attempt; contradictory costs at the same timestamp are
+rejected instead of silently choosing one.
+
+The report shows missing attempt receipts and artifacts excluded from another
+campaign. Legacy CI evidence can be grouped by workflow run URL. Unscoped local
+artifacts cannot establish campaign membership, so only the selected result is
+used. Older normalized bundles without receipts are labeled `selected_results`;
+rendering them again cannot recover lost retry costs.
+
+Billing reads decimal `costUsdExact` receipts and separates model estimates
+from provider-reported amounts. Explicit zero usage differs from a missing or
+unfinished receipt. Coverage is limited to captured usage. It does not certify
+setup probes, subscription allocations, external tools, CI, storage, or provider
+invoice reconciliation. These require separate receipts. Do not use the current
+numeric report subtotals as an enforced spending limit.
+
The trusted publisher discovers display-only entries for selected execution IDs
absent from its local catalog, so branch-only suites remain visible in the
dashboard, filters, gallery, and summary image. It validates execution identity
@@ -2748,3 +2771,173 @@ The arithmetic journey records four delivered agent handoffs and one completed
terminal intent. The final case is `done` at handoff version 5, while the same
four provider work runs and unchanged workflow document revision finish the task.
The oracle rejects a missing or pending terminal intent.
+
+### Complaints and suggestions
+
+Each ordinary live attempt collects company-wide saved feedback after retiring
+runs and before fixture teardown, even when cancellation or grading fails. This
+includes feedback from child and background runs. Full paginated bodies and
+native tool invocation outcomes are checkpointed in the access-controlled
+`snapshots/agent-feedback.json` artifact. Collection has a 30-second deadline;
+missing detail pages, truncated run lists, active work, and unavailable APIs
+remain explicit partial/unavailable observations. Legacy shell helpers do not
+have complete native invocation coverage. An empty saved-feedback list does not
+prove that a tool was available, and a completed tool receipt does not by itself
+prove a saved submission.
+
+Results and retained-attempt receipts contain only counts, closed statuses, and
+body fingerprints. Campaign reports group identical fingerprints across attempts
+into distinct items awaiting review. Feedback presence does not change product
+grades. Inspect full bodies in private evidence and record a disposition before
+considering the review complete; collection itself performs no model call or
+semantic triage. Historical artifacts without feedback fields are labeled as
+uncaptured. A killed worker that never reaches finalization is also uncaptured,
+not evidence of zero feedback.
+
+Create a private review template from the merged `normalized-results.json`:
+
+```sh
+pnpm exec tsx tests/runner-e2e/feedback-dispositions.ts /private/report/normalized-results.json /private/feedback-review.json
+```
+
+Inspect the full bodies in the retained attempt snapshots. Set `reviewedAt` to
+an ISO timestamp with an explicit timezone, such as `2026-10-20T01:00:00Z`,
+and `reviewedBy` to `human` or `agent`. Timestamp comments and free-form dates
+are rejected so private notes cannot enter public report metadata. Assign each fingerprint
+one disposition: `actionable`, `duplicate`, `already_fixed`, `unsupported`, or
+`needs_investigation`. A duplicate must name a non-duplicate fingerprint in the
+same campaign via `duplicateOf`. Optional `issueNumber` links a public
+`paperclipai/paperclip` issue; keep private rationale outside this file.
+
+Import a completed review into an existing bundle without rerunning agents:
+
+```sh
+pnpm test:e2e:runner:dashboard -- /private/report --feedback-review /private/feedback-review.json
+```
+
+Alternatively, rebuild the same campaign and selection with
+`PAPERCLIP_E2E_FEEDBACK_REVIEW=/private/feedback-review.json`. Keep its original
+campaign ID. The importer requires exactly one decision per collected group
+and rejects a review if retained attempts or feedback coverage changed. Blank
+templates cannot count as reviewed. Dashboard regeneration and publication
+preserve the review. Regeneration refreshes the review status in an existing
+`summary.md` while preserving its other measurements and links. An empty review
+can record that no saved items were found; it never closes collection gaps,
+failed submissions, or changes product grades.
+This workflow makes no additional model call. Review performed by a separate
+paid model must be budgeted and recorded as `feedback_review` work separately.
+
+### Cost-ledger integration contract
+
+`cost-ledger.ts` provides the request-level journal used by the eval operations
+integration. It is a library; the existing launcher is not yet universally routed
+through it, and merely setting a campaign ID does not enforce a spending cap.
+Every paid dispatch must await a durable reservation before contacting a provider.
+The caller supplies an explicit campaign ceiling, deadline, request count, and
+concurrency. Monetary decisions use integer nanodollars. Model list prices remain
+estimates with a pinned version; they do not become provider invoices.
+
+The journal retains setup/agent/judge/tool/infrastructure attribution separately
+from heartbeat IDs. Multiple sources for the same request are retained but only
+one cost contributes to the total: reconciled, reported, estimated, then allocated.
+Missing receipts and cancellations retain their reservation. Contradictory final
+receipts stop new admissions. An observed overshoot is retained in full and stops
+further work; local estimated limits are not provider-enforced dollar limits.
+A request bound must account for input/context, output limits, cache-write pricing,
+and any paid tools before claiming a bounded overshoot.
+
+A single writer serializes admissions and fsyncs each journal record. Read-only
+inspection survives process loss; truncated/tampered records fail closed. A stale
+`.lock` requires inspection of the recorded owner PID before operator recovery;
+never remove a live writer's lock. Resuming a campaign cannot silently change its
+limits. The ledger rejects unreviewed fields and contains no prompts, responses,
+headers, or credentials. Cross-source request joins must use a verified request
+identity; ambiguous provider-request joins are rejected rather than double counted.
+
+### Opt-in Braintrust traces
+
+Set `PAPERCLIP_E2E_BRAINTRUST_PROJECT_ID` to the destination project UUID and
+provide `BRAINTRUST_API_KEY` securely to the launcher. An ambient key alone does
+not enable export. The launcher includes this key in secret scans and removes
+Braintrust variables from the server's inherited environment. Fixture setup
+saves the company's native observability configuration through the public,
+revision-fenced API using a non-Playwright sensitive request, before credential,
+agent, and setup-probe creation. The onboarding flow installs it immediately
+after company creation. The existing company setting controls native capture.
+
+Each ordinary `runner.spec.ts` attempt records orchestration spans for setup,
+execution, feedback, and cleanup, and exports protobuf over HTTPS with a ten
+second deadline and no redirect/retry. `snapshots/braintrust-trace.json` checkpoints
+the identifiers/configuration and final export receipt. Normalized results retain
+the receipt, including failed exports. Export failure does not rewrite the product
+verdict; operational qualification must separately require accepted and verified
+traces. A killed worker can leave a pending receipt, not a claimed successful export.
+
+`accepted` means the OTLP receiver accepted the batch; it does not establish that
+Braintrust indexed it or that native agent spans arrived. Search the project for
+`paperclip.eval.attempt_id` and its trace ID, then verify native records using the
+company ID and `paperclip.eval.run_ids`. The disposable company uses the custom
+OTLP destination at `https://api.braintrust.dev/otel` with a private `x-bt-parent`
+header containing the exported attempt parent. Braintrust attaches independent
+native trace roots beneath that attempt; native children keep their own hierarchy.
+This uses the existing company observability API without changing production
+trace parentage or enabling export on other companies.
+
+The parent encoder uses Braintrust's V4 project-log wire format with JSON identity
+fields (version 4, project-log type 2, zero compressed fields). Its fixture was
+round-tripped through the public `SpanComponentsV4.fromStr` export in
+`braintrust/util` 3.38.0. The implementation is specified in the
+[official SDK source](https://github.com/braintrustdata/braintrust-sdk-javascript/blob/403ae94fb3f1916a101d8d84de4c4871e4ae3bec/js/util/span_identifier_v4.ts).
+A model-free live probe verified the indexed child's `root_span_id` and
+`span_parents` in Braintrust; a native-agent run must separately qualify the path.
+The parent contains identifiers only and is installed through the sensitive
+credential-header path, outside browser evidence.
+
+Attempt spans include suite ID and definition hash, profile, case, configured
+provider/model, source revision (or explicit `unknown`), and the runner-spec grader
+path. The source revision identifies that grader's implementation. Worker exports
+are marked `paperclip.eval.outcome_stage=worker`. After evidence packaging,
+gateway shutdown and outer cleanup, the supervisor updates the same attempt root
+with its final outcome and adds a finalization span. This preserves failures first
+discovered after the worker returned. The result retains a separate finalization
+export receipt; old or killed attempts can explicitly lack one. Export failure
+remains separate from the product grade. Orchestration spans omit model usage and
+cost metrics. Gateway model logging remains a separate observation lane;
+its model costs must not be added to native costs without a verified request join.
+The separate provider connection suite and standalone judges do not yet use this
+attempt exporter.
+
+### Bounded local gateway pilot
+
+[OPERATIONS.md](./OPERATIONS.md) documents the opt-in operations manifest, exact
+selection admission, durable request reservations, local Responses/Messages proxy,
+and Braintrust routing. Current support is deliberately limited to local core
+Codex/Claude cells. The document distinguishes implemented accounting controls
+from live qualification and lists remaining billing/reporting coverage gaps.
+
+## Feedback persistence (explicit only)
+
+`--suite feedback-persistence` selects two local native Claude/Codex cells.
+Each creates one standard task through the browser, asks for two distinct
+synthetic feedback items, and requires the real `submit_complaint` and
+`submit_suggestion` tools. The five-minute case expects one task work run and
+disables automatic retries. Company and agent accounted-work budgets are $10.
+Setup probes and any background work must also pass through the separately
+budgeted gateway when using an operations manifest. Local compute is unmetered.
+Cost and duration forecasts remain unmeasured until a retained live attempt.
+The runner E2E maintainers own this manual case; `--all` excludes it.
+
+After collection and fixture teardown, the oracle reads the retained evidence
+file and requires exact nonce-bearing bodies, the original company/agent/task
+and observed run, plus one completed native receipt for each tool. Missing or
+partial collection, prose-only claims, foreign attribution, duplicate records,
+and failed submissions cannot pass. Full bodies stay in private evidence;
+public reports retain counts, fingerprints, and closed oracle checks. The usual
+result screenshot, trace, secret scanning and supervisor cleanup still apply.
+This case proves explicit submission and persistence, not spontaneous feedback
+quality or survival after an abrupt worker kill.
+
+```sh
+pnpm test:e2e:runner -- --list --suite feedback-persistence
+pnpm test:e2e:runner -- --id feedback-persistence.runner-acpx-claude.local.complaint-and-suggestion --max-automatic-retries 0 --max-parallel 1
+```
diff --git a/tests/runner-e2e/SECURITY.md b/tests/runner-e2e/SECURITY.md
index 093f2066130..2a84a315e51 100644
--- a/tests/runner-e2e/SECURITY.md
+++ b/tests/runner-e2e/SECURITY.md
@@ -423,6 +423,66 @@ Two passive seed cases support browser list assertions; screenshots remain on
the exact primary case/task routes and document dialogs preserve that case URL.
The public screenshot allowlist is unchanged.
+## Eval feedback capture
+
+`snapshots/agent-feedback.json` retains company-wide complaint/suggestion bodies
+and attributed invocation outcomes in the access-controlled Actions artifact.
+It passes the existing credential scanner and is excluded from public history
+like other snapshots. Normalized results and campaign attempt receipts contain
+only numeric counts, closed collection statuses, and SHA-256 fingerprints for
+review grouping. Public reports never copy feedback bodies, previews, tool
+arguments, or free-form collection errors. Collecting feedback performs no new
+model call and does not forward it to an external service.
+
+Feedback capture retains at most 1,000 full entries and 8 MiB of serialized
+entries/invocations per attempt. Event inspection stops at 10,000 events or
+16 MiB of serialized event data. Limits produce partial collection and a closed
+`evidence_limit_reached` diagnostic; omitted evidence is never reported as absent.
+Checkpoints are considered every ten details, at page boundaries, and at stage
+completion; geometric data growth bounds repeated writes. Reads stop before the
+overall deadline, reserving up to five seconds (20% of shorter windows) to persist
+the collected prefix. Persistence shares the overall collection deadline and
+receives an abort signal. The fixture writes an abortable temporary
+snapshot before replacing the last complete checkpoint, so a stalled write does
+not overwrite earlier evidence with a partially written JSON file.
+Review counts and fingerprints advance only after a checkpoint succeeds. A later
+write failure or expired deadline retains the last saved bodies and marks coverage
+incomplete.
+
+Braintrust export requires an explicit eval project UUID. Native capture includes
+prompts, responses, and tool content only after that opt-in configures the isolated
+company's observability destination. The ingestion credential crosses the public API via
+Node fetch, outside Playwright tracing, and is included in launcher evidence
+redaction. Orchestration spans contain identifiers, phase timing, outcome, and
+feedback counts; they never contain provider credentials or feedback bodies.
+
+With a bounded operations manifest, the ingestion credential is an expiring
+per-attempt relay token. The organization-wide Braintrust key stays in the
+controller. Only authenticated POSTs to the relay's trace/log ingest paths are
+allowed, with a fixed HTTPS upstream and redirects disabled. Header and payload
+`braintrust.parent` values must reference the selected project. Duplicate protobuf
+singular/oneof fields are refused, and JSON is canonicalized to remove duplicate
+key ambiguity. The token cannot call model, read, delete, or other API endpoints;
+it can still write records within the selected project until its finite request,
+byte or time allowance is exhausted. It is not an attempt-level row permission.
+
+The worker environment removes upstream provider/Braintrust credentials before
+inserting the model and OTLP relay tokens. Ambient relay URLs are discarded, and
+the fixture server receives the selected endpoint through company settings.
+Both tokens participate in evidence redaction. Private OTLP receipts retain
+metadata only, and persistence errors stop further admission. The relay closes
+with the model gateway before controller finalization. The controller's direct
+finalization export and tracing without an operations manifest retain the
+existing direct credential path. Remote relay publication is not enabled here.
+
+The bounded operations manifest is credential-free and private. Its parent-owned
+loopback gateway uses an ephemeral per-attempt token. Upstream credentials stay
+in the launcher, are included in secret scans, and are removed from the child
+provider environment. Only the selected Responses or Messages endpoint is
+forwarded to a fixed provider or Braintrust HTTPS origin; request headers do not
+select arbitrary destinations. Gateway response caching and fallback are refused.
+Request journal and usage receipts contain no prompt/response bodies or keys.
+The manifest, journal, and receipts are not public report projections.
## Configured repository and conversion proof
diff --git a/tests/runner-e2e/api.ts b/tests/runner-e2e/api.ts
index 398691d1e27..5032c3e56f9 100644
--- a/tests/runner-e2e/api.ts
+++ b/tests/runner-e2e/api.ts
@@ -58,15 +58,27 @@ export class RunnerApi {
* never serialized into trace/blob evidence before Paperclip encrypts it.
*/
async postSensitive(path: string, data: unknown): Promise {
- const response = await fetch(new URL(path, this.baseURL), {
- method: "POST",
- headers: { "content-type": "application/json", ...this.sessionOptions().headers },
- body: JSON.stringify(data),
- });
+ return this.sensitive("POST", path, data);
+ }
+
+ async putSensitive(path: string, data: unknown): Promise {
+ return this.sensitive("PUT", path, data);
+ }
+
+ private async sensitive(method: "POST" | "PUT", path: string, data: unknown): Promise {
+ const url = new URL(path, this.baseURL);
+ if (url.origin !== this.baseURL || !url.pathname.startsWith("/api/")) throw new Error("Sensitive request must use the fixture API");
+ let response: Response;
+ try {
+ response = await fetch(url, {
+ method, redirect: "error", signal: AbortSignal.timeout(15_000),
+ headers: { "content-type": "application/json", ...this.sessionOptions().headers },
+ body: JSON.stringify(data),
+ });
+ } catch { throw new Error(`Sensitive ${method} request failed; details withheld`); }
if (!response.ok) {
- throw new Error(
- `Sensitive POST ${path} returned ${response.status}; response body withheld`,
- );
+ await response.body?.cancel();
+ throw new Error(`Sensitive ${method} ${url.pathname} returned ${response.status}; response body withheld`);
}
return response.json() as Promise;
}
diff --git a/tests/runner-e2e/artifact-budget.mjs b/tests/runner-e2e/artifact-budget.mjs
new file mode 100644
index 00000000000..cf84bbcb8ad
--- /dev/null
+++ b/tests/runner-e2e/artifact-budget.mjs
@@ -0,0 +1,163 @@
+import { constants } from "node:fs";
+import { createHash } from "node:crypto";
+import { lstat, mkdir, open, opendir, realpath, rm, writeFile } from "node:fs/promises";
+import path from "node:path";
+import { pathToFileURL } from "node:url";
+
+export const ARTIFACT_BOUNDS = Object.freeze({
+ lock: 16 * 1024 * 1024,
+ build: 1024 * 1024 * 1024,
+ evidence: 256 * 1024 * 1024,
+ report: 256 * 1024 * 1024,
+ receipt: 8 * 1024 * 1024,
+ entries: 4096,
+});
+
+export async function checkArtifactFiles(kind, files) {
+ if (!["lock", "build"].includes(kind) || !files.length) throw new Error("Invalid artifact size check");
+ let bytes = 0;
+ for (const file of files) {
+ const info = await lstat(file);
+ if (!info.isFile() || info.isSymbolicLink() || info.size === 0) throw new Error("Invalid artifact input file");
+ bytes += info.size;
+ if (!Number.isSafeInteger(bytes) || bytes > ARTIFACT_BOUNDS[kind]) throw new Error(`${kind} artifact exceeds its source-byte allowance`);
+ }
+ return { kind, files: files.length, sourceBytes: bytes, limitBytes: ARTIFACT_BOUNDS[kind] };
+}
+
+function priority(relative) {
+ if (relative === "scheduled" || relative.startsWith("scheduled/")) return 0;
+ if (/\.(json|jsonl)$/.test(relative)) return 1;
+ if (/\.(md|html|xml|txt|log)$/.test(relative)) return 2;
+ return 3;
+}
+
+export async function stageBoundedArtifact(kind, source, destination, bounds = ARTIFACT_BOUNDS) {
+ if (!["evidence", "report"].includes(kind) || !Number.isSafeInteger(bounds[kind])
+ || !Number.isSafeInteger(bounds.receipt) || bounds[kind] <= bounds.receipt
+ || !Number.isSafeInteger(bounds.entries) || bounds.entries < 1 || bounds.entries > ARTIFACT_BOUNDS.entries) {
+ throw new Error("Invalid artifact staging allowance");
+ }
+ const root = path.resolve(source), out = path.resolve(destination);
+ if (out === root || out.startsWith(`${root}${path.sep}`) || root.startsWith(`${out}${path.sep}`)) {
+ throw new Error("Artifact staging paths must be separate");
+ }
+ await mkdir(out, { mode: 0o700 });
+ const receipt = { version: 1, kind, sourceComplete: true, inventoryComplete: true,
+ scope: "Artifact evidence; regular playwright-output/.last-run.json bookkeeping is excluded",
+ sourceBytes: 0, retainedBytes: 0, limitBytes: bounds[kind], entryLimit: bounds.entries,
+ findings: [], files: [] };
+ const candidates = [];
+ let visited = 0;
+ const finding = code => { receipt.sourceComplete = false; if (!receipt.findings.includes(code)) receipt.findings.push(code); };
+ const skipped = (relative, reason, subtree = false) => {
+ const shortened = relative.length > 512 || Buffer.byteLength(JSON.stringify(relative)) > 1024;
+ receipt.files.push({ path: shortened ? `${relative.slice(0, 160)}…${relative.slice(-80)}` : relative,
+ ...(shortened ? { pathTruncated: true, pathSha256: createHash("sha256").update(relative).digest("hex") } : {}),
+ ...(subtree ? { scope: "subtree" } : {}), retained: false, reason });
+ };
+ const scan = async () => {
+ const pending = [{ absolute: root, relative: ".", depth: 0, order: 0 }];
+ while (pending.length) {
+ pending.sort((a, b) => a.order - b.order || a.depth - b.depth || a.relative.localeCompare(b.relative));
+ const directory = pending.shift();
+ if (visited >= bounds.entries || directory.depth > 24) {
+ const reason = visited >= bounds.entries ? "entry_limit" : "directory_depth_limit";
+ receipt.inventoryComplete = false; finding(reason); skipped(directory.relative, reason, true); continue;
+ }
+ try {
+ const handle = await opendir(directory.absolute);
+ for await (const entry of handle) {
+ if (visited >= bounds.entries) {
+ receipt.inventoryComplete = false; finding("entry_limit"); skipped(directory.relative, "entry_limit", true); break;
+ }
+ visited++;
+ const absolute = path.join(directory.absolute, entry.name), relative = path.relative(root, absolute).split(path.sep).join("/");
+ if (relative.length > 512 || Buffer.byteLength(JSON.stringify(relative)) > 1024) {
+ receipt.inventoryComplete = false; finding("path_length_limit"); skipped(relative, "path_length_limit", entry.isDirectory()); continue;
+ }
+ if (entry.isFile() && /(?:^|\/)playwright-output\/\.last-run\.json$/.test(relative)) {
+ skipped(relative, "playwright_bookkeeping_excluded"); continue;
+ }
+ if (entry.name.startsWith(".")) {
+ receipt.inventoryComplete = false; finding("hidden_path_excluded"); skipped(relative, "hidden_path_excluded", entry.isDirectory()); continue;
+ }
+ if (entry.isDirectory()) {
+ const diagnostic = /(?:^|\/)(?:blob-report|html-report|playwright-output)(?:\/|$)/.test(relative);
+ pending.push({ absolute, relative, depth: directory.depth + 1,
+ order: diagnostic ? 3 : relative === "scheduled" || relative.startsWith("scheduled/") ? 0 : 1 });
+ } else if (entry.isFile()) candidates.push({ absolute, relative });
+ else { finding("non_regular_file"); skipped(relative, "non_regular_file"); }
+ }
+ } catch { receipt.inventoryComplete = false; finding("inventory_failed"); skipped(directory.relative, "inventory_failed", true); }
+ }
+ };
+ try {
+ const metadata = await lstat(root);
+ if (!metadata.isDirectory() || metadata.isSymbolicLink()) throw new Error("Invalid evidence root");
+ await scan();
+ } catch { receipt.inventoryComplete = false; finding("inventory_failed"); skipped(".", "inventory_failed", true); }
+ const canonicalRoot = await realpath(root).catch(() => root);
+ candidates.sort((a, b) => priority(a.relative) - priority(b.relative) || a.relative.localeCompare(b.relative));
+ for (const { absolute, relative } of candidates) {
+ let input, output;
+ const target = path.join(out, relative);
+ const row = { path: relative, retained: false };
+ receipt.files.push(row);
+ try {
+ if (relative === "artifact-budget-receipt.json") throw new Error("reserved_path");
+ const canonical = await realpath(absolute);
+ if (!canonical.startsWith(`${canonicalRoot}${path.sep}`)) throw new Error("source_escaped");
+ input = await open(absolute, constants.O_RDONLY | constants.O_NOFOLLOW);
+ const before = await input.stat();
+ if (!before.isFile()) throw new Error("non_regular_file");
+ row.sourceBytes = before.size;
+ receipt.sourceBytes += before.size;
+ if (!Number.isSafeInteger(receipt.sourceBytes)) throw new Error("source_size_overflow");
+ if (receipt.retainedBytes + before.size > bounds[kind] - bounds.receipt) throw new Error("byte_limit");
+ await mkdir(path.dirname(target), { recursive: true, mode: 0o700 });
+ output = await open(target, "wx", 0o600);
+ const buffer = Buffer.alloc(64 * 1024), hash = createHash("sha256");
+ let copied = 0;
+ while (copied < before.size) {
+ const read = await input.read(buffer, 0, Math.min(buffer.length, before.size - copied), copied);
+ if (!read.bytesRead) throw new Error("source_changed");
+ hash.update(buffer.subarray(0, read.bytesRead));
+ let written = 0;
+ while (written < read.bytesRead) {
+ const part = await output.write(buffer, written, read.bytesRead - written, copied + written);
+ if (!part.bytesWritten) throw new Error("copy_failed");
+ written += part.bytesWritten;
+ }
+ copied += read.bytesRead;
+ }
+ const after = await input.stat();
+ if (before.size !== after.size || before.mtimeMs !== after.mtimeMs || before.ctimeMs !== after.ctimeMs) throw new Error("source_changed");
+ await output.sync();
+ row.retained = true; row.sha256 = hash.digest("hex");
+ receipt.retainedBytes += copied;
+ } catch (error) {
+ const code = ["reserved_path", "source_escaped", "non_regular_file", "source_size_overflow", "byte_limit", "source_changed"].includes(error.message) ? error.message : "copy_failed";
+ row.reason = code; finding(code);
+ } finally {
+ await input?.close(); await output?.close();
+ if (!row.retained) await rm(target, { force: true });
+ }
+ }
+ const serialized = `${JSON.stringify(receipt, null, 2)}\n`;
+ if (Buffer.byteLength(serialized) > bounds.receipt) throw new Error("Artifact receipt exceeds reserved allowance");
+ await writeFile(path.join(out, "artifact-budget-receipt.json"), serialized, { mode: 0o600, flag: "wx" });
+ return { ...receipt, stagedBytes: receipt.retainedBytes + Buffer.byteLength(serialized) };
+}
+
+if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
+ const [command, kind, ...args] = process.argv.slice(2);
+ if (command === "check") console.log(JSON.stringify(await checkArtifactFiles(kind, args)));
+ else if (command === "stage" && args.length === 2) {
+ const receipt = await stageBoundedArtifact(kind, args[0], args[1]);
+ if (process.env.GITHUB_OUTPUT) await writeFile(process.env.GITHUB_OUTPUT, `complete=${receipt.sourceComplete}\n`, { flag: "a" });
+ const summary = `Eval ${kind}: retained ${receipt.retainedBytes} source bytes; staged ${receipt.stagedBytes} bytes; complete=${receipt.sourceComplete}. Findings: ${receipt.findings.join(", ") || "none"}.`;
+ if (process.env.GITHUB_STEP_SUMMARY) await writeFile(process.env.GITHUB_STEP_SUMMARY, `${summary}\n`, { flag: "a" });
+ console.log(summary);
+ } else throw new Error("Usage: artifact-budget.mjs check | stage ");
+}
diff --git a/tests/runner-e2e/artifact-budget.test.mjs b/tests/runner-e2e/artifact-budget.test.mjs
new file mode 100644
index 00000000000..06493ca2a64
--- /dev/null
+++ b/tests/runner-e2e/artifact-budget.test.mjs
@@ -0,0 +1,120 @@
+import assert from "node:assert/strict";
+import { createHash } from "node:crypto";
+import { mkdtemp, mkdir, readFile, rm, stat, symlink, truncate, writeFile } from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import { test } from "node:test";
+import { ARTIFACT_BOUNDS, checkArtifactFiles, stageBoundedArtifact } from "./artifact-budget.mjs";
+
+const bounds = { ...ARTIFACT_BOUNDS, evidence: 12_288, receipt: 8192, entries: 32 };
+async function fixture(run) {
+ const root = await mkdtemp(path.join(os.tmpdir(), "eval-artifacts-"));
+ try { const source = path.join(root, "source"); await mkdir(source); await run({ root, source, out: path.join(root, "out") }); }
+ finally { await rm(root, { recursive: true, force: true }); }
+}
+
+test("retains exact accounting and feedback before oversized diagnostics and keeps the source intact", async () => fixture(async ({ source, out }) => {
+ await mkdir(path.join(source, "scheduled"));
+ await writeFile(path.join(source, "scheduled/ledger.jsonl"), '{"reserved":"4.16"}\n');
+ await writeFile(path.join(source, "feedback.json"), '{"complaint":"retained"}\n');
+ await writeFile(path.join(source, "trace.zip"), Buffer.alloc(5000, 1));
+ const result = await stageBoundedArtifact("evidence", source, out, bounds);
+ assert.equal(result.sourceComplete, false); assert.equal(result.inventoryComplete, true);
+ assert.deepEqual(result.findings, ["byte_limit"]);
+ assert.equal(await readFile(path.join(out, "scheduled/ledger.jsonl"), "utf8"), '{"reserved":"4.16"}\n');
+ assert.equal(await readFile(path.join(out, "feedback.json"), "utf8"), '{"complaint":"retained"}\n');
+ assert.equal((await stat(path.join(source, "trace.zip"))).size, 5000);
+ await assert.rejects(stat(path.join(out, "trace.zip")), { code: "ENOENT" });
+ const receipt = JSON.parse(await readFile(path.join(out, "artifact-budget-receipt.json"), "utf8"));
+ assert.equal(receipt.files.find(row => row.path === "trace.zip").reason, "byte_limit");
+ assert.match(receipt.files.find(row => row.path === "feedback.json").sha256, /^[a-f0-9]{64}$/);
+ assert.ok(result.stagedBytes <= bounds.evidence);
+}));
+
+test("keeps complete evidence unchanged at the exact payload boundary and refuses destination reuse", async () => fixture(async ({ source, out }) => {
+ const content = Buffer.alloc(bounds.evidence - bounds.receipt, 42);
+ await writeFile(path.join(source, "result.json"), content);
+ const result = await stageBoundedArtifact("evidence", source, out, bounds);
+ assert.equal(result.sourceComplete, true); assert.equal(result.retainedBytes, content.length);
+ assert.deepEqual(await readFile(path.join(out, "result.json")), content);
+ await assert.rejects(stageBoundedArtifact("evidence", source, out, bounds), { code: "EEXIST" });
+}));
+
+test("refuses symlinks and records missing evidence without reading outside the source", async () => fixture(async ({ root, source, out }) => {
+ await writeFile(path.join(root, "private"), "secret outside source");
+ await symlink(path.join(root, "private"), path.join(source, "result.json"));
+ const result = await stageBoundedArtifact("evidence", source, out, bounds);
+ assert.equal(result.sourceComplete, false); assert.deepEqual(result.findings, ["non_regular_file"]);
+ assert.equal(result.retainedBytes, 0);
+ const missing = await stageBoundedArtifact("evidence", path.join(root, "missing"), path.join(root, "missing-out"), bounds);
+ assert.equal(missing.inventoryComplete, false); assert.deepEqual(missing.findings, ["inventory_failed"]);
+}));
+
+test("bounds entry inventory and exposes partial traversal", async () => fixture(async ({ source, out }) => {
+ for (let i = 0; i < 5; i++) await writeFile(path.join(source, `${i}.json`), "{}");
+ const result = await stageBoundedArtifact("evidence", source, out, { ...bounds, entries: 3 });
+ assert.equal(result.inventoryComplete, false); assert.equal(result.sourceComplete, false);
+ assert.equal(result.files.filter(row => row.retained).length, 3); assert.deepEqual(result.findings, ["entry_limit"]);
+ assert.deepEqual(result.files.find(row => row.reason === "entry_limit"), { path: ".", scope: "subtree", retained: false, reason: "entry_limit" });
+}));
+
+test("keeps hidden files excluded from upload and exposes that incomplete scope", async () => fixture(async ({ source, out }) => {
+ await mkdir(path.join(source, ".private"));
+ await writeFile(path.join(source, ".private/token"), "fixture");
+ const result = await stageBoundedArtifact("evidence", source, out, bounds);
+ assert.equal(result.inventoryComplete, false); assert.equal(result.sourceComplete, false);
+ assert.deepEqual(result.findings, ["hidden_path_excluded"]);
+ assert.deepEqual(result.files, [{ path: ".private", scope: "subtree", retained: false, reason: "hidden_path_excluded" }]);
+ await assert.rejects(stat(path.join(out, ".private")), { code: "ENOENT" });
+}));
+
+test("excludes ordinary packaged Playwright bookkeeping without failing passing evidence", async () => fixture(async ({ source, out }) => {
+ const folder = "core-compatibility/runner-codex/local/message-marker/attempt-1";
+ await mkdir(path.join(source, folder, "playwright-output"), { recursive: true });
+ await writeFile(path.join(source, folder, "playwright-output/.last-run.json"), '{"status":"passed","failedTests":[]}');
+ await writeFile(path.join(source, folder, "result.json"), '{"status":"passed"}');
+ const result = await stageBoundedArtifact("evidence", source, out, bounds);
+ assert.equal(result.sourceComplete, true); assert.equal(result.inventoryComplete, true);
+ assert.deepEqual(result.findings, []);
+ assert.equal(result.files.find(row => row.path.endsWith(".last-run.json")).reason, "playwright_bookkeeping_excluded");
+ assert.equal(await readFile(path.join(out, folder, "result.json"), "utf8"), '{"status":"passed"}');
+}));
+
+test("discovers snapshots in all attempts before archive directories can consume the entry limit", async () => fixture(async ({ source, out }) => {
+ for (const attempt of ["attempt-a", "attempt-b"]) {
+ await mkdir(path.join(source, attempt, "blob-report"), { recursive: true });
+ await mkdir(path.join(source, attempt, "snapshots"));
+ await writeFile(path.join(source, attempt, "snapshots/feedback.json"), '{"saved":true}');
+ for (let i = 0; i < 20; i++) await writeFile(path.join(source, attempt, `blob-report/${i}.zip`), "archive");
+ }
+ const result = await stageBoundedArtifact("evidence", source, out, { ...bounds, entries: 12 });
+ assert.equal(result.sourceComplete, false); assert.equal(result.inventoryComplete, false);
+ for (const attempt of ["attempt-a", "attempt-b"]) {
+ assert.equal(await readFile(path.join(out, attempt, "snapshots/feedback.json"), "utf8"), '{"saved":true}');
+ assert.ok(result.files.some(row => row.path === `${attempt}/blob-report` && row.scope === "subtree" && row.reason === "entry_limit"));
+ }
+}));
+
+test("identifies an overlong skipped subtree with a bounded preview and full path digest", async () => fixture(async ({ source, out }) => {
+ const segments = Array.from({ length: 6 }, (_, index) => `${index}-${"x".repeat(90)}`);
+ await mkdir(path.join(source, ...segments), { recursive: true });
+ await writeFile(path.join(source, ...segments, "feedback.json"), "{}");
+ const result = await stageBoundedArtifact("evidence", source, out, bounds);
+ const omitted = result.files.find(row => row.reason === "path_length_limit");
+ assert.equal(result.sourceComplete, false); assert.equal(result.inventoryComplete, false);
+ assert.equal(omitted.scope, "subtree"); assert.equal(omitted.pathTruncated, true);
+ assert.match(omitted.pathSha256, /^[a-f0-9]{64}$/); assert.ok(omitted.path.startsWith(segments[0]));
+ assert.ok(omitted.path.endsWith(segments[5].slice(-80)));
+ assert.equal(omitted.pathSha256, createHash("sha256").update(segments.join("/")).digest("hex"));
+ assert.ok(result.stagedBytes <= bounds.evidence);
+}));
+
+test("checks immutable build and lock source totals before upload without reading their contents", async () => fixture(async ({ source }) => {
+ const a = path.join(source, "a"), b = path.join(source, "b");
+ await writeFile(a, "a"); await writeFile(b, "b");
+ await truncate(a, ARTIFACT_BOUNDS.lock - 1);
+ assert.equal((await checkArtifactFiles("lock", [a, b])).sourceBytes, ARTIFACT_BOUNDS.lock);
+ await truncate(a, ARTIFACT_BOUNDS.lock);
+ await assert.rejects(checkArtifactFiles("lock", [a, b]), /exceeds/);
+ await assert.rejects(checkArtifactFiles("other", [a]), /Invalid/);
+}));
diff --git a/tests/runner-e2e/attempt-accounting.test.ts b/tests/runner-e2e/attempt-accounting.test.ts
new file mode 100644
index 00000000000..e7335879eef
--- /dev/null
+++ b/tests/runner-e2e/attempt-accounting.test.ts
@@ -0,0 +1,87 @@
+import { describe, expect, it } from "vitest";
+import { buildCampaignAccounting, isSelectedCarryForward, retainedCampaignAttempts } from "./attempt-accounting.js";
+import { buildRunnerCampaign, campaignHistoryRecord } from "./history.js";
+import type { RunnerE2EResult } from "./types.js";
+
+function result(attempt: number, cost: number, overrides: Partial = {}): RunnerE2EResult {
+ return {
+ schema: "paperclip.runner-e2e.result/v2",
+ accountingIdentity: { campaignId: "campaign", attemptId: `attempt-${attempt}` },
+ executionId: "core-compatibility.runner-codex.local.message-marker", suiteId: "core-compatibility",
+ attempt, status: attempt === 1 ? "failed" : "passed", profileId: "runner-codex",
+ environmentId: "local", caseId: "message-marker", provider: "codex", model: "fixture", runtimeMode: "native",
+ startedAt: `2026-10-20T00:0${attempt}:00Z`, finishedAt: `2026-10-20T00:0${attempt}:01Z`, durationMs: 1000,
+ runIds: [`run-${attempt}`], usage: { inputTokens: 100, outputTokens: 10, costUsd: cost }, cleanup: "passed",
+ ...overrides,
+ };
+}
+
+describe("retained attempt accounting", () => {
+ it("adds failed and successful attempts exactly once without changing the chosen verdict", () => {
+ const failed = result(1, 0.01);
+ const passed = result(2, 0.02);
+ const campaign = buildRunnerCampaign({ campaignId: "campaign", generatedAt: passed.finishedAt,
+ expected: [passed.executionId], results: [passed], attempts: [failed, passed, structuredClone(passed)] });
+ expect(campaign).toMatchObject({ passed: 1, failed: 0, selected: 1,
+ accounting: { scope: "retained_attempts", missingAttempts: 0, attempts: [{ attempt: 1 }, { attempt: 2 }] },
+ billing: { testCount: 1, testsWithCompleteBilling: 1, reportedLlmCostUsd: 0.03,
+ llm: { runCount: 2, inputTokens: 200, outputTokens: 20 } } });
+ expect(campaign.results[0]!.billing!.reportedCostUsd).toBe(0.02);
+ expect(campaign.suites[0]!.billing.reportedLlmCostUsd).toBe(0.03);
+ expect(campaignHistoryRecord(campaign, "https://example.test").billing.reportedLlmCostUsd).toBe(0.03);
+ });
+
+ it("retains known costs from interrupted attempts and marks missing attempts", () => {
+ const interrupted = result(2, 0.02, { status: "failed", cleanup: "failed" });
+ const campaign = buildRunnerCampaign({ campaignId: "campaign", generatedAt: interrupted.finishedAt,
+ expected: [interrupted.executionId], results: [interrupted], attempts: [interrupted] });
+ expect(campaign.accounting!.missingAttempts).toBe(1);
+ expect(campaign.billing.reportedLlmCostUsd).toBe(0.02);
+ expect(campaign.billing.testsWithCompleteBilling).toBe(0);
+ const missing = result(0, 0, { runIds: [], usage: null });
+ expect(buildCampaignAccounting({ results: [missing], attempts: [] })).toMatchObject({ missingAttempts: 1, attempts: [] });
+ });
+
+ it("labels legacy selected-only bundles without inventing receipts for retries", () => {
+ const selected = result(2, 0.02, { accountingIdentity: undefined });
+ expect(buildCampaignAccounting({ results: [selected] })).toMatchObject({
+ scope: "selected_results", missingAttempts: 1, attempts: [{ attempt: 2 }],
+ });
+ });
+
+ it("excludes other campaigns and ambiguous local artifacts", () => {
+ const selected = result(2, 0.02);
+ const other = result(1, 0.9, { accountingIdentity: { campaignId: "other", attemptId: "attempt-1" } });
+ const first = result(1, 0.01);
+ expect(retainedCampaignAttempts([selected], [first, selected, other])).toEqual({ attempts: [first, selected], excludedArtifacts: 1 });
+ const legacy = { ...selected, accountingIdentity: undefined };
+ expect(retainedCampaignAttempts([legacy], [legacy, { ...legacy }])).toEqual({ attempts: [legacy], excludedArtifacts: 1 });
+ });
+
+ it("uses the final supervisor receipt but rejects contradictory same-version costs", () => {
+ const selected = result(1, 0.02);
+ const early = { ...selected, finishedAt: selected.startedAt, usage: { inputTokens: 20, costUsd: 0.01 } };
+ expect(buildCampaignAccounting({ results: [selected], attempts: [selected, early] }).attempts).toMatchObject([
+ { billing: { reportedCostUsd: 0.02 } },
+ ]);
+ expect(() => buildCampaignAccounting({ results: [selected], attempts: [selected,
+ { ...selected, usage: { inputTokens: 20, costUsd: 0.1 } }] })).toThrow("Conflicting accounting receipts");
+ });
+});
+
+it("requires exact selector provenance before admitting a carried workflow result", () => {
+ const retained = result(1, 0.01, { accountingIdentity: { campaignId: "gha-123-1", attemptId: "a" },
+ source: { sha: "source", ref: "master", workflowRunUrl: "https://github.com/paperclipai/paperclip/actions/runs/123" } });
+ const artifactName = `runner-e2e-123-1-${retained.executionId}`;
+ const input = { requestedCampaignId: "gha-123-2", result: retained,
+ relativeResultPath: `${artifactName}/gha-123-1-${retained.executionId}/attempt-1/result.json`,
+ selection: { version: 1, workflowRunId: "123", workflowRunAttempt: 2, sourceSha: retained.source!.sha,
+ sourceRef: retained.source!.ref, workflowRunUrl: retained.source!.workflowRunUrl,
+ selections: [{ executionId: retained.executionId, workflowAttempt: 1, artifactName }] } };
+ expect(isSelectedCarryForward(input)).toBe(true);
+ expect(isSelectedCarryForward({ ...input, selection: undefined })).toBe(false);
+ expect(isSelectedCarryForward({ ...input, requestedCampaignId: "gha-999-2" })).toBe(false);
+ expect(isSelectedCarryForward({ ...input, relativeResultPath: `unselected/${input.relativeResultPath}` })).toBe(false);
+ expect(isSelectedCarryForward({ ...input, result: { ...retained, source: { ...retained.source!, sha: "other" } } })).toBe(false);
+ expect(isSelectedCarryForward({ ...input, result: { ...retained, accountingIdentity: { campaignId: "gha-123-3", attemptId: "a" } } })).toBe(false);
+});
diff --git a/tests/runner-e2e/attempt-accounting.ts b/tests/runner-e2e/attempt-accounting.ts
new file mode 100644
index 00000000000..3fea6442d7f
--- /dev/null
+++ b/tests/runner-e2e/attempt-accounting.ts
@@ -0,0 +1,99 @@
+import { createHash } from "node:crypto";
+import { summarizeExecutionBilling } from "./billing.js";
+import type { RunnerE2EAttemptReceipt, RunnerE2ECampaignAccounting, RunnerE2EResult } from "./types.js";
+
+export function isSelectedCarryForward(input: {
+ requestedCampaignId: string; result: RunnerE2EResult; relativeResultPath: string; selection: unknown;
+}): boolean {
+ const s = input.selection as Record | null;
+ if (!s || s.version !== 1 || typeof s.workflowRunId !== "string" || !/^[0-9]+$/.test(s.workflowRunId)
+ || !Number.isSafeInteger(s.workflowRunAttempt) || Number(s.workflowRunAttempt) < 1
+ || input.requestedCampaignId !== `gha-${s.workflowRunId}-${s.workflowRunAttempt}` || !Array.isArray(s.selections)) return false;
+ const source = input.result.source;
+ if (!source || !source.sha || !source.ref || !source.workflowRunUrl
+ || source.sha !== s.sourceSha || source.ref !== s.sourceRef || source.workflowRunUrl !== s.workflowRunUrl) return false;
+ if (input.relativeResultPath.split("/").some(part => !part || part === "." || part === "..")) return false;
+ return s.selections.some(value => {
+ if (!value || typeof value !== "object") return false;
+ const selected = value as Record;
+ const attempt = selected.workflowAttempt;
+ const campaign = `gha-${s.workflowRunId}-${attempt}`;
+ const artifact = `runner-e2e-${s.workflowRunId}-${attempt}-${input.result.executionId}`;
+ return selected.executionId === input.result.executionId && Number.isSafeInteger(attempt)
+ && Number(attempt) >= 1 && Number(attempt) < Number(s.workflowRunAttempt)
+ && selected.artifactName === artifact && input.result.accountingIdentity?.campaignId === campaign
+ && input.relativeResultPath.startsWith(`${artifact}/${campaign}-${input.result.executionId}/`);
+ });
+}
+
+function scope(result: RunnerE2EResult): string | undefined {
+ return result.accountingIdentity?.campaignId ?? result.source?.workflowRunUrl ?? undefined;
+}
+
+export function retainedCampaignAttempts(
+ selected: readonly RunnerE2EResult[],
+ candidates: readonly RunnerE2EResult[],
+): { attempts: RunnerE2EResult[]; excludedArtifacts: number } {
+ const attempts = candidates.filter(candidate => selected.some(result =>
+ result.executionId === candidate.executionId &&
+ (scope(result) ? scope(result) === scope(candidate) : result === candidate),
+ ));
+ return { attempts, excludedArtifacts: candidates.length - attempts.length };
+}
+
+function receipt(result: RunnerE2EResult): RunnerE2EAttemptReceipt {
+ const identity = [scope(result) ?? null, result.executionId,
+ result.accountingIdentity?.attemptId ?? [result.attempt, result.startedAt]];
+ return {
+ id: createHash("sha256").update(JSON.stringify(identity)).digest("hex"),
+ executionId: result.executionId,
+ attempt: result.attempt,
+ startedAt: result.startedAt,
+ finishedAt: result.finishedAt,
+ durationMs: result.durationMs,
+ ...(result.workerTiming ? { workerTiming: result.workerTiming } : {}),
+ status: result.status,
+ cleanup: result.cleanup,
+ billing: summarizeExecutionBilling(result),
+ ...(result.feedback ? { feedback: result.feedback } : {}),
+ ...(result.tracing ? { tracing: result.tracing } : {}),
+ };
+}
+
+export function buildCampaignAccounting(input: {
+ results: readonly RunnerE2EResult[];
+ attempts?: readonly RunnerE2EResult[];
+ excludedArtifacts?: number;
+}): RunnerE2ECampaignAccounting {
+ const receipts = new Map();
+ for (const result of input.attempts ?? input.results) {
+ if (result.attempt === 0) continue; // Missing result placeholders are not free attempts.
+ const next = receipt(result);
+ const previous = receipts.get(next.id);
+ if (previous && previous.finishedAt === next.finishedAt &&
+ JSON.stringify(previous.billing) !== JSON.stringify(next.billing)) {
+ throw new Error(`Conflicting accounting receipts for ${result.executionId} attempt ${result.attempt}`);
+ }
+ if (previous && previous.finishedAt === next.finishedAt && previous.workerTiming && next.workerTiming &&
+ JSON.stringify(previous.workerTiming) !== JSON.stringify(next.workerTiming)) {
+ throw new Error(`Conflicting timing receipts for ${result.executionId} attempt ${result.attempt}`);
+ }
+ if (previous && previous.finishedAt === next.finishedAt && previous.workerTiming && !next.workerTiming) next.workerTiming = previous.workerTiming;
+ if (!previous || Date.parse(next.finishedAt) >= Date.parse(previous.finishedAt)) receipts.set(next.id, next);
+ }
+ const attempts = [...receipts.values()].sort((a, b) =>
+ a.executionId.localeCompare(b.executionId) || a.attempt - b.attempt || a.id.localeCompare(b.id));
+ let missingAttempts = 0;
+ for (const result of input.results) {
+ const retained = attempts.filter(attempt => attempt.executionId === result.executionId);
+ const latest = Math.max(result.attempt, ...retained.map(attempt => attempt.attempt), 1);
+ missingAttempts += Math.max(0, latest - new Set(retained.map(attempt => attempt.attempt)).size);
+ }
+ return {
+ version: 1,
+ scope: input.attempts ? "retained_attempts" : "selected_results",
+ attempts,
+ missingAttempts,
+ excludedArtifacts: input.excludedArtifacts ?? 0,
+ };
+}
diff --git a/tests/runner-e2e/attempt-phase.ts b/tests/runner-e2e/attempt-phase.ts
new file mode 100644
index 00000000000..1c8204e4c66
--- /dev/null
+++ b/tests/runner-e2e/attempt-phase.ts
@@ -0,0 +1,34 @@
+import { randomUUID } from "node:crypto";
+import { readFile, rename, rm, writeFile } from "node:fs/promises";
+import type { EvalChargeIdentity } from "./cost-ledger.js";
+
+type Identity = Pick;
+interface Checkpoint extends Identity { version: 1; phase: "setup" | "agent" }
+
+export async function createEvalAttemptPhase(file: string, identity: Identity) {
+ await writeFile(file, JSON.stringify({ version: 1, ...identity, phase: "setup" }), { mode: 0o600, flag: "wx" });
+}
+
+export async function readEvalAttemptPhase(file: string, expected: Identity): Promise {
+ const text = await readFile(file, "utf8");
+ if (text.length > 4096) throw new Error("Invalid eval phase checkpoint");
+ const value = JSON.parse(text) as Checkpoint;
+ if (!value || typeof value !== "object" || Array.isArray(value)
+ || Object.keys(value).some(key => !["version", "executionId", "attemptId", "phase"].includes(key))
+ || value.version !== 1 || value.executionId !== expected.executionId || value.attemptId !== expected.attemptId
+ || !["setup", "agent"].includes(value.phase)) throw new Error("Invalid eval phase checkpoint");
+ return { ...expected, phase: value.phase };
+}
+
+export async function beginEvalAgentPhase(executionId: string, env: NodeJS.ProcessEnv = process.env) {
+ const file = env.PAPERCLIP_E2E_GATEWAY_PHASE_PATH;
+ if (!file && !env.PAPERCLIP_E2E_GATEWAY_BASE_URL) return;
+ if (!file || !env.PAPERCLIP_E2E_ACCOUNTING_ATTEMPT_ID) throw new Error("Missing eval phase checkpoint");
+ const identity = await readEvalAttemptPhase(file, { executionId, attemptId: env.PAPERCLIP_E2E_ACCOUNTING_ATTEMPT_ID });
+ if (identity.phase === "agent") return;
+ const temporary = `${file}.${randomUUID()}.tmp`;
+ try {
+ await writeFile(temporary, JSON.stringify({ version: 1, ...identity, phase: "agent" }), { mode: 0o600, flag: "wx" });
+ await rename(temporary, file);
+ } finally { await rm(temporary, { force: true }); }
+}
diff --git a/tests/runner-e2e/billing.test.ts b/tests/runner-e2e/billing.test.ts
index f16e2d9a60d..48352815c0e 100644
--- a/tests/runner-e2e/billing.test.ts
+++ b/tests/runner-e2e/billing.test.ts
@@ -6,6 +6,8 @@ import {
summarizeExecutionBilling,
} from "./billing.js";
import type { RunnerE2EResult } from "./types.js";
+import { durationLabel, llmCostLabel } from "./report-format.js";
+import { validateRetainedRunnerResult } from "./result-validation.js";
function result(overrides: Partial = {}): RunnerE2EResult {
return {
@@ -29,6 +31,80 @@ function result(overrides: Partial = {}): RunnerE2EResult {
}
describe("runner E2E billing summaries", () => {
+ it("keeps missing agent time separate from attempt duration and measured zero", () => {
+ const interrupted = result({ status: "failed", durationMs: 108_850 });
+ expect(summarizeExecutionBilling(interrupted).runtime.agentRunDurationMs).toBeNull();
+ const noCapturedRuns = result({ ...interrupted, runIds: [],
+ runtimeUsage: buildRuntimeUsage({ environmentId: "local", runs: [] }) });
+ expect(noCapturedRuns.runtimeUsage?.agentRunDurationMs).toBeNull();
+ expect(() => validateRetainedRunnerResult(noCapturedRuns)).not.toThrow();
+ const measured = result({ runtimeUsage: buildRuntimeUsage({ environmentId: "local", runs: [
+ { startedAt: "2026-10-21T00:00:00Z", finishedAt: "2026-10-21T00:00:04Z" },
+ { startedAt: "2026-10-21T00:00:01Z", finishedAt: "2026-10-21T00:00:05Z" },
+ ] }) });
+ expect(aggregateCampaignBilling([measured]).agentRunDurationMs).toBe(8000);
+ expect(aggregateCampaignBilling([measured, interrupted]).agentRunDurationMs).toBeNull();
+ expect(aggregateCampaignBilling([measured, noCapturedRuns]).agentRunDurationMs).toBeNull();
+ expect(durationLabel(aggregateCampaignBilling([measured, interrupted]).agentRunDurationMs)).toBe("Unknown");
+ const zero = buildRuntimeUsage({ environmentId: "local", runs: [
+ { startedAt: "2026-10-21T00:00:00Z", finishedAt: "2026-10-21T00:00:00Z" },
+ ] });
+ expect(zero.agentRunDurationMs).toBe(0);
+ expect(durationLabel(zero.agentRunDurationMs)).toBe("0ms");
+ });
+
+ it.each([undefined, "not-a-date", "2026-10-20T23:59:59Z"])("retains unknown time for a missing or invalid end timestamp: %s", finishedAt => {
+ const runtimeUsage = buildRuntimeUsage({ environmentId: "daytona", runs: [
+ { startedAt: "2026-10-21T00:00:00Z", finishedAt: "2026-10-21T00:00:04Z" },
+ { startedAt: "2026-10-21T00:00:00Z", finishedAt },
+ ], leases: [{ acquiredAt: "2026-10-21T00:00:00Z", releasedAt: finishedAt,
+ metadata: { cpu: 4, memory: 4, disk: 10 } }] });
+ expect(runtimeUsage).toMatchObject({ agentRunDurationMs: null, leaseDurationMs: null, costStatus: "unavailable" });
+ expect(runtimeUsage.estimatedListCostUsd).toBeUndefined();
+ const retained = result({ environmentId: "daytona", runtimeUsage });
+ expect(() => validateRetainedRunnerResult(retained)).not.toThrow();
+ expect(aggregateCampaignBilling([retained, result()]).leaseDurationMs).toBeNull();
+ expect(aggregateCampaignBilling([result()]).leaseDurationMs).toBe(0);
+ });
+
+ it("reads precise provider amounts without relabeling estimates or unpriced usage", () => {
+ const precise = summarizeExecutionBilling(result({ usage: {
+ inputTokens: 0, outputTokens: 0, cachedInputTokens: 5,
+ costUsdExact: "0.000000123", costStatus: "reported", accountingReceiptReady: true,
+ } }));
+ expect(precise.llm).toMatchObject({ reportedCostUsd: 0.000000123, runsWithReportedCost: 1, runsWithTokenUsage: 1 });
+ const estimated = summarizeExecutionBilling(result({ usage: {
+ inputTokens: 100, outputTokens: 10, costUsdExact: "0.012345678", costStatus: "estimated",
+ } }));
+ expect(estimated.llm).toMatchObject({ reportedCostUsd: 0, estimatedCostUsd: 0.012345678,
+ runsWithReportedCost: 0, runsWithEstimatedCost: 1, costStatus: "estimated" });
+ expect(estimated.observedAndEstimatedCostUsd).toBe(0.012345678);
+ const unpriced = summarizeExecutionBilling(result({ usage: {
+ inputTokens: 100, outputTokens: 10, costUsd: 0, costStatus: "unpriced",
+ } }));
+ expect(unpriced.llm).toMatchObject({ runsWithReportedCost: 0, costStatus: "unpriced" });
+ });
+
+ it("distinguishes explicit zero usage from a partial or missing receipt", () => {
+ const zero = summarizeExecutionBilling(result({ usage: { inputTokens: 0, outputTokens: 0, costUsdExact: "0", accountingReceiptReady: true } }));
+ expect(zero.complete).toBe(true);
+ const partial = summarizeExecutionBilling(result({ usage: { inputTokens: 100, outputTokens: 10, costUsd: 0.01, accountingReceiptReady: false } }));
+ expect(partial.reportedCostUsd).toBe(0.01);
+ expect(partial.complete).toBe(false);
+ expect(summarizeExecutionBilling(result({ usage: null })).complete).toBe(false);
+ });
+ it("retains a known OpenCode retry subtotal while the missing turn price stays incomplete", () => {
+ const partial = result({ usage: { inputTokens: 30, outputTokens: 6, costUsd: 0.5, costStatus: "unpriced", accountingReceiptReady: true } });
+ const billing = summarizeExecutionBilling(partial);
+ expect(billing.llm).toMatchObject({ reportedCostUsd: 0.5, runsWithReportedCost: 1, runsWithPartialReportedCost: 1, costStatus: "partial" });
+ expect(billing.observedAndEstimatedCostUsd).toBe(0.5);
+ expect(billing.complete).toBe(false);
+ expect(llmCostLabel(billing.reportedCostUsd, billing.llm)).toBe("$0.5000 (partial amount)");
+ const campaign = aggregateCampaignBilling([partial, result({ executionId: "other", usage: { inputTokens: 1, outputTokens: 1, costUsd: 0.2 } })]);
+ expect(campaign).toMatchObject({ reportedLlmCostUsd: 0.7, observedAndEstimatedCostUsd: 0.7, testsWithCompleteBilling: 1,
+ llm: { runsWithReportedCost: 2, runsWithPartialReportedCost: 1, costStatus: "partial" } });
+ expect(llmCostLabel(campaign.reportedLlmCostUsd, campaign.llm)).toBe("$0.7000 (partial amount)");
+ });
it("counts completion judge reservations and preserves unknown spend after interruption", () => {
const pending = { ...reserveCompletionQuality({ sourceId: "chat", marker: "x", worker: { id: "task", status: "done", completedAt: "2026-09-01" }, documents: [{ id: "doc", issueId: "task", body: "result" }], comments: [{ id: "reply", issueId: "chat", authorAgentId: "agent", createdAt: "2026-09-02", body: "ready" }], runs: [] }, 0.5), name: "completion", expectedPass: true };
const unknown = summarizeExecutionBilling(result({ completionQuality: [pending] }));
diff --git a/tests/runner-e2e/billing.ts b/tests/runner-e2e/billing.ts
index 4681696a7ee..35f70e83569 100644
--- a/tests/runner-e2e/billing.ts
+++ b/tests/runner-e2e/billing.ts
@@ -32,21 +32,30 @@ export interface CampaignBillingSummary {
assistant?: import("./types.js").RunnerE2EAggregateBillingSummary["assistant"];
judge?: import("./types.js").RunnerE2EJudgeBillingSummary;
testCount: number;
- agentRunDurationMs: number;
- leaseDurationMs: number;
+ agentRunDurationMs: number | null;
+ leaseDurationMs: number | null;
llm: RunnerE2EBillingSummary["llm"];
reportedLlmCostUsd: number;
+ estimatedLlmCostUsd?: number;
estimatedRuntimeCostUsd: number;
observedAndEstimatedCostUsd: number | null;
testsWithCompleteBilling: number;
}
/** Render observed subtotals without presenting absent measurements as zero. */
-export function billingCoverageLabel(value: string, covered: number, total: number): string {
+export function billingCoverageLabel(value: string, covered: number, total: number, incompleteAmount = false): string {
if (!Number.isInteger(covered) || !Number.isInteger(total) || covered <= 0 || total <= 0 || covered > total) {
return "Unavailable";
}
- return covered < total ? `${value} (partial: ${covered}/${total} runs)` : value;
+ return covered < total ? `${value} (partial: ${covered}/${total} runs)` : incompleteAmount ? `${value} (partial amount)` : value;
+}
+
+function costCoverageStatus(runs: number, tokens: number, reported: number, estimated: number, partial = 0): RunnerE2EBillingSummary["llm"]["costStatus"] {
+ if (partial > 0) return "partial";
+ if (runs > 0 && reported === runs) return "reported";
+ if (runs > 0 && reported + estimated === runs) return "estimated";
+ if (reported + estimated > 0) return "partial";
+ return tokens > 0 ? "unpriced" : "unavailable";
}
function record(value: unknown): Record {
@@ -92,24 +101,25 @@ function usageMeasurement(usage: Record) {
}
function usageCostUsd(usage: Record) {
+ if (usage.billingType === "subscription") return undefined;
const measurement = usageMeasurement(usage);
+ const exact = typeof usage.costUsdExact === "string" && /^\d+(?:\.\d+)?$/.test(usage.costUsdExact)
+ ? finiteNumber(Number(usage.costUsdExact)) : undefined;
const direct =
- firstNumber(usage, [
- "cacheAdjustedCostUsd",
- "costUsd",
- "providerCostUsd",
- ]) ??
+ firstNumber(usage, ["cacheAdjustedCostUsd"]) ?? exact ??
+ firstNumber(usage, ["costUsd", "providerCostUsd"]) ??
firstNumber(measurement, [
"cacheAdjustedCostUsd",
"costUsd",
"providerCostUsd",
]);
- if (direct !== undefined) return direct;
+ if (direct !== undefined) return usage.costStatus === "unpriced" && direct === 0 ? undefined : direct;
const cost = record(usage.cost);
const currency =
typeof cost.currency === "string" ? cost.currency.toUpperCase() : "USD";
if (currency !== "USD") return undefined;
- return firstNumber(cost, ["amount", "total"]);
+ const amount = firstNumber(cost, ["amount", "total"]);
+ return usage.costStatus === "unpriced" && amount === 0 ? undefined : amount;
}
function usageEntries(
@@ -144,9 +154,13 @@ function durationBetween(
) {
const started = startedAt ? new Date(startedAt).getTime() : Number.NaN;
const finished = finishedAt ? new Date(finishedAt).getTime() : Number.NaN;
- return Number.isFinite(started) && Number.isFinite(finished)
- ? Math.max(0, finished - started)
- : 0;
+ return Number.isFinite(started) && Number.isFinite(finished) && finished >= started
+ ? finished - started
+ : null;
+}
+
+function sumDurations(values: readonly (number | null)[]): number | null {
+ return values.some(value => value === null) ? null : values.reduce((total, value) => total + value!, 0);
}
function resourceValue(
@@ -162,10 +176,9 @@ export function buildRuntimeUsage(input: {
leases?: readonly RuntimeLeaseUsageInput[];
fallbackFinishedAt?: string | Date;
}): RunnerE2ERuntimeUsage {
- const agentRunDurationMs = input.runs.reduce(
- (total, run) => total + durationBetween(run.startedAt, run.finishedAt),
- 0,
- );
+ const agentRunDurationMs = input.runs.length > 0
+ ? sumDurations(input.runs.map(run => durationBetween(run.startedAt, run.finishedAt)))
+ : null;
if (input.environmentId === "local") {
return {
provider: "local",
@@ -179,6 +192,7 @@ export function buildRuntimeUsage(input: {
const leases = input.leases ?? [];
let leaseDurationMs = 0;
+ let leaseDurationsComplete = leases.length > 0;
let estimatedListCostUsd = 0;
let resourcesComplete = leases.length > 0;
const resourceRows: Array<{ cpu: number; memory: number; disk: number }> = [];
@@ -186,6 +200,11 @@ export function buildRuntimeUsage(input: {
const finishedAt =
lease.releasedAt ?? input.fallbackFinishedAt ?? lease.updatedAt ?? null;
const durationMs = durationBetween(lease.acquiredAt, finishedAt);
+ if (durationMs === null) {
+ leaseDurationsComplete = false;
+ resourcesComplete = false;
+ continue;
+ }
leaseDurationMs += durationMs;
const cpu = resourceValue(lease.metadata, "cpu");
const memory = resourceValue(lease.metadata, "memory");
@@ -209,7 +228,7 @@ export function buildRuntimeUsage(input: {
return {
provider: "daytona",
agentRunDurationMs,
- leaseDurationMs: leases.length > 0 ? leaseDurationMs : null,
+ leaseDurationMs: leaseDurationsComplete ? leaseDurationMs : null,
leaseCount: leases.length,
...(commonResource("cpu") === undefined
? {}
@@ -243,7 +262,7 @@ export function fallbackRuntimeUsage(
if (result.runtimeUsage) return result.runtimeUsage;
return {
provider: result.environmentId,
- agentRunDurationMs: result.durationMs,
+ agentRunDurationMs: null,
leaseDurationMs: null,
leaseCount: 0,
costStatus:
@@ -269,7 +288,11 @@ export function summarizeExecutionBilling(
let cachedInputTokens = 0;
let runsWithTokenUsage = 0;
let runsWithReportedCost = 0;
+ let runsWithEstimatedCost = 0;
+ let runsWithPartialReportedCost = 0;
let reportedCostUsd = 0;
+ let estimatedCostUsd = 0;
+ let incompleteReceipt = false;
for (const usage of entries) {
if (!usage) continue;
const measurement = usageMeasurement(usage);
@@ -290,22 +313,24 @@ export function summarizeExecutionBilling(
inputTokens += input;
outputTokens += output;
cachedInputTokens += cached;
- if (input > 0 || output > 0 || cached > 0) runsWithTokenUsage += 1;
+ const completeTokens = firstNumber(measurement, ["inputTokens", "input", "promptTokens"]) !== undefined &&
+ firstNumber(measurement, ["outputTokens", "output", "completionTokens"]) !== undefined && usage.accountingReceiptReady !== false;
+ if (completeTokens) runsWithTokenUsage += 1;
+ if (usage.accountingReceiptReady === false) incompleteReceipt = true;
const costUsd = usageCostUsd(usage);
- if (costUsd !== undefined && (costUsd > 0 || input > 0 || output > 0)) {
- runsWithReportedCost += 1;
- reportedCostUsd += costUsd;
+ if (costUsd !== undefined) {
+ if (usage.costStatus === "estimated") {
+ runsWithEstimatedCost += 1;
+ estimatedCostUsd += costUsd;
+ } else {
+ runsWithReportedCost += 1;
+ if (usage.costStatus === "unpriced") runsWithPartialReportedCost += 1;
+ reportedCostUsd += costUsd;
+ }
}
}
const runCount = Math.max(requestedRunCount, entries.length);
- const costStatus =
- runsWithReportedCost === runCount
- ? "reported"
- : runsWithReportedCost > 0
- ? "partial"
- : runsWithTokenUsage > 0
- ? "unpriced"
- : "unavailable";
+ const costStatus = costCoverageStatus(runCount, runsWithTokenUsage, runsWithReportedCost, runsWithEstimatedCost, runsWithPartialReportedCost);
const runtime = { ...fallbackRuntimeUsage(result) };
if (browserOnly) runtime.agentRunDurationMs = 0;
const estimatedRuntimeCostUsd = runtime.estimatedListCostUsd ?? 0;
@@ -315,12 +340,13 @@ export function summarizeExecutionBilling(
// The operations receipt covers setup and agent requests for ONE owning attempt.
// Descendants reuse the fixture context without copying its receipt or charges.
const operations = result.productJourney?.operations;
- const modelSubtotal = operations ? operations.modelCostUsd : reportedCostUsd;
+ const modelSubtotal = operations ? operations.modelCostUsd : reportedCostUsd + estimatedCostUsd;
const complete =
+ !incompleteReceipt && runsWithPartialReportedCost === 0 &&
(!quality || quality.estimatedCostUsd !== null) &&
(!operations || operations.modelCostUsd !== null) &&
runsWithTokenUsage === runCount &&
- runsWithReportedCost === runCount &&
+ runsWithReportedCost + runsWithEstimatedCost === runCount &&
runtime.costStatus !== "unavailable";
return {
...(result.publicMcp ? { assistant: result.publicMcp } : {}),
@@ -328,11 +354,14 @@ export function summarizeExecutionBilling(
runCount,
runsWithTokenUsage,
runsWithReportedCost,
+ runsWithEstimatedCost,
+ runsWithPartialReportedCost,
inputTokens,
outputTokens,
cachedInputTokens,
totalTokens: inputTokens + cachedInputTokens + outputTokens,
reportedCostUsd,
+ estimatedCostUsd,
costStatus,
},
runtime,
@@ -347,7 +376,16 @@ export function summarizeExecutionBilling(
export function aggregateCampaignBilling(
results: readonly RunnerE2EResult[],
): CampaignBillingSummary {
- const summaries = results.map(summarizeExecutionBilling);
+ return aggregateBillingSummaries(results.map(result => ({
+ executionId: result.executionId, billing: summarizeExecutionBilling(result),
+ })));
+}
+
+export function aggregateBillingSummaries(
+ entries: readonly { executionId: string; billing: RunnerE2EBillingSummary }[],
+): CampaignBillingSummary {
+ const summaries = entries.map(entry => entry.billing);
+ const executionIds = [...new Set(entries.map(entry => entry.executionId))];
const runCount = summaries.reduce(
(total, summary) => total + summary.llm.runCount,
0,
@@ -364,6 +402,9 @@ export function aggregateCampaignBilling(
(total, summary) => total + summary.reportedCostUsd,
0,
);
+ const runsWithEstimatedCost = summaries.reduce((sum, summary) => sum + (summary.llm.runsWithEstimatedCost ?? 0), 0);
+ const runsWithPartialReportedCost = summaries.reduce((sum, summary) => sum + (summary.llm.runsWithPartialReportedCost ?? 0), 0);
+ const estimatedLlmCostUsd = summaries.reduce((sum, summary) => sum + (summary.llm.estimatedCostUsd ?? 0), 0);
const estimatedRuntimeCostUsd = summaries.reduce(
(total, summary) => total + summary.estimatedRuntimeCostUsd,
0,
@@ -386,19 +427,16 @@ export function aggregateCampaignBilling(
reservedCostUsd: judges.reduce((n, q) => n + q.reservedCostUsd, 0),
attemptsWithUnknownUsage: judges.filter(q => q.estimatedCostUsd === null).length,
} } : {}),
- testCount: results.length,
- agentRunDurationMs: summaries.reduce(
- (total, summary) => total + summary.runtime.agentRunDurationMs,
- 0,
- ),
- leaseDurationMs: summaries.reduce(
- (total, summary) => total + (summary.runtime.leaseDurationMs ?? 0),
- 0,
- ),
+ testCount: executionIds.length,
+ agentRunDurationMs: sumDurations(summaries.map(summary => summary.runtime.agentRunDurationMs)),
+ leaseDurationMs: sumDurations(summaries.filter(summary => summary.runtime.provider === "daytona")
+ .map(summary => summary.runtime.leaseDurationMs)),
llm: {
runCount,
runsWithTokenUsage,
runsWithReportedCost,
+ runsWithEstimatedCost,
+ runsWithPartialReportedCost,
inputTokens: summaries.reduce(
(total, summary) => total + summary.llm.inputTokens,
0,
@@ -416,19 +454,14 @@ export function aggregateCampaignBilling(
0,
),
reportedCostUsd: reportedLlmCostUsd,
- costStatus:
- runsWithReportedCost === runCount
- ? "reported"
- : runsWithReportedCost > 0
- ? "partial"
- : runsWithTokenUsage > 0
- ? "unpriced"
- : "unavailable",
+ estimatedCostUsd: estimatedLlmCostUsd,
+ costStatus: costCoverageStatus(runCount, runsWithTokenUsage, runsWithReportedCost, runsWithEstimatedCost, runsWithPartialReportedCost),
},
reportedLlmCostUsd,
+ estimatedLlmCostUsd,
estimatedRuntimeCostUsd,
observedAndEstimatedCostUsd: summaries.some(s => s.observedAndEstimatedCostUsd === null) ? null : summaries.reduce((total, summary) => total + (summary.observedAndEstimatedCostUsd ?? 0), 0),
- testsWithCompleteBilling: summaries.filter((summary) => summary.complete)
- .length,
+ testsWithCompleteBilling: executionIds.filter(id =>
+ entries.filter(entry => entry.executionId === id).every(entry => entry.billing.complete)).length,
};
}
diff --git a/tests/runner-e2e/braintrust-experiment-import.ts b/tests/runner-e2e/braintrust-experiment-import.ts
new file mode 100644
index 00000000000..eef8f3598c5
--- /dev/null
+++ b/tests/runner-e2e/braintrust-experiment-import.ts
@@ -0,0 +1,45 @@
+import { readFile, rename, writeFile } from "node:fs/promises";
+import path from "node:path";
+import { randomUUID } from "node:crypto";
+import { evalBraintrustConfig } from "./braintrust.js";
+import { buildBraintrustExperiment, importBraintrustExperiment } from "./braintrust-experiment.js";
+
+const [campaignFile, originalPathsFile, outputFile, mode] = process.argv.slice(2);
+if (!campaignFile || !originalPathsFile || !outputFile || ![undefined, "--dry-run"].includes(mode)) {
+ throw new Error("Usage: braintrust-experiment-import.ts [--dry-run]");
+}
+let totalBytes = 0;
+const readJson = async (file: string) => {
+ const bytes = await readFile(file);
+ totalBytes += bytes.length;
+ if (bytes.length > 8 * 1024 * 1024 || totalBytes > 64 * 1024 * 1024) throw new Error("Experiment input exceeds its file or campaign size limit");
+ return JSON.parse(bytes.toString("utf8"));
+};
+const paths = await readJson(originalPathsFile);
+if (!Array.isArray(paths) || paths.length === 0 || paths.length > 10_000
+ || paths.some(file => typeof file !== "string" || !path.isAbsolute(file))) throw new Error("Expected explicit absolute original result paths");
+if ([campaignFile, originalPathsFile, ...paths].some(file => path.resolve(file) === path.resolve(outputFile))) {
+ throw new Error("Experiment output must not replace input evidence");
+}
+const campaign = await readJson(campaignFile), originals = [];
+for (const file of paths) originals.push(await readJson(file));
+const projectId = process.env.PAPERCLIP_E2E_BRAINTRUST_PROJECT_ID ?? "";
+const plan = buildBraintrustExperiment(projectId, campaign, originals);
+const persist = async (value: unknown) => {
+ const temporary = `${outputFile}.${randomUUID()}.tmp`;
+ await writeFile(temporary, `${JSON.stringify(value, null, 2)}\n`, { flag: "wx", mode: 0o600 });
+ await rename(temporary, outputFile);
+};
+if (mode === "--dry-run") {
+ await persist(plan);
+ console.log(`Prepared ${plan.events.length} retained rows; no network requests made.`);
+} else {
+ const config = evalBraintrustConfig(process.env);
+ if (!config) throw new Error("Braintrust project and ingestion credential required");
+ const previous = await readJson(outputFile).catch(error => {
+ if (error?.code === "ENOENT") return undefined;
+ throw error;
+ });
+ const receipt = await importBraintrustExperiment(config, campaign, originals, persist, fetch, previous);
+ console.log(`Braintrust accepted ${receipt.acceptedRows} retained rows in experiment ${receipt.experimentId}; indexed verification remains required.`);
+}
diff --git a/tests/runner-e2e/braintrust-experiment.test.ts b/tests/runner-e2e/braintrust-experiment.test.ts
new file mode 100644
index 00000000000..41c61fc5b20
--- /dev/null
+++ b/tests/runner-e2e/braintrust-experiment.test.ts
@@ -0,0 +1,328 @@
+import { expect, it, vi } from "vitest";
+import { execFile } from "node:child_process";
+import { promisify } from "node:util";
+import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import path from "node:path";
+import { buildBraintrustExperiment, importBraintrustExperiment, renderBraintrustExperiment, type BraintrustExperimentReceipt } from "./braintrust-experiment.js";
+import { regenerateRunnerDashboard } from "./dashboard-regenerate.js";
+import { buildRunnerCampaign } from "./history.js";
+import { runnerMatrix } from "./catalog.js";
+import { emptyFeedbackSummary } from "./feedback.js";
+import { openExperimentIngestion } from "./experiment-ingestion.js";
+import type { RunnerE2ECampaign, RunnerE2EResult } from "./types.js";
+
+const projectId = "11111111-1111-4111-8111-111111111111";
+const experimentId = "22222222-2222-4222-8222-222222222222";
+function fixture() {
+ const cell = runnerMatrix.find(cell => cell.id === "core-compatibility.runner-codex.local.message-marker")!;
+ const first: RunnerE2EResult = { schema: "paperclip.runner-e2e.result/v2", executionId: cell.id,
+ suiteId: cell.suite.id, suiteDefinitionHash: cell.suiteDefinitionHash,
+ accountingIdentity: { campaignId: "retained-pilot", attemptId: "first" }, source: { sha: "a".repeat(40), ref: null, workflowRunUrl: null },
+ attempt: 1, status: "failed", cleanup: "passed", profileId: cell.profile.id, caseId: cell.task.id,
+ environmentId: cell.environment.id, provider: cell.profile.provider, model: cell.profile.model, runtimeMode: cell.profile.expectedRuntimeMode,
+ startedAt: "2026-10-20T00:00:00.000Z", finishedAt: "2026-10-20T00:00:01.000Z", durationMs: 1000,
+ error: "private-canary-error", usage: { privateCanary: "private-canary-usage" }, feedback: emptyFeedbackSummary() };
+ const second: RunnerE2EResult = { ...structuredClone(first), source: { ...first.source!, sha: "b".repeat(40) },
+ accountingIdentity: { campaignId: "retained-pilot", attemptId: "second" }, attempt: 2, status: "passed", error: undefined,
+ startedAt: "2026-10-20T00:01:00.000Z", finishedAt: "2026-10-20T00:01:01.000Z",
+ tracing: { projectId, traceId: "c".repeat(32), spanId: "d".repeat(16), status: "accepted", verification: "unverified", nativeConfiguration: "configured" } };
+ const originals = [first, second];
+ const campaign = buildRunnerCampaign({ campaignId: "retained-pilot", generatedAt: second.finishedAt,
+ expected: [cell.id], results: [second], attempts: originals });
+ return { campaign, originals };
+}
+
+it("imports all attempts with their own source and original grades, excluding private evidence and duplicate cost metrics", () => {
+ const { campaign, originals } = fixture();
+ const plan = buildBraintrustExperiment(projectId, campaign, originals);
+ expect(plan.events).toHaveLength(2);
+ expect(plan.events.map(row => row.scores.product_pass).sort()).toEqual([0, 1]);
+ expect(plan.events.map(row => row.input.source_revision).sort()).toEqual(["a".repeat(40), "b".repeat(40)]);
+ expect(JSON.stringify(plan)).not.toMatch(/private-canary|estimated_cost|prompt_tokens/);
+ expect(plan.events.every(row => row.metadata.request_ledger === null)).toBe(true);
+ expect(plan.events.find(row => row.metadata.attempt_id === "second")!.metadata.trace).toMatchObject({ root_span_id: "c".repeat(32) });
+ expect(buildBraintrustExperiment(projectId, { ...campaign, generatedAt: "2026-10-21T00:00:00Z" }, [...originals].reverse())).toEqual(plan);
+ expect(buildBraintrustExperiment(projectId, campaign, [...originals, originals[0]!])).toEqual(plan);
+});
+
+it("imports retained grades through the scheduled transport allowance without changing the evidence", async () => {
+ const root = await mkdtemp(path.join(tmpdir(), "bounded-experiment-import-"));
+ const { campaign, originals } = fixture(), before = structuredClone({ campaign, originals });
+ const plan = buildBraintrustExperiment(projectId, campaign, originals), receiptPath = path.join(root, "ingestion.jsonl");
+ const meter = await openExperimentIngestion({ projectId, campaignId: campaign.campaignId, receiptPath, deadlineAt: Date.now() + 60_000,
+ transport: async (url, init) => String(url).endsWith("/experiment")
+ ? Response.json({ ...JSON.parse(String(init!.body)), id: experimentId })
+ : String(url).endsWith("/insert") ? Response.json({ row_ids: plan.events.map(row => row.id) }) : Response.json({}) });
+ try {
+ expect(await importBraintrustExperiment({ projectId, apiKey: "private-fixture" }, campaign, originals, async () => {}, meter.transport))
+ .toMatchObject({ status: "accepted", acceptedRows: 2, expectedRows: 2 });
+ await meter.close();
+ const rows = (await readFile(receiptPath, "utf8")).trim().split("\n").map(line => JSON.parse(line));
+ expect(rows.at(-1)).toMatchObject({ admittedRequests: 3, admittedScores: 4, unresolvedRequests: 0 });
+ expect({ campaign, originals }).toEqual(before);
+ } finally { await meter.close(); await rm(root, { recursive: true, force: true }); }
+});
+
+it.each(["missing", "foreign", "modified-grade", "ambiguous-source", "missing-source", "foreign-trace", "missing-hash"])("rejects %s evidence before network requests", async (kind) => {
+ const { campaign, originals } = fixture();
+ if (kind === "missing") originals.pop();
+ if (kind === "foreign") originals[0]!.accountingIdentity!.campaignId = "other";
+ if (kind === "modified-grade") originals[0]!.status = "passed";
+ if (kind === "ambiguous-source") originals.push({ ...originals[0]!, source: { ...originals[0]!.source!, sha: "c".repeat(40) } });
+ if (kind === "missing-source") originals[0]!.source = undefined;
+ if (kind === "foreign-trace") originals[1]!.tracing!.projectId = experimentId;
+ if (kind === "missing-hash") originals[0]!.suiteDefinitionHash = undefined;
+ const transport = vi.fn();
+ await expect(importBraintrustExperiment({ projectId, apiKey: "key" }, campaign, originals, async () => {}, transport)).rejects.toThrow();
+ expect(transport).not.toHaveBeenCalled();
+});
+
+it("changes experiment identity for changed evidence while keeping unknown accounting unknown", () => {
+ const { campaign, originals } = fixture();
+ const before = buildBraintrustExperiment(projectId, campaign, originals);
+ campaign.accounting!.missingAttempts = 1;
+ const after = buildBraintrustExperiment(projectId, campaign, originals);
+ expect(after.payloadHash).not.toBe(before.payloadHash);
+ expect(after.metadata.request_ledger).toBeNull();
+ expect(after.metadata.missing_attempts).toBe(1);
+});
+
+it("imports carried-forward receipts with their original campaign, revision, and grades", () => {
+ const { campaign, originals } = fixture();
+ const report = { ...campaign, campaignId: "later-ci-attempt" };
+ const before = structuredClone(originals);
+ const plan = buildBraintrustExperiment(projectId, report, originals);
+ expect(plan.metadata.campaign_id).toBe("later-ci-attempt");
+ expect(plan.events.every(row => row.metadata.campaign_id === "retained-pilot")).toBe(true);
+ expect(plan.events.map(row => row.input.source_revision).sort()).toEqual(["a".repeat(40), "b".repeat(40)]);
+ expect(plan.events.map(row => row.scores.product_pass).sort()).toEqual([0, 1]);
+ expect(plan.events.map(row => row.id)).toEqual(buildBraintrustExperiment(projectId, campaign, originals).events.map(row => row.id));
+ expect(originals).toEqual(before);
+ const wrong = structuredClone(originals);
+ wrong[0]!.accountingIdentity!.campaignId = report.campaignId;
+ expect(() => buildBraintrustExperiment(projectId, report, wrong)).toThrow();
+});
+
+it("imports original grades when report evidence validation changes the selected outcome", () => {
+ const { campaign, originals } = fixture();
+ const adjusted = structuredClone(campaign);
+ adjusted.results[0]!.status = "failed";
+ adjusted.results[0]!.error = "Missing final-state.png";
+ const plan = buildBraintrustExperiment(projectId, adjusted, originals);
+ expect(plan.events.find(row => row.metadata.attempt_id === "second")!.scores.product_pass).toBe(1);
+ expect(plan).toEqual(buildBraintrustExperiment(projectId, campaign, originals));
+ expect(adjusted.results[0]!.status).toBe("failed");
+ const changed = structuredClone(originals);
+ changed[1]!.status = "failed";
+ expect(() => buildBraintrustExperiment(projectId, adjusted, changed)).toThrow();
+});
+
+it("accepts current catalog model names including Cursor qualification parameters", () => {
+ for (const model of new Set(runnerMatrix.map(cell => cell.profile.model))) {
+ const { campaign, originals } = fixture();
+ const changed = originals.map(original => ({ ...original, model }));
+ const selected = { ...campaign, results: campaign.results.map(result => ({ ...result, model })) };
+ const plan = buildBraintrustExperiment(projectId, selected, changed);
+ expect(plan.events.every(row => row.input.model === model)).toBe(true);
+ }
+});
+
+it.each(["", "model\nprivate", "model\u0000private", "model", "a".repeat(257)])("rejects malformed model names %j", model => {
+ const { campaign, originals } = fixture();
+ expect(() => buildBraintrustExperiment(projectId, campaign, originals.map(result => ({ ...result, model })))).toThrow();
+});
+
+it("keeps precise ledger amounts and reservations separate from trace metrics and unobserved attempts", () => {
+ const { campaign, originals } = fixture();
+ campaign.operations = { version: 1, scope: "shared_gateway_ledger", ledgerCampaignId: campaign.campaignId,
+ journalHash: "c".repeat(64), records: 5, deadlineAt: "2026-10-21T01:00:00Z", limitUsd: "100.000000000",
+ knownUsd: "0.000000001", heldUsd: "4.000000000", committedUsd: "4.000000001", requests: 1,
+ unresolvedRequests: 1, conflictingRequests: 0, inflight: 0,
+ breakdown: { estimated: "0.000000001", reported: "0.000000000", reconciled: "0.000000000", allocated: "0.000000000" },
+ attempts: [{ executionId: originals[0]!.executionId, attemptId: "first", requests: 1, unresolvedRequests: 1,
+ knownUsd: "0.000000001", heldUsd: "4.000000000" }] };
+ const plan = buildBraintrustExperiment(projectId, campaign, originals);
+ expect(plan.metadata.request_ledger).toMatchObject({ known_usd: "0.000000001", additional_held_usd: "4.000000000" });
+ expect(plan.events.find(row => row.metadata.attempt_id === "first")!.metadata.request_ledger).toMatchObject({ unresolved_requests: 1 });
+ expect(plan.events.find(row => row.metadata.attempt_id === "second")!.metadata.request_ledger).toBeNull();
+ expect(plan.events.every(row => Object.keys(row.metrics).sort().join() === "end,start")).toBe(true);
+ const carried = buildBraintrustExperiment(projectId, {
+ ...campaign, campaignId: "later-ci-attempt",
+ operations: { ...campaign.operations, ledgerCampaignId: "later-ci-attempt" },
+ }, originals);
+ expect(carried.metadata.request_ledger).toMatchObject({ requests: 1 });
+ expect(carried.events.every(row => row.metadata.request_ledger === null)).toBe(true);
+});
+
+function receiver(campaign: ReturnType["campaign"], originals: RunnerE2EResult[]) {
+ const plan = buildBraintrustExperiment(projectId, campaign, originals);
+ return vi.fn(async (url, init) => {
+ expect(String(url).startsWith("https://api.braintrust.dev/v1/")).toBe(true);
+ expect(init?.redirect).toBe("error");
+ if (String(url).endsWith("/summarize")) return Response.json({ experiment_name: plan.name,
+ project_url: "https://www.braintrust.dev/app/Example/p/Evals", experiment_url: `https://www.braintrust.dev/app/Example/p/Evals/experiments/${plan.name}` });
+ const sent = JSON.parse(String(init?.body));
+ if (String(url).endsWith("/experiment")) {
+ expect(sent).toMatchObject({ public: false, ensure_new: false, project_id: projectId, name: plan.name });
+ return Response.json({ ...sent, id: experimentId });
+ }
+ return Response.json({ row_ids: sent.events.map((row: { id: string }) => row.id) });
+ });
+}
+
+it("retries identical evidence using the same private experiment and stable row IDs without invoking agents", async () => {
+ const { campaign, originals } = fixture();
+ const transport = receiver(campaign, originals), receipts: BraintrustExperimentReceipt[] = [];
+ for (let i = 0; i < 2; i++) await importBraintrustExperiment({ projectId, apiKey: "private-key" }, campaign, originals,
+ async receipt => { receipts.push(receipt); }, transport);
+ expect(receipts.map(receipt => receipt.status)).toEqual(["created", "accepted", "accepted", "created", "accepted", "accepted"]);
+ expect(receipts[2]).toEqual(receipts[5]);
+ expect(receipts[1]).toMatchObject({ acceptedRows: 2, expectedRows: 2, verification: "unverified" });
+ expect(transport.mock.calls[0]![1]!.body).toBe(transport.mock.calls[3]![1]!.body);
+ expect(transport.mock.calls[1]![1]!.body).toBe(transport.mock.calls[4]![1]!.body);
+});
+
+it("preserves an evidence-bound experiment link through report regeneration and rejects stale or unsafe links", async () => {
+ const { campaign, originals } = fixture();
+ const receipt = await importBraintrustExperiment({ projectId, apiKey: "key" }, campaign, originals, async () => {}, receiver(campaign, originals));
+ campaign.braintrustExperiment = receipt;
+ const root = await mkdtemp(path.join(tmpdir(), "experiment-report-"));
+ try {
+ await writeFile(path.join(root, "normalized-results.json"), JSON.stringify(campaign));
+ await regenerateRunnerDashboard({ bundle: root, historyFile: null });
+ expect(JSON.parse(await readFile(path.join(root, "normalized-results.json"), "utf8")).braintrustExperiment).toEqual(receipt);
+ expect(await readFile(path.join(root, "dashboard.html"), "utf8")).toContain(receipt.experimentUrl);
+ expect(() => renderBraintrustExperiment({ ...campaign, braintrustExperiment: { ...receipt, experimentUrl: "https://untrusted.example/app/secret" } })).toThrow();
+ expect(() => renderBraintrustExperiment({ ...campaign, braintrustExperiment: { ...receipt, privateNote: "private-canary" } as never })).toThrow();
+ campaign.accounting!.missingAttempts++;
+ expect(() => renderBraintrustExperiment(campaign)).toThrow();
+ } finally { await rm(root, { recursive: true, force: true }); }
+});
+
+it("rebuilds a report with a missing cell and its saved experiment without rerunning retained attempts", async () => {
+ const root = await mkdtemp(path.join(tmpdir(), "experiment-missing-report-"));
+ const repositoryRoot = path.resolve(import.meta.dirname, "../..");
+ const original = fixture().originals[1]!;
+ const missing = runnerMatrix.find(cell => cell.id !== original.executionId)!;
+ const input = path.join(root, "input"), attempt = path.join(input, "attempt");
+ const receiptPath = path.join(root, "experiment.json");
+ async function report(output: string, savedReceipt?: string) {
+ const result = await promisify(execFile)(process.execPath,
+ [path.join(repositoryRoot, "cli/node_modules/tsx/dist/cli.mjs"), path.join(repositoryRoot, "tests/runner-e2e/report.ts")], {
+ cwd: repositoryRoot,
+ env: { ...process.env, PAPERCLIP_RUNNER_E2E_REPORT_ROOT: input, PAPERCLIP_RUNNER_E2E_REPORT_OUT: output,
+ PAPERCLIP_E2E_CAMPAIGN_ID: "retained-pilot", PAPERCLIP_RUNNER_E2E_EXPECTED_IDS: JSON.stringify([original.executionId, missing.id]),
+ PAPERCLIP_E2E_BRAINTRUST_EXPERIMENT: savedReceipt },
+ }).catch(error => error as { code: number; stderr: string });
+ // Missing coverage still fails the campaign, but must not prevent rendering it.
+ expect(result).toMatchObject({ code: 1 });
+ expect(result.stderr).not.toContain("Invalid retained experiment evidence");
+ return JSON.parse(await readFile(path.join(output, "normalized-results.json"), "utf8")) as RunnerE2ECampaign;
+ }
+ try {
+ await mkdir(attempt, { recursive: true });
+ const raw = JSON.stringify(original);
+ await writeFile(path.join(attempt, "result.json"), raw);
+ await writeFile(path.join(attempt, "final-state.png"), "fixture-png");
+ await writeFile(path.join(attempt, "evidence-manifest.json"), JSON.stringify({ files: ["final-state.png"], leaks: [], missing: [] }));
+ const before = await report(path.join(root, "before"));
+ const receipt = await importBraintrustExperiment({ projectId, apiKey: "fixture-key" }, before, [original], async () => {}, receiver(before, [original]));
+ await writeFile(receiptPath, JSON.stringify(receipt));
+ const after = await report(path.join(root, "after"), receiptPath);
+ expect(after.results.find(result => result.attempt === 0)!.startedAt).not.toBe(before.results.find(result => result.attempt === 0)!.startedAt);
+ expect(after.braintrustExperiment).toEqual(receipt);
+ expect(after.accounting).toEqual(before.accounting);
+ expect(await readFile(path.join(root, "after", "index.html"), "utf8")).toContain(receipt.experimentUrl);
+ expect(await readFile(path.join(attempt, "result.json"), "utf8")).toBe(raw);
+ const changed = structuredClone(after);
+ changed.results.find(result => result.attempt > 0)!.startedAt = "2026-10-20T00:00:30.000Z";
+ expect(() => renderBraintrustExperiment(changed)).toThrow();
+ } finally { await rm(root, { recursive: true, force: true }); }
+});
+
+it("checkpoints accepted batches before a later insert fails", async () => {
+ const base = fixture().originals[0]!;
+ const originals = Array.from({ length: 101 }, (_, i) => ({ ...base, attempt: i + 1,
+ accountingIdentity: { campaignId: "retained-pilot", attemptId: `attempt-${i}` } }));
+ const campaign = buildRunnerCampaign({ campaignId: "retained-pilot", generatedAt: base.finishedAt,
+ expected: [base.executionId], results: [originals[100]!], attempts: originals });
+ const normal = receiver(campaign, originals), receipts: BraintrustExperimentReceipt[] = [];
+ let calls = 0;
+ await expect(importBraintrustExperiment({ projectId, apiKey: "key" }, campaign, originals,
+ async receipt => { receipts.push(receipt); }, async (url, init) => {
+ if (++calls === 3) throw new Error("private-network-error");
+ return normal(url, init);
+ })).rejects.toThrow("Remote details withheld");
+ expect(receipts).toHaveLength(2);
+ expect(receipts[1]).toMatchObject({ status: "created", acceptedRows: 100, expectedRows: 101 });
+ const previous = structuredClone(receipts[1]!);
+ calls = 0;
+ await expect(importBraintrustExperiment({ projectId, apiKey: "key" }, campaign, originals,
+ async receipt => { receipts.push(receipt); }, async (url, init) => {
+ if (++calls === 2) throw new Error("retry insert failed");
+ return normal(url, init);
+ }, previous)).rejects.toThrow("Remote details withheld");
+ expect(receipts).toHaveLength(2);
+ expect(receipts.at(-1)).toEqual(previous);
+ await importBraintrustExperiment({ projectId, apiKey: "key" }, campaign, originals,
+ async receipt => { receipts.push(receipt); }, normal, previous);
+ expect(receipts.slice(2).map(receipt => receipt.acceptedRows)).toEqual([100, 101, 101]);
+ expect(receipts.at(-1)!.status).toBe("accepted");
+});
+
+it.each(["source", "definition"])("rejects a saved link and original import after selected %s provenance changes", async kind => {
+ const { campaign, originals } = fixture();
+ const receipt = await importBraintrustExperiment({ projectId, apiKey: "key" }, campaign, originals, async () => {}, receiver(campaign, originals));
+ const changed = structuredClone(campaign);
+ changed.braintrustExperiment = receipt;
+ if (kind === "source") changed.results[0]!.source!.sha = "e".repeat(40);
+ else changed.results[0]!.suiteDefinitionHash = "f".repeat(64);
+ expect(() => renderBraintrustExperiment(changed)).toThrow();
+ expect(() => buildBraintrustExperiment(projectId, changed, originals)).toThrow();
+ expect(buildBraintrustExperiment(projectId, campaign, originals).events.map(row => row.input.source_revision).sort())
+ .toEqual(["a".repeat(40), "b".repeat(40)]);
+});
+
+it("propagates a failed receipt save after successful summary lookup while retaining the accepted checkpoint", async () => {
+ const { campaign, originals } = fixture(), receipts: BraintrustExperimentReceipt[] = [];
+ await expect(importBraintrustExperiment({ projectId, apiKey: "key" }, campaign, originals, async receipt => {
+ if (receipt.experimentUrl) throw new Error("disk write failed");
+ receipts.push(receipt);
+ }, receiver(campaign, originals))).rejects.toThrow("import incomplete");
+ expect(receipts.at(-1)).toMatchObject({ status: "accepted", acceptedRows: 2 });
+ expect(receipts.at(-1)!.experimentUrl).toBeUndefined();
+});
+
+it("rejects conflicting retry receipts before overwriting saved evidence", async () => {
+ const { campaign, originals } = fixture();
+ const receipt = await importBraintrustExperiment({ projectId, apiKey: "key" }, campaign, originals, async () => {}, receiver(campaign, originals));
+ const persist = vi.fn(), transport = vi.fn();
+ await expect(importBraintrustExperiment({ projectId, apiKey: "key" }, campaign, originals, persist, transport,
+ { ...receipt, projectId: experimentId })).rejects.toThrow();
+ expect(persist).not.toHaveBeenCalled();
+ expect(transport).not.toHaveBeenCalled();
+});
+
+it.each(["public", "wrong-project", "wrong-hash", "partial-insert", "receiver-error", "oversized"])("retains safe incomplete status for %s", async kind => {
+ const { campaign, originals } = fixture(), receipts: BraintrustExperimentReceipt[] = [];
+ const normal = receiver(campaign, originals);
+ const transport: typeof fetch = async (url, init) => {
+ const response = await normal(url, init), value = await response.json() as Record;
+ if (String(url).endsWith("/experiment")) {
+ if (kind === "public") value.public = true;
+ if (kind === "wrong-project") value.project_id = experimentId;
+ if (kind === "wrong-hash") value.metadata = { payload_hash: "a".repeat(64) };
+ if (kind === "oversized") return new Response("x".repeat(1_048_577));
+ } else {
+ if (kind === "partial-insert") value.row_ids = [];
+ if (kind === "receiver-error") return new Response("private-upstream-failure", { status: 503 });
+ }
+ return Response.json(value);
+ };
+ await expect(importBraintrustExperiment({ projectId, apiKey: "private-key" }, campaign, originals,
+ async receipt => { receipts.push(receipt); }, transport)).rejects.toThrow("Remote details withheld");
+ expect(receipts.some(receipt => receipt.status === "accepted")).toBe(false);
+ expect(JSON.stringify(receipts)).not.toMatch(/private-key|private-upstream/);
+ if (["partial-insert", "receiver-error"].includes(kind)) expect(receipts[0]).toMatchObject({ status: "created", acceptedRows: 0 });
+});
diff --git a/tests/runner-e2e/braintrust-experiment.ts b/tests/runner-e2e/braintrust-experiment.ts
new file mode 100644
index 00000000000..4b2859e17d8
--- /dev/null
+++ b/tests/runner-e2e/braintrust-experiment.ts
@@ -0,0 +1,207 @@
+import { createHash } from "node:crypto";
+import { buildCampaignAccounting } from "./attempt-accounting.js";
+import { exportBraintrustAttemptParent, type EvalBraintrustConfig } from "./braintrust.js";
+import { validateCampaignFeedbackReview } from "./feedback-dispositions.js";
+import { validateEvalOperationsReport } from "./operations-report.js";
+import { html, isPublicReportTimestamp } from "./report-format.js";
+import type { RunnerE2ECampaign, RunnerE2EResult } from "./types.js";
+
+const uuid = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
+const digest = (value: unknown) => createHash("sha256").update(JSON.stringify(value)).digest("hex");
+const identifier = (value: unknown): value is string => typeof value === "string" && /^[A-Za-z0-9][A-Za-z0-9._:/-]{0,255}$/.test(value);
+const modelName = (value: unknown): value is string => typeof value === "string" && /^[A-Za-z0-9][A-Za-z0-9._:/@=,\[\]-]{0,255}$/.test(value);
+const count = (value: unknown): value is number => Number.isSafeInteger(value) && Number(value) >= 0;
+const invalid = (): never => { throw new Error("Invalid retained experiment evidence"); };
+
+function attemptProjection(result: RunnerE2EResult, projectId: string) {
+ if (result.schema !== "paperclip.runner-e2e.result/v2"
+ || ![result.executionId, result.suiteId, result.profileId, result.caseId, result.provider,
+ result.accountingIdentity?.campaignId, result.accountingIdentity?.attemptId].every(identifier)
+ || !modelName(result.model)
+ || !/^[0-9a-f]{64}$/.test(result.suiteDefinitionHash ?? "") || !/^[0-9a-f]{40}$/.test(result.source?.sha ?? "")
+ || !count(result.attempt) || result.attempt < 1 || !count(result.durationMs)
+ || !["passed", "failed"].includes(result.status) || !["passed", "failed", "not_started"].includes(result.cleanup)
+ || !["native", "legacy"].includes(result.runtimeMode) || !["local", "daytona"].includes(result.environmentId)
+ || !isPublicReportTimestamp(result.startedAt) || !isPublicReportTimestamp(result.finishedAt)
+ || Date.parse(result.finishedAt) < Date.parse(result.startedAt)) invalid();
+ const feedback = result.feedback;
+ if (feedback && (![feedback.complaints, feedback.suggestions, feedback.invocations?.failed, feedback.invocations?.unknown].every(count)
+ || !["complete", "partial", "unavailable", "not_started"].includes(feedback.collection)
+ || !["complete", "partial", "unavailable", "not_started"].includes(feedback.invocationCollection))) invalid();
+ const trace = result.tracing;
+ if (trace) {
+ exportBraintrustAttemptParent(trace);
+ if (trace.projectId !== projectId || !["pending", "accepted", "failed"].includes(trace.status)) invalid();
+ }
+ return {
+ input: { execution_id: result.executionId, suite_id: result.suiteId, profile_id: result.profileId,
+ case_id: result.caseId, provider: result.provider, model: result.model, environment: result.environmentId,
+ runtime: result.runtimeMode, source_revision: result.source!.sha, definition_hash: result.suiteDefinitionHash },
+ output: { product: result.status, cleanup: result.cleanup },
+ scores: { product_pass: Number(result.status === "passed"), cleanup_pass: Number(result.cleanup === "passed") },
+ metrics: { start: Date.parse(result.startedAt) / 1000, end: Date.parse(result.finishedAt) / 1000 },
+ metadata: { campaign_id: result.accountingIdentity!.campaignId, attempt_id: result.accountingIdentity!.attemptId,
+ attempt_number: result.attempt, wall_duration_ms: result.durationMs, grader: "tests/runner-e2e/runner.spec.ts",
+ feedback: feedback ? { collection: feedback.collection, invocation_collection: feedback.invocationCollection,
+ complaints: feedback.complaints, suggestions: feedback.suggestions,
+ failed_submissions: feedback.invocations.failed, unresolved_submissions: feedback.invocations.unknown } : null,
+ trace: trace ? { project_id: trace.projectId, root_span_id: trace.traceId, span_id: trace.spanId,
+ export_status: trace.status, indexed_verification: "unverified" } : null },
+ span_attributes: { name: result.executionId, type: "eval" },
+ };
+}
+
+export function buildBraintrustExperiment(projectId: string, campaign: RunnerE2ECampaign, originals: readonly RunnerE2EResult[]) {
+ if (!uuid.test(projectId) || campaign.schema !== "paperclip.runner-e2e.campaign/v2" || !identifier(campaign.campaignId)
+ || campaign.accounting?.scope !== "retained_attempts" || !Array.isArray(campaign.accounting.attempts)
+ || !count(campaign.accounting.missingAttempts) || !Array.isArray(campaign.expected)
+ || campaign.expected.length === 0 || !campaign.expected.every(identifier)
+ || new Set(campaign.expected).size !== campaign.expected.length || !Array.isArray(originals)
+ || originals.length > 10_000 || campaign.accounting.attempts.length === 0) invalid();
+ if (campaign.operations) validateEvalOperationsReport(campaign.operations);
+ if (campaign.feedbackReview) validateCampaignFeedbackReview(campaign, campaign.feedbackReview);
+ const accounting = campaign.accounting!;
+ const operations = campaign.operations?.status !== "unavailable" ? campaign.operations : undefined;
+ if (operations && operations.ledgerCampaignId !== campaign.campaignId) invalid();
+ const candidates = new Map }>();
+ for (const original of originals) {
+ const projection = attemptProjection(original, projectId);
+ if (!campaign.expected.includes(original.executionId)) invalid();
+ const receipt = buildCampaignAccounting({ results: [original] }).attempts[0]!;
+ const retained = accounting.attempts.find(attempt => attempt.id === receipt.id);
+ if (!retained || JSON.stringify(retained) !== JSON.stringify(receipt)) invalid();
+ const previous = candidates.get(receipt.id);
+ if (previous && digest(previous.projection) !== digest(projection)) invalid();
+ candidates.set(receipt.id, { original, projection });
+ }
+ if (candidates.size !== accounting.attempts.length) invalid();
+ for (const selected of campaign.results.filter(result => result.attempt > 0)) {
+ const id = buildCampaignAccounting({ results: [selected] }).attempts[0]!.id;
+ const original = candidates.get(id);
+ const selectedProjection = attemptProjection(selected, projectId);
+ if (!original || digest(original.projection.input) !== digest(selectedProjection.input)
+ || digest(original.projection.metrics) !== digest(selectedProjection.metrics)
+ || digest(original.projection.metadata) !== digest(selectedProjection.metadata)) invalid();
+ }
+ const events = [...candidates.entries()].sort(([a], [b]) => a.localeCompare(b)).map(([id, { original, projection }]) => {
+ const cost = operations && original.accountingIdentity!.campaignId === operations.ledgerCampaignId
+ ? operations.attempts.find(row => row.executionId === original.executionId && row.attemptId === original.accountingIdentity!.attemptId)
+ : undefined;
+ return { ...projection, id, span_id: id, root_span_id: id,
+ metadata: { ...projection.metadata, request_ledger: cost ? {
+ known_usd: cost.knownUsd, additional_held_usd: cost.heldUsd, requests: cost.requests,
+ unresolved_requests: cost.unresolvedRequests, journal_hash: operations!.journalHash,
+ } : null } };
+ });
+ const metadata = { importer: "paperclip-retained-attempts-v1", campaign_id: campaign.campaignId,
+ expected_execution_ids: [...campaign.expected].sort(), retained_attempts: events.length,
+ missing_attempts: accounting.missingAttempts,
+ request_ledger: operations ? { journal_hash: operations.journalHash, known_usd: operations.knownUsd,
+ additional_held_usd: operations.heldUsd, committed_usd: operations.committedUsd,
+ requests: operations.requests, unresolved_requests: operations.unresolvedRequests, breakdown: operations.breakdown } : null,
+ feedback_review: campaign.feedbackReview ?? null,
+ accounting_note: "Ledger amounts are separate observations, not additive trace metrics. Held reservations are not charges. Requests outside the gateway and infrastructure require separate accounting." };
+ const payloadHash = digest({ projectId, metadata, events });
+ return { projectId, payloadHash, name: `${campaign.campaignId}-${payloadHash}`, metadata: { ...metadata, payload_hash: payloadHash }, events };
+}
+
+export interface BraintrustExperimentReceipt {
+ version: 1; projectId: string; experimentId: string; name: string; payloadHash: string;
+ status: "created" | "accepted"; acceptedRows: number; expectedRows: number; verification: "unverified";
+ evidenceHash: string; projectUrl?: string; experimentUrl?: string;
+}
+
+export function experimentEvidenceHash(campaign: RunnerE2ECampaign) {
+ return digest({ campaignId: campaign.campaignId, expected: [...campaign.expected].sort(),
+ selected: campaign.results.map(result => ({ executionId: result.executionId, accountingIdentity: result.accountingIdentity,
+ attempt: result.attempt, startedAt: result.attempt === 0 ? null : result.startedAt, sourceRevision: result.source?.sha ?? null,
+ suiteDefinitionHash: result.suiteDefinitionHash ?? null, suiteId: result.suiteId, profileId: result.profileId,
+ caseId: result.caseId, provider: result.provider, model: result.model, environmentId: result.environmentId,
+ runtimeMode: result.runtimeMode })).sort((a, b) => digest(a).localeCompare(digest(b))),
+ accounting: campaign.accounting, operations: campaign.operations ?? null, feedbackReview: campaign.feedbackReview ?? null });
+}
+
+function safeBraintrustUrl(value: unknown): value is string {
+ if (typeof value !== "string") return false;
+ try {
+ const url = new URL(value);
+ return url.protocol === "https:" && ["braintrust.dev", "www.braintrust.dev"].includes(url.host)
+ && !url.username && !url.password && !url.search && !url.hash && url.pathname.startsWith("/app/");
+ } catch { return false; }
+}
+
+export function validateBraintrustExperimentReceipt(campaign: RunnerE2ECampaign, receipt: BraintrustExperimentReceipt): void {
+ if (!receipt || Object.keys(receipt).some(key => !["version", "projectId", "experimentId", "name", "payloadHash", "status", "acceptedRows", "expectedRows", "verification", "evidenceHash", "projectUrl", "experimentUrl"].includes(key))
+ || receipt.version !== 1 || !uuid.test(receipt.projectId) || !uuid.test(receipt.experimentId)
+ || receipt.evidenceHash !== experimentEvidenceHash(campaign) || !/^[a-f0-9]{64}$/.test(receipt.payloadHash)
+ || receipt.name !== `${campaign.campaignId}-${receipt.payloadHash}` || !["created", "accepted"].includes(receipt.status)
+ || !count(receipt.acceptedRows) || receipt.expectedRows !== campaign.accounting?.attempts.length
+ || receipt.acceptedRows > receipt.expectedRows || receipt.verification !== "unverified"
+ || receipt.status === "accepted" && receipt.acceptedRows !== receipt.expectedRows
+ || receipt.experimentUrl !== undefined && !safeBraintrustUrl(receipt.experimentUrl)
+ || receipt.projectUrl !== undefined && !safeBraintrustUrl(receipt.projectUrl)) invalid();
+}
+
+export function renderBraintrustExperiment(campaign: RunnerE2ECampaign): string {
+ const receipt = campaign.braintrustExperiment;
+ if (!receipt) return "";
+ validateBraintrustExperimentReceipt(campaign, receipt);
+ return `
Braintrust experiment
${receipt.experimentUrl
+ ? `Open retained attempt scores in Braintrust`
+ : `Experiment ${html(receipt.experimentId)}`}. ${receipt.acceptedRows}/${receipt.expectedRows} rows accepted.
+ Original product and cleanup grades are separate. Indexing requires independent verification; no agents were rerun for this import.
Model spend is the provider-reported subtotal; unpriced or unavailable runs are excluded, never counted as free. Daytona runtime is a public-list-price estimate from captured lease time and pinned resources, before credits, discounts, storage allowance, or invoice adjustments. Local execution has no external runtime meter.
-
+
Campaign costs include retained attempts, including failed retries. Result cards show the selected attempt. Provider-reported amounts and model estimates are separate; unpriced or unavailable runs are excluded, never counted as free. Coverage describes captured usage; unobserved setup, subscriptions, tools, CI and storage require additional accounting. Daytona runtime is a public-list-price estimate from captured lease time and pinned resources, before credits, discounts, storage allowance, or invoice adjustments. Local execution has no external runtime meter.
Full bodies and review rationale stay in private evidence. Exact duplicate bodies are grouped once; related causes can be linked by a reviewed duplicate disposition.