Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-10-22 16:51:28 +02:00
Code Issues Packages Projects Releases Wiki Activity

test(runner): add full-stack acceptance and eval gates (#12700)

Browse Source 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The runner subsystem executes agent work across local and managed
provider backends.
> - The lower pull requests restore the task runtime, provider backends,
and managed-provider control plane.
> - The restored system needs repeatable full-stack checks before it can
ship safely.
> - Paid live checks also need clear access, cost, and secret controls.
> - This pull request adds acceptance, live evaluation, chaos, and
release gates for the restored runner stack.
> - The benefit is measurable runner parity with safer release
decisions.

## Linked Issues or Issue Description

**Subsystem affected**

Cross-cutting. This change covers runner tests, release workflows,
server contracts, and evaluation tools.

**Problem or motivation**

The runner stack did not have one complete acceptance surface for native
Codex, ACPX, Claude Managed, and AWS AgentCore. Release checks could
miss provider drift, task-view regressions, cost-policy errors, and
destructive cleanup errors.

**Proposed solution**

Add a 57-cell full-stack catalog, a Daytona image, and opt-in paid
workflows. Add live evaluation, chaos, cost-limit, redaction, and
release contract checks. Add AWS AgentCore infrastructure and guarded
provisioning tools. Keep the native runner experimental flag off by
default.

**Alternatives considered**

We considered manual smoke tests only. They do not give repeatable
evidence and they do not protect release branches. We also considered
one large pull request. The stacked pull requests keep each review below
the Greptile file limit.

**Roadmap alignment**

This work supports the shipped Cloud / Sandbox agents milestone and the
shipped Agent evals & feedback milestone in `ROADMAP.md`.

Related stack:

- #12699 adds managed provider backends and lifecycle support.
- #12691 adds qualified OpenCode and ACPX provider backends.
- #12685 restores task runtime rendering and steering.

## What Changed

- Add the runner full-stack harness with 57 catalog cells and 60 unit
tests.
- Add a Daytona runner image with digest-pinned base images and
base-aware image-content checks.
- Add guarded live evaluation and chaos workflows with a fixed
40-execution matrix; live and full-stack paid schedules now run only on
Sundays or by manual dispatch.
- Add in-flight reported-usage cost stops, post-turn cost caps,
exact-threshold failure classification, secret redaction, retry
classification, and actor authorization.
- Reattach stream and hard-budget listeners before restart-recovery
continuations so restored paid sessions cannot bypass in-flight
interruption.
- Preserve OpenCode usage and cost across tool-loop messages and turns
while exposing an explicit current-run delta to durable accounting.
- Keep PNG/WebM evidence in access-controlled artifacts only, reject
SVG, and publish only pruned inert structured per-attempt evidence.
- Add AWS AgentCore infrastructure, provisioning checks, and smoke
tools; reject unsafe model identifiers, require exact stack ownership
markers, and make failed-stack replacement explicit.
- Add evaluation-session contracts and capability reports.
- Add release workflow checks for immutable action pins, frozen
dependency installs, exact weekly cron shape, paid-run guards,
provider-secret isolation, and chaos test paths.
- Reauthorize the original and triggering numeric actor IDs as the first
step of every provider-secret job, including partial reruns, before
checkout or provider access.
- Give each full-stack matrix cell only its matching provider
credential, expose Daytona only to Daytona cells, and disable shared
dependency caches anywhere paid credentials or OIDC write access are
present.
- Protect the legacy manual E2E workflow with the same default-branch,
allowlist, environment, and per-job authorization boundary.
- Rotate live-eval candidates by week and retain 120 days of compatible
history so the seven-week trend window remains viable.
- Restore the root runner-acceptance commands and reconcile reported
snapshots,
raw receipts, and terminal usage without double counting or losing late
usage.
- Mark ACPX token deltas exact only when every budget field is present,
keep
cumulative cost/request authority separate, reject non-USD cost
labeling,
  and include thought tokens in output-token budgets.
- Keep `enableNativeRunner` off by default. The acceptance harness
enables it only in its isolated test instance.

## Verification

Passed locally:

- `pnpm --filter @paperclipai/paperclip-runner typecheck`
- `pnpm test:runner-acceptance:typecheck`
- `pnpm test:runner-acceptance` (19 tests)
- focused OpenCode proxy, driver, runnerd transport, live-session, and
turn-stream tests (106 tests)
- `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/live/clean-room-server.test.ts` (22 tests)
- `pnpm test:e2e:runner:typecheck`
- `pnpm test:e2e:runner:unit` (62 tests)
- `node --test scripts/__tests__/release-verify-workflow.test.mjs`
- `pnpm --filter @paperclipai/paperclip-runner
test:runner-workflow-evals` (22 tests)
- `pnpm -r typecheck`
- `pnpm build`
- `node --test
packages/paperclip-runner/scripts/aws-agentcore-provisioning.test.mjs`
(6 tests)
- `git diff --check`
- `cargo test --manifest-path
packages/paperclip-runner/runner/Cargo.toml -p paperclip-runner-core
--lib --locked` (161 tests)
- focused ACPX provider-event tests (10 tests)
- The rebased PR changes 92 files. `pnpm-lock.yaml` is unchanged.

I did not run paid live provider jobs or provision AWS resources. Those
checks need credentials and can create cost.

## Risks

The paid workflows can create provider cost. They require an allowlisted
original and triggering actor, the protected `runner-e2e-paid`
environment, explicit opt-in variables, and cost limits. The four
provider credentials exist only in that master-only environment, which
requires allowlisted reviewer approval and disables administrator
bypass; repository and organization Actions scopes contain no copies.

Provider usage arrives after a billable request, so the live guard
cannot prevent one request from crossing a threshold. It interrupts
immediately on the first reported threshold hit and permits no
continuation.

Visual evidence can contain secrets rendered as pixels. PNG/WebM remain
only in access-controlled workflow artifacts; SVG and per-attempt XML
are excluded, and S3/Pages receive a pruned structured dashboard.

The AWS scripts can create cloud resources. They use explicit commands,
least-privilege roles, KMS encryption, saved nonsecret metadata, and
explicit teardown.

This pull request does not enable the experimental native runner for
existing instances.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

OpenAI Codex with GPT-5. The model used extended reasoning, tool use,
code execution, and parallel subagents.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
This commit is contained in:
Dotta authored and GitHub committed 2026-09-02 08:55:08 -05:00
1 parent 109d81db4f
commit 5716fe907e
92 files changed
+19537 -285
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

tests/runner-e2e/FIXTURES.md
+149
View File
@@ -0,0 +1,149 @@
# Runner E2E fixture authoring
The fixture catalog is executable production-contract data. Keep it small,
typed, deterministic, and free of raw credentials.
## Suites and matrices
A `RunnerSuiteFixture` declares one durable testing purpose: stable ID, label,
description, profiles, environments, cases, expected size, and definition or
ranking metadata. Its execution IDs are globally prefixed as
`<suite>.<profile>.<environment>.<case>`. Add a new suite when the testing
purpose or desired cross-product differs; do not inflate an existing suite with
unrelated dimensions.
The suite definition fingerprint is historical comparison metadata. Any
profile, model qualification, environment, task, or ranking-snapshot change
must change that fingerprint automatically so the dashboard can annotate the
boundary instead of silently joining unlike totals.
## Agent profiles
Add `RunnerProfileFixture` entries in `catalog.ts`. A profile declares:
- a stable ID and searchable groups;
- legacy or native generation;
- adapter/provider and required credential;
- a model imported from its adapter constant or qualified runner profile;
- supported environment IDs;
- expected runtime metadata; and
- an agent payload factory.
Do not duplicate model IDs, qualification decisions, CLI versions, or runner
artifact rules. Codex profiles import `DEFAULT_CODEX_LOCAL_MODEL`, OpenCode
profiles import `QUALIFIED_OPENCODE_MODEL`, and ACPX profiles import
`QUALIFIED_ACPX_PROFILES`. Add or qualify models at their owning production
source first.
OpenRouter breadth profiles are generated from `openrouter-models.json`, not
written by hand. That reviewed snapshot must contain exactly five unique,
available, tool-capable models with rank, canonical ID, display name, supported
parameters, source URL, capture time, and verified content hash. Refresh it
manually with `pnpm test:e2e:runner:models:update`; nightly campaigns never
change fixture definitions.
Agent `adapterConfig.env` values must be `{type:"secret_ref", secretId,
version:"latest"}` objects supplied to the factory. A fixture source containing
a raw secret-looking value is rejected by catalog validation.
## Environments
An `EnvironmentFixture` declares driver/provider, credential requirements,
attempt deadline, lifecycle behavior, expected execution target, and a payload
factory validated by the shared environment schema.
The local environment is instance-managed: company creation ensures it exists,
and the public API intentionally rejects a second local environment. The setup
registry therefore discovers that row through the public environments API.
This still provides full isolation because every cell starts a new Paperclip
instance and database.
Daytona creates a sandbox environment through the public API. Keep
`reuseLease:false`, `runnerLifecycleMode:"per_turn"`, short provider cleanup
backstops, a Daytona secret reference, and an immutable image digest. Teardown
must delete the environment with reusable-lease destruction and must fail the
cell if cleanup cannot be confirmed. Keep CPU, memory, and disk explicit: lease
metadata and the per-test public-list-price runtime estimate depend on that
pinned billable resource shape. Changing it requires updating billing tests and
reviewing the versioned Daytona rates in `billing.ts`.
## Usage and billing data
Do not add fixture-authored token or dollar expectations. The live harness
reads usage from selected public heartbeat-run records and records coverage per
run. Provider-reported dollars remain distinct from runtime estimates. A zero
or missing native usage payload is `unavailable` unless a real token-bearing
receipt or provider cost proves otherwise. New execution environments must
provide lease/resource metadata for a runtime estimate or explicitly remain
`unavailable`; never infer that missing billing data means free execution.
Future providers (SSH, E2B, Modal, Cloudflare, Kubernetes, Novita, exe.dev)
should implement the same setup/probe/cleanup contract before being added to a
matrix. Unsupported profile/environment combinations belong in
`supportedEnvironments`, not in ad hoc test conditionals.
## Task cases and matchers
A `RunnerTaskFixture` owns a work mode, a typed flow, expected run count,
nonce-based title/prompt/marker factories, per-environment attempt deadlines,
deterministic matchers, and expected terminal state. Single-turn prompts should
make one bounded request with observable output and no nondeterministic judging.
The `plan_revision_acceptance` flow must also provide revision-request and Plan
marker factories. `question_resume_completion` must define the deterministic
browser answer and prove exactly two successful runs with no pending
interaction. `plan_approval_completion` must target the exact two-step
canonical Plan revision, capture its pending UI, approve in the browser, and
prove exactly two successful runs.
Every selected case runs in its own isolated Paperclip process, and independent
cases may run concurrently. Follow-up turns inside one case retain their shared
task state. Each case creates and tears down its own company, secrets,
environment selection, agent, and browser-created task. The current plan case
proves three runs on the same issue: publish a two-step Plan,
request a three-step revision through the UI, and accept the exact new revision
through the UI before verifying implementation and Done.
The matcher union supports message exact/contains/regex/ordered checks, issue
and run state, runtime/environment metadata, files, artifacts, JSON paths, and
JSON Schema. The initial cases use normalized `message_contains` plus state,
runtime, and environment assertions; the plan flow additionally verifies
canonical document revision IDs, bodies, step counts, interaction targets, and
visible previews. Add matcher behavior and credential-free tests together.
Adding a task expands its suite's matrix. Update the suite's intentional size,
the complete-catalog size, and credential-free unit tests in the same change.
Paid tests never silently skip a missing credential or unsupported artifact.
## New Paperclip object fixtures
Register new objects in `live-fixtures.ts` with explicit dependencies in
`FixtureRegistry`. Setup must use a public API. Teardown runs in reverse order
and is invoked after partial setup failures. Direct database writes and private
test-only runner endpoints are prohibited.
The expected dependency shape is:
```text
company
└── encrypted secrets
└── environment
└── agent
└── browser-created task
```
Projects, goals, apps, and configuration fixtures can be inserted into that
graph without changing the launcher. Keep returned fixture state to IDs and
sanitized metadata; never retain raw secret values.
## Required checks
Run before a fixture change is reviewed:
```bash
pnpm test:e2e:runner:unit
pnpm test:e2e:runner:typecheck
pnpm test:e2e:runner -- --list
```
Then run the narrowest paid cell that exercises the fixture. A full matrix is a
manual or scheduled campaign, not a PR requirement.
tests/runner-e2e/README.md
+328
View File
@@ -0,0 +1,328 @@
# Paid runner full-stack E2E
This is the billable browser acceptance campaign system for Paperclip runner
profiles. It is deliberately separate from `tests/e2e`: every independently
scheduled execution gets
a fresh Paperclip home, embedded Postgres database, instance configuration,
port, workspace, company, encrypted secrets, environment, and agent.
The vocabulary is: a **campaign** is one workflow invocation against one SHA; a
**suite** is a durable testing purpose; a **matrix** is that suite's profiles ×
environments × cases; an **execution/cell** is one parallel job; and an
**attempt** is one isolated harness run, including an infrastructure retry.
The browser creates and assigns the task. The harness does not call a private
runner hook or write fixtures directly to the database.
## Credentials
Copy `.env.runner-e2e.example` to `.env.runner-e2e.local` and fill only the
credentials needed by the selected cells:
```bash
cp .env.runner-e2e.example .env.runner-e2e.local
chmod 600 .env.runner-e2e.local
```
Shell variables take precedence over the local file. The recognized names are:
- `OPENAI_API_KEY`
- `ANTHROPIC_API_KEY`
- `OPENROUTER_API_KEY`
- `DAYTONA_API_KEY`
- `PAPERCLIP_E2E_DAYTONA_IMAGE` (Daytona only)
The image must be an immutable `image@sha256:...` reference. The launcher
reports missing variable names but never prints values. It passes raw provider
keys only to Playwright, which posts each value once to the company-secrets API.
Paperclip receives secret references in agent/environment payloads. Provider
keys, Daytona keys, `DATABASE_URL`, and `DATABASE_MIGRATION_URL` are removed
from the Paperclip child process.
Never put credentials in `catalog.ts`, screenshots, fixture metadata, workflow
inputs, or a tracked env file.
## Local commands
Install dependencies and Chromium once. Native local cells also need the local
runner binaries:
```bash
pnpm install
pnpm exec playwright install chromium
pnpm --filter @paperclipai/paperclip-runner build:runner-binaries
```
List cells without loading credentials or starting Paperclip:
```bash
pnpm test:e2e:runner -- --list
```
Examples of explicit billable runs:
```bash
pnpm test:e2e:runner -- --id core-compatibility.legacy-codex.local.message-marker --headed
pnpm test:e2e:runner -- --suite openrouter-model-breadth --case hello-complete
pnpm test:e2e:runner -- --group native --environment local
pnpm test:e2e:runner -- --profile runner-codex --case message-marker
pnpm test:e2e:runner -- --case plan-revise-accept --group local
pnpm test:e2e:runner -- --case ask-question --group native
pnpm test:e2e:runner -- --all
```
The catalog contains two suites. `core-compatibility` (**Core Runner
Compatibility**) is seven major runner profiles × local/Daytona × three
workflows: 42 cells. Its cases are:
- `message-marker`: one basic visible response and Done transition;
- `plan-revise-accept`: an initial Plan, a browser-requested revision on the
same Plan, browser acceptance of the new revision, and verified execution;
- `ask-question`: a direct answer from a task created in Ask mode.
`openrouter-model-breadth` (**OpenRouter Model Breadth**) is five models from
the tracked weekly tool-capable ranking snapshot × native OpenCode × local ×
three workflows: 15 cells. Its cases are:
- `hello-complete`: a basic nonce response and explicit Done transition;
- `question-resume-complete`: one structured question, browser selection of
“Cobalt,” then a resumed completion on the same task; and
- `plan-approve-complete`: one exact two-step Plan, browser approval of that
revision, then a resumed completion on the same task.
The complete catalog is 57 cells and 95 expected paid agent turns. Follow-up
steps remain ordered within their cell; all other cells are independent.
Narrow selectors are strongly recommended while developing fixtures.
`--suite`, `--group`, `--profile`, `--environment`, and `--case` are repeatable. Repeated
values in one dimension use OR semantics; dimensions and repeated groups use
AND semantics. `--id` is exclusive with dimension selectors and `--all`.
`--headed`, `--ui`, and `--debug` are forwarded to Playwright. An unknown
selector, an empty selection, or a run with no explicit selector exits before
Paperclip starts. `--max-parallel <n>` controls the number of isolated
profile/environment/case harnesses that can overlap (default 1, also configurable
with `PAPERCLIP_E2E_MAX_PARALLEL`). Headed/UI/debug runs are forced to one worker.
The Plan case is still sequential internally because its turns share one task;
it runs in parallel with unrelated scenarios.
Use a single `--id` smoke test for routine local verification. Full-matrix
parallelism is intended for GitHub Actions; raising local parallelism starts
multiple Paperclip/Postgres/Chromium stacks and can consume substantial CPU and
memory.
Credential-free checks are:
```bash
pnpm test:e2e:runner:unit
pnpm test:e2e:runner:typecheck
```
The OpenRouter ranking snapshot is tracked in `openrouter-models.json`; nightly
runs never mutate it. Refresh it deliberately, review the source/capture/hash
diff, and rerun credential-free checks:
```bash
pnpm test:e2e:runner:models:update
```
## Daytona image
Use the immutable digest printed by the `Publish verified Daytona image` job,
or publish the current source locally:
```bash
content_id="$(pnpm --silent test:e2e:runner:image-id)"
source_revision="$(git rev-parse HEAD)"
image="ghcr.io/paperclipai/paperclip-daytona-runner:e2e-content-${content_id}"
if ! docker buildx imagetools inspect "$image" >/dev/null 2>&1; then
docker buildx build \
--platform linux/amd64 \
--build-arg "PAPERCLIP_RUNNER_CONTENT_ID=${content_id}" \
--build-arg "PAPERCLIP_RUNNER_SOURCE_REVISION=${source_revision}" \
--file docker/daytona-runner/Dockerfile \
--tag "$image" \
--push \
.
fi
docker buildx imagetools inspect "$image"
```
The content ID hashes the audited image inputs, including the Dockerfile,
platform, root package/lock/build configuration, dependency patches,
`paperclip-eval-kernel`, and `paperclip-runner`. Changes elsewhere in the
repository keep the same tag and reuse the already signed image. The Git SHA is
stored separately as image provenance. CI reads that provenance back from a
reused image when it builds the controller-side provider pack, preserving the
exact manifest match required to avoid restaging the pack into Daytona.
Resolve the manifest digest and set `PAPERCLIP_E2E_DAYTONA_IMAGE` to
`ghcr.io/paperclipai/paperclip-daytona-runner@sha256:...`. The repository
workflow signs that digest with Cosign/OIDC and verifies that it is publicly
pullable, includes the provider pack, and advertises `dial_ws_loopback`,
`dial_wss`, and `listen_ws`. The GHCR package must be configured as public;
the image job deliberately fails its anonymous-pull check otherwise. Existing
content tags are never rebuilt or overwritten by the workflow.
## Evidence and cleanup
Packaged, access-controlled evidence is written beneath
`tests/runner-e2e/results/<campaign>/...`. Passing attempts include
`final-state.png`, Plan draft/revision screenshots when applicable, matcher
outcomes, sanitized fixture/API metadata, a result record, JUnit, HTML, and a
blob report. Failures additionally retain the Playwright trace/video, browser
diagnostics, failure screenshot, and sanitized Paperclip/run logs when
produced. PNG and WebM files are not pixel-inspected, so they are suitable only
for the local results directory and access-controlled GitHub Actions artifacts.
SVG is active content and is rejected from the packaged evidence entirely.
Every completed local campaign also writes
`tests/runner-e2e/results/<campaign>/dashboard.html`. The self-contained page
shows the complete profile/environment grid with screenshot thumbnails.
Expanding a case shows its matchers, pass/fail details, provider/model/runtime,
timings, token and cost accounting, and evidence links. The campaign header
aggregates input, output, and cached tokens, provider-reported LLM spend,
Daytona list-price runtime estimates, and pricing coverage. Missing provider
usage is labeled `unavailable` or `unpriced`; it is never presented as zero
cost. The CI report job stages the same portable site at
`normalized/index.html` inside the access-controlled merged report artifact.
Permanent public history has a narrower boundary. Before uploading to S3 or
packaging the optional GitHub Pages artifact, the publisher removes raster and
video evidence, archives, and the generated Playwright/blob/HTML report trees.
It then regenerates the dashboard against only the remaining allowlisted,
inert structured per-attempt evidence (`.json`, `.log`, `.md`, and `.txt`).
Per-attempt XML is excluded because browsers can process XML/XSLT. The root
`junit.xml` remains public because the report aggregator builds it from fixed
markup and XML-escaped fields. Public dashboards therefore contain results and
accounting but no attempt screenshots, videos, traces, or generated Playwright
reports.
### Billing interpretation
Each result contains raw sanitized `usage`, normalized `billing`, and
`runtimeUsage`:
- LLM token and dollar values come from the persisted heartbeat-run usage. A
multi-turn case aggregates every selected run and records how many runs
supplied tokens and provider-reported cost.
- Local execution records agent run time but is `not_metered` because there is
no external environment provider charge to attribute.
- Daytona records every public-API lease window and its pinned 4 vCPU, 4 GiB
RAM, and 10 GiB disk allocation. Its runtime dollar value is an estimate at
the versioned public list rates in `billing.ts`, not an invoice amount.
Credits, discounts, the storage allowance, and delayed billing adjustments
can make the eventual Daytona charge lower.
`normalized-results.json` uses the v2 campaign schema and includes per-test,
per-suite, and overall billing. The compact `history.json` index retains the
same metrics per campaign/suite/execution, source SHA/ref, definition
fingerprints, completeness, retries, and cleanup. Trend charts compare only
complete campaigns by default; partial/manual selections remain browsable.
`summary.md` carries the current totals into the GitHub Actions job summary.
### Iterate on a published dashboard without rerunning paid tests
Download and extract the `github-pages` artifact from an existing workflow run,
then regenerate only its HTML from the retained `normalized-results.json` and
public structured evidence files. The Pages artifact has already had private
visual and generated report evidence removed:
```bash
gh run download <run-id> --repo paperclipai/paperclip --name github-pages --dir /tmp/runner-e2e-pages
mkdir /tmp/runner-e2e-site
tar -xf /tmp/runner-e2e-pages/artifact.tar -C /tmp/runner-e2e-site
pnpm test:e2e:runner:dashboard -- /tmp/runner-e2e-site
# Optionally use a downloaded history index:
pnpm test:e2e:runner:dashboard -- /tmp/runner-e2e-site --history /tmp/history.json
```
Serve that directory with any static file server. This path does not start
Paperclip, invoke an agent, create a Daytona lease, or consume provider tokens.
Before an access-controlled evidence artifact is uploaded, the launcher:
1. copies only allowlisted file types;
2. scans raw API snapshots before sanitizing them;
3. scans the closed Paperclip home/database and workspace as streams;
4. redacts loaded exact values and known provider-key shapes from text;
5. expands ZIP reports for secret scanning;
6. rejects SVG and other unsafe files and fails the cell if a leak is detected;
and
7. verifies that a passing attempt has its final-state screenshot.
The temporary Paperclip home, embedded database, raw workspace, master key,
and unredacted logs are removed after each attempt. Daytona teardown destroys
the environment and any reusable leases through the public API; provider-side
auto-stop/archive/delete values remain as cancellation backstops.
## GitHub Actions
`Runner Full-Stack E2E` has only `schedule` and `workflow_dispatch` triggers; it
never runs for a pull request or ordinary push. Because this repository is
public, manual campaigns fail before checkout unless they run from the default
branch and both the original actor and rerun actor have numeric GitHub user IDs
in the non-empty JSON-array repository variable
`RUNNER_E2E_ALLOWED_ACTOR_IDS`. Usernames are intentionally not trusted.
The first scheduled attempt is trusted automation; any human rerun of a
scheduled campaign must pass the triggering-actor allowlist.
Create a protected `runner-e2e-paid` GitHub environment, restrict it to the
default branch, limit environment administration to trusted maintainers, and
store the four provider secrets there. This is a second authorization boundary:
the pre-check prevents unauthorized scheduling, while the environment prevents
secret release if the workflow gate is accidentally weakened. Also restrict
Actions to approved actions and require review of `.github/workflows/**` and
`tests/runner-e2e/**` through CODEOWNERS and branch protection. Manual inputs
accept comma-separated values for repeatable dimensions.
The nightly cron is `08:47 UTC`, but scheduled execution is intentionally gated
by the repository variable `RUNNER_FULL_STACK_E2E_NIGHTLY_ENABLED=true`. Set it
only after the live acceptance ladder in the architecture plan is green.
Set `RUNNER_E2E_MAX_PARALLEL` to an integer from 1–57 (default 32). Paid cells
run on `ubuntu-latest-m`; multi-turn steps are sequential inside their cell
while independent cells overlap. Artifacts and merged HTML/JUnit/normalized
reports are retained for 30 days.
Restrict the `ubuntu-latest-m` runner group to this workflow and the selected
repository. Do not let pull-request or fork-triggered workflows target that
group, do not mix it with untrusted workloads, and use ephemeral/reimaged
runners so one paid cell cannot leave state for the next. These runner-group
controls are external GitHub settings and are as important as the workflow
checks in a public repository.
GitHub Actions artifacts are access-controlled 30-day operational copies, not
the permanent public history. They retain packaged PNG/WebM and generated
reports for debugging. Create a second protected `runner-e2e-history`
environment, restricted to the default branch and trusted environment
administrators, then configure these repository variables:
- `RUNNER_E2E_HISTORY_AWS_ROLE_ARN`
- `RUNNER_E2E_HISTORY_AWS_REGION`
- `RUNNER_E2E_HISTORY_S3_BUCKET`
- `RUNNER_E2E_HISTORY_PUBLIC_BASE_URL`
- optional `RUNNER_E2E_HISTORY_PREFIX` (default `runner-e2e`)
The job exchanges GitHub OIDC for short-lived AWS credentials; never add AWS
access-key secrets. Its IAM role must trust only
`repo:paperclipai/paperclip:environment:runner-e2e-history`, and permit only
Get/List/Put under the configured prefix—never Delete. Enable S3 versioning and
Block Public Access. CloudFront reads the private bucket through Origin Access
Control. Immutable campaign bundles live under `campaigns/<run-id>-<attempt>/`;
mutable `history.json`, `latest.json`, and `latest-green.json` are updated by a
globally serialized publisher. An existing campaign key with a different
bundle digest fails closed.
GitHub Pages remains the stable latest dashboard. Enable Pages with GitHub
Actions as its source and set `RUNNER_FULL_STACK_E2E_PUBLISH_PAGES=true`.
The publisher prunes screenshots, video, archives, and generated report trees,
then regenerates the public dashboard before either the CloudFront-backed S3
history or optional Pages artifact is created. Public per-attempt evidence is
limited to allowlisted inert structured text. Databases, Paperclip homes,
workspaces, raw/unredacted logs, credentials, and visual evidence are never
published. Sanitized allowlisted `.log` copies may be public only after
exact-value/key-shape scanning and redaction.
See [FIXTURES.md](./FIXTURES.md) before adding or changing a profile,
environment, task, matcher, or future Paperclip object fixture.
See [SECURITY.md](./SECURITY.md) before enabling paid dispatch, the runner
group, or permanent public history in this public repository.
tests/runner-e2e/SECURITY.md
+165
View File
@@ -0,0 +1,165 @@
# Runner E2E security for a public repository
This suite can spend provider money, expose four API credentials to isolated
test processes, publish a container, retain private visual evidence, and write
public structured evidence. Treat changes to the workflow, harness, fixture
prompts, evidence packager, and publisher as security-sensitive production
changes.
## GitHub authorization
Set `RUNNER_E2E_ALLOWED_ACTOR_IDS` to a non-empty JSON array of numeric GitHub
user IDs, for example `[123456,789012]`. Resolve each ID from the authenticated
CLI and verify the login before adding it:
```bash
gh api users/LOGIN --jq '{login,id}'
```
The paid workflows reject manual dispatches outside the default branch before
checkout. They verify both the original actor and triggering actor for every
scheduled or manual attempt, including human reruns. Every
secret-bearing job repeats this check as its first step so GitHub's partial-job
rerun feature cannot bypass a successful predecessor authorization job. The
legacy manually dispatched E2E workflow uses the same gate. Numeric IDs are
stable across username changes and prevent lookalike-name authorization.
The full-stack and live campaigns have one Sunday UTC schedule each and also
support explicit manual dispatch. Their legacy-named nightly repository
variables remain independent kill switches. Neither paid workflow accepts
pull-request, push, workflow-run, or reusable-workflow triggers.
Protect the default branch, require review for workflow/harness paths, restrict
workflow dispatch permission, and restrict repository variable/environment
administration to the same trusted maintainers. Configure the organization to
allow only approved GitHub Actions. A malicious change merged into the default
branch executes with the same authority as the suite.
Every external action in the paid workflow is pinned to a full commit SHA. Keep
the adjacent major-version comment for update tooling, and resolve and review a
new immutable SHA before upgrading an action. The credential-free security test
rejects mutable tag or branch references.
## Secrets and protected environments
Create `runner-e2e-paid`, restrict deployments to the default branch, and put
only `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `OPENROUTER_API_KEY`, and
`DAYTONA_API_KEY` in it. Do not duplicate these credentials as repository- or
organization-level Actions secrets: environment scoping is the boundary that
prevents branch or pull-request jobs from requesting them. Require approval
from an account in `RUNNER_E2E_ALLOWED_ACTOR_IDS` for this environment and
disable administrator bypass. The authorize,
catalog, image, report, history, and Pages jobs receive none of these secrets.
Each full-stack matrix cell receives only its selected profile credential, plus
Daytona only for Daytona cells. Secret-bearing and OIDC jobs use frozen installs
without a shared dependency cache.
The Paperclip server process also receives none; the browser posts each value
once to the encrypted company secret API and agents/environments retain only
secret references.
Create `runner-e2e-history`, also default-branch-only, for the OIDC publishing
job. It contains no long-lived AWS key. Required reviewers may be added when a
human approval on every nightly publication is acceptable; otherwise rely on
the actor gate, environment branch restriction, and protected default branch.
## Runner group isolation
Restrict the `ubuntu-latest-m` runner group to `paperclipai/paperclip` and, when
the GitHub plan supports selected-workflow restrictions, to
`.github/workflows/runner-full-stack-e2e.yml` on the default branch. Never let
fork or pull-request workflows target the group. Use ephemeral runners, or
guaranteed reimaging between jobs, and do not share this group with untrusted
workloads. Disable interactive SSH/debug access for paid jobs unless a separate
incident procedure explicitly authorizes it.
## AWS OIDC and S3
The AWS role trust policy should accept only GitHub's OIDC audience and the
publishing environment subject:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::ACCOUNT_ID:oidc-provider/token.actions.githubusercontent.com"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
"token.actions.githubusercontent.com:sub": "repo:paperclipai/paperclip:environment:runner-e2e-history"
}
}
}
]
}
```
Grant only List on the bucket prefix and Get/Put on its objects. Do not grant
Delete, ACL, bucket-policy, or wildcard-resource permissions:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::BUCKET",
"Condition": {
"StringLike": { "s3:prefix": ["runner-e2e", "runner-e2e/*"] }
}
},
{
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:PutObject"],
"Resource": "arn:aws:s3:::BUCKET/runner-e2e/*"
}
]
}
```
Enable S3 versioning, default encryption, and Block Public Access. Disable
object ACLs. CloudFront receives read-only access through Origin Access Control;
the bucket itself stays private. Log S3 data writes and alert on attempts to
write outside the prefix or assume the role with a different subject.
Campaign prefixes are content-digested and immutable. The publisher refuses a
different digest at an existing campaign key. Only the compact history and
latest pointers are mutable, and S3 versioning makes those updates recoverable.
## Public evidence boundary
CloudFront and GitHub Pages are public. Fixture identifiers, timing, token
usage, costs, normalized results, and allowlisted inert structured per-attempt
evidence are expected public data. Screenshots, video, archives, generated
Playwright/blob/HTML report trees, credentials, Paperclip homes, databases,
workspaces, master keys, raw/unredacted logs, and unallowlisted files are not.
Only allowlisted `.log` copies that passed exact-value/key-shape scanning and
redaction may cross the public boundary.
The packaged evidence uploaded as a 30-day GitHub Actions artifact has a
different, access-controlled boundary. Text is exact-value and key-shape
scanned and redacted. PNG and WebM are raw-byte scanned but cannot be inspected
for credentials rendered as pixels, so they remain only in local evidence and
the access-controlled artifact. SVG is rejected during packaging because it is
active content.
Before permanent publication, the campaign publisher prunes raster/video
files, archives, and generated report trees. It then regenerates the dashboard
from the remaining allowlisted `.json`, `.log`, `.md`, and `.txt` evidence and
accepts only that dashboard, normalized JSON/JUnit/summary, fixed
branding assets, and the inert structured evidence paths. Per-attempt XML is
excluded because browsers can process XML/XSLT; the only public XML is the
root `junit.xml`, which the report aggregator constructs from fixed markup and
XML-escaped fields. The same pruned tree feeds both S3/CloudFront history and
the optional GitHub Pages artifact. A leak fails the cell and withholds the
unsafe file.
Rotate the affected credential immediately if a secret-scanning failure or
unexpected public object is observed. Preserve the access-controlled Actions
artifact and S3 object versions for incident analysis; do not weaken scanning
to make a campaign publish.
tests/runner-e2e/api.ts
+109
View File
@@ -0,0 +1,109 @@
import type { APIRequestContext, APIResponse } from "@playwright/test";
export interface JsonRecord {
[key: string]: unknown;
}
async function failureMessage(response: APIResponse, method: string) {
const text = await response.text().catch(() => "");
return `${method} ${response.url()} returned ${response.status()}${text ? `: ${text}` : ""}`;
}
export class RunnerApi {
readonly baseURL: string;
constructor(readonly request: APIRequestContext) {
const port = process.env.PAPERCLIP_RUNNER_E2E_PORT?.trim();
if (!port) throw new Error("PAPERCLIP_RUNNER_E2E_PORT is required");
this.baseURL = `http://127.0.0.1:${port}`;
}
async get<T>(path: string): Promise<T> {
const response = await this.request.get(path);
if (!response.ok()) throw new Error(await failureMessage(response, "GET"));
return response.json() as Promise<T>;
}
async post<T>(path: string, data?: unknown): Promise<T> {
const response = await this.request.post(path, { data });
if (!response.ok()) throw new Error(await failureMessage(response, "POST"));
return response.json() as Promise<T>;
}
/**
* Playwright traces APIRequestContext request bodies. Secret creation must
* still use the public API, but it goes through Node fetch so plaintext is
* never serialized into trace/blob evidence before Paperclip encrypts it.
*/
async postSensitive<T>(path: string, data: unknown): Promise<T> {
const response = await fetch(new URL(path, this.baseURL), {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify(data),
});
if (!response.ok) {
throw new Error(
`Sensitive POST ${path} returned ${response.status}; response body withheld`,
);
}
return response.json() as Promise<T>;
}
async patch<T>(path: string, data: unknown): Promise<T> {
const response = await this.request.patch(path, { data });
if (!response.ok())
throw new Error(await failureMessage(response, "PATCH"));
return response.json() as Promise<T>;
}
async delete(
path: string,
options?: { allowNotFound?: boolean },
): Promise<void> {
const response = await this.request.delete(path);
if (response.ok() || (options?.allowNotFound && response.status() === 404))
return;
throw new Error(await failureMessage(response, "DELETE"));
}
}
export async function pollUntil<T>(input: {
label: string;
deadlineAt: number;
load: () => Promise<T>;
accept: (value: T) => boolean;
reject?: (value: T) => string | undefined;
intervalMs?: number;
}): Promise<T> {
let last: T | undefined;
let lastError: unknown;
while (Date.now() < input.deadlineAt) {
try {
last = await input.load();
if (input.accept(last)) return last;
const rejection = input.reject?.(last);
if (rejection) {
throw new Error(`Stopped waiting for ${input.label}: ${rejection}`);
}
lastError = undefined;
} catch (error) {
if (
error instanceof Error &&
error.message.startsWith(`Stopped waiting for ${input.label}:`)
) {
throw error;
}
lastError = error;
}
await new Promise((resolve) =>
setTimeout(resolve, input.intervalMs ?? 2_000),
);
}
const detail =
lastError instanceof Error
? lastError.message
: last === undefined
? "no observation"
: JSON.stringify(last);
throw new Error(`Timed out waiting for ${input.label}: ${detail}`);
}
tests/runner-e2e/billing.test.ts
+162
View File
@@ -0,0 +1,162 @@
import { describe, expect, it } from "vitest";
import {
aggregateCampaignBilling,
buildRuntimeUsage,
summarizeExecutionBilling,
} from "./billing.js";
import type { RunnerE2EResult } from "./types.js";
function result(overrides: Partial<RunnerE2EResult> = {}): RunnerE2EResult {
return {
schema: "paperclip.runner-e2e.result/v1",
executionId: "legacy-claude.local.message-marker",
attempt: 1,
status: "passed",
profileId: "legacy-claude",
environmentId: "local",
caseId: "message-marker",
provider: "anthropic",
model: "fixture-model",
runtimeMode: "legacy",
runIds: ["run-1"],
startedAt: "2026-08-27T00:00:00.000Z",
finishedAt: "2026-08-27T00:00:02.000Z",
durationMs: 2_000,
cleanup: "passed",
...overrides,
};
}
describe("runner E2E billing summaries", () => {
it("summarizes provider-reported token usage and cost", () => {
const billing = summarizeExecutionBilling(
result({
usage: {
inputTokens: 12_000,
outputTokens: 420,
cachedInputTokens: 5_000,
cacheAdjustedCostUsd: 0.08125,
costStatus: "reported",
},
}),
);
expect(billing.llm).toMatchObject({
runCount: 1,
runsWithTokenUsage: 1,
runsWithReportedCost: 1,
inputTokens: 12_000,
outputTokens: 420,
cachedInputTokens: 5_000,
totalTokens: 17_420,
reportedCostUsd: 0.08125,
costStatus: "reported",
});
expect(billing.runtime.costStatus).toBe("not_metered");
expect(billing.complete).toBe(true);
});
it("labels missing and unpriced runs instead of treating them as free", () => {
const billing = summarizeExecutionBilling(
result({
runIds: ["run-1", "run-2", "run-3"],
usage: {
runs: [
{
runId: "run-1",
usage: {
inputTokens: 1_000,
outputTokens: 100,
costUsd: 0.01,
},
},
{
runId: "run-2",
usage: {
inputTokens: 2_000,
outputTokens: 200,
costStatus: "unpriced",
},
},
{ runId: "run-3", usage: null },
],
},
}),
);
expect(billing.llm).toMatchObject({
runCount: 3,
runsWithTokenUsage: 2,
runsWithReportedCost: 1,
reportedCostUsd: 0.01,
costStatus: "partial",
});
expect(billing.complete).toBe(false);
});
it("estimates Daytona list price from captured lease seconds and resources", () => {
const runtime = buildRuntimeUsage({
environmentId: "daytona",
runs: [
{
startedAt: "2026-08-27T00:00:05.000Z",
finishedAt: "2026-08-27T00:30:05.000Z",
},
],
leases: [
{
acquiredAt: "2026-08-27T00:00:00.000Z",
releasedAt: "2026-08-27T01:00:00.000Z",
metadata: { cpu: 4, memory: 4, disk: 10 },
},
],
});
expect(runtime).toMatchObject({
provider: "daytona",
agentRunDurationMs: 1_800_000,
leaseDurationMs: 3_600_000,
leaseCount: 1,
cpuCores: 4,
memoryGiB: 4,
diskGiB: 10,
costStatus: "estimated",
estimatedListCostUsd: 0.26748,
});
});
it("aggregates tokens, reported spend, runtime estimates, and coverage", () => {
const local = result({
usage: { inputTokens: 100, outputTokens: 20, costUsd: 0.004 },
});
const daytona = result({
executionId: "runner-acpx-claude.daytona.message-marker",
profileId: "runner-acpx-claude",
environmentId: "daytona",
runtimeMode: "native",
usage: { inputTokens: 200, outputTokens: 30 },
runtimeUsage: {
provider: "daytona",
agentRunDurationMs: 30_000,
leaseDurationMs: 40_000,
leaseCount: 1,
cpuCores: 4,
memoryGiB: 4,
diskGiB: 10,
estimatedListCostUsd: 0.002972,
costStatus: "estimated",
costSource: "daytona_public_list_price",
},
});
expect(aggregateCampaignBilling([local, daytona])).toMatchObject({
testCount: 2,
reportedLlmCostUsd: 0.004,
estimatedRuntimeCostUsd: 0.002972,
llm: {
runCount: 2,
runsWithTokenUsage: 2,
runsWithReportedCost: 1,
inputTokens: 300,
outputTokens: 50,
costStatus: "partial",
},
});
});
});
tests/runner-e2e/billing.ts
+394
View File
@@ -0,0 +1,394 @@
import type {
RunnerE2EBillingSummary,
RunnerE2EResult,
RunnerE2ERuntimeUsage,
RunnerEnvironmentId,
} from "./types.js";
// Public list prices are deliberately versioned here instead of being treated
// as provider-reported charges. Credits, discounts, and Daytona's first 5 GiB
// storage allowance can make the invoice amount lower than this estimate.
export const DAYTONA_LIST_PRICING = {
asOf: "2026-08-27",
url: "https://www.daytona.io/pricing",
cpuCoreHourUsd: 0.0504,
memoryGiBHourUsd: 0.0162,
diskGiBHourUsd: 0.000108,
} as const;
export interface RuntimeLeaseUsageInput {
acquiredAt?: string | Date | null;
releasedAt?: string | Date | null;
updatedAt?: string | Date | null;
metadata?: Record<string, unknown> | null;
}
export interface AgentRunUsageInput {
startedAt?: string | Date | null;
finishedAt?: string | Date | null;
}
export interface CampaignBillingSummary {
testCount: number;
agentRunDurationMs: number;
leaseDurationMs: number;
llm: RunnerE2EBillingSummary["llm"];
reportedLlmCostUsd: number;
estimatedRuntimeCostUsd: number;
observedAndEstimatedCostUsd: number;
testsWithCompleteBilling: number;
}
function record(value: unknown): Record<string, unknown> {
return value && typeof value === "object" && !Array.isArray(value)
? (value as Record<string, unknown>)
: {};
}
function finiteNumber(value: unknown): number | undefined {
return typeof value === "number" && Number.isFinite(value) && value >= 0
? value
: undefined;
}
function firstNumber(source: Record<string, unknown>, keys: readonly string[]) {
for (const key of keys) {
const value = finiteNumber(source[key]);
if (value !== undefined) return value;
}
return undefined;
}
function usageMeasurement(usage: Record<string, unknown>) {
const candidates = [
record(usage.runDelta),
record(usage.total),
record(usage.cumulative),
usage,
];
return (
candidates.find(
(candidate) =>
firstNumber(candidate, [
"inputTokens",
"input",
"promptTokens",
"outputTokens",
"output",
"completionTokens",
]) !== undefined,
) ?? usage
);
}
function usageCostUsd(usage: Record<string, unknown>) {
const measurement = usageMeasurement(usage);
const direct =
firstNumber(usage, [
"cacheAdjustedCostUsd",
"costUsd",
"providerCostUsd",
]) ??
firstNumber(measurement, [
"cacheAdjustedCostUsd",
"costUsd",
"providerCostUsd",
]);
if (direct !== undefined) return direct;
const cost = record(usage.cost);
const currency =
typeof cost.currency === "string" ? cost.currency.toUpperCase() : "USD";
if (currency !== "USD") return undefined;
return firstNumber(cost, ["amount", "total"]);
}
function usageEntries(
rawUsage: Record<string, unknown> | null | undefined,
runCount: number,
) {
const runs = Array.isArray(rawUsage?.runs) ? rawUsage.runs : null;
if (runs) {
const entries = runs.map((entry) => {
const candidate = record(entry);
const usage = candidate.usage;
return usage && typeof usage === "object" && !Array.isArray(usage)
? (usage as Record<string, unknown>)
: null;
});
return [
...entries,
...Array.from(
{ length: Math.max(0, runCount - entries.length) },
() => null,
),
];
}
return Array.from({ length: Math.max(1, runCount) }, (_, index) =>
index === 0 && rawUsage ? rawUsage : null,
);
}
function durationBetween(
startedAt: string | Date | null | undefined,
finishedAt: string | Date | null | undefined,
) {
const started = startedAt ? new Date(startedAt).getTime() : Number.NaN;
const finished = finishedAt ? new Date(finishedAt).getTime() : Number.NaN;
return Number.isFinite(started) && Number.isFinite(finished)
? Math.max(0, finished - started)
: 0;
}
function resourceValue(
metadata: Record<string, unknown> | null | undefined,
key: "cpu" | "memory" | "disk",
) {
return finiteNumber(metadata?.[key]);
}
export function buildRuntimeUsage(input: {
environmentId: RunnerEnvironmentId;
runs: readonly AgentRunUsageInput[];
leases?: readonly RuntimeLeaseUsageInput[];
fallbackFinishedAt?: string | Date;
}): RunnerE2ERuntimeUsage {
const agentRunDurationMs = input.runs.reduce(
(total, run) => total + durationBetween(run.startedAt, run.finishedAt),
0,
);
if (input.environmentId === "local") {
return {
provider: "local",
agentRunDurationMs,
leaseDurationMs: null,
leaseCount: 0,
costStatus: "not_metered",
costSource: "local_not_metered",
};
}
const leases = input.leases ?? [];
let leaseDurationMs = 0;
let estimatedListCostUsd = 0;
let resourcesComplete = leases.length > 0;
const resourceRows: Array<{ cpu: number; memory: number; disk: number }> = [];
for (const lease of leases) {
const finishedAt =
lease.releasedAt ?? input.fallbackFinishedAt ?? lease.updatedAt ?? null;
const durationMs = durationBetween(lease.acquiredAt, finishedAt);
leaseDurationMs += durationMs;
const cpu = resourceValue(lease.metadata, "cpu");
const memory = resourceValue(lease.metadata, "memory");
const disk = resourceValue(lease.metadata, "disk");
if (cpu === undefined || memory === undefined || disk === undefined) {
resourcesComplete = false;
continue;
}
resourceRows.push({ cpu, memory, disk });
const hours = durationMs / 3_600_000;
estimatedListCostUsd +=
hours *
(cpu * DAYTONA_LIST_PRICING.cpuCoreHourUsd +
memory * DAYTONA_LIST_PRICING.memoryGiBHourUsd +
disk * DAYTONA_LIST_PRICING.diskGiBHourUsd);
}
const commonResource = (key: "cpu" | "memory" | "disk") => {
const values = new Set(resourceRows.map((row) => row[key]));
return values.size === 1 ? resourceRows[0]?.[key] : undefined;
};
return {
provider: "daytona",
agentRunDurationMs,
leaseDurationMs: leases.length > 0 ? leaseDurationMs : null,
leaseCount: leases.length,
...(commonResource("cpu") === undefined
? {}
: { cpuCores: commonResource("cpu") }),
...(commonResource("memory") === undefined
? {}
: { memoryGiB: commonResource("memory") }),
...(commonResource("disk") === undefined
? {}
: { diskGiB: commonResource("disk") }),
...(resourcesComplete ? { estimatedListCostUsd } : {}),
costStatus: resourcesComplete ? "estimated" : "unavailable",
costSource: resourcesComplete
? "daytona_public_list_price"
: "provider_cost_unavailable",
...(resourcesComplete
? {
pricingAsOf: DAYTONA_LIST_PRICING.asOf,
pricingUrl: DAYTONA_LIST_PRICING.url,
}
: {}),
};
}
export function fallbackRuntimeUsage(
result: Pick<
RunnerE2EResult,
"environmentId" | "durationMs" | "runtimeUsage"
>,
) {
if (result.runtimeUsage) return result.runtimeUsage;
return {
provider: result.environmentId,
agentRunDurationMs: result.durationMs,
leaseDurationMs: null,
leaseCount: 0,
costStatus:
result.environmentId === "local" ? "not_metered" : "unavailable",
costSource:
result.environmentId === "local"
? "local_not_metered"
: "provider_cost_unavailable",
} satisfies RunnerE2ERuntimeUsage;
}
export function summarizeExecutionBilling(
result: Pick<
RunnerE2EResult,
"runIds" | "usage" | "environmentId" | "durationMs" | "runtimeUsage"
>,
): RunnerE2EBillingSummary {
const requestedRunCount = Math.max(result.runIds?.length ?? 0, 1);
const entries = usageEntries(result.usage, requestedRunCount);
let inputTokens = 0;
let outputTokens = 0;
let cachedInputTokens = 0;
let runsWithTokenUsage = 0;
let runsWithReportedCost = 0;
let reportedCostUsd = 0;
for (const usage of entries) {
if (!usage) continue;
const measurement = usageMeasurement(usage);
const input =
firstNumber(measurement, ["inputTokens", "input", "promptTokens"]) ?? 0;
const output =
firstNumber(measurement, [
"outputTokens",
"output",
"completionTokens",
]) ?? 0;
const cached =
firstNumber(measurement, [
"cachedInputTokens",
"cacheReadTokens",
"cachedReadTokens",
]) ?? 0;
inputTokens += input;
outputTokens += output;
cachedInputTokens += cached;
if (input > 0 || output > 0 || cached > 0) runsWithTokenUsage += 1;
const costUsd = usageCostUsd(usage);
if (costUsd !== undefined && (costUsd > 0 || input > 0 || output > 0)) {
runsWithReportedCost += 1;
reportedCostUsd += costUsd;
}
}
const runCount = Math.max(requestedRunCount, entries.length);
const costStatus =
runsWithReportedCost === runCount
? "reported"
: runsWithReportedCost > 0
? "partial"
: runsWithTokenUsage > 0
? "unpriced"
: "unavailable";
const runtime = fallbackRuntimeUsage(result);
const estimatedRuntimeCostUsd = runtime.estimatedListCostUsd ?? 0;
const complete =
runsWithTokenUsage === runCount &&
runsWithReportedCost === runCount &&
runtime.costStatus !== "unavailable";
return {
llm: {
runCount,
runsWithTokenUsage,
runsWithReportedCost,
inputTokens,
outputTokens,
cachedInputTokens,
totalTokens: inputTokens + cachedInputTokens + outputTokens,
reportedCostUsd,
costStatus,
},
runtime,
reportedCostUsd,
estimatedRuntimeCostUsd,
observedAndEstimatedCostUsd: reportedCostUsd + estimatedRuntimeCostUsd,
complete,
};
}
export function aggregateCampaignBilling(
results: readonly RunnerE2EResult[],
): CampaignBillingSummary {
const summaries = results.map(summarizeExecutionBilling);
const runCount = summaries.reduce(
(total, summary) => total + summary.llm.runCount,
0,
);
const runsWithTokenUsage = summaries.reduce(
(total, summary) => total + summary.llm.runsWithTokenUsage,
0,
);
const runsWithReportedCost = summaries.reduce(
(total, summary) => total + summary.llm.runsWithReportedCost,
0,
);
const reportedLlmCostUsd = summaries.reduce(
(total, summary) => total + summary.reportedCostUsd,
0,
);
const estimatedRuntimeCostUsd = summaries.reduce(
(total, summary) => total + summary.estimatedRuntimeCostUsd,
0,
);
return {
testCount: results.length,
agentRunDurationMs: summaries.reduce(
(total, summary) => total + summary.runtime.agentRunDurationMs,
0,
),
leaseDurationMs: summaries.reduce(
(total, summary) => total + (summary.runtime.leaseDurationMs ?? 0),
0,
),
llm: {
runCount,
runsWithTokenUsage,
runsWithReportedCost,
inputTokens: summaries.reduce(
(total, summary) => total + summary.llm.inputTokens,
0,
),
outputTokens: summaries.reduce(
(total, summary) => total + summary.llm.outputTokens,
0,
),
cachedInputTokens: summaries.reduce(
(total, summary) => total + summary.llm.cachedInputTokens,
0,
),
totalTokens: summaries.reduce(
(total, summary) => total + summary.llm.totalTokens,
0,
),
reportedCostUsd: reportedLlmCostUsd,
costStatus:
runsWithReportedCost === runCount
? "reported"
: runsWithReportedCost > 0
? "partial"
: runsWithTokenUsage > 0
? "unpriced"
: "unavailable",
},
reportedLlmCostUsd,
estimatedRuntimeCostUsd,
observedAndEstimatedCostUsd: reportedLlmCostUsd + estimatedRuntimeCostUsd,
testsWithCompleteBilling: summaries.filter((summary) => summary.complete)
.length,
};
}
tests/runner-e2e/catalog.test.ts
+292
View File
@@ -0,0 +1,292 @@
import { describe, expect, it } from "vitest";
import {
runnerEnvironments,
runnerMatrix,
openRouterBreadthProfiles,
openRouterBreadthTasks,
runnerProfiles,
runnerSuites,
runnerTasks,
isImmutableDaytonaImage,
validateRunnerCatalog,
} from "./catalog.js";
import {
buildMatrixJobs,
parseRunnerSelectors,
RunnerSelectorError,
selectRunnerExecutions,
} from "./selectors.js";
describe("runner E2E catalog", () => {
it("validates the 42-cell core and 15-cell breadth suites", () => {
expect(runnerProfiles).toHaveLength(7);
expect(openRouterBreadthProfiles).toHaveLength(5);
expect(runnerEnvironments).toHaveLength(2);
expect(runnerTasks).toHaveLength(3);
expect(openRouterBreadthTasks).toHaveLength(3);
expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
42, 15,
]);
expect(validateRunnerCatalog()).toHaveLength(57);
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(57);
expect(
runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
).toHaveLength(42);
expect(
runnerMatrix.filter(
(entry) => entry.suite.id === "openrouter-model-breadth",
),
).toHaveLength(15);
expect(
runnerMatrix.reduce(
(total, execution) => total + execution.task.expectedRunCount,
0,
),
).toBe(95);
});
it("derives five local native OpenCode profiles from the ranked snapshot", () => {
expect(
openRouterBreadthProfiles.map((profile) => profile.ranking?.rank),
).toEqual([1, 2, 3, 4, 5]);
expect(
openRouterBreadthProfiles.every(
(profile) =>
profile.adapterType === "paperclip_runner" &&
profile.provider === "opencode" &&
profile.model.startsWith("openrouter/") &&
profile.supportedEnvironments.join(",") === "local" &&
profile.modelQualification.source === "openrouter_rankings_snapshot",
),
).toBe(true);
});
it("defines deterministic two-run question and plan state machines", () => {
const question = openRouterBreadthTasks.find(
(task) => task.id === "question-resume-complete",
);
const plan = openRouterBreadthTasks.find(
(task) => task.id === "plan-approve-complete",
);
expect(question).toMatchObject({
flow: "question_resume_completion",
expectedRunCount: 2,
});
expect(question?.buildQuestionAnswer?.("nonce")).toMatchObject({
optionLabel: "Cobalt",
});
expect(plan).toMatchObject({
flow: "plan_approval_completion",
expectedRunCount: 2,
});
expect(plan?.buildPrompt("nonce")).toContain("exactly two numbered steps");
});
it("uses only declared secret references in generated payloads", () => {
expect(
runnerMatrix.every((entry) =>
entry.requiredCredentials.includes(entry.profile.credential),
),
).toBe(true);
expect(
runnerMatrix
.filter((entry) => entry.environment.id === "daytona")
.every((entry) =>
entry.requiredCredentials.includes("DAYTONA_API_KEY"),
),
).toBe(true);
});
it("pins legacy Codex and Claude to their classic CLI engines", () => {
for (const profileId of ["legacy-codex", "legacy-claude"]) {
const execution = runnerMatrix.find(
(candidate) =>
candidate.profile.id === profileId &&
candidate.environment.id === "local",
);
expect(execution).toBeDefined();
expect(
execution!.profile.buildAgent({
environmentId: "11111111-1111-4111-8111-111111111111",
environmentFixtureId: "local",
workspacePath: "/tmp/runner-e2e-workspace",
secretRefs: {
[execution!.profile.credential]: {
type: "secret_ref",
secretId: "22222222-2222-4222-8222-222222222222",
version: "latest",
},
},
executionId: execution!.id,
}),
).toMatchObject({ adapterConfig: { engine: "cli" } });
}
});
it("binds native Codex automation auth to the encrypted OpenAI secret", () => {
const execution = runnerMatrix.find(
(candidate) =>
candidate.id === "core-compatibility.runner-codex.local.message-marker",
);
expect(execution).toBeDefined();
const secretRef = {
type: "secret_ref" as const,
secretId: "22222222-2222-4222-8222-222222222222",
version: "latest" as const,
};
const agent = execution!.profile.buildAgent({
environmentId: "11111111-1111-4111-8111-111111111111",
environmentFixtureId: "local",
workspacePath: "/tmp/runner-e2e-workspace",
secretRefs: { OPENAI_API_KEY: secretRef },
executionId: execution!.id,
});
expect(agent.adapterConfig).toMatchObject({
env: {
OPENAI_API_KEY: secretRef,
CODEX_API_KEY: secretRef,
},
});
});
it("gives legacy planning agents a direct bounded API recipe", () => {
const task = runnerTasks.find(
(candidate) => candidate.id === "plan-revise-accept",
);
const execution = runnerMatrix.find(
(candidate) =>
candidate.profile.id === "legacy-claude" &&
candidate.environment.id === "local" &&
candidate.task.id === "plan-revise-accept",
);
expect(task).toBeDefined();
expect(execution).toBeDefined();
const agent = execution!.profile.buildAgent({
environmentId: "11111111-1111-4111-8111-111111111111",
environmentFixtureId: "local",
workspacePath: "/tmp/runner-e2e-workspace",
secretRefs: {
ANTHROPIC_API_KEY: {
type: "secret_ref",
secretId: "22222222-2222-4222-8222-222222222222",
version: "latest",
},
},
executionId: execution!.id,
});
expect(agent.adapterConfig).toMatchObject({ maxTurnsPerRun: 24 });
expect(agent.instructionsBundle).toMatchObject({
files: { "AGENTS.md": expect.stringContaining("/interactions") },
});
expect(task!.buildPrompt("nonce")).toContain("request_confirmation");
expect(task!.buildPrompt("nonce")).toContain("baseRevisionId");
expect(task!.buildRevisionRequest?.("nonce")).toContain("baseRevisionId");
});
it("accepts only complete immutable Daytona digests", () => {
expect(
isImmutableDaytonaImage(
`ghcr.io/paperclipai/paperclip-daytona-runner@sha256:${"a".repeat(64)}`,
),
).toBe(true);
expect(
isImmutableDaytonaImage(
"ghcr.io/paperclipai/paperclip-daytona-runner@sha256:REPLACE_ME",
),
).toBe(false);
expect(
isImmutableDaytonaImage(
"ghcr.io/paperclipai/paperclip-daytona-runner:e2e-latest",
),
).toBe(false);
});
});
describe("runner E2E selectors", () => {
it("requires an explicit billable selector", () => {
expect(() => parseRunnerSelectors([])).toThrow(RunnerSelectorError);
});
it("selects dimensions with OR within a dimension and AND across dimensions", () => {
const options = parseRunnerSelectors([
"--profile",
"legacy-codex",
"--profile",
"runner-codex",
"--environment",
"local",
]);
expect(selectRunnerExecutions(options).map((entry) => entry.id)).toEqual([
"core-compatibility.legacy-codex.local.message-marker",
"core-compatibility.legacy-codex.local.plan-revise-accept",
"core-compatibility.legacy-codex.local.ask-question",
"core-compatibility.runner-codex.local.message-marker",
"core-compatibility.runner-codex.local.plan-revise-accept",
"core-compatibility.runner-codex.local.ask-question",
]);
});
it("selects a suite without exploding its environment matrix", () => {
const selected = selectRunnerExecutions(
parseRunnerSelectors(["--suite", "openrouter-model-breadth"]),
);
expect(selected).toHaveLength(15);
expect(
selected.every(
(entry) =>
entry.suite.id === "openrouter-model-breadth" &&
entry.environment.id === "local",
),
).toBe(true);
});
it("combines repeated groups with AND semantics", () => {
const options = parseRunnerSelectors([
"--group",
"native",
"--group",
"daytona",
]);
const selected = selectRunnerExecutions(options);
expect(selected).toHaveLength(12);
expect(
selected.every(
(entry) =>
entry.profile.generation === "native" &&
entry.environment.id === "daytona",
),
).toBe(true);
});
it("rejects groups outside the advertised four", () => {
const options = parseRunnerSelectors(["--group", "codex"]);
expect(() => selectRunnerExecutions(options)).toThrow("Unknown group");
});
it("emits one independently schedulable job per scenario", () => {
const jobs = buildMatrixJobs(
selectRunnerExecutions(parseRunnerSelectors(["--all"])),
);
expect(jobs).toHaveLength(57);
expect(jobs.filter((job) => job.needsDaytona)).toHaveLength(21);
expect(new Set(jobs.map((job) => job.executionId)).size).toBe(57);
expect(
jobs.every((job) =>
runnerMatrix.some(
(execution) =>
execution.id === job.executionId &&
execution.profile.credential === job.credentialName,
),
),
).toBe(true);
});
it("validates bounded local parallelism", () => {
expect(
parseRunnerSelectors(["--all", "--max-parallel", "8"]).maxParallel,
).toBe(8);
expect(() =>
parseRunnerSelectors(["--all", "--max-parallel", "0"]),
).toThrow("positive integer");
});
});
tests/runner-e2e/catalog.ts
+857
View File
@@ -0,0 +1,857 @@
import { createHash } from "node:crypto";
import { createAgentSchema } from "../../packages/shared/src/validators/agent.js";
import { createEnvironmentSchema } from "../../packages/shared/src/validators/environment.js";
import { DEFAULT_CODEX_LOCAL_MODEL } from "../../packages/adapters/codex-local/src/index.js";
import { models as claudeModels } from "../../packages/adapters/claude-local/src/index.js";
import { QUALIFIED_ACPX_PROFILES } from "../../packages/paperclip-runner/src/drivers/acpx/qualified-profiles.js";
import { QUALIFIED_OPENCODE_MODEL } from "../../packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.js";
import { CREDENTIAL_NAMES } from "./types.js";
import {
openRouterProfileId,
openRouterRankingSnapshot,
} from "./openrouter-ranking.js";
import type {
AgentFixtureBuildInput,
EnvironmentFixture,
EnvironmentFixtureBuildInput,
MatrixExecution,
RunnerProfileFixture,
RunnerTaskFixture,
RunnerSuiteFixture,
SecretReference,
} from "./types.js";
const ENVIRONMENT_IDS = ["local", "daytona"] as const;
const SELECTABLE_GROUPS = [
"legacy",
"native",
"local",
"daytona",
"core",
"breadth",
] as const;
const SAMPLE_UUID = "11111111-1111-4111-8111-111111111111";
export function isImmutableDaytonaImage(value: string | undefined) {
return /^.+@sha256:[0-9a-f]{64}$/i.test(value ?? "");
}
function requiredSecret(
input: AgentFixtureBuildInput,
name: RunnerProfileFixture["credential"],
): SecretReference {
const value = input.secretRefs[name];
if (!value) throw new Error(`Missing fixture secret reference ${name}`);
return value;
}
function commonAgent(
input: AgentFixtureBuildInput,
fixtureId: string,
adapterType: string,
adapterConfig: Record<string, unknown>,
) {
return {
name: `Runner E2E ${fixtureId} ${input.executionId}`,
role: "qa",
title: "Paid full-stack runner acceptance fixture",
capabilities:
"Completes deterministic standard, planning, and ask-mode runner acceptance tasks.",
adapterType,
adapterConfig,
defaultEnvironmentId: input.environmentId,
budgetMonthlyCents: 0,
instructionsBundle: {
entryFile: "AGENTS.md",
files: {
"AGENTS.md": [
"You are running a paid Paperclip end-to-end acceptance fixture.",
"Follow the assigned task and its Paperclip work mode literally.",
"For standard and ask tasks, publish the requested visible answer and mark the task done.",
"For planning tasks, publish or revise the canonical Plan document and its revision-bound request_confirmation, then wait. Only implement after that exact plan is accepted.",
"Invoke assigned tools only through the runtime's real tool-call channel. Never print XML, DSML, JSON, or other tool-call markup as assistant text.",
"Legacy adapters must use the public Paperclip API and the injected PAPERCLIP_API_URL, PAPERCLIP_API_KEY, PAPERCLIP_TASK_ID, and PAPERCLIP_RUN_ID values for comments, documents, interactions, and status changes.",
...(adapterType === "paperclip_runner"
? []
: [
'For a planning task, do not inspect the OpenAPI schema. PUT /api/issues/$PAPERCLIP_TASK_ID/documents/plan with {title:"Plan",format:"markdown",body,changeSummary}; read latestRevisionId and latestRevisionNumber from that response. Then POST /api/issues/$PAPERCLIP_TASK_ID/interactions with {kind:"request_confirmation",continuationPolicy:"wake_assignee",payload:{version:1,prompt,acceptLabel:"Approve",rejectLabel:"Reject",rejectRequiresReason:true,target:{type:"issue_document",key:"plan",revisionId,revisionNumber}}}, and PATCH the issue to {status:"in_review"}. Include Authorization and X-Paperclip-Run-Id on every write.',
]),
"Never print, persist, or expose credential values, and never create unrelated work.",
].join("\n"),
},
},
runtimeConfig: {},
};
}
function legacyProfile(input: {
id: string;
label: string;
adapterType: "codex_local" | "claude_local" | "opencode_local";
provider: string;
model: string;
credential: RunnerProfileFixture["credential"];
extraConfig?: Record<string, unknown>;
}): RunnerProfileFixture {
return {
...input,
modelQualification: {
source: "adapter_constant",
qualificationId: `${input.adapterType}:default-model`,
},
generation: "legacy",
groups: ["legacy"],
supportedEnvironments: ENVIRONMENT_IDS,
expectedRuntimeMode: "legacy",
expectedRuntimeMetadata: {
adapterType: input.adapterType,
provider: input.provider,
},
buildAgent(buildInput) {
return commonAgent(buildInput, input.id, input.adapterType, {
// Remote adapters must inherit the lease's provider-owned remoteCwd.
// A host path here would override that mapping inside the sandbox.
...(buildInput.environmentFixtureId === "local"
? { cwd: buildInput.workspacePath }
: {}),
model: input.model,
timeoutSec: buildInput.environmentFixtureId === "daytona" ? 780 : 360,
dangerouslySkipPermissions: true,
...input.extraConfig,
env: {
[input.credential]: requiredSecret(buildInput, input.credential),
},
});
},
};
}
function nativeProfile(input: {
id: string;
label: string;
provider: "codex" | "opencode" | "acpx";
model: string;
credential: RunnerProfileFixture["credential"];
acpxAgent?: "claude" | "codex";
supportedEnvironments?: readonly (typeof ENVIRONMENT_IDS)[number][];
modelQualification?: RunnerProfileFixture["modelQualification"];
ranking?: RunnerProfileFixture["ranking"];
}): RunnerProfileFixture {
return {
...input,
adapterType: "paperclip_runner",
generation: "native",
groups: ["native"],
supportedEnvironments: input.supportedEnvironments ?? ENVIRONMENT_IDS,
expectedRuntimeMode: "native",
modelQualification: input.modelQualification ?? {
source:
input.provider === "acpx"
? "qualified_runner_profile"
: "adapter_constant",
qualificationId:
input.provider === "acpx"
? `acpx:${input.acpxAgent}`
: `${input.provider}:qualified-model`,
},
...(input.ranking ? { ranking: input.ranking } : {}),
expectedRuntimeMetadata: {
adapterType: "paperclip_runner",
provider: input.provider,
},
buildAgent(buildInput) {
const credentialRef = requiredSecret(buildInput, input.credential);
const permissionConfig =
input.provider === "codex"
? { codexPermissionMode: "never" }
: input.provider === "opencode"
? { opencodePermissionMode: "allow" }
: { acpxPermissionMode: "approve-all", acpxAgent: input.acpxAgent };
return commonAgent(buildInput, input.id, "paperclip_runner", {
provider: input.provider,
model: input.model,
lifecycleMode: "per_turn",
idleTimeoutMs: 300_000,
...permissionConfig,
env: {
[input.credential]: credentialRef,
// Codex's supported automation credential is CODEX_API_KEY. Keep
// OPENAI_API_KEY as the operator-facing fixture secret name and bind
// the same encrypted reference to the runtime-specific alias.
...(input.provider === "codex"
? { CODEX_API_KEY: credentialRef }
: {}),
},
});
},
};
}
const claudeLegacyModel = "claude-sonnet-4-6";
if (!claudeModels.some((model) => model.id === claudeLegacyModel)) {
throw new Error(
`Claude adapter does not expose the qualified ${claudeLegacyModel} model`,
);
}
export const runnerProfiles: readonly RunnerProfileFixture[] = [
legacyProfile({
id: "legacy-codex",
label: "Legacy Codex",
adapterType: "codex_local",
provider: "codex",
model: DEFAULT_CODEX_LOCAL_MODEL,
credential: "OPENAI_API_KEY",
// Keep this fixture on the classic adapter/CLI lane. ACP execution is
// covered independently by the native runner ACPX profiles below.
extraConfig: { engine: "cli" },
}),
legacyProfile({
id: "legacy-claude",
label: "Legacy Claude",
adapterType: "claude_local",
provider: "claude",
model: claudeLegacyModel,
credential: "ANTHROPIC_API_KEY",
extraConfig: {
engine: "cli",
// Plan fixtures need enough tool turns to read the issue, write the
// canonical Plan document, and request confirmation. Four turns caused
// the Claude CLI to terminate correctly but prematurely with
// `max_turns_exhausted` during the revision flow.
maxTurnsPerRun: 24,
},
}),
legacyProfile({
id: "legacy-opencode",
label: "Legacy OpenCode",
adapterType: "opencode_local",
provider: "opencode",
model: QUALIFIED_OPENCODE_MODEL,
credential: "OPENROUTER_API_KEY",
}),
nativeProfile({
id: "runner-codex",
label: "Runner Codex",
provider: "codex",
model: DEFAULT_CODEX_LOCAL_MODEL,
credential: "OPENAI_API_KEY",
}),
nativeProfile({
id: "runner-opencode",
label: "Runner OpenCode",
provider: "opencode",
model: QUALIFIED_OPENCODE_MODEL,
credential: "OPENROUTER_API_KEY",
}),
nativeProfile({
id: "runner-acpx-claude",
label: "Runner ACPX Claude",
provider: "acpx",
acpxAgent: "claude",
model: QUALIFIED_ACPX_PROFILES.claude.qualificationModel,
credential: "ANTHROPIC_API_KEY",
}),
nativeProfile({
id: "runner-acpx-codex",
label: "Runner ACPX Codex",
provider: "acpx",
acpxAgent: "codex",
model: QUALIFIED_ACPX_PROFILES.codex.qualificationModel,
credential: "OPENAI_API_KEY",
}),
] as const;
export const openRouterBreadthProfiles: readonly RunnerProfileFixture[] =
openRouterRankingSnapshot.models.map((rankedModel) =>
nativeProfile({
id: openRouterProfileId(rankedModel.id),
label: `#${rankedModel.rank} ${rankedModel.name}`,
provider: "opencode",
model: `openrouter/${rankedModel.id}`,
credential: "OPENROUTER_API_KEY",
supportedEnvironments: ["local"],
modelQualification: {
source: "openrouter_rankings_snapshot",
qualificationId: `${openRouterRankingSnapshot.snapshotId}:${rankedModel.rank}`,
},
ranking: {
rank: rankedModel.rank,
canonicalModelId: rankedModel.id,
snapshotId: openRouterRankingSnapshot.snapshotId,
capturedAt: openRouterRankingSnapshot.capturedAt,
sourceUrl: openRouterRankingSnapshot.sourceUrl,
},
}),
);
function requiredDaytonaSecret(input: EnvironmentFixtureBuildInput) {
const apiKey = input.secretRefs.DAYTONA_API_KEY;
if (!apiKey)
throw new Error("Missing fixture secret reference DAYTONA_API_KEY");
return apiKey;
}
export const runnerEnvironments: readonly EnvironmentFixture[] = [
{
id: "local",
label: "Isolated local",
groups: ["local"],
driver: "local",
provider: "local",
lifecycle: {
setup: "instance_managed",
probe: "run_context_via_api",
cleanup: "instance_shutdown",
},
expectedExecutionTarget: { kind: "local" },
buildEnvironment(input) {
return {
name: `Runner E2E local ${input.executionId}`,
description: "Ephemeral local runner E2E environment",
driver: "local",
config: {},
envVars: {},
};
},
},
{
id: "daytona",
label: "Daytona sandbox",
groups: ["daytona"],
driver: "sandbox",
provider: "daytona",
credential: "DAYTONA_API_KEY",
lifecycle: {
setup: "create_via_api",
probe: "run_context_via_api",
cleanup: "delete_via_api_and_destroy_leases",
},
expectedExecutionTarget: { kind: "remote", transport: "sandbox" },
buildEnvironment(input) {
if (!isImmutableDaytonaImage(input.daytonaImage)) {
throw new Error(
"PAPERCLIP_E2E_DAYTONA_IMAGE must be an immutable image digest",
);
}
return {
name: `Runner E2E Daytona ${input.executionId}`,
description: "Ephemeral Daytona runner E2E environment",
driver: "sandbox",
config: {
provider: "daytona",
apiKey: requiredDaytonaSecret(input),
image: input.daytonaImage,
// Pin the billable resource shape so per-test runtime list-price
// estimates remain reproducible when provider defaults change.
cpu: 4,
memory: 4,
disk: 10,
reuseLease: false,
runnerLifecycleMode: "per_turn",
autoStopInterval: 5,
autoArchiveInterval: 15,
autoDeleteInterval: 60,
timeoutMs: 300_000,
livenessTimeoutMs: 30_000,
},
envVars: {},
};
},
},
] as const;
export const runnerTasks: readonly RunnerTaskFixture[] = [
{
id: "message-marker",
label: "Basic response",
groups: [],
workMode: "standard",
flow: "single_turn",
expectedRunCount: 1,
attemptTimeoutMs: {
local: 8 * 60_000,
daytona: 15 * 60_000,
},
expectedTerminalState: { issue: "done", run: "succeeded" },
buildTitle: (nonce) => `Runner E2E PAPERCLIP_E2E_OK_${nonce}`,
buildVisibleMarker: (nonce) => `PAPERCLIP_E2E_OK_${nonce}`,
buildPrompt: (nonce) =>
[
"Complete this task in a single run.",
`The exact marker also appears unescaped in the task title: PAPERCLIP_E2E_OK_${nonce}`,
`Your final visible task-thread response must be exactly this marker: PAPERCLIP_E2E_OK_${nonce}`,
`In a native runner, first emit that exact marker as the complete user-facing final response, then use paperclip_finish with the same marker as its summary and an objective-satisfied claim for the supplied contract revision.`,
`In a legacy runner, post a task comment containing exactly that marker and mark the task Done through the public API.`,
"The visible task-thread response is asserted; hidden reasoning or provider terminal output alone does not count.",
"Use underscore characters exactly as shown and do not insert backslashes.",
"Do not create files, ask questions, start additional tasks, or include any credentials.",
].join("\n"),
buildMatchers(nonce, execution) {
return [
{ kind: "message_contains", expected: `PAPERCLIP_E2E_OK_${nonce}` },
{
kind: "issue_status",
expected: execution.task.expectedTerminalState.issue,
},
{
kind: "run_status",
expected: execution.task.expectedTerminalState.run,
},
{
kind: "runtime_mode",
expected: execution.profile.expectedRuntimeMode,
},
{ kind: "environment", expected: execution.environment.id },
];
},
},
{
id: "plan-revise-accept",
label: "Plan, revise, accept, implement",
groups: [],
workMode: "planning",
flow: "plan_revision_acceptance",
expectedRunCount: 3,
attemptTimeoutMs: {
local: 20 * 60_000,
daytona: 35 * 60_000,
},
expectedTerminalState: { issue: "done", run: "succeeded" },
buildTitle: (nonce) => `Runner E2E plan lifecycle ${nonce}`,
buildVisibleMarker: (nonce) => `PAPERCLIP_E2E_PLAN_DONE_${nonce}`,
buildPlanMarkers: (nonce) => ({
draft: `PAPERCLIP_E2E_PLAN_DRAFT_${nonce}`,
revised: `PAPERCLIP_E2E_PLAN_REVISED_${nonce}`,
}),
buildRevisionRequest: (nonce) =>
[
"Revise this same plan; do not implement it yet.",
`Remove PAPERCLIP_E2E_PLAN_DRAFT_${nonce} and include PAPERCLIP_E2E_PLAN_REVISED_${nonce}.`,
"Change the plan from two steps to exactly three numbered steps, with verification as step 3.",
"Publish the revised canonical Plan revision and request confirmation for that new revision.",
"In a native runner, call write_document for key `plan`, then call request_human_input exactly once with interactionKind `confirmation`, targetRevisionId set to the returned latest Plan revision, and continuationPolicy `wake_assignee`; do not call paperclip_finish while waiting.",
"In a legacy runner, first GET the current `plan` issue document, then PUT the revised Plan with `baseRevisionId` set to that response's `latestRevisionId`; after the update succeeds, create the equivalent request_confirmation targeting the newly returned `latestRevisionId` through the public Paperclip API.",
].join(" "),
buildPrompt: (nonce) =>
[
"This is a planning-mode lifecycle acceptance task.",
"First, create a small canonical Plan with exactly two numbered steps and request approval; do not implement it.",
`The initial Plan body must contain PAPERCLIP_E2E_PLAN_DRAFT_${nonce}.`,
"If the plan is sent back, revise that same Plan document according to the revision note, publish a new revision-bound confirmation, and still do not implement.",
"Only after the revised plan is accepted, implement it by posting one final visible task-thread response containing exactly " +
`PAPERCLIP_E2E_PLAN_DONE_${nonce}` +
" and mark the task Done.",
"For a native runner, remain in the requested planning collaboration mode. Call write_document for key `plan`, then call request_human_input exactly once with interactionKind `confirmation`, targetRevisionId set to the returned latest Plan revision, and continuationPolicy `wake_assignee`. Do not call paperclip_finish while waiting for either Plan confirmation.",
"For a legacy runner, use the public Paperclip API. The first PUT of the `plan` issue document creates it. For every later PUT, first GET the current document and set `baseRevisionId` to its `latestRevisionId`; a 409 means you must GET again and retry with the new latest revision. Create a `request_confirmation` targeting the successful PUT response's `latestRevisionId` with `continuationPolicy: wake_assignee`, and move the issue to `in_review` while waiting.",
"Do not create files, child tasks, or unrelated work, and do not expose credentials.",
].join("\n"),
buildMatchers(nonce, execution) {
return [
{
kind: "message_contains",
expected: `PAPERCLIP_E2E_PLAN_DONE_${nonce}`,
},
{
kind: "issue_status",
expected: execution.task.expectedTerminalState.issue,
},
{
kind: "run_status",
expected: execution.task.expectedTerminalState.run,
},
{
kind: "runtime_mode",
expected: execution.profile.expectedRuntimeMode,
},
{ kind: "environment", expected: execution.environment.id },
];
},
},
{
id: "ask-question",
label: "Ask mode question",
groups: [],
workMode: "ask",
flow: "single_turn",
expectedRunCount: 1,
attemptTimeoutMs: {
local: 8 * 60_000,
daytona: 15 * 60_000,
},
expectedTerminalState: { issue: "done", run: "succeeded" },
buildTitle: (nonce) => `Runner E2E ask mode ${nonce}`,
buildVisibleMarker: (nonce) => `E2E_ASK_12_${nonce}`,
buildPrompt: (nonce) =>
[
"Answer this question directly in Ask mode: what is seven plus five?",
`Your final visible task-thread answer must contain this complete marker, including its final suffix: E2E_ASK_12_${nonce}.`,
"Do not create or modify files, do not create a plan or additional work, and do not expose credentials.",
"After posting the direct answer, mark this task Done.",
`In a native runner, emit the answer and finish with E2E_ASK_12_${nonce} as the summary.`,
`In a legacy runner, post a task comment containing E2E_ASK_12_${nonce} and mark the task Done through the public API.`,
].join("\n"),
buildMatchers(nonce, execution) {
return [
{
kind: "message_contains",
expected: `E2E_ASK_12_${nonce}`,
},
{
kind: "issue_status",
expected: execution.task.expectedTerminalState.issue,
},
{
kind: "run_status",
expected: execution.task.expectedTerminalState.run,
},
{
kind: "runtime_mode",
expected: execution.profile.expectedRuntimeMode,
},
{ kind: "environment", expected: execution.environment.id },
];
},
},
] as const;
function terminalMatchers(
nonceMarker: string,
execution: MatrixExecution,
): readonly ReturnType<RunnerTaskFixture["buildMatchers"]>[number][] {
return [
{ kind: "message_contains", expected: nonceMarker },
{
kind: "issue_status",
expected: execution.task.expectedTerminalState.issue,
},
{ kind: "run_status", expected: execution.task.expectedTerminalState.run },
{ kind: "runtime_mode", expected: execution.profile.expectedRuntimeMode },
{ kind: "environment", expected: execution.environment.id },
];
}
function breadthMarker(phase: "H" | "Q_C" | "P_READY" | "P_OK", nonce: string) {
// Keep the complete visible marker comfortably below the shortest output
// fragments observed across ranked models while retaining the attempt nonce.
return `PC_${phase}_${nonce}`;
}
export const openRouterBreadthTasks: readonly RunnerTaskFixture[] = [
{
id: "hello-complete",
label: "Hello and complete",
groups: [],
workMode: "standard",
flow: "single_turn",
expectedRunCount: 1,
attemptTimeoutMs: { local: 8 * 60_000, daytona: 8 * 60_000 },
expectedTerminalState: { issue: "done", run: "succeeded" },
buildTitle: (nonce) => `OpenRouter breadth hello ${nonce}`,
buildVisibleMarker: (nonce) => breadthMarker("H", nonce),
buildPrompt: (nonce) =>
[
"Complete this deterministic hello task in one turn.",
`Return ${breadthMarker("H", nonce)} as the complete visible response.`,
`Call paperclip_finish with ${breadthMarker("H", nonce)} as its summary and mark the task Done.`,
"Do not create files, plans, interactions, or additional work.",
].join("\n"),
buildMatchers: (nonce, execution) =>
terminalMatchers(breadthMarker("H", nonce), execution),
},
{
id: "question-resume-complete",
label: "Ask, answer, resume",
groups: [],
workMode: "standard",
flow: "question_resume_completion",
expectedRunCount: 2,
attemptTimeoutMs: { local: 12 * 60_000, daytona: 12 * 60_000 },
expectedTerminalState: { issue: "done", run: "succeeded" },
buildTitle: (nonce) => `OpenRouter breadth question ${nonce}`,
buildVisibleMarker: (nonce) => breadthMarker("Q_C", nonce),
buildQuestionAnswer: (nonce) => ({
optionLabel: "Cobalt",
expectedMarker: breadthMarker("Q_C", nonce),
}),
buildPrompt: (nonce) =>
[
"Ask the user one structured question before completing this task.",
"Call request_human_input exactly once with interactionKind `questions`, title `Verification word`, prompt `Choose the verification word`, continuationPolicy `wake_assignee`, and payload {version:1,questions:[{id:`verification-word`,prompt:`Choose the verification word.`,selectionMode:`single`,required:true,options:[{id:`cobalt`,label:`Cobalt`},{id:`amber`,label:`Amber`}]}]}.",
"Do not call paperclip_finish while the question is pending.",
`After the answer arrives, if it is Cobalt, return ${breadthMarker("Q_C", nonce)} visibly and call paperclip_finish with that marker as the summary.`,
"Do not create files, plans, or additional work.",
].join("\n"),
buildMatchers: (nonce, execution) =>
terminalMatchers(breadthMarker("Q_C", nonce), execution),
},
{
id: "plan-approve-complete",
label: "Plan, approve, complete",
groups: [],
workMode: "planning",
flow: "plan_approval_completion",
expectedRunCount: 2,
attemptTimeoutMs: { local: 15 * 60_000, daytona: 15 * 60_000 },
expectedTerminalState: { issue: "done", run: "succeeded" },
buildTitle: (nonce) => `OpenRouter breadth plan ${nonce}`,
buildVisibleMarker: (nonce) => breadthMarker("P_OK", nonce),
buildPlanMarkers: (nonce) => ({
draft: breadthMarker("P_READY", nonce),
revised: breadthMarker("P_OK", nonce),
}),
buildPrompt: (nonce) =>
[
"Create a canonical Plan with exactly two numbered steps and request approval; do not implement before approval.",
`The Plan body must contain ${breadthMarker("P_READY", nonce)}.`,
"Call write_document for key `plan`, then call request_human_input exactly once with interactionKind `confirmation`, targetRevisionId set to the returned latest Plan revision, and continuationPolicy `wake_assignee`.",
"Do not call paperclip_finish while confirmation is pending.",
`After that exact Plan revision is accepted, return ${breadthMarker("P_OK", nonce)} visibly and call paperclip_finish with that marker as the summary.`,
"Do not create files, child tasks, or unrelated work.",
].join("\n"),
buildMatchers: (nonce, execution) =>
terminalMatchers(breadthMarker("P_OK", nonce), execution),
},
] as const;
const localEnvironment = runnerEnvironments.find(
(environment) => environment.id === "local",
)!;
export const runnerSuites: readonly RunnerSuiteFixture[] = [
{
id: "core-compatibility",
label: "Core Runner Compatibility",
description:
"Major provider, runtime generation, and execution-environment compatibility.",
groups: ["core"],
profiles: runnerProfiles,
environments: runnerEnvironments,
tasks: runnerTasks,
expectedMatrixSize: 42,
},
{
id: "openrouter-model-breadth",
label: "OpenRouter Model Breadth",
description:
"Weekly-ranked tool-capable OpenRouter models through native OpenCode on isolated local workspaces.",
groups: ["breadth"],
profiles: openRouterBreadthProfiles,
environments: [localEnvironment],
tasks: openRouterBreadthTasks,
expectedMatrixSize: 15,
definitionMetadata: {
rankingSnapshotId: openRouterRankingSnapshot.snapshotId,
rankingContentHash: openRouterRankingSnapshot.contentHash,
rankingCapturedAt: openRouterRankingSnapshot.capturedAt,
rankingSourceUrl: openRouterRankingSnapshot.sourceUrl,
},
},
] as const;
export function suiteDefinitionHash(suite: RunnerSuiteFixture) {
return createHash("sha256")
.update(
JSON.stringify({
id: suite.id,
profiles: suite.profiles.map((profile) => ({
id: profile.id,
model: profile.model,
qualification: profile.modelQualification,
})),
environments: suite.environments.map((environment) => environment.id),
tasks: suite.tasks.map((task) => ({
id: task.id,
flow: task.flow,
expectedRunCount: task.expectedRunCount,
})),
metadata: suite.definitionMetadata ?? null,
}),
)
.digest("hex");
}
export function buildRunnerMatrix(
suites: readonly RunnerSuiteFixture[] = runnerSuites,
): MatrixExecution[] {
return suites.flatMap((suite) =>
suite.profiles.flatMap((profile) =>
suite.environments
.filter((environment) =>
profile.supportedEnvironments.includes(environment.id),
)
.flatMap((environment) =>
suite.tasks.map((task) => ({
id: `${suite.id}.${profile.id}.${environment.id}.${task.id}`,
suite,
suiteDefinitionHash: suiteDefinitionHash(suite),
profile,
environment,
task,
groups: [
...new Set([
...suite.groups,
...profile.groups,
...environment.groups,
...task.groups,
]),
],
requiredCredentials: [
profile.credential,
...(environment.credential ? [environment.credential] : []),
],
})),
),
),
);
}
function duplicateIds(values: readonly { id: string }[]) {
const seen = new Set<string>();
return values
.map((value) => value.id)
.filter((id) => {
if (seen.has(id)) return true;
seen.add(id);
return false;
});
}
function assertNoRawSecretValues(value: unknown, label: string) {
if (typeof value === "string") {
if (/\b(?:sk-(?:proj-)?|sk-ant-)[A-Za-z0-9_-]{12,}\b/.test(value)) {
throw new Error(`${label} contains a raw secret-looking value`);
}
return;
}
if (Array.isArray(value)) {
value.forEach((entry) => assertNoRawSecretValues(entry, label));
return;
}
if (value && typeof value === "object") {
Object.entries(value).forEach(([key, entry]) => {
if (
typeof entry === "string" &&
/(?:api.?key|access.?token|credential|secret)$/i.test(key) &&
entry.trim()
) {
throw new Error(`${label} contains a raw credential at ${key}`);
}
assertNoRawSecretValues(entry, label);
});
}
}
export function validateRunnerCatalog(): MatrixExecution[] {
const allProfiles = [...runnerProfiles, ...openRouterBreadthProfiles];
const allTasks = [...runnerTasks, ...openRouterBreadthTasks];
for (const [label, values] of [
["suite", runnerSuites],
["profile", allProfiles],
["environment", runnerEnvironments],
["task", allTasks],
] as const) {
const duplicates = duplicateIds(values);
if (duplicates.length > 0)
throw new Error(
`Duplicate ${label} fixture ids: ${duplicates.join(", ")}`,
);
}
const selectableGroups = new Set<string>(SELECTABLE_GROUPS);
for (const fixture of [
...runnerSuites,
...allProfiles,
...runnerEnvironments,
...allTasks,
]) {
const unknownGroups = fixture.groups.filter(
(group) => !selectableGroups.has(group),
);
if (unknownGroups.length > 0) {
throw new Error(
`Fixture ${fixture.id} declares unknown groups: ${unknownGroups.join(", ")}`,
);
}
}
const sampleRefs = Object.fromEntries(
[
"OPENAI_API_KEY",
"ANTHROPIC_API_KEY",
"OPENROUTER_API_KEY",
"DAYTONA_API_KEY",
].map((name, index) => [
name,
{
type: "secret_ref" as const,
secretId: `${String(index + 1).padStart(8, "0")}-1111-4111-8111-111111111111`,
version: "latest" as const,
},
]),
);
for (const environment of runnerEnvironments) {
const payload = environment.buildEnvironment({
secretRefs: sampleRefs,
daytonaImage:
"ghcr.io/paperclipai/paperclip-daytona-runner@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
executionId: "schema-validation",
});
createEnvironmentSchema.parse(payload);
assertNoRawSecretValues(payload, `environment ${environment.id}`);
}
for (const profile of allProfiles) {
if (!CREDENTIAL_NAMES.includes(profile.credential)) {
throw new Error(
`Profile ${profile.id} declares unknown credential ${profile.credential}`,
);
}
const unsupportedEnvironmentIds = profile.supportedEnvironments.filter(
(environmentId) => !ENVIRONMENT_IDS.includes(environmentId),
);
if (unsupportedEnvironmentIds.length > 0) {
throw new Error(
`Profile ${profile.id} declares unknown environments: ${unsupportedEnvironmentIds.join(", ")}`,
);
}
const payload = profile.buildAgent({
environmentId: SAMPLE_UUID,
environmentFixtureId: "local",
workspacePath: "/tmp/paperclip-runner-e2e-schema",
secretRefs: sampleRefs,
executionId: "schema-validation",
});
createAgentSchema.parse(payload);
assertNoRawSecretValues(payload, `profile ${profile.id}`);
}
const matrix = buildRunnerMatrix();
const duplicateMatrixIds = duplicateIds(matrix);
if (duplicateMatrixIds.length > 0) {
throw new Error(
`Duplicate matrix execution ids: ${duplicateMatrixIds.join(", ")}`,
);
}
for (const suite of runnerSuites) {
const suiteSize = matrix.filter(
(execution) => execution.suite.id === suite.id,
).length;
if (suiteSize !== suite.expectedMatrixSize) {
throw new Error(
`Expected ${suite.expectedMatrixSize} ${suite.id} executions; received ${suiteSize}`,
);
}
}
if (matrix.length !== 57)
throw new Error(`Expected 57 runner executions; received ${matrix.length}`);
return matrix;
}
export const runnerMatrix = validateRunnerCatalog();
export function runnerExecutionById(id: string): MatrixExecution {
const execution = runnerMatrix.find((candidate) => candidate.id === id);
if (!execution) throw new Error(`Unknown runner E2E execution id: ${id}`);
return execution;
}
tests/runner-e2e/dashboard-regenerate.ts
+203
View File
@@ -0,0 +1,203 @@
import path from "node:path";
import { mkdir, readFile, readdir, writeFile } from "node:fs/promises";
import { runnerMatrix } from "./catalog.js";
import { renderRunnerE2EDashboard } from "./dashboard.js";
import {
buildRunnerCampaign,
canonicalExecutionId,
upgradeRunnerResult,
} from "./history.js";
import type {
RunnerE2ECampaign,
RunnerE2EHistoryIndex,
RunnerE2EResult,
} from "./types.js";
interface PublishedResult extends RunnerE2EResult {
evidenceValid?: boolean;
evidenceErrors?: string[];
}
interface PublishedCampaign {
schema?: string;
campaignId?: string;
generatedAt: string;
expected: string[];
results: PublishedResult[];
}
async function relativeFiles(root: string, current = root): Promise<string[]> {
const entries = await readdir(current, { withFileTypes: true }).catch(
() => [],
);
const files: string[] = [];
for (const entry of entries) {
const absolute = path.join(current, entry.name);
if (entry.isDirectory())
files.push(...(await relativeFiles(root, absolute)));
if (entry.isFile()) {
files.push(path.relative(root, absolute).split(path.sep).join("/"));
}
}
return files;
}
async function readOptionalHistory(file: string) {
return readFile(file, "utf8")
.then((value) => JSON.parse(value) as RunnerE2EHistoryIndex)
.then((value) =>
value.schema === "paperclip.runner-e2e.history/v1" ? value : undefined,
)
.catch(() => undefined);
}
export async function regenerateRunnerDashboard(input: {
bundle: string;
historyFile?: string | null;
outputDirectory?: string;
evidenceHrefPrefix?: string;
}) {
const bundle = path.resolve(input.bundle);
const outputDirectory = path.resolve(input.outputDirectory ?? bundle);
const evidenceHrefPrefix = input.evidenceHrefPrefix?.replace(
/^\/+|\/+$/g,
"",
);
if (
evidenceHrefPrefix &&
(evidenceHrefPrefix.includes("\\") ||
evidenceHrefPrefix
.split("/")
.some((segment) => !segment || segment === "." || segment === ".."))
) {
throw new Error("evidenceHrefPrefix must be a safe relative URL path");
}
const normalized = JSON.parse(
await readFile(path.join(bundle, "normalized-results.json"), "utf8"),
) as PublishedCampaign;
if (
!Array.isArray(normalized.expected) ||
!Array.isArray(normalized.results) ||
typeof normalized.generatedAt !== "string"
) {
throw new Error("Published bundle has an invalid normalized-results.json");
}
const results = normalized.results.map(upgradeRunnerResult);
const expected = normalized.expected.map(canonicalExecutionId);
const campaign: RunnerE2ECampaign = buildRunnerCampaign({
campaignId:
normalized.campaignId ??
`legacy-${normalized.generatedAt.replace(/[:.]/g, "-")}`,
generatedAt: normalized.generatedAt,
expected,
results,
});
const history =
input.historyFile === null
? undefined
: await readOptionalHistory(
input.historyFile ?? path.join(bundle, "history.json"),
);
const entries = await Promise.all(
normalized.results.map(async (publishedResult, index) => {
const result = results[index]!;
const originalExecutionId = publishedResult.executionId;
const evidenceIds = [
originalExecutionId,
...(originalExecutionId.startsWith("core-compatibility.")
? [originalExecutionId.slice("core-compatibility.".length)]
: []),
];
let evidenceBaseHref = "";
let evidenceFiles: string[] = [];
for (const evidenceId of evidenceIds) {
const candidate = [
"evidence",
evidenceId,
`attempt-${result.attempt}`,
].join("/");
const files = await relativeFiles(
path.join(bundle, ...candidate.split("/")),
);
if (files.length === 0) continue;
evidenceBaseHref = [evidenceHrefPrefix, candidate]
.filter(Boolean)
.join("/");
evidenceFiles = files;
break;
}
return {
result,
valid:
publishedResult.evidenceValid ??
(result.status === "passed" && result.cleanup === "passed"),
errors: publishedResult.evidenceErrors ?? [],
evidenceBaseHref,
evidenceFiles,
};
}),
);
const dashboard = renderRunnerE2EDashboard({
title: "Runner Full-Stack E2E",
generatedAt: normalized.generatedAt,
expected,
catalog: runnerMatrix,
entries,
campaign,
history,
});
const upgraded = {
...campaign,
results: campaign.results.map((result, index) => ({
...result,
evidenceValid:
normalized.results[index]?.evidenceValid ??
(result.status === "passed" && result.cleanup === "passed"),
evidenceErrors: normalized.results[index]?.evidenceErrors ?? [],
})),
};
await mkdir(outputDirectory, { recursive: true });
await Promise.all([
writeFile(path.join(outputDirectory, "index.html"), dashboard, "utf8"),
writeFile(path.join(outputDirectory, "dashboard.html"), dashboard, "utf8"),
writeFile(
path.join(outputDirectory, "normalized-results.json"),
`${JSON.stringify(upgraded, null, 2)}\n`,
"utf8",
),
]);
console.log(
`Regenerated dashboard from ${normalized.results.length} retained result(s) in ${outputDirectory}`,
);
}
async function main() {
const arguments_ = process.argv
.slice(2)
.filter((argument) => argument !== "--");
const bundleArgument = arguments_.find(
(argument) => !argument.startsWith("--"),
);
if (!bundleArgument) {
throw new Error(
"Usage: pnpm test:e2e:runner:dashboard -- <published-bundle-directory> [--history <history.json>]",
);
}
const historyIndex = arguments_.indexOf("--history");
const historyFile =
historyIndex >= 0 ? arguments_[historyIndex + 1] : undefined;
if (historyIndex >= 0 && !historyFile)
throw new Error("--history requires a path");
await regenerateRunnerDashboard({ bundle: bundleArgument, historyFile });
}
if (
process.argv[1] &&
path.resolve(process.argv[1]) === path.resolve(import.meta.filename)
) {
await main().catch((error) => {
console.error(error instanceof Error ? error.message : String(error));
process.exitCode = 1;
});
}
tests/runner-e2e/dashboard.ts
+1059
View File
File diff suppressed because it is too large. Load diff
tests/runner-e2e/daytona-image-content.ts
+242
View File
@@ -0,0 +1,242 @@
import { createHash } from "node:crypto";
import { lstat, readdir, readFile, readlink } from "node:fs/promises";
import path from "node:path";
import { fileURLToPath, pathToFileURL } from "node:url";
const repositoryRoot = path.resolve(import.meta.dirname, "../..");
export const DAYTONA_IMAGE_CONTENT_SCHEMA =
"paperclip-daytona-runner-image-content/v3";
export const DAYTONA_IMAGE_PLATFORM = "linux/amd64";
export const DAYTONA_IMAGE_DOCKERFILE_PATH = "docker/daytona-runner/Dockerfile";
// This is the audited dependency closure of docker/daytona-runner/Dockerfile.
// Keep it conservative: a false positive only rebuilds the image, while a
// missing input could incorrectly reuse an incompatible paid-test image.
export const DAYTONA_IMAGE_INPUT_PATHS = [
".dockerignore",
".npmrc",
"docker/daytona-runner/Dockerfile",
"package.json",
"patches",
"pnpm-lock.yaml",
"pnpm-workspace.yaml",
"scripts/link-plugin-dev-sdk.mjs",
"tsconfig.base.json",
"packages/paperclip-eval-kernel",
"packages/paperclip-runner",
] as const;
const ignoredDirectoryPaths = new Set([
"packages/paperclip-runner/dist",
"packages/paperclip-runner/runner/target",
]);
export interface DaytonaImageContentOptions {
repositoryRoot?: string;
inputPaths?: readonly string[];
platform?: string;
baseImages?: readonly string[];
frontendDigest?: string;
}
function compareNames(left: string, right: string): number {
return left < right ? -1 : left > right ? 1 : 0;
}
function normalizedRelativePath(value: string): string {
return value.split(path.sep).join("/");
}
function shouldIgnore(relativePath: string): boolean {
if (ignoredDirectoryPaths.has(relativePath)) return true;
return relativePath.split("/").includes("node_modules");
}
function updateRecord(
hash: ReturnType<typeof createHash>,
kind: string,
relativePath: string,
payload = "",
): void {
hash.update(kind);
hash.update("\0");
hash.update(relativePath);
hash.update("\0");
hash.update(payload);
hash.update("\0");
}
function assertPinnedBaseImage(reference: string): void {
if (!/@sha256:[0-9a-f]{64}$/.test(reference)) {
throw new Error(
`Daytona image base must use an immutable sha256 digest: ${reference}`,
);
}
}
function assertPinnedFrontendDigest(digest: string): void {
if (!/^sha256:[0-9a-f]{64}$/.test(digest)) {
throw new Error(
`Daytona Dockerfile frontend must use an immutable sha256 digest: ${digest}`,
);
}
}
export function extractDaytonaDockerfileFrontendDigest(
dockerfile: string,
): string {
const firstLine = dockerfile.split(/\r?\n/, 1)[0] ?? "";
const match = /^#\s*syntax=\S+@(sha256:[0-9a-f]{64})\s*$/.exec(firstLine);
if (!match) {
throw new Error(
"Daytona Dockerfile first line must pin its syntax frontend to an immutable sha256 digest",
);
}
const digest = match[1]!;
assertPinnedFrontendDigest(digest);
return digest;
}
export function extractDaytonaBaseImages(dockerfile: string): string[] {
const baseImages: string[] = [];
const stageAliases = new Set<string>();
for (const line of dockerfile.split(/\r?\n/)) {
if (!/^\s*FROM\s/i.test(line)) continue;
const match = /^\s*FROM(?:\s+--\S+)*\s+(\S+)(?:\s+AS\s+(\S+))?\s*$/i.exec(
line,
);
if (!match) {
throw new Error(`Cannot parse Daytona Dockerfile FROM line: ${line}`);
}
const reference = match[1]!;
if (!stageAliases.has(reference)) {
assertPinnedBaseImage(reference);
baseImages.push(reference);
}
if (match[2]) stageAliases.add(match[2]);
}
if (baseImages.length === 0) {
throw new Error("Daytona Dockerfile does not declare a base image");
}
return baseImages;
}
async function resolveBaseImages(
root: string,
explicitBaseImages?: readonly string[],
): Promise<string[]> {
const baseImages = explicitBaseImages
? [...explicitBaseImages]
: extractDaytonaBaseImages(
await readFile(path.join(root, DAYTONA_IMAGE_DOCKERFILE_PATH), "utf8"),
);
if (baseImages.length === 0) {
throw new Error("Daytona image content identity requires a base image");
}
for (const reference of baseImages) assertPinnedBaseImage(reference);
return baseImages.sort(compareNames);
}
async function resolveFrontendDigest(
root: string,
explicitFrontendDigest?: string,
): Promise<string> {
const digest =
explicitFrontendDigest ??
extractDaytonaDockerfileFrontendDigest(
await readFile(path.join(root, DAYTONA_IMAGE_DOCKERFILE_PATH), "utf8"),
);
assertPinnedFrontendDigest(digest);
return digest;
}
async function hashEntry(
hash: ReturnType<typeof createHash>,
root: string,
relativePath: string,
): Promise<void> {
const normalizedPath = normalizedRelativePath(relativePath);
if (shouldIgnore(normalizedPath)) return;
const absolutePath = path.resolve(root, relativePath);
const relativeFromRoot = path.relative(root, absolutePath);
if (
relativeFromRoot.startsWith(`..${path.sep}`) ||
relativeFromRoot === ".." ||
path.isAbsolute(relativeFromRoot)
) {
throw new Error(
`Daytona image input escapes repository root: ${relativePath}`,
);
}
const stats = await lstat(absolutePath);
if (stats.isDirectory()) {
updateRecord(hash, "directory", normalizedPath);
const entries = await readdir(absolutePath, { withFileTypes: true });
entries.sort((left, right) => compareNames(left.name, right.name));
for (const entry of entries) {
await hashEntry(hash, root, path.join(relativePath, entry.name));
}
return;
}
if (stats.isSymbolicLink()) {
updateRecord(hash, "symlink", normalizedPath, await readlink(absolutePath));
return;
}
if (stats.isFile()) {
const executable =
(stats.mode & 0o111) === 0 ? "non-executable" : "executable";
updateRecord(hash, "file", normalizedPath, executable);
hash.update(await readFile(absolutePath));
hash.update("\0");
return;
}
throw new Error(`Unsupported Daytona image input type: ${relativePath}`);
}
export async function computeDaytonaImageContentId(
options: DaytonaImageContentOptions = {},
): Promise<string> {
const root = path.resolve(options.repositoryRoot ?? repositoryRoot);
const inputPaths = [
...(options.inputPaths ?? DAYTONA_IMAGE_INPUT_PATHS),
].sort(compareNames);
const hash = createHash("sha256");
updateRecord(
hash,
"contract",
DAYTONA_IMAGE_CONTENT_SCHEMA,
options.platform ?? DAYTONA_IMAGE_PLATFORM,
);
updateRecord(
hash,
"dockerfile-frontend",
await resolveFrontendDigest(root, options.frontendDigest),
);
for (const baseImage of await resolveBaseImages(root, options.baseImages)) {
updateRecord(hash, "base-image", baseImage);
}
for (const inputPath of inputPaths) {
await hashEntry(hash, root, inputPath);
}
return hash.digest("hex");
}
const invokedPath = process.argv[1] ? path.resolve(process.argv[1]) : null;
if (invokedPath && import.meta.url === pathToFileURL(invokedPath).href) {
computeDaytonaImageContentId()
.then((contentId) => process.stdout.write(`${contentId}\n`))
.catch((error: unknown) => {
process.stderr.write(
`${error instanceof Error ? error.message : String(error)}\n`,
);
process.exitCode = 1;
});
}
export const DAYTONA_IMAGE_CONTENT_SCRIPT = fileURLToPath(import.meta.url);
tests/runner-e2e/daytona-image.test.ts
+203
View File
@@ -0,0 +1,203 @@
import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import path from "node:path";
import { describe, expect, it } from "vitest";
import {
computeDaytonaImageContentId,
DAYTONA_IMAGE_DOCKERFILE_PATH,
DAYTONA_IMAGE_INPUT_PATHS,
extractDaytonaBaseImages,
extractDaytonaDockerfileFrontendDigest,
} from "./daytona-image-content.js";
const repositoryRoot = path.resolve(import.meta.dirname, "../..");
describe("runner E2E Daytona image contract", () => {
it("builds runnerd and the provider pack and verifies every required transport", async () => {
const [dockerfile, dockerignore, workflow] = await Promise.all([
readFile(
path.join(repositoryRoot, "docker/daytona-runner/Dockerfile"),
"utf8",
),
readFile(path.join(repositoryRoot, ".dockerignore"), "utf8"),
readFile(
path.join(
repositoryRoot,
".github/workflows/runner-full-stack-e2e.yml",
),
"utf8",
),
]);
expect(dockerfile).toContain("--bin paperclip-runnerd");
expect(dockerfile).toContain("build-provider-pack.mjs /provider-pack");
expect(dockerfile).toContain(
"/opt/paperclip-runner/provider-pack/provider-pack.json",
);
expect(dockerfile).toContain(
"${PAPERCLIP_RUNNER_PROVIDER_PACK_ROOT}/node_modules/.bin",
);
for (const command of ["acpx", "claude-agent-acp", "codex-acp"]) {
expect(dockerfile).toContain(command);
}
for (const transport of ["dial_ws_loopback", "dial_wss", "listen_ws"]) {
expect(dockerfile).toContain(transport);
}
expect(dockerfile).toContain(
'metadata="$(paperclip-runnerd --build-metadata)"',
);
expect(dockerfile).toContain("provider-pack.json");
expect(dockerfile).toContain("io.paperclip.runner.content-id");
expect(dockerfile).toContain("org.opencontainers.image.revision");
expect(extractDaytonaDockerfileFrontendDigest(dockerfile)).toBe(
"sha256:a57df69d0ea827fb7266491f2813635de6f17269be881f696fbfdf2d83dda33e",
);
expect(extractDaytonaBaseImages(dockerfile)).toEqual([
"rust:1.97-bookworm@sha256:408fe88047cef61a2087653b0c5255fa51c0f2d6d94ddedd7a2562a9b91a46f6",
"node:24-bookworm@sha256:9137a20e25879e0b557227b57e3ee4e9af4bde29eb3db66134cd1723e84f830b",
"daytonaio/sandbox:0.8.0@sha256:eadf88e4391072b7ad4bed27d9cadfc9fe9d8ed375d9219d34c2ccb518f213e3",
]);
expect(dockerignore).toContain("**/node_modules");
expect(dockerignore).toContain("packages/paperclip-runner/dist");
expect(dockerignore).toContain("packages/paperclip-runner/runner/target");
expect(workflow).toContain("--platform linux/amd64");
expect(workflow).toContain(
"Compute Daytona image content ID with pinned bases",
);
expect(workflow).toContain(
"e2e-content-${{ needs.catalog.outputs.daytona_image_content_id }}",
);
expect(workflow).toContain(
'--build-arg "PAPERCLIP_RUNNER_CONTENT_ID=${IMAGE_CONTENT_ID}"',
);
expect(workflow).not.toContain("e2e-git-${{ github.sha }}");
expect(workflow).toContain("cosign sign --yes");
expect(workflow).toContain("docker image inspect");
expect(workflow).toContain('.Config.User == "daytona"');
expect(workflow).toContain("PAPERCLIP_RUNNER_PROVIDER_PACK_ROOT=");
expect(workflow).toContain(
"pnpm --filter @paperclipai/paperclip-runner build:provider-pack",
);
expect(workflow).toContain(
"PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH: ${{ github.workspace }}/packages/paperclip-runner/provider-pack",
);
expect(workflow).toContain(
"PAPERCLIP_RUNNER_SOURCE_REVISION: ${{ needs.daytona_image.outputs.source_revision }}",
);
expect(workflow).toContain("anonymous_config");
});
it("hashes the audited image dependency closure rather than the repository revision", async () => {
for (const requiredPath of [
".dockerignore",
"docker/daytona-runner/Dockerfile",
"pnpm-lock.yaml",
"patches",
"packages/paperclip-eval-kernel",
"packages/paperclip-runner",
]) {
expect(DAYTONA_IMAGE_INPUT_PATHS).toContain(requiredPath);
}
expect(DAYTONA_IMAGE_DOCKERFILE_PATH).toBe(
"docker/daytona-runner/Dockerfile",
);
const contentId = await computeDaytonaImageContentId();
expect(contentId).toMatch(/^[0-9a-f]{64}$/);
});
it("changes only when an image input, frontend, base, or platform changes", async () => {
const root = await mkdtemp(
path.join(tmpdir(), "paperclip-daytona-image-id-"),
);
try {
await mkdir(path.join(root, "image-input"));
await writeFile(
path.join(root, "image-input", "runner.ts"),
"version one\n",
);
const baseline = await computeDaytonaImageContentId({
repositoryRoot: root,
inputPaths: ["image-input"],
baseImages: [`example.test/base:1@sha256:${"a".repeat(64)}`],
frontendDigest: `sha256:${"c".repeat(64)}`,
});
expect(
await computeDaytonaImageContentId({
repositoryRoot: root,
inputPaths: ["image-input"],
baseImages: [`example.test/base:1@sha256:${"b".repeat(64)}`],
frontendDigest: `sha256:${"c".repeat(64)}`,
}),
).not.toBe(baseline);
expect(
await computeDaytonaImageContentId({
repositoryRoot: root,
inputPaths: ["image-input"],
baseImages: [`example.test/base:1@sha256:${"a".repeat(64)}`],
frontendDigest: `sha256:${"d".repeat(64)}`,
}),
).not.toBe(baseline);
await writeFile(
path.join(root, "unrelated.txt"),
"does not enter the image\n",
);
expect(
await computeDaytonaImageContentId({
repositoryRoot: root,
inputPaths: ["image-input"],
baseImages: [`example.test/base:1@sha256:${"a".repeat(64)}`],
frontendDigest: `sha256:${"c".repeat(64)}`,
}),
).toBe(baseline);
await writeFile(
path.join(root, "image-input", "runner.ts"),
"version two\n",
);
expect(
await computeDaytonaImageContentId({
repositoryRoot: root,
inputPaths: ["image-input"],
baseImages: [`example.test/base:1@sha256:${"a".repeat(64)}`],
frontendDigest: `sha256:${"c".repeat(64)}`,
}),
).not.toBe(baseline);
expect(
await computeDaytonaImageContentId({
repositoryRoot: root,
inputPaths: ["image-input"],
platform: "linux/arm64",
baseImages: [`example.test/base:1@sha256:${"a".repeat(64)}`],
frontendDigest: `sha256:${"c".repeat(64)}`,
}),
).not.toBe(baseline);
} finally {
await rm(root, { recursive: true, force: true });
}
});
it("rejects mutable Docker base references", () => {
expect(() => extractDaytonaBaseImages("FROM node:24-bookworm\n")).toThrow(
"must use an immutable sha256 digest",
);
});
it("rejects a mutable or missing Dockerfile syntax frontend", async () => {
expect(() =>
extractDaytonaDockerfileFrontendDigest(
"# syntax=docker/dockerfile:1.7\nFROM scratch\n",
),
).toThrow("must pin its syntax frontend");
expect(() =>
extractDaytonaDockerfileFrontendDigest("FROM scratch\n"),
).toThrow("must pin its syntax frontend");
await expect(
computeDaytonaImageContentId({
inputPaths: [],
baseImages: [`example.test/base:1@sha256:${"a".repeat(64)}`],
frontendDigest: "sha256:mutable",
}),
).rejects.toThrow("must use an immutable sha256 digest");
});
});
tests/runner-e2e/evidence.ts
+232
View File
@@ -0,0 +1,232 @@
import { spawn } from "node:child_process";
import {
mkdir,
readFile,
readdir,
rm,
writeFile,
copyFile,
} from "node:fs/promises";
import path from "node:path";
import {
assertSecretFree,
findSecretLeak,
findSecretLeakInJsonValues,
redactText,
sanitizeJson,
} from "./redaction.js";
const TEXT_EXTENSIONS = new Set([
".json",
".xml",
".html",
".css",
".js",
".txt",
".log",
".md",
]);
const BINARY_EXTENSIONS = new Set([".png", ".webm"]);
const ACTIVE_CONTENT_EXTENSIONS = new Set([".svg"]);
const ALLOWED_DIRECTORIES = new Set([
"snapshots",
"playwright-output",
"blob-report",
"html-report",
]);
const ALLOWED_ROOT_FILES = new Set([
"result.json",
"final-state.png",
"failure.png",
"server.log",
"playwright.log",
"junit.xml",
]);
const REQUIRED_PASS_FILES = [
"final-state.png",
"result.json",
"junit.xml",
path.join("html-report", "index.html"),
path.join("snapshots", "fixtures.json"),
path.join("snapshots", "api-state.json"),
] as const;
export interface EvidencePackageResult {
files: string[];
leaks: Array<{ file: string; reason: string }>;
missing: string[];
}
async function walk(root: string, relative = ""): Promise<string[]> {
const directory = path.join(root, relative);
const entries = await readdir(directory, { withFileTypes: true }).catch(
() => [],
);
const files: string[] = [];
for (const entry of entries) {
const next = path.join(relative, entry.name);
if (entry.isDirectory()) files.push(...(await walk(root, next)));
else if (entry.isFile()) files.push(next);
}
return files;
}
function isAllowed(relative: string) {
const segments = relative.split(path.sep);
if (segments.length === 1)
return (
ALLOWED_ROOT_FILES.has(relative) ||
/^(?:plan|question)-[a-z0-9-]+\.png$/.test(relative)
);
return ALLOWED_DIRECTORIES.has(segments[0]);
}
async function inspectZip(source: string, secrets: readonly string[]) {
// Failure traces can exceed Node's child-process output buffer. Stream the
// expanded archive through the exact-value scanner with enough overlap to
// detect a credential split across stdout chunks, without retaining the
// archive in memory or weakening fail-closed evidence publication.
const overlap = Math.max(
256,
...secrets.map((secret) => Buffer.byteLength(secret, "utf8") + 16),
);
return new Promise<string | null>((resolve) => {
const unzip = spawn("unzip", ["-p", source], {
stdio: ["ignore", "pipe", "pipe"],
});
let carry = Buffer.alloc(0);
let leak: string | null = null;
let spawnError: Error | null = null;
unzip.stdout.on("data", (chunk: Buffer) => {
if (leak) return;
const data = Buffer.concat([carry, chunk]);
leak = findSecretLeak(data, secrets, { includeShapes: false });
carry = data.subarray(Math.max(0, data.length - overlap));
if (leak) unzip.kill();
});
// Drain diagnostics so a noisy unzip cannot block. Error text is withheld
// because archive paths and contents belong to private attempt evidence.
unzip.stderr.resume();
unzip.on("error", (error) => {
spawnError = error;
});
unzip.on("close", (code) => {
if (leak) return resolve(leak);
if (spawnError) {
return resolve(`zip could not be inspected: ${spawnError.message}`);
}
return resolve(
code === 0
? null
: `zip could not be inspected: unzip exited with code ${String(code)}`,
);
});
});
}
export async function packageEvidence(input: {
privateDir: string;
uploadDir: string;
secrets: readonly string[];
expectPassScreenshot: boolean;
}): Promise<EvidencePackageResult> {
await rm(input.uploadDir, { recursive: true, force: true });
await mkdir(input.uploadDir, { recursive: true });
const files: string[] = [];
const leaks: EvidencePackageResult["leaks"] = [];
const available = await walk(input.privateDir);
for (const relative of available.filter(isAllowed)) {
const source = path.join(input.privateDir, relative);
const extension = path.extname(relative).toLowerCase();
const destination = path.join(input.uploadDir, relative);
await mkdir(path.dirname(destination), { recursive: true });
if (ACTIVE_CONTENT_EXTENSIONS.has(extension)) {
// SVG can execute script when opened directly from an artifact. Keep the
// source in the disposable private attempt directory, but never admit it
// to the sanitized CI artifact.
continue;
} else if (TEXT_EXTENSIONS.has(extension)) {
const raw = await readFile(source, "utf8");
const parsed = extension === ".json" ? JSON.parse(raw) : null;
const leak =
extension === ".json"
? findSecretLeakInJsonValues(parsed, input.secrets)
: findSecretLeak(raw, input.secrets);
if (leak) leaks.push({ file: relative, reason: leak });
// Redacting an already serialized JSON string can change escape
// boundaries around shell commands. Sanitize parsed values instead so
// uploaded snapshots stay valid JSON.
const safe =
extension === ".json"
? `${JSON.stringify(sanitizeJson(parsed, input.secrets), null, 2)}\n`
: redactText(raw, input.secrets);
if (extension === ".json") {
const safeLeak = findSecretLeakInJsonValues(
JSON.parse(safe),
input.secrets,
);
if (safeLeak)
throw new Error(`Secret leak in ${relative}: ${safeLeak}`);
} else {
assertSecretFree(safe, input.secrets, relative);
}
await writeFile(destination, safe, "utf8");
files.push(relative);
} else if (extension === ".zip") {
const leak = await inspectZip(source, input.secrets);
if (leak) {
leaks.push({ file: relative, reason: leak });
continue;
}
await copyFile(source, destination);
files.push(relative);
} else if (BINARY_EXTENSIONS.has(extension)) {
// This raw-byte scan catches embedded plaintext credentials, but cannot
// inspect rendered pixels. Raster/video files are retained in the
// access-controlled CI artifact and stripped at the public-history
// boundary by history-publish.ts.
const raw = await readFile(source);
const leak = findSecretLeak(raw, input.secrets);
if (leak) {
leaks.push({ file: relative, reason: leak });
continue;
}
await copyFile(source, destination);
files.push(relative);
}
}
const missing = input.expectPassScreenshot
? [
...REQUIRED_PASS_FILES.filter((required) => !files.includes(required)),
...(files.some(
(file) =>
file.startsWith(`blob-report${path.sep}`) && file.endsWith(".zip"),
)
? []
: [path.join("blob-report", "*.zip")]),
]
: [];
const manifest = {
schema: "paperclip.runner-e2e.evidence/v1",
files: [...files].sort(),
leaks,
missing,
};
const manifestText = `${JSON.stringify(sanitizeJson(manifest, input.secrets), null, 2)}\n`;
const manifestLeak = findSecretLeakInJsonValues(
JSON.parse(manifestText),
input.secrets,
);
if (manifestLeak)
throw new Error(`Secret leak in evidence-manifest.json: ${manifestLeak}`);
await writeFile(
path.join(input.uploadDir, "evidence-manifest.json"),
manifestText,
"utf8",
);
files.push("evidence-manifest.json");
return { files, leaks, missing };
}
tests/runner-e2e/failure-classifier.ts
+30
View File
@@ -0,0 +1,30 @@
import type { FailureClass } from "./types.js";
const TRANSIENT =
/(?:\b429\b|\b5\d\d\b|rate.?limit|ECONN(?:RESET|REFUSED)|socket hang up|network (?:error|interruption|timeout)|service unavailable|(?:provider|server|bootstrap|browser|webserver|health|daytona|sandbox|ingress|preview|connection|harness).*(?:temporar|timed? out|timeout|closed|failed|unavailable|interrupt|reset|refused|create|start|connect)|(?:timed? out|timeout).*(?:provider|server|bootstrap|browser|webserver|health|daytona|sandbox|ingress|preview|connection|harness))/i;
const PERMANENT =
/(?:missing (?:credential|fixture secret)|invalid.*(?:credential|api key)|unauthorized|forbidden|qualification|model.*(?:unsupported|incompatible)|artifact.*incompatible|runner_remote_.*(?:incompatible|unavailable)|immutable image digest)/i;
const CANDIDATE =
/(?:matcher|expected.*observed|marker|issue status|run status|runtime mode|wrong output|missing output)/i;
export function classifyFailure(error: unknown): FailureClass {
const message =
error instanceof Error ? `${error.name}: ${error.message}` : String(error);
if (/browser bootstrap failed before task creation/i.test(message))
return "transient_infrastructure";
if (/secret.*(?:leak|plaintext|redaction)/i.test(message))
return "secret_leak";
if (/cleanup|teardown|lease.*release/i.test(message)) {
return TRANSIENT.test(message)
? "transient_infrastructure"
: "cleanup_failure";
}
if (PERMANENT.test(message)) return "permanent_infrastructure";
if (TRANSIENT.test(message)) return "transient_infrastructure";
if (CANDIDATE.test(message)) return "candidate_failure";
return "candidate_failure";
}
export function shouldRetryFailure(failureClass: FailureClass) {
return failureClass === "transient_infrastructure";
}
tests/runner-e2e/fixture-registry.ts
+74
View File
@@ -0,0 +1,74 @@
export interface FixtureDefinition<T = unknown> {
id: string;
dependencies?: readonly string[];
setup(resolved: ReadonlyMap<string, unknown>): Promise<T>;
teardown?(value: T, resolved: ReadonlyMap<string, unknown>): Promise<void>;
}
export class FixtureRegistry {
readonly #definitions = new Map<string, FixtureDefinition>();
register<T>(definition: FixtureDefinition<T>) {
if (this.#definitions.has(definition.id))
throw new Error(`Duplicate fixture id: ${definition.id}`);
this.#definitions.set(definition.id, definition as FixtureDefinition);
return this;
}
async setupAll() {
const resolved = new Map<string, unknown>();
const completed: FixtureDefinition[] = [];
const visiting = new Set<string>();
const setup = async (id: string): Promise<void> => {
if (resolved.has(id)) return;
if (visiting.has(id))
throw new Error(`Fixture dependency cycle at ${id}`);
const definition = this.#definitions.get(id);
if (!definition) throw new Error(`Unknown fixture dependency: ${id}`);
visiting.add(id);
for (const dependency of definition.dependencies ?? [])
await setup(dependency);
visiting.delete(id);
const value = await definition.setup(resolved);
resolved.set(id, value);
completed.push(definition);
};
try {
for (const id of this.#definitions.keys()) await setup(id);
} catch (setupError) {
try {
await this.#teardown(completed, resolved);
} catch (teardownError) {
throw new AggregateError(
[setupError, teardownError],
`Fixture setup failed and cleanup failed: ${teardownError instanceof Error ? teardownError.message : String(teardownError)}`,
);
}
throw setupError;
}
return {
values: resolved,
teardown: () => this.#teardown(completed, resolved),
};
}
async #teardown(
completed: FixtureDefinition[],
resolved: Map<string, unknown>,
) {
const errors: Error[] = [];
for (const definition of [...completed].reverse()) {
if (!definition.teardown) continue;
try {
await definition.teardown(resolved.get(definition.id), resolved);
} catch (error) {
errors.push(error instanceof Error ? error : new Error(String(error)));
}
}
if (errors.length > 0)
throw new AggregateError(errors, "Fixture teardown failed");
}
}
tests/runner-e2e/harness-env.ts
+91
View File
@@ -0,0 +1,91 @@
import { CREDENTIAL_NAMES } from "./types.js";
const DATABASE_KEYS = ["DATABASE_URL", "DATABASE_MIGRATION_URL"] as const;
const AMBIENT_PAPERCLIP_CREDENTIAL_KEYS = [
"PAPERCLIP_API_KEY",
"PAPERCLIP_AGENT_API_KEY",
"PAPERCLIP_TASK_BRIDGE_TOKEN",
"PAPERCLIP_SETUP_TOKEN",
"PAPERCLIP_SECRETS_MASTER_KEY",
"PAPERCLIP_SECRETS_MASTER_KEY_FILE",
] as const;
const GENERATED_SERVER_SECRET_KEYS = [
"PAPERCLIP_AGENT_JWT_SECRET",
"PAPERCLIP_DECISION_SIGNING_SECRET",
"PAPERCLIP_TOOL_ACTION_SIGNING_SECRET",
"BETTER_AUTH_SECRET",
] as const;
const AMBIENT_EXTERNAL_STATE_KEYS = [
"PAPERCLIP_STORAGE_S3_BUCKET",
"PAPERCLIP_STORAGE_S3_REGION",
"PAPERCLIP_STORAGE_S3_ENDPOINT",
"PAPERCLIP_STORAGE_S3_PREFIX",
"PAPERCLIP_STORAGE_S3_FORCE_PATH_STYLE",
] as const;
const PROVIDER_SECRET_KEY = /^(?:OPENAI|ANTHROPIC|OPENROUTER|DAYTONA)(?:_|$)/;
/**
* Build the environment inherited by the Paperclip server. Paid credentials
* deliberately stay in the launcher/Playwright process and cross the server
* boundary only once, in the encrypted company-secrets API request.
*/
export function buildPaperclipServerEnvironment(
source: NodeJS.ProcessEnv,
overrides: NodeJS.ProcessEnv = {},
): NodeJS.ProcessEnv {
const result = { ...source };
for (const key of Object.keys(result)) {
if (PROVIDER_SECRET_KEY.test(key)) delete result[key];
}
for (const key of [
...CREDENTIAL_NAMES,
...DATABASE_KEYS,
...AMBIENT_PAPERCLIP_CREDENTIAL_KEYS,
...AMBIENT_EXTERNAL_STATE_KEYS,
]) {
delete result[key];
}
for (const key of GENERATED_SERVER_SECRET_KEYS) delete result[key];
Object.assign(result, overrides);
return result;
}
export function assertIsolatedServerEnvironment(
env: NodeJS.ProcessEnv,
expected: {
temporaryRoot: string;
paperclipHome: string;
configPath: string;
},
) {
const home = env.PAPERCLIP_HOME;
const config = env.PAPERCLIP_CONFIG;
if (home !== expected.paperclipHome || config !== expected.configPath) {
throw new Error(
"Paperclip server environment does not use the allocated home/config paths",
);
}
if (
!home.startsWith(`${expected.temporaryRoot}/`) ||
!config.startsWith(`${expected.temporaryRoot}/`)
) {
throw new Error(
"Paperclip server paths escape the isolated temporary root",
);
}
for (const key of [
...CREDENTIAL_NAMES,
...DATABASE_KEYS,
...AMBIENT_PAPERCLIP_CREDENTIAL_KEYS,
...AMBIENT_EXTERNAL_STATE_KEYS,
]) {
if (env[key])
throw new Error(
`Paperclip server environment unexpectedly contains ${key}`,
);
}
for (const key of GENERATED_SERVER_SECRET_KEYS) {
if (!env[key])
throw new Error(`Paperclip server environment is missing ${key}`);
}
}
tests/runner-e2e/history-index.ts
+221
View File
@@ -0,0 +1,221 @@
import type {
RunnerE2EHistoryCampaign,
RunnerE2EHistoryIndex,
} from "./types.js";
function html(value: unknown) {
return String(value ?? "")
.replaceAll("&", "&amp;")
.replaceAll("<", "&lt;")
.replaceAll(">", "&gt;")
.replaceAll('"', "&quot;")
.replaceAll("'", "&#39;");
}
function number(value: number) {
return new Intl.NumberFormat("en-US").format(value);
}
function usd(value: number) {
return `$${value.toFixed(value < 0.01 ? 6 : 2)}`;
}
function duration(durationMs: number) {
const seconds = Math.round(durationMs / 1_000);
if (seconds < 60) return `${seconds}s`;
const minutes = Math.floor(seconds / 60);
if (minutes < 60) return `${minutes}m ${seconds % 60}s`;
return `${Math.floor(minutes / 60)}h ${minutes % 60}m`;
}
function date(value: string) {
return new Intl.DateTimeFormat("en-US", {
dateStyle: "medium",
timeStyle: "short",
timeZone: "UTC",
}).format(new Date(value));
}
function campaignStatus(campaign: RunnerE2EHistoryCampaign) {
return campaign.failed === 0 &&
campaign.passed === campaign.selected &&
campaign.executed === campaign.selected &&
campaign.cleanupPassed
? "passed"
: "failed";
}
function sourceCell(campaign: RunnerE2EHistoryCampaign) {
const sha = campaign.source.sha;
const shortSha = sha?.slice(0, 8) ?? "Unknown";
const shaLabel = /^[0-9a-f]{40}$/i.test(sha ?? "")
? `<a href="https://github.com/paperclipai/paperclip/commit/${html(sha)}">${html(shortSha)}</a>`
: html(shortSha);
const workflow = campaign.source.workflowRunUrl
? `<a class="secondary-link" href="${html(campaign.source.workflowRunUrl)}">Workflow run</a>`
: "";
return `${shaLabel}<small>${html(campaign.source.ref ?? "Unknown ref")}${workflow}</small>`;
}
function campaignRow(campaign: RunnerE2EHistoryCampaign) {
const status = campaignStatus(campaign);
const suites = campaign.suites
.map(
(suite) =>
`<span>${html(suite.suiteId)} · ${suite.passed}/${suite.selected}</span>`,
)
.join("");
const billing = campaign.billing;
return `<tr>
<td data-label="Campaign">
<a class="campaign-link" href="${html(campaign.publicUrl)}">${html(campaign.campaignId)}</a>
<small>${html(date(campaign.generatedAt))} UTC</small>
</td>
<td data-label="Status">
<span class="status status-${status}">${status}</span>
<small>${campaign.complete ? "Complete campaign" : "Partial campaign"}${campaign.retries > 0 ? ` · ${campaign.retries} retries` : ""}</small>
</td>
<td data-label="Source">${sourceCell(campaign)}</td>
<td data-label="Suites"><div class="suite-list">${suites}</div></td>
<td data-label="Tests">
<strong>${campaign.passed}/${campaign.selected} passed</strong>
<small>${campaign.executed} executed · ${campaign.failed} failed</small>
</td>
<td data-label="Tokens">
<strong>${html(number(billing.llm.totalTokens))}</strong>
<small>${html(number(billing.llm.inputTokens))} in · ${html(number(billing.llm.outputTokens))} out · ${html(number(billing.llm.cachedInputTokens))} cached</small>
</td>
<td data-label="Cost">
<strong>${html(usd(billing.observedAndEstimatedCostUsd))}</strong>
<small>${html(usd(billing.reportedLlmCostUsd))} LLM · ${html(usd(billing.estimatedRuntimeCostUsd))} runtime</small>
</td>
<td data-label="Time">
<strong>${html(duration(billing.agentRunDurationMs))}</strong>
<small>${html(duration(billing.leaseDurationMs))} Daytona lease</small>
</td>
<td class="open-cell"><a href="${html(campaign.publicUrl)}" aria-label="Open campaign ${html(campaign.campaignId)}">Open report&nbsp;→</a></td>
</tr>`;
}
export function renderRunnerHistoryIndex(history: RunnerE2EHistoryIndex) {
const campaigns = [...history.campaigns].sort((left, right) =>
right.generatedAt.localeCompare(left.generatedAt),
);
const passed = campaigns.filter(
(campaign) => campaignStatus(campaign) === "passed",
).length;
const latest = campaigns.find(
(campaign) => campaign.campaignId === history.latestCampaignId,
);
const latestGreen = campaigns.find(
(campaign) => campaign.campaignId === history.latestGreenCampaignId,
);
const totalCost = campaigns.reduce(
(sum, campaign) => sum + campaign.billing.observedAndEstimatedCostUsd,
0,
);
const rows =
campaigns.length > 0
? campaigns.map(campaignRow).join("")
: `<tr><td class="empty" colspan="9">No campaigns have been published yet.</td></tr>`;
return `<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="color-scheme" content="light dark">
<meta name="theme-color" content="#ffffff" media="(prefers-color-scheme: light)">
<meta name="theme-color" content="#141413" media="(prefers-color-scheme: dark)">
<link rel="icon" href="assets/favicon.svg" type="image/svg+xml">
<title>Runner E2E Campaigns · Paperclip</title>
<style>
@font-face { font-family: "Paperclip Inter"; src: url("assets/InterVariable.woff2") format("woff2"); font-style: normal; font-weight: 100 900; font-display: swap; }
:root { color-scheme: light dark; --background:#fff; --foreground:#0a0a0a; --muted:#60666a; --border:#e5e5e5; --raised:#fafafa; --pass:#188a3c; --pass-bg:#dcfce7; --fail:#991b1b; --fail-bg:#fee2e2; --font:"Paperclip Inter",Inter,ui-sans-serif,system-ui,-apple-system,sans-serif; --mono:ui-monospace,SFMono-Regular,Menlo,monospace; }
@media (prefers-color-scheme: dark) { :root { --background:#141413; --foreground:#fafafa; --muted:#a3a3a3; --border:rgb(255 255 255 / 12%); --raised:#1c1c1b; --pass:#34d06f; --pass-bg:#22c55e1f; --fail:#ef4444; --fail-bg:#dc26262e; } }
* { box-sizing:border-box; }
body { margin:0; background:var(--background); color:var(--foreground); font:14px/1.45 var(--font); }
a { color:inherit; text-underline-offset:3px; }
.brand-bar { min-height:64px; display:flex; align-items:center; justify-content:space-between; gap:24px; padding:0 max(24px,calc((100vw - 1560px)/2)); border-bottom:1px solid var(--border); }
.brand-lockup { display:inline-flex; align-items:center; gap:10px; font-size:16px; font-weight:650; text-decoration:none; }
.brand-lockup svg { width:24px; height:24px; }
.brand-context { color:var(--muted); font-size:12px; }
main { width:min(1560px,calc(100% - 48px)); margin:48px auto 72px; }
header { display:flex; align-items:end; justify-content:space-between; gap:32px; margin-bottom:36px; }
.eyebrow { margin:0 0 10px; color:var(--muted); font:500 11px/1.4 var(--mono); letter-spacing:.08em; text-transform:uppercase; }
h1 { margin:0; font-size:clamp(34px,4vw,56px); line-height:1.04; letter-spacing:-.04em; }
.lede { max-width:680px; margin:14px 0 0; color:var(--muted); font-size:16px; }
.summary { display:flex; border:1px solid var(--border); border-radius:8px; background:var(--raised); }
.metric { min-width:112px; padding:13px 16px; border-right:1px solid var(--border); }
.metric:last-child { border:0; }
.metric strong,.metric span { display:block; }
.metric strong { font:600 19px/1.2 var(--mono); }
.metric span { margin-top:4px; color:var(--muted); font-size:10px; letter-spacing:.06em; text-transform:uppercase; }
.pointers { display:flex; flex-wrap:wrap; gap:10px; margin-bottom:16px; }
.pointers a { padding:8px 11px; border:1px solid var(--border); border-radius:7px; background:var(--raised); font-size:12px; text-decoration:none; }
.pointers a:hover,.campaign-link:hover,.open-cell a:hover { text-decoration:underline; }
.table-wrap { border-top:1px solid var(--border); border-bottom:1px solid var(--border); }
table { width:100%; border-collapse:collapse; }
th,td { padding:14px 12px; border-bottom:1px solid var(--border); text-align:left; vertical-align:top; }
tr:last-child td { border-bottom:0; }
th { background:var(--raised); color:var(--muted); font-size:10px; letter-spacing:.06em; text-transform:uppercase; white-space:nowrap; }
td strong,.campaign-link { font-weight:650; }
td small { display:block; margin-top:4px; color:var(--muted); font-size:10px; }
td a.secondary-link { display:block; margin-top:3px; }
.campaign-link { font:600 11px/1.4 var(--mono); overflow-wrap:anywhere; }
.status { display:inline-block; padding:3px 8px; border-radius:999px; font-size:10px; font-weight:700; letter-spacing:.05em; text-transform:uppercase; }
.status-passed { color:var(--pass); background:var(--pass-bg); }
.status-failed { color:var(--fail); background:var(--fail-bg); }
.suite-list { display:grid; gap:3px; font-size:11px; white-space:nowrap; }
.open-cell { text-align:right; white-space:nowrap; font-weight:650; }
.empty { padding:48px; color:var(--muted); text-align:center; }
footer { display:flex; justify-content:space-between; gap:16px; margin-top:20px; color:var(--muted); font-size:11px; }
@media (max-width:1100px) { header { align-items:start; flex-direction:column; } .summary { width:100%; } .metric { flex:1; } }
@media (max-width:860px) {
.brand-context { display:none; } main { width:min(100% - 28px,1560px); margin-top:32px; }
.summary { display:grid; grid-template-columns:repeat(3,1fr); } .metric { min-width:0; border-bottom:1px solid var(--border); }
.table-wrap { border:0; } thead { display:none; } table,tbody,tr,td { display:block; width:100%; }
tr { margin-bottom:14px; padding:8px 14px; border:1px solid var(--border); border-radius:8px; background:var(--raised); }
td { display:grid; grid-template-columns:105px minmax(0,1fr); gap:12px; padding:10px 0; }
td::before { content:attr(data-label); color:var(--muted); font-size:10px; letter-spacing:.06em; text-transform:uppercase; }
.open-cell { display:block; padding-top:14px; text-align:left; } .open-cell::before { display:none; }
}
@media (max-width:520px) { .summary { grid-template-columns:1fr; } .metric { border-right:0; } footer { flex-direction:column; } }
</style>
</head>
<body>
<div class="brand-bar">
<a class="brand-lockup" href="https://paperclip.ing" aria-label="Paperclip home">
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="m16 6-8.414 8.586a2 2 0 0 0 2.829 2.829l8.414-8.586a4 4 0 1 0-5.657-5.657l-8.379 8.551a6 6 0 1 0 8.485 8.485l8.379-8.551"/></svg>
<span>Paperclip</span>
</a>
<span class="brand-context">Quality engineering · Runner acceptance</span>
</div>
<main>
<header>
<div>
<p class="eyebrow">Historical test reporting</p>
<h1>Runner E2E campaigns</h1>
<p class="lede">Each row is one workflow campaign against a Paperclip revision. Open a report for its configuration matrices, matchers, per-test billing, and sanitized structured evidence. Visual evidence remains in access-controlled workflow artifacts.</p>
</div>
<div class="summary" aria-label="History summary">
<div class="metric"><strong>${campaigns.length}</strong><span>Campaigns</span></div>
<div class="metric"><strong>${passed}</strong><span>Passed</span></div>
<div class="metric"><strong>${html(usd(totalCost))}</strong><span>Recorded cost</span></div>
</div>
</header>
<nav class="pointers" aria-label="Campaign pointers">
${latest ? `<a href="${html(latest.publicUrl)}">Latest run · ${html(latest.campaignId)}</a>` : ""}
${latestGreen ? `<a href="${html(latestGreen.publicUrl)}">Latest complete green · ${html(latestGreen.campaignId)}</a>` : ""}
</nav>
<div class="table-wrap">
<table>
<thead><tr><th>Campaign</th><th>Status</th><th>Source</th><th>Suites</th><th>Tests</th><th>Tokens</th><th>Cost</th><th>Agent time</th><th></th></tr></thead>
<tbody>${rows}</tbody>
</table>
</div>
<footer><span>Updated ${html(date(history.updatedAt))} UTC</span><span>Immutable campaign reports · Inert structured public evidence</span></footer>
</main>
</body>
</html>`;
}
tests/runner-e2e/history-publish.ts
+499
View File
@@ -0,0 +1,499 @@
import { createHash } from "node:crypto";
import { execFile } from "node:child_process";
import path from "node:path";
import { promisify } from "node:util";
import {
mkdtemp,
lstat,
readFile,
readdir,
rm,
stat,
writeFile,
} from "node:fs/promises";
import os from "node:os";
import { regenerateRunnerDashboard } from "./dashboard-regenerate.js";
import { renderRunnerHistoryIndex } from "./history-index.js";
import {
campaignHistoryRecord,
emptyRunnerHistory,
mergeRunnerHistory,
} from "./history.js";
import type { RunnerE2ECampaign, RunnerE2EHistoryIndex } from "./types.js";
const execFileAsync = promisify(execFile);
const MUTABLE_HISTORY_FILES = new Set([
"history.json",
"latest.json",
"latest-green.json",
]);
const PUBLISH_ROOT_FILES = new Set([
"dashboard.html",
"index.html",
"junit.xml",
"normalized-results.json",
"summary.md",
]);
const PUBLIC_EVIDENCE_EXTENSIONS = new Set([".json", ".log", ".md", ".txt"]);
const PRIVATE_EVIDENCE_DIRECTORIES = new Set([
"blob-report",
"html-report",
"playwright-output",
]);
function publicEvidencePath(relative: string) {
const match = relative.match(
/^evidence\/[A-Za-z0-9._-]+\/attempt-[1-9][0-9]*\/(.+)$/,
);
if (!match) return false;
const evidencePath = match[1]!;
const segments = evidencePath.split("/");
if (
segments.some(
(segment) =>
!segment ||
segment === "." ||
segment === ".." ||
PRIVATE_EVIDENCE_DIRECTORIES.has(segment),
)
) {
return false;
}
return PUBLIC_EVIDENCE_EXTENSIONS.has(
path.posix.extname(evidencePath).toLowerCase(),
);
}
export function isHistoricalBundlePathAllowed(relative: string) {
if (
relative.includes("\\") ||
relative.startsWith("/") ||
relative.includes("..")
) {
return false;
}
if (PUBLISH_ROOT_FILES.has(relative)) return true;
if (
relative === "assets/favicon.svg" ||
relative === "assets/InterVariable.woff2"
) {
return true;
}
return publicEvidencePath(relative);
}
async function pruneEvidenceDirectory(root: string, current: string) {
const entries = await readdir(current, { withFileTypes: true });
for (const entry of entries) {
const absolute = path.join(current, entry.name);
if (entry.isDirectory()) {
await pruneEvidenceDirectory(root, absolute);
if ((await readdir(absolute)).length === 0) {
await rm(absolute, { recursive: true });
}
continue;
}
const relative = path.relative(root, absolute).split(path.sep).join("/");
if (!entry.isFile() || !publicEvidencePath(relative)) {
await rm(absolute, { force: true });
}
}
}
export async function prunePrivateHistoryEvidence(root: string) {
const evidenceRoot = path.join(root, "evidence");
const metadata = await lstat(evidenceRoot).catch(() => null);
if (!metadata) return;
if (!metadata.isDirectory()) {
throw new Error("Historical evidence root must be a directory");
}
await pruneEvidenceDirectory(root, evidenceRoot);
}
interface BundleManifest {
schema: "paperclip.runner-e2e.bundle/v1";
campaignId: string;
bundleDigest: string;
files: Array<{ path: string; sha256: string; bytes: number }>;
}
function json(value: unknown) {
return `${JSON.stringify(value, null, 2)}\n`;
}
export function validateHistoryDestination(input: {
bucket: string;
prefix: string;
publicBaseUrl: string;
}) {
if (!/^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$/.test(input.bucket)) {
throw new Error("RUNNER_E2E_HISTORY_S3_BUCKET is not a valid bucket name");
}
const prefix = input.prefix.replace(/^\/+|\/+$/g, "");
if (
!prefix ||
prefix
.split("/")
.some((segment) => !segment || segment === "." || segment === "..")
) {
throw new Error(
"RUNNER_E2E_HISTORY_PREFIX must be a safe non-empty key prefix",
);
}
const publicUrl = new URL(input.publicBaseUrl);
if (
publicUrl.protocol !== "https:" ||
publicUrl.username ||
publicUrl.password ||
publicUrl.search ||
publicUrl.hash
) {
throw new Error(
"RUNNER_E2E_HISTORY_PUBLIC_BASE_URL must be a credential-free HTTPS URL",
);
}
return { prefix, publicBaseUrl: publicUrl.href.replace(/\/$/, "") };
}
async function relativeFiles(root: string, current = root): Promise<string[]> {
const entries = await readdir(current, { withFileTypes: true });
const files: string[] = [];
for (const entry of entries) {
const absolute = path.join(current, entry.name);
if (entry.isSymbolicLink()) {
throw new Error(`Refusing to publish symbolic link ${entry.name}`);
}
if (entry.isDirectory()) {
files.push(...(await relativeFiles(root, absolute)));
} else if (entry.isFile()) {
const relative = path.relative(root, absolute).split(path.sep).join("/");
if (MUTABLE_HISTORY_FILES.has(relative)) continue;
if (!isHistoricalBundlePathAllowed(relative)) {
throw new Error(
`Refusing non-allowlisted historical bundle path ${relative}`,
);
}
files.push(relative);
}
}
return files;
}
export async function createBundleManifest(
root: string,
campaignId: string,
): Promise<BundleManifest> {
if (!/^[A-Za-z0-9][A-Za-z0-9._-]{0,199}$/.test(campaignId)) {
throw new Error("Campaign ID is unsafe for immutable object storage");
}
const files = await Promise.all(
(await relativeFiles(root)).sort().map(async (relative) => {
const absolute = path.join(root, ...relative.split("/"));
const [content, metadata] = await Promise.all([
readFile(absolute),
stat(absolute),
]);
return {
path: relative,
sha256: createHash("sha256").update(content).digest("hex"),
bytes: metadata.size,
};
}),
);
const bundleDigest = createHash("sha256")
.update(JSON.stringify(files))
.digest("hex");
return {
schema: "paperclip.runner-e2e.bundle/v1",
campaignId,
bundleDigest,
files,
};
}
export function buildHistoryPointers(history: RunnerE2EHistoryIndex) {
const byCampaign = new Map(
history.campaigns.map((campaign) => [campaign.campaignId, campaign]),
);
const pointer = (campaignId: string | null | undefined) => {
const campaign = campaignId ? byCampaign.get(campaignId) : undefined;
return campaign
? {
campaignId: campaign.campaignId,
generatedAt: campaign.generatedAt,
publicUrl: campaign.publicUrl,
sha: campaign.source.sha,
}
: null;
};
return {
latest: {
schema: "paperclip.runner-e2e.pointer/v1",
updatedAt: history.updatedAt,
overall: pointer(history.latestCampaignId),
suites: Object.fromEntries(
Object.entries(history.latestBySuite).map(([suiteId, campaignId]) => [
suiteId,
pointer(campaignId),
]),
),
},
latestGreen: {
schema: "paperclip.runner-e2e.pointer/v1",
updatedAt: history.updatedAt,
overall: pointer(history.latestGreenCampaignId),
suites: Object.fromEntries(
Object.entries(history.latestGreenBySuite).map(
([suiteId, campaignId]) => [suiteId, pointer(campaignId)],
),
),
},
};
}
function awsObject(bucket: string, key: string) {
return `s3://${bucket}/${key}`;
}
async function objectExists(bucket: string, key: string) {
try {
await execFileAsync("aws", [
"s3api",
"head-object",
"--bucket",
bucket,
"--key",
key,
]);
return true;
} catch (error) {
const detail = String(
(error as { stderr?: string }).stderr ??
(error instanceof Error ? error.message : error),
);
if (/\b(?:404|Not Found|NoSuchKey)\b/i.test(detail)) return false;
throw new Error(
`Unable to inspect historical object: ${detail.slice(0, 400)}`,
);
}
}
async function downloadJson<T>(
bucket: string,
key: string,
destination: string,
) {
if (!(await objectExists(bucket, key))) return null;
await execFileAsync("aws", [
"s3",
"cp",
awsObject(bucket, key),
destination,
"--only-show-errors",
]);
return JSON.parse(await readFile(destination, "utf8")) as T;
}
async function uploadJson(
bucket: string,
key: string,
file: string,
cacheControl: string,
) {
await uploadFile(bucket, key, file, "application/json", cacheControl);
}
async function uploadFile(
bucket: string,
key: string,
file: string,
contentType: string,
cacheControl: string,
) {
await execFileAsync("aws", [
"s3",
"cp",
file,
awsObject(bucket, key),
"--only-show-errors",
"--content-type",
contentType,
"--cache-control",
cacheControl,
]);
}
async function main() {
const reportRoot = path.resolve(
process.env.PAPERCLIP_RUNNER_E2E_REPORT_DIR ??
"runner-e2e-merged-report/normalized",
);
const bucket = process.env.RUNNER_E2E_HISTORY_S3_BUCKET ?? "";
const destination = validateHistoryDestination({
bucket,
prefix: process.env.RUNNER_E2E_HISTORY_PREFIX ?? "runner-e2e",
publicBaseUrl: process.env.RUNNER_E2E_HISTORY_PUBLIC_BASE_URL ?? "",
});
const campaign = JSON.parse(
await readFile(path.join(reportRoot, "normalized-results.json"), "utf8"),
) as RunnerE2ECampaign;
if (campaign.schema !== "paperclip.runner-e2e.campaign/v2") {
throw new Error("Historical publishing requires a v2 normalized campaign");
}
const temporary = await mkdtemp(
path.join(os.tmpdir(), "runner-e2e-history-"),
);
const historyKey = `${destination.prefix}/history.json`;
const current =
(await downloadJson<RunnerE2EHistoryIndex>(
bucket,
historyKey,
path.join(temporary, "current-history.json"),
)) ?? emptyRunnerHistory();
const history = mergeRunnerHistory(
current,
campaignHistoryRecord(
campaign,
`${destination.publicBaseUrl}/${destination.prefix}`,
),
);
// Campaign bundles are immutable and must not capture a mutable history
// file left in a reused local directory. The root landing page below is the
// only dashboard that embeds navigation across campaigns.
// Raster/video pixels are not OCR-scanned for secrets, and generated HTML,
// archives, and SVG may contain or execute active/private content. Preserve
// those in the access-controlled workflow artifact but remove them from the
// directory shared by public S3 and Pages publication.
await prunePrivateHistoryEvidence(reportRoot);
await regenerateRunnerDashboard({ bundle: reportRoot, historyFile: null });
const manifest = await createBundleManifest(reportRoot, campaign.campaignId);
const campaignPrefix = `${destination.prefix}/campaigns/${campaign.campaignId}`;
const manifestKey = `${campaignPrefix}/bundle-manifest.json`;
const existingManifest = await downloadJson<BundleManifest>(
bucket,
manifestKey,
path.join(temporary, "existing-manifest.json"),
);
if (
existingManifest &&
existingManifest.bundleDigest !== manifest.bundleDigest
) {
throw new Error(
`Immutable campaign ${campaign.campaignId} already exists with a different digest`,
);
}
if (!existingManifest) {
await execFileAsync("aws", [
"s3",
"cp",
reportRoot,
awsObject(bucket, campaignPrefix),
"--recursive",
"--only-show-errors",
"--exclude",
"history.json",
"--exclude",
"latest.json",
"--exclude",
"latest-green.json",
"--cache-control",
"public,max-age=31536000,immutable",
]);
const manifestFile = path.join(temporary, "bundle-manifest.json");
await writeFile(manifestFile, json(manifest), "utf8");
await uploadJson(
bucket,
manifestKey,
manifestFile,
"public,max-age=31536000,immutable",
);
}
const pointers = buildHistoryPointers(history);
const historyFile = path.join(reportRoot, "history.json");
const latestFile = path.join(reportRoot, "latest.json");
const latestGreenFile = path.join(reportRoot, "latest-green.json");
await Promise.all([
writeFile(historyFile, json(history), "utf8"),
writeFile(latestFile, json(pointers.latest), "utf8"),
writeFile(latestGreenFile, json(pointers.latestGreen), "utf8"),
]);
await regenerateRunnerDashboard({ bundle: reportRoot, historyFile });
const landingDirectory = path.join(temporary, "landing");
await regenerateRunnerDashboard({
bundle: reportRoot,
historyFile,
outputDirectory: landingDirectory,
evidenceHrefPrefix: `campaigns/${campaign.campaignId}`,
});
await writeFile(
path.join(landingDirectory, "index.html"),
renderRunnerHistoryIndex(history),
"utf8",
);
await Promise.all([
uploadJson(bucket, historyKey, historyFile, "no-cache"),
uploadJson(
bucket,
`${destination.prefix}/latest.json`,
latestFile,
"no-cache",
),
uploadJson(
bucket,
`${destination.prefix}/latest-green.json`,
latestGreenFile,
"no-cache",
),
uploadFile(
bucket,
`${destination.prefix}/index.html`,
path.join(landingDirectory, "index.html"),
"text/html; charset=utf-8",
"no-cache",
),
uploadFile(
bucket,
`${destination.prefix}/dashboard.html`,
path.join(landingDirectory, "dashboard.html"),
"text/html; charset=utf-8",
"no-cache",
),
uploadFile(
bucket,
`${destination.prefix}/normalized-results.json`,
path.join(landingDirectory, "normalized-results.json"),
"application/json",
"no-cache",
),
uploadFile(
bucket,
`${destination.prefix}/assets/favicon.svg`,
path.join(reportRoot, "assets", "favicon.svg"),
"image/svg+xml",
"public,max-age=86400",
),
uploadFile(
bucket,
`${destination.prefix}/assets/InterVariable.woff2`,
path.join(reportRoot, "assets", "InterVariable.woff2"),
"font/woff2",
"public,max-age=86400",
),
]);
console.log(
`Published immutable campaign ${campaign.campaignId} (${manifest.bundleDigest}) and ${history.campaigns.length} history record(s)`,
);
}
if (
process.argv[1] &&
path.resolve(process.argv[1]) === path.resolve(import.meta.filename)
) {
await main().catch((error) => {
console.error(error instanceof Error ? error.message : String(error));
process.exitCode = 1;
});
}
tests/runner-e2e/history.test.ts
+363
View File
@@ -0,0 +1,363 @@
import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { afterEach, describe, expect, it } from "vitest";
import { runnerMatrix } from "./catalog.js";
import { regenerateRunnerDashboard } from "./dashboard-regenerate.js";
import { renderRunnerE2EDashboard } from "./dashboard.js";
import {
buildHistoryPointers,
createBundleManifest,
isHistoricalBundlePathAllowed,
prunePrivateHistoryEvidence,
validateHistoryDestination,
} from "./history-publish.js";
import {
buildRunnerCampaign,
campaignHistoryRecord,
canonicalExecutionId,
emptyRunnerHistory,
mergeRunnerHistory,
} from "./history.js";
import { renderRunnerHistoryIndex } from "./history-index.js";
import type { MatrixExecution, RunnerE2EResult } from "./types.js";
const temporaryDirectories: string[] = [];
afterEach(async () => {
await Promise.all(
temporaryDirectories
.splice(0)
.map((directory) => rm(directory, { recursive: true, force: true })),
);
});
function result(execution: MatrixExecution, status: "passed" | "failed") {
return {
schema: "paperclip.runner-e2e.result/v2",
executionId: execution.id,
suiteId: execution.suite.id,
suiteDefinitionHash: execution.suiteDefinitionHash,
attempt: 1,
status,
profileId: execution.profile.id,
environmentId: execution.environment.id,
caseId: execution.task.id,
provider: execution.profile.provider,
model: execution.profile.model,
runtimeMode: execution.profile.expectedRuntimeMode,
runIds: ["run-1"],
usage: { inputTokens: 100, outputTokens: 25, costUsd: 0.01 },
startedAt: "2026-08-28T00:00:00.000Z",
finishedAt: "2026-08-28T00:00:01.000Z",
durationMs: 1_000,
cleanup: "passed",
} satisfies RunnerE2EResult;
}
describe("runner E2E campaign history", () => {
it("migrates v1 execution IDs and keeps partial suite runs out of overall trends", () => {
expect(canonicalExecutionId("legacy-codex.local.message-marker")).toBe(
"core-compatibility.legacy-codex.local.message-marker",
);
const breadth = runnerMatrix.filter(
(execution) => execution.suite.id === "openrouter-model-breadth",
);
const campaign = buildRunnerCampaign({
campaignId: "breadth-smoke",
generatedAt: "2026-08-28T00:01:00.000Z",
expected: breadth.map((execution) => execution.id),
results: breadth.map((execution) => result(execution, "passed")),
});
expect(campaign).toMatchObject({ complete: false, passed: 15, failed: 0 });
expect(campaign.suites[0]).toMatchObject({
suiteId: "openrouter-model-breadth",
complete: true,
selected: 15,
});
expect(campaign.billing).toMatchObject({
reportedLlmCostUsd: 0.15,
llm: { inputTokens: 1_500, outputTokens: 375 },
});
const history = mergeRunnerHistory(
emptyRunnerHistory(),
campaignHistoryRecord(campaign, "https://history.example/runner-e2e"),
);
expect(history.latestGreenCampaignId).toBeNull();
expect(history.latestGreenBySuite).toEqual({
"openrouter-model-breadth": "breadth-smoke",
});
});
it("retains latest and latest-green pointers independently", () => {
const green = buildRunnerCampaign({
campaignId: "complete-green",
generatedAt: "2026-08-28T00:01:00.000Z",
expected: runnerMatrix.map((execution) => execution.id),
results: runnerMatrix.map((execution) => result(execution, "passed")),
});
const red = buildRunnerCampaign({
campaignId: "complete-red",
generatedAt: "2026-08-28T01:01:00.000Z",
expected: runnerMatrix.map((execution) => execution.id),
results: runnerMatrix.map((execution, index) =>
result(execution, index === 0 ? "failed" : "passed"),
),
});
let history = mergeRunnerHistory(
emptyRunnerHistory(),
campaignHistoryRecord(green, "https://history.example/runner-e2e"),
);
history = mergeRunnerHistory(
history,
campaignHistoryRecord(red, "https://history.example/runner-e2e"),
);
const pointers = buildHistoryPointers(history);
expect(pointers.latest.overall).toMatchObject({
campaignId: "complete-red",
});
expect(pointers.latestGreen.overall).toMatchObject({
campaignId: "complete-green",
});
expect(history.campaigns).toHaveLength(2);
const dashboard = renderRunnerE2EDashboard({
title: "Runner Full-Stack E2E",
generatedAt: red.generatedAt,
expected: red.expected,
catalog: runnerMatrix,
campaign: red,
history,
entries: red.results.map((campaignResult) => ({
result: campaignResult,
valid: campaignResult.status === "passed",
errors: campaignResult.status === "passed" ? [] : ["failed"],
})),
});
expect(dashboard).toContain("Campaign trends");
expect(dashboard).toContain("data-history-from");
expect(dashboard).toContain("data-history-through");
expect(dashboard).toContain(
'data-history-suite-trends="core-compatibility"',
);
expect(dashboard).toContain(
'data-history-suite-trends="openrouter-model-breadth"',
);
expect(dashboard).toContain("Suite pass rate");
expect(dashboard).toContain("lines break at definition changes");
expect(dashboard).toContain("cleanup passed");
const index = renderRunnerHistoryIndex(history);
expect(index).toContain("Runner E2E campaigns");
expect(index).toContain("complete-green");
expect(index).toContain("complete-red");
expect(index).toContain("57/57 passed");
expect(index).toContain("56/57 passed");
expect(index).toContain("Open report&nbsp;→");
expect(index).toContain(
"Visual evidence remains in access-controlled workflow artifacts",
);
expect(index).toContain("Inert structured public evidence");
expect(index).not.toContain("data-gallery-dialog");
expect(index).not.toContain("Configuration matrix");
});
});
describe("historical publication security", () => {
it("keeps visual and active evidence private when building the public dashboard", async () => {
const root = await mkdtemp(path.join(os.tmpdir(), "runner-landing-test-"));
const output = path.join(root, "landing");
temporaryDirectories.push(root);
const execution = runnerMatrix[0]!;
const campaignResult = {
...result(execution, "passed"),
screenshots: [
{
id: "final-state",
label: "Final state",
file: "final-state.png",
},
],
} satisfies RunnerE2EResult;
const campaign = buildRunnerCampaign({
campaignId: "campaign-1",
generatedAt: "2026-08-28T00:01:00.000Z",
expected: [execution.id],
results: [campaignResult],
});
const evidenceDirectory = path.join(
root,
"evidence",
execution.id,
"attempt-1",
);
await mkdir(evidenceDirectory, { recursive: true });
await writeFile(
path.join(root, "normalized-results.json"),
JSON.stringify(campaign),
);
await writeFile(path.join(evidenceDirectory, "final-state.png"), "png");
await writeFile(path.join(evidenceDirectory, "failure.webm"), "webm");
await writeFile(path.join(evidenceDirectory, "unsafe.svg"), "<svg />");
await writeFile(
path.join(evidenceDirectory, "junit.xml"),
"<?xml-stylesheet href='https://example.test/private.xsl'?>",
);
await writeFile(path.join(evidenceDirectory, "result.json"), "{}\n");
await mkdir(path.join(evidenceDirectory, "snapshots"));
await writeFile(
path.join(evidenceDirectory, "snapshots", "api-state.json"),
"{}\n",
);
await mkdir(path.join(evidenceDirectory, "html-report"));
await writeFile(
path.join(evidenceDirectory, "html-report", "index.html"),
"<img src='data:image/png;base64,cHJpdmF0ZQ==' />",
);
await mkdir(path.join(evidenceDirectory, "blob-report"));
await writeFile(
path.join(evidenceDirectory, "blob-report", "report.zip"),
"private archive",
);
await prunePrivateHistoryEvidence(root);
await regenerateRunnerDashboard({
bundle: root,
outputDirectory: output,
evidenceHrefPrefix: "campaigns/campaign-1",
});
const dashboard = await readFile(path.join(output, "index.html"), "utf8");
expect(dashboard).not.toContain(
`campaigns/campaign-1/evidence/${execution.id}/attempt-1/final-state.png`,
);
expect(dashboard).toContain("Visual evidence · workflow artifact only");
expect(dashboard).toContain(
"public history contains inert structured evidence only",
);
expect(dashboard).toContain("Public history excludes visual evidence");
await expect(
readFile(path.join(evidenceDirectory, "final-state.png")),
).rejects.toThrow();
await expect(
readFile(path.join(evidenceDirectory, "failure.webm")),
).rejects.toThrow();
await expect(
readFile(path.join(evidenceDirectory, "unsafe.svg")),
).rejects.toThrow();
await expect(
readFile(path.join(evidenceDirectory, "junit.xml")),
).rejects.toThrow();
await expect(
readFile(path.join(evidenceDirectory, "html-report", "index.html")),
).rejects.toThrow();
await expect(
readFile(path.join(evidenceDirectory, "blob-report", "report.zip")),
).rejects.toThrow();
await expect(
readFile(path.join(evidenceDirectory, "result.json"), "utf8"),
).resolves.toBe("{}\n");
await expect(
readFile(
path.join(evidenceDirectory, "snapshots", "api-state.json"),
"utf8",
),
).resolves.toBe("{}\n");
expect(
JSON.parse(
await readFile(path.join(output, "normalized-results.json"), "utf8"),
).schema,
).toBe("paperclip.runner-e2e.campaign/v2");
await expect(
regenerateRunnerDashboard({
bundle: root,
outputDirectory: output,
evidenceHrefPrefix: "../unsafe",
}),
).rejects.toThrow("safe relative URL path");
});
it("requires a private-origin-compatible destination shape", () => {
expect(
validateHistoryDestination({
bucket: "paperclip-runner-e2e-history",
prefix: "/runner-e2e/",
publicBaseUrl: "https://history.paperclip.ai/",
}),
).toEqual({
prefix: "runner-e2e",
publicBaseUrl: "https://history.paperclip.ai",
});
expect(() =>
validateHistoryDestination({
bucket: "paperclip-runner-e2e-history",
prefix: "../unsafe",
publicBaseUrl: "https://history.paperclip.ai/",
}),
).toThrow("safe non-empty key prefix");
expect(() =>
validateHistoryDestination({
bucket: "paperclip-runner-e2e-history",
prefix: "runner-e2e",
publicBaseUrl: "http://history.paperclip.ai/",
}),
).toThrow("credential-free HTTPS");
});
it("rejects non-allowlisted files and fingerprints an immutable bundle", async () => {
expect(isHistoricalBundlePathAllowed("normalized-results.json")).toBe(true);
expect(isHistoricalBundlePathAllowed("junit.xml")).toBe(true);
expect(
isHistoricalBundlePathAllowed(
"evidence/core-compatibility.profile.local.case/attempt-1/final-state.png",
),
).toBe(false);
expect(
isHistoricalBundlePathAllowed(
"evidence/core-compatibility.profile.local.case/attempt-1/failure.webm",
),
).toBe(false);
expect(
isHistoricalBundlePathAllowed(
"evidence/core-compatibility.profile.local.case/attempt-1/unsafe.svg",
),
).toBe(false);
expect(
isHistoricalBundlePathAllowed(
"evidence/core-compatibility.profile.local.case/attempt-1/junit.xml",
),
).toBe(false);
expect(
isHistoricalBundlePathAllowed(
"evidence/core-compatibility.profile.local.case/attempt-1/blob-report/report.zip",
),
).toBe(false);
expect(
isHistoricalBundlePathAllowed(
"evidence/core-compatibility.profile.local.case/attempt-1/html-report/index.html",
),
).toBe(false);
expect(
isHistoricalBundlePathAllowed(
"evidence/core-compatibility.profile.local.case/attempt-1/result.json",
),
).toBe(true);
expect(
isHistoricalBundlePathAllowed(
"evidence/core-compatibility.profile.local.case/attempt-1/snapshots/api-state.json",
),
).toBe(true);
expect(isHistoricalBundlePathAllowed("paperclip-home/database")).toBe(
false,
);
const root = await mkdtemp(path.join(os.tmpdir(), "runner-history-test-"));
temporaryDirectories.push(root);
await mkdir(path.join(root, "assets"));
await writeFile(path.join(root, "index.html"), "safe");
await writeFile(path.join(root, "assets", "favicon.svg"), "safe");
const first = await createBundleManifest(root, "campaign-1");
const second = await createBundleManifest(root, "campaign-1");
expect(first.bundleDigest).toBe(second.bundleDigest);
await writeFile(path.join(root, "database.sqlite"), "unsafe");
await expect(createBundleManifest(root, "campaign-1")).rejects.toThrow(
"non-allowlisted",
);
});
});
tests/runner-e2e/history.ts
+245
View File
@@ -0,0 +1,245 @@
import { runnerMatrix, runnerSuites } from "./catalog.js";
import {
aggregateCampaignBilling,
summarizeExecutionBilling,
} from "./billing.js";
import type {
RunnerE2ECampaign,
RunnerE2EHistoryCampaign,
RunnerE2EHistoryIndex,
RunnerE2EResult,
} from "./types.js";
export function canonicalExecutionId(id: string) {
if (runnerMatrix.some((execution) => execution.id === id)) return id;
const coreId = `core-compatibility.${id}`;
return runnerMatrix.some((execution) => execution.id === coreId)
? coreId
: id;
}
export function upgradeRunnerResult(result: RunnerE2EResult): RunnerE2EResult {
const executionId = canonicalExecutionId(result.executionId);
const execution = runnerMatrix.find(
(candidate) => candidate.id === executionId,
);
if (!execution) return result;
return {
...result,
executionId,
suiteId: result.suiteId ?? execution.suite.id,
suiteDefinitionHash:
result.suiteDefinitionHash ?? execution.suiteDefinitionHash,
...(result.schema === "paperclip.runner-e2e.result/v1"
? {
source: result.source ?? {
sha: null,
ref: null,
workflowRunUrl: null,
},
}
: {}),
};
}
export function buildRunnerCampaign(input: {
campaignId: string;
generatedAt: string;
expected: readonly string[];
results: readonly RunnerE2EResult[];
eventName?: string | null;
}): RunnerE2ECampaign {
const expected = input.expected.map(canonicalExecutionId);
const results = input.results.map((result) => ({
...upgradeRunnerResult(result),
billing: result.billing ?? summarizeExecutionBilling(result),
}));
const resultSource = results.find((result) => result.source)?.source;
const source = {
sha: resultSource?.sha ?? process.env.GITHUB_SHA ?? null,
ref: resultSource?.ref ?? process.env.GITHUB_REF ?? null,
workflowRunUrl:
resultSource?.workflowRunUrl ??
(process.env.GITHUB_SERVER_URL &&
process.env.GITHUB_REPOSITORY &&
process.env.GITHUB_RUN_ID
? `${process.env.GITHUB_SERVER_URL}/${process.env.GITHUB_REPOSITORY}/actions/runs/${process.env.GITHUB_RUN_ID}`
: null),
eventName: input.eventName ?? process.env.GITHUB_EVENT_NAME ?? null,
};
const suites = runnerSuites
.map((suite) => {
const suiteExpected = expected.filter((id) =>
id.startsWith(`${suite.id}.`),
);
if (suiteExpected.length === 0) return null;
const suiteResults = results.filter(
(result) => result.suiteId === suite.id,
);
const passed = suiteResults.filter(
(result) => result.status === "passed" && result.cleanup === "passed",
).length;
const executed = suiteResults.filter(
(result) => result.attempt > 0,
).length;
return {
suiteId: suite.id,
suiteDefinitionHash:
suiteResults[0]?.suiteDefinitionHash ??
runnerMatrix.find((execution) => execution.suite.id === suite.id)!
.suiteDefinitionHash,
expected: suite.expectedMatrixSize,
selected: suiteExpected.length,
executed,
passed,
failed: suiteExpected.length - passed,
retries: suiteResults.reduce(
(total, result) => total + Math.max(0, result.attempt - 1),
0,
),
cleanupPassed: suiteResults.every(
(result) => result.cleanup === "passed",
),
complete: suiteExpected.length === suite.expectedMatrixSize,
durationMs: suiteResults.reduce(
(total, result) => total + result.durationMs,
0,
),
billing: aggregateCampaignBilling(suiteResults),
};
})
.filter((suite): suite is NonNullable<typeof suite> => Boolean(suite));
const passed = results.filter(
(result) => result.status === "passed" && result.cleanup === "passed",
).length;
const rankingSnapshots = [
...new Map(
results.flatMap((result) =>
result.rankingSnapshot
? [
[
result.rankingSnapshot.snapshotId,
{
snapshotId: result.rankingSnapshot.snapshotId,
capturedAt: result.rankingSnapshot.capturedAt,
sourceUrl: result.rankingSnapshot.sourceUrl,
},
] as const,
]
: [],
),
).values(),
];
return {
schema: "paperclip.runner-e2e.campaign/v2",
campaignId: input.campaignId,
generatedAt: input.generatedAt,
source,
expected,
complete:
suites.length === runnerSuites.length &&
suites.every((suite) => suite.complete),
selected: expected.length,
executed: results.filter((result) => result.attempt > 0).length,
passed,
failed: expected.length - passed,
retries: results.reduce(
(total, result) => total + Math.max(0, result.attempt - 1),
0,
),
cleanupPassed: results.every((result) => result.cleanup === "passed"),
rankingSnapshots,
billing: aggregateCampaignBilling(results),
suites,
results,
};
}
export function campaignHistoryRecord(
campaign: RunnerE2ECampaign,
publicBaseUrl: string,
): RunnerE2EHistoryCampaign {
const publicUrl = `${publicBaseUrl.replace(/\/$/, "")}/campaigns/${encodeURIComponent(campaign.campaignId)}/`;
return {
campaignId: campaign.campaignId,
generatedAt: campaign.generatedAt,
source: campaign.source,
complete: campaign.complete,
selected: campaign.selected,
executed: campaign.executed,
passed: campaign.passed,
failed: campaign.failed,
retries: campaign.retries,
cleanupPassed: campaign.cleanupPassed,
publicUrl,
billing: campaign.billing,
suites: campaign.suites,
executions: campaign.results.map((result) => ({
executionId: result.executionId,
suiteId: result.suiteId ?? "core-compatibility",
profileId: result.profileId,
environmentId: result.environmentId,
caseId: result.caseId,
provider: result.provider,
model: result.model,
status: result.status,
durationMs: result.durationMs,
attempt: result.attempt,
cleanup: result.cleanup,
billing: result.billing ?? summarizeExecutionBilling(result),
})),
};
}
export function emptyRunnerHistory(): RunnerE2EHistoryIndex {
return {
schema: "paperclip.runner-e2e.history/v1",
updatedAt: new Date(0).toISOString(),
latestCampaignId: null,
latestGreenCampaignId: null,
latestBySuite: {},
latestGreenBySuite: {},
campaigns: [],
};
}
export function mergeRunnerHistory(
current: RunnerE2EHistoryIndex | null | undefined,
campaign: RunnerE2EHistoryCampaign,
): RunnerE2EHistoryIndex {
const base =
current?.schema === "paperclip.runner-e2e.history/v1"
? current
: emptyRunnerHistory();
const campaigns = [
campaign,
...base.campaigns.filter(
(candidate) => candidate.campaignId !== campaign.campaignId,
),
].sort(
(left, right) =>
Date.parse(right.generatedAt) - Date.parse(left.generatedAt),
);
const latestBySuite: Record<string, string> = {};
const latestGreenBySuite: Record<string, string> = {};
for (const candidate of campaigns) {
for (const suite of candidate.suites) {
latestBySuite[suite.suiteId] ??= candidate.campaignId;
if (suite.complete && suite.failed === 0) {
latestGreenBySuite[suite.suiteId] ??= candidate.campaignId;
}
}
}
return {
schema: "paperclip.runner-e2e.history/v1",
updatedAt: new Date().toISOString(),
latestCampaignId: campaigns[0]?.campaignId ?? null,
latestGreenCampaignId:
campaigns.find(
(candidate) => candidate.complete && candidate.failed === 0,
)?.campaignId ?? null,
latestBySuite,
latestGreenBySuite,
campaigns,
};
}
tests/runner-e2e/instance-isolation.ts
+69
View File
@@ -0,0 +1,69 @@
import path from "node:path";
import { readFile, stat } from "node:fs/promises";
export async function assertEmbeddedDatabaseIsolation(
configPath: string,
temporaryRoot: string,
) {
const configText = await readFile(configPath, "utf8");
const config = JSON.parse(configText) as {
database?: {
mode?: string;
connectionString?: string;
embeddedPostgresDataDir?: string;
backup?: { dir?: string };
};
logging?: { logDir?: string };
storage?: {
provider?: string;
localDisk?: { baseDir?: string };
};
secrets?: {
provider?: string;
strictMode?: boolean;
localEncrypted?: { keyFilePath?: string };
};
};
if (
config.database?.mode !== "embedded-postgres" ||
config.database.connectionString
) {
throw new Error(
"Runner E2E Paperclip instance did not use its embedded database",
);
}
if (
config.storage?.provider !== "local_disk" ||
config.secrets?.provider !== "local_encrypted" ||
config.secrets.strictMode !== true
) {
throw new Error(
"Runner E2E instance did not use isolated local storage and strict encrypted secrets",
);
}
const isolatedPaths = {
database: config.database.embeddedPostgresDataDir,
backup: config.database.backup?.dir,
logs: config.logging?.logDir,
storage: config.storage.localDisk?.baseDir,
secretsKey: config.secrets.localEncrypted?.keyFilePath,
};
for (const [label, configuredPath] of Object.entries(isolatedPaths)) {
if (!configuredPath) {
throw new Error(`Runner E2E config omitted its ${label} path`);
}
const resolved = path.resolve(configuredPath);
if (!resolved.startsWith(`${temporaryRoot}${path.sep}`)) {
throw new Error(
`Runner E2E ${label} path escaped the isolated root: ${resolved}`,
);
}
}
const databasePath = path.resolve(isolatedPaths.database!);
const databaseStat = await stat(databasePath);
if (!databaseStat.isDirectory())
throw new Error("Embedded database path is not a directory");
const keyStat = await stat(path.resolve(isolatedPaths.secretsKey!));
if (!keyStat.isFile())
throw new Error("Encrypted-secrets master key path is not a file");
}
tests/runner-e2e/launch.ts
+921
View File
@@ -0,0 +1,921 @@
import { randomBytes } from "node:crypto";
import { spawn } from "node:child_process";
import { createWriteStream } from "node:fs";
import { createRequire } from "node:module";
import { createServer } from "node:net";
import os from "node:os";
import path from "node:path";
import {
chmod,
cp,
lstat,
mkdir,
mkdtemp,
readFile,
readdir,
rm,
symlink,
writeFile,
} from "node:fs/promises";
import { isImmutableDaytonaImage, runnerMatrix } from "./catalog.js";
import { renderRunnerE2EDashboard } from "./dashboard.js";
import { packageEvidence } from "./evidence.js";
import { classifyFailure, shouldRetryFailure } from "./failure-classifier.js";
import { buildRunnerCampaign } from "./history.js";
import { assertEmbeddedDatabaseIsolation } from "./instance-isolation.js";
import {
assertSecretFree,
findSecretLeakInDirectory,
isEphemeralCodexRuntimeAuthFile,
normalizedSecrets,
sanitizeJson,
} from "./redaction.js";
import {
buildMatrixJobs,
parseRunnerSelectors,
RunnerSelectorError,
selectRunnerExecutions,
} from "./selectors.js";
import {
CREDENTIAL_NAMES,
type MatrixExecution,
type RunnerE2EResult,
} from "./types.js";
import {
reapNewDetachedDarwinSharedMemory,
snapshotDarwinSharedMemory,
} from "./shared-memory.js";
const repositoryRoot = path.resolve(import.meta.dirname, "../..");
const localEnvPath = path.join(repositoryRoot, ".env.runner-e2e.local");
const resultsRoot = path.join(repositoryRoot, "tests/runner-e2e/results");
const activeProcessGroups = new Set<number>();
const activeProcessCleanup = new Map<number, Promise<string | null>>();
let cancelled = false;
function cleanId(value: string) {
const result = value
.replace(/[^A-Za-z0-9_.-]+/g, "-")
.replace(/^-+|-+$/g, "");
if (!result)
throw new Error(
`Invalid empty identifier derived from ${JSON.stringify(value)}`,
);
return result;
}
function secret(bytes = 32) {
return randomBytes(bytes).toString("base64url");
}
async function makeDirectoryTreeRemovable(directory: string): Promise<void> {
await chmod(directory, 0o700);
const entries = await readdir(directory, { withFileTypes: true });
await Promise.all(
entries
.filter((entry) => entry.isDirectory() && !entry.isSymbolicLink())
.map((entry) =>
makeDirectoryTreeRemovable(path.join(directory, entry.name)),
),
);
}
function stopProcessGroup(pid: number, signal: NodeJS.Signals = "SIGTERM") {
try {
if (process.platform === "win32") process.kill(pid, signal);
else process.kill(-pid, signal);
} catch {
// The process tree already exited.
}
}
function processGroupIsAlive(pid: number) {
try {
process.kill(process.platform === "win32" ? pid : -pid, 0);
return true;
} catch {
return false;
}
}
async function terminateProcessGroup(pid: number) {
stopProcessGroup(pid, "SIGTERM");
const gracefulDeadline = Date.now() + 5_000;
while (processGroupIsAlive(pid) && Date.now() < gracefulDeadline) {
await new Promise((resolve) => setTimeout(resolve, 50));
}
if (!processGroupIsAlive(pid)) return null;
stopProcessGroup(pid, "SIGKILL");
const forcedDeadline = Date.now() + 5_000;
while (processGroupIsAlive(pid) && Date.now() < forcedDeadline) {
await new Promise((resolve) => setTimeout(resolve, 50));
}
return processGroupIsAlive(pid)
? `Paperclip/Playwright process group ${pid} survived SIGKILL`
: null;
}
for (const signal of ["SIGINT", "SIGTERM", "SIGHUP"] as const) {
process.on(signal, () => {
cancelled = true;
for (const pid of activeProcessGroups) {
if (activeProcessCleanup.has(pid)) {
stopProcessGroup(pid, "SIGKILL");
continue;
}
activeProcessCleanup.set(pid, terminateProcessGroup(pid));
}
});
}
async function loadLocalEnvironment(target: NodeJS.ProcessEnv) {
const contents = await readFile(localEnvPath, "utf8").catch(
(error: NodeJS.ErrnoException) => {
if (error.code === "ENOENT") return null;
throw error;
},
);
if (contents === null) return;
for (const [index, rawLine] of contents.split(/\r?\n/).entries()) {
const line = rawLine.trim();
if (!line || line.startsWith("#")) continue;
const match = /^(?:export\s+)?([A-Za-z_][A-Za-z0-9_]*)=(.*)$/.exec(line);
if (!match)
throw new Error(
`Invalid ${path.basename(localEnvPath)} line ${index + 1}`,
);
if (target[match[1]] !== undefined) continue;
let value = match[2].trim();
if (
(value.startsWith('"') && value.endsWith('"')) ||
(value.startsWith("'") && value.endsWith("'"))
) {
value = value.slice(1, -1);
}
target[match[1]] = value;
}
}
async function reservePort() {
const server = createServer();
await new Promise<void>((resolve, reject) => {
server.once("error", reject);
server.listen(0, "127.0.0.1", resolve);
});
const address = server.address();
if (!address || typeof address === "string")
throw new Error("Failed to reserve a loopback port");
await new Promise<void>((resolve, reject) =>
server.close((error) => (error ? reject(error) : resolve())),
);
return address.port;
}
async function prepareProviderPath(
temporaryRoot: string,
inheritedPath: string | undefined,
) {
const toolBin = path.join(temporaryRoot, "provider-bin");
await mkdir(toolBin, { recursive: true });
const runnerRequire = createRequire(
path.join(repositoryRoot, "packages/paperclip-runner/package.json"),
);
const codexAcpPackage = runnerRequire.resolve(
"@agentclientprotocol/codex-acp/package.json",
);
const codexRequire = createRequire(codexAcpPackage);
const codexPackage = codexRequire.resolve("@openai/codex/package.json");
const codexManifest = JSON.parse(await readFile(codexPackage, "utf8")) as {
bin?: string | Record<string, string>;
};
const codexBin =
typeof codexManifest.bin === "string"
? codexManifest.bin
: codexManifest.bin?.codex;
if (!codexBin)
throw new Error(
"Production Codex ACP dependency does not expose its pinned Codex executable",
);
await symlink(
path.resolve(path.dirname(codexPackage), codexBin),
path.join(toolBin, "codex"),
);
const packageBin = path.join(
repositoryRoot,
"packages/paperclip-runner/node_modules/.bin",
);
return [toolBin, packageBin, inheritedPath]
.filter(Boolean)
.join(path.delimiter);
}
async function runProcess(
args: string[],
env: NodeJS.ProcessEnv,
timeoutMs: number | null,
logPath: string,
) {
const log = createWriteStream(logPath, { flags: "a", mode: 0o600 });
const child = spawn("pnpm", args, {
cwd: repositoryRoot,
env,
stdio: ["inherit", "pipe", "pipe"],
detached: process.platform !== "win32",
});
if (!child.pid) throw new Error("Failed to start Playwright");
activeProcessGroups.add(child.pid);
let outputTail = "";
const recordOutput = (chunk: Buffer, destination: NodeJS.WriteStream) => {
destination.write(chunk);
log.write(chunk);
outputTail += chunk.toString("utf8");
if (outputTail.length > 1024 * 1024)
outputTail = outputTail.slice(-1024 * 1024);
};
child.stdout?.on("data", (chunk: Buffer) =>
recordOutput(chunk, process.stdout),
);
child.stderr?.on("data", (chunk: Buffer) =>
recordOutput(chunk, process.stderr),
);
let timedOut = false;
const timer =
timeoutMs === null
? undefined
: setTimeout(() => {
timedOut = true;
stopProcessGroup(child.pid!, "SIGTERM");
setTimeout(
() => stopProcessGroup(child.pid!, "SIGKILL"),
10_000,
).unref();
}, timeoutMs);
timer?.unref();
let spawnError: string | null = null;
const exitCode = await new Promise<number>((resolve, reject) => {
child.once("error", reject);
child.once("exit", (code) => resolve(code ?? 1));
}).catch((error) => {
spawnError = error instanceof Error ? error.message : String(error);
return 1;
});
if (timer) clearTimeout(timer);
const processCleanupError = child.pid
? await (activeProcessCleanup.get(child.pid) ??
terminateProcessGroup(child.pid))
: null;
if (child.pid) {
activeProcessGroups.delete(child.pid);
activeProcessCleanup.delete(child.pid);
}
await new Promise<void>((resolve) => log.end(resolve));
return {
exitCode,
timedOut,
processCleanupError,
spawnError,
outputTail,
};
}
function syntheticResult(
execution: MatrixExecution,
attempt: number,
startedAtMs: number,
error: string,
failureClass: RunnerE2EResult["failureClass"] = "transient_infrastructure",
): RunnerE2EResult {
const finishedAtMs = Date.now();
return {
schema: "paperclip.runner-e2e.result/v2",
executionId: execution.id,
suiteId: execution.suite.id,
suiteDefinitionHash: execution.suiteDefinitionHash,
source: {
sha: process.env.GITHUB_SHA ?? null,
ref: process.env.GITHUB_REF ?? null,
workflowRunUrl:
process.env.GITHUB_SERVER_URL &&
process.env.GITHUB_REPOSITORY &&
process.env.GITHUB_RUN_ID
? `${process.env.GITHUB_SERVER_URL}/${process.env.GITHUB_REPOSITORY}/actions/runs/${process.env.GITHUB_RUN_ID}`
: null,
},
...(execution.profile.ranking
? { rankingSnapshot: execution.profile.ranking }
: {}),
attempt,
status: "failed",
failureClass,
error,
profileId: execution.profile.id,
environmentId: execution.environment.id,
caseId: execution.task.id,
provider: execution.profile.provider,
model: execution.profile.model,
runtimeMode: execution.profile.expectedRuntimeMode,
startedAt: new Date(startedAtMs).toISOString(),
finishedAt: new Date(finishedAtMs).toISOString(),
durationMs: finishedAtMs - startedAtMs,
cleanup: "not_started",
};
}
async function readResult(resultPath: string, fallback: RunnerE2EResult) {
try {
return JSON.parse(await readFile(resultPath, "utf8")) as RunnerE2EResult;
} catch {
return fallback;
}
}
async function copySharedEvidence(privateRoot: string, casePrivateDir: string) {
for (const relative of [
"server.log",
"playwright.log",
"junit.xml",
"html-report",
"blob-report",
"playwright-output",
]) {
await cp(
path.join(privateRoot, relative),
path.join(casePrivateDir, relative),
{ recursive: true, force: true },
).catch((error: NodeJS.ErrnoException) => {
if (error.code !== "ENOENT") throw error;
});
}
}
async function runAttempt(input: {
executions: readonly MatrixExecution[];
attempt: number;
campaignId: string;
options: ReturnType<typeof parseRunnerSelectors>;
}): Promise<RunnerE2EResult[]> {
const { executions, attempt, campaignId, options } = input;
const execution = executions[0];
if (!execution) throw new Error("A runner E2E cell must contain a task case");
if (
executions.some(
(candidate) =>
candidate.profile.id !== execution.profile.id ||
candidate.environment.id !== execution.environment.id,
)
) {
throw new Error(
"One isolated runner E2E harness cannot mix profiles or environments",
);
}
const startedAtMs = Date.now();
const sharedMemoryBaseline = snapshotDarwinSharedMemory();
const temporaryRoot = await mkdtemp(
path.join(os.tmpdir(), "paperclip-runner-e2e-"),
);
const publishedResults: RunnerE2EResult[] = [];
const publishedResultPaths = new Map<string, string>();
let attemptSecrets: string[] = [];
try {
const paperclipHome = path.join(temporaryRoot, "paperclip-home");
const workspace = path.join(temporaryRoot, "workspace");
const privateDir = path.join(temporaryRoot, "artifacts-private");
const instanceId = `runner-e2e-${randomBytes(8).toString("hex")}`;
const configPath = path.join(
paperclipHome,
"instances",
instanceId,
"config.json",
);
const port = await reservePort();
await Promise.all([
mkdir(paperclipHome, { recursive: true }),
mkdir(workspace, { recursive: true }),
mkdir(privateDir, { recursive: true }),
]);
const providerPath = await prepareProviderPath(
temporaryRoot,
process.env.PATH,
);
const agentJwtSecret = secret(48);
const decisionSigningSecret = secret(48);
const toolActionSigningSecret = secret(48);
const betterAuthSecret = secret(48);
const credentials = normalizedSecrets([
...CREDENTIAL_NAMES.map((name) => process.env[name]),
agentJwtSecret,
decisionSigningSecret,
toolActionSigningSecret,
betterAuthSecret,
]);
attemptSecrets = credentials;
const childEnv: NodeJS.ProcessEnv = {
...process.env,
PATH: providerPath,
PAPERCLIP_RUNNER_E2E_EXECUTION_IDS: JSON.stringify(
executions.map((candidate) => candidate.id),
),
PAPERCLIP_RUNNER_E2E_ATTEMPT: String(attempt),
PAPERCLIP_RUNNER_E2E_PORT: String(port),
PAPERCLIP_RUNNER_E2E_TEMP_ROOT: temporaryRoot,
PAPERCLIP_RUNNER_E2E_PRIVATE_DIR: privateDir,
PAPERCLIP_RUNNER_E2E_WORKSPACE: workspace,
PAPERCLIP_RUNNER_E2E_SERVER_LOG: path.join(privateDir, "server.log"),
// Vite's optimized dependency cache embeds revision query strings. A
// private per-attempt cache prevents an earlier cell or local rebuild
// from producing `504 Outdated Optimize Dep` during browser bootstrap.
PAPERCLIP_VITE_CACHE_DIR: path.join(temporaryRoot, "vite-cache"),
PAPERCLIP_RUNNER_E2E_TEST_TIMEOUT_MS: String(
Math.max(
...executions.map(
(candidate) =>
candidate.task.attemptTimeoutMs[candidate.environment.id],
),
) + 90_000,
),
PAPERCLIP_HOME: paperclipHome,
PAPERCLIP_INSTANCE_ID: instanceId,
PAPERCLIP_CONFIG: configPath,
PAPERCLIP_AGENT_JWT_SECRET: agentJwtSecret,
PAPERCLIP_DECISION_SIGNING_SECRET: decisionSigningSecret,
PAPERCLIP_TOOL_ACTION_SIGNING_SECRET: toolActionSigningSecret,
BETTER_AUTH_SECRET: betterAuthSecret,
};
// The database URLs are stripped here and again at the Playwright web-server
// boundary so a developer's shell can never redirect this paid test.
delete childEnv.DATABASE_URL;
delete childEnv.DATABASE_MIGRATION_URL;
const playwrightArgs = [
"exec",
"playwright",
"test",
"--config",
"tests/runner-e2e/playwright.config.ts",
...(options.headed ? ["--headed"] : []),
...(options.ui ? ["--ui"] : []),
...(options.debug ? ["--debug"] : []),
];
const watchdog =
options.ui || options.debug
? null
: executions.reduce(
(total, candidate) =>
total +
candidate.task.attemptTimeoutMs[candidate.environment.id] +
90_000,
0,
) +
5 * 60_000;
const processResult = await runProcess(
playwrightArgs,
childEnv,
watchdog,
path.join(privateDir, "playwright.log"),
);
const processFailure = processResult.spawnError
? `Playwright failed to start: ${processResult.spawnError}`
: processResult.timedOut
? `Harness process exceeded ${Math.round((watchdog ?? 0) / 1000)} seconds`
: `Playwright exited ${processResult.exitCode} before writing a result`;
const processFailureClass =
processResult.spawnError || processResult.timedOut
? "transient_infrastructure"
: classifyFailure(
new Error(
processResult.outputTail
? `${processFailure}\n${processResult.outputTail}`
: processFailure,
),
);
const results = await Promise.all(
executions.map(async (candidate) => {
const resultPath = path.join(
privateDir,
"cases",
candidate.task.id,
"result.json",
);
const fallback = syntheticResult(
candidate,
attempt,
startedAtMs,
processFailure,
processFailureClass,
);
const result = await readResult(resultPath, fallback);
return processResult.processCleanupError
? {
...result,
status: "failed" as const,
failureClass: "cleanup_failure" as const,
error: processResult.processCleanupError,
cleanup: "failed" as const,
}
: result;
}),
);
let isolationError: unknown;
try {
await assertEmbeddedDatabaseIsolation(configPath, temporaryRoot);
} catch (error) {
isolationError = error;
}
let persistedStateError: unknown;
try {
const expectedEphemeralCredentials = new Set<string>();
for (const [label, directory] of [
["Paperclip home", paperclipHome],
["workspace", workspace],
] as const) {
while (true) {
// The managed Codex home may legitimately contain upstream source-code
// fixtures with fake `sk-*` strings. Reject exact campaign credentials.
const leak = await findSecretLeakInDirectory(directory, credentials, {
includeShapes: false,
ignoreFile: (file) => expectedEphemeralCredentials.has(file),
});
if (!leak) break;
const isManagedCodexRuntimeAuth =
label === "Paperclip home" &&
isEphemeralCodexRuntimeAuthFile(paperclipHome, leak.file);
if (isManagedCodexRuntimeAuth) {
const metadata = await lstat(leak.file);
if (metadata.isFile() && (metadata.mode & 0o777) === 0o600) {
// Codex CLI API-key mode requires this one runtime auth file. It
// lives only in the disposable cell root, is never published,
// must be owner-only, and is removed with the root below.
expectedEphemeralCredentials.add(leak.file);
continue;
}
}
throw new Error(
`Secret leak in persisted ${label} state at ${path.relative(temporaryRoot, leak.file)}: ${leak.reason}`,
);
}
}
} catch (error) {
persistedStateError = error;
}
for (const [index, candidate] of executions.entries()) {
let result = results[index];
if (
isolationError &&
result.failureClass !== "cleanup_failure" &&
result.failureClass !== "secret_leak"
) {
const isolationMessage =
isolationError instanceof Error
? isolationError.message
: String(isolationError);
const configWasUnavailableDuringTransientBootstrap =
result.failureClass === "transient_infrastructure" &&
typeof isolationError === "object" &&
isolationError !== null &&
"code" in isolationError &&
isolationError.code === "ENOENT";
result = {
...result,
status: "failed",
failureClass: configWasUnavailableDuringTransientBootstrap
? result.failureClass
: "permanent_infrastructure",
error: configWasUnavailableDuringTransientBootstrap
? `${result.error}; isolated config could not be inspected after bootstrap failure: ${isolationMessage}`
: isolationMessage,
};
}
if (persistedStateError) {
const persistedStateMessage =
persistedStateError instanceof Error
? persistedStateError.message
: String(persistedStateError);
const persistedStateClass = classifyFailure(persistedStateError);
if (
persistedStateClass === "secret_leak" ||
(result.failureClass !== "secret_leak" &&
result.failureClass !== "cleanup_failure")
) {
result = {
...result,
status: "failed",
failureClass:
persistedStateClass === "secret_leak"
? "secret_leak"
: "permanent_infrastructure",
error: persistedStateMessage,
};
} else {
result = {
...result,
error: `${result.error}; persisted-state scan also failed: ${persistedStateMessage}`,
};
}
}
const casePrivateDir = path.join(privateDir, "cases", candidate.task.id);
const resultPath = path.join(casePrivateDir, "result.json");
const uploadDir = path.join(
resultsRoot,
campaignId,
candidate.suite.id,
candidate.profile.id,
candidate.environment.id,
candidate.task.id,
`attempt-${attempt}`,
);
await mkdir(casePrivateDir, { recursive: true });
await copySharedEvidence(privateDir, casePrivateDir);
await writeFile(
resultPath,
`${JSON.stringify(sanitizeJson(result, credentials), null, 2)}\n`,
"utf8",
);
let evidence = await packageEvidence({
privateDir: casePrivateDir,
uploadDir,
secrets: credentials,
expectPassScreenshot: result.status === "passed",
});
if (evidence.leaks.length > 0 || evidence.missing.length > 0) {
result = {
...result,
status: "failed",
failureClass:
evidence.leaks.length > 0
? "secret_leak"
: "permanent_infrastructure",
error:
evidence.leaks.length > 0
? `Secret leak rejected from evidence: ${evidence.leaks.map((leak) => leak.file).join(", ")}`
: `Required evidence missing: ${evidence.missing.join(", ")}`,
};
await writeFile(
resultPath,
`${JSON.stringify(sanitizeJson(result, credentials), null, 2)}\n`,
"utf8",
);
evidence = await packageEvidence({
privateDir: casePrivateDir,
uploadDir,
secrets: credentials,
expectPassScreenshot: false,
});
}
console.log(
`${result.status === "passed" ? "PASS" : "FAIL"} ${candidate.id} attempt ${attempt} -> ${uploadDir}`,
);
publishedResults.push(result);
publishedResultPaths.set(
candidate.id,
path.join(uploadDir, "result.json"),
);
}
return [...publishedResults];
} finally {
reapNewDetachedDarwinSharedMemory(sharedMemoryBaseline);
let cleanupError: unknown;
if (
temporaryRoot.startsWith(`${os.tmpdir()}${path.sep}paperclip-runner-e2e-`)
) {
for (let cleanupAttempt = 1; cleanupAttempt <= 3; cleanupAttempt += 1) {
try {
await rm(temporaryRoot, { recursive: true, force: true });
cleanupError = undefined;
break;
} catch (error) {
cleanupError = error;
if (cleanupAttempt < 3) {
await makeDirectoryTreeRemovable(temporaryRoot).catch(() => {});
await new Promise((resolve) =>
setTimeout(resolve, cleanupAttempt * 250),
);
}
}
}
} else {
cleanupError = new Error(
`Refusing to remove unexpected temporary path ${temporaryRoot}`,
);
}
if (cleanupError) {
const message = `Temporary runner E2E state cleanup failed at ${temporaryRoot}: ${cleanupError instanceof Error ? cleanupError.message : String(cleanupError)}`;
for (const publishedResult of publishedResults) {
const publishedResultPath = publishedResultPaths.get(
publishedResult.executionId,
);
if (!publishedResultPath) continue;
Object.assign(publishedResult, {
status: "failed",
failureClass: "cleanup_failure",
error: message,
cleanup: "failed",
} satisfies Partial<RunnerE2EResult>);
const safeResult = `${JSON.stringify(
sanitizeJson(publishedResult, attemptSecrets),
null,
2,
)}\n`;
assertSecretFree(safeResult, attemptSecrets, publishedResultPath);
await writeFile(publishedResultPath, safeResult, "utf8");
}
// Cleanup failure is a failed cell, but must not suppress later cells in
// the same campaign. The result above carries the terminal failure.
console.error(message);
}
}
}
function printList(executions: readonly MatrixExecution[]) {
console.log("ID\tSUITE\tGENERATION\tPROVIDER\tMODEL\tCREDENTIALS");
for (const execution of executions) {
console.log(
[
execution.id,
execution.suite.id,
execution.profile.generation,
execution.profile.provider,
execution.profile.model,
execution.requiredCredentials.join(","),
].join("\t"),
);
}
}
async function runExecutionWithRetry(input: {
execution: MatrixExecution;
campaignId: string;
options: ReturnType<typeof parseRunnerSelectors>;
}): Promise<RunnerE2EResult> {
const { execution, campaignId, options } = input;
const [firstResult] = await runAttempt({
executions: [execution],
attempt: 1,
campaignId,
options,
});
if (!firstResult) throw new Error(`No result produced for ${execution.id}`);
if (
options.ui ||
options.debug ||
firstResult.status !== "failed" ||
!firstResult.failureClass ||
!shouldRetryFailure(firstResult.failureClass)
) {
return firstResult;
}
if (cancelled) throw new Error("Runner E2E campaign cancelled");
console.warn(
`Retrying ${execution.id} in a fresh isolated harness after transient infrastructure failure`,
);
const [retryResult] = await runAttempt({
executions: [execution],
attempt: 2,
campaignId,
options,
});
if (!retryResult)
throw new Error(`No retry result produced for ${execution.id}`);
return retryResult;
}
async function runWithConcurrency<T, R>(
values: readonly T[],
concurrency: number,
worker: (value: T) => Promise<R>,
): Promise<R[]> {
const results = new Array<R>(values.length);
let cursor = 0;
const workers = Array.from(
{ length: Math.min(concurrency, values.length) },
async () => {
while (true) {
const index = cursor;
cursor += 1;
if (index >= values.length) return;
if (cancelled) throw new Error("Runner E2E campaign cancelled");
results[index] = await worker(values[index]!);
}
},
);
await Promise.all(workers);
return results;
}
async function main() {
let options: ReturnType<typeof parseRunnerSelectors>;
try {
options = parseRunnerSelectors(process.argv.slice(2));
} catch (error) {
if (error instanceof RunnerSelectorError)
throw new Error(`${error.message}\nUse --list to inspect valid cells.`);
throw error;
}
const executions = selectRunnerExecutions(options, runnerMatrix);
if (options.list) {
printList(executions);
return;
}
if (options.matrixJson) {
const jobs = buildMatrixJobs(executions);
console.log(
JSON.stringify({
include: jobs,
needsDaytona: jobs.some((job) => job.needsDaytona),
executionIds: executions.map((execution) => execution.id),
}),
);
return;
}
await loadLocalEnvironment(process.env);
const missingCredentials = [
...new Set(
executions.flatMap((execution) => execution.requiredCredentials),
),
].filter((name) => !process.env[name]?.trim());
if (missingCredentials.length > 0) {
throw new Error(
`Missing runner E2E credentials: ${missingCredentials.join(", ")}`,
);
}
if (
executions.some((execution) => execution.environment.id === "daytona") &&
!isImmutableDaytonaImage(process.env.PAPERCLIP_E2E_DAYTONA_IMAGE)
) {
throw new Error(
"PAPERCLIP_E2E_DAYTONA_IMAGE must be an immutable image@sha256 digest for Daytona cells",
);
}
const campaignId = cleanId(
process.env.PAPERCLIP_E2E_CAMPAIGN_ID ??
`local-${new Date().toISOString().replace(/[:.]/g, "-")}`,
);
const requestedParallelism =
options.headed || options.ui || options.debug ? 1 : options.maxParallel;
console.log(
`Running ${executions.length} isolated execution(s) with max parallelism ${requestedParallelism}`,
);
const finalResults = await runWithConcurrency(
executions,
requestedParallelism,
(execution) => runExecutionWithRetry({ execution, campaignId, options }),
);
const generatedAt = new Date().toISOString();
const campaign = buildRunnerCampaign({
campaignId,
generatedAt,
expected: executions.map((execution) => execution.id),
results: finalResults,
});
const summaryDir = path.join(resultsRoot, campaignId);
await mkdir(summaryDir, { recursive: true });
const campaignSecrets = normalizedSecrets(
CREDENTIAL_NAMES.map((name) => process.env[name]),
);
const safeCampaign = sanitizeJson(
campaign,
campaignSecrets,
) as typeof campaign;
const campaignText = `${JSON.stringify(safeCampaign, null, 2)}\n`;
assertSecretFree(campaignText, campaignSecrets, "campaign.json");
await writeFile(path.join(summaryDir, "campaign.json"), campaignText, "utf8");
const dashboard = renderRunnerE2EDashboard({
title: `Runner E2E · ${campaignId}`,
generatedAt,
expected: executions.map((execution) => execution.id),
catalog: runnerMatrix,
campaign: safeCampaign,
entries: safeCampaign.results.map((result) => ({
result,
valid: result.status === "passed" && result.cleanup === "passed",
errors:
result.status === "passed" && result.cleanup === "passed"
? []
: [
result.error ??
result.failureClass ??
`cleanup=${result.cleanup}`,
],
evidenceBaseHref: [
result.suiteId ?? "core-compatibility",
result.profileId,
result.environmentId,
result.caseId,
`attempt-${result.attempt}`,
].join("/"),
})),
});
assertSecretFree(dashboard, campaignSecrets, "dashboard.html");
await writeFile(path.join(summaryDir, "dashboard.html"), dashboard, "utf8");
console.log(
`Campaign ${campaignId}: ${safeCampaign.passed}/${safeCampaign.selected} passed`,
);
if (safeCampaign.failed > 0) process.exitCode = 1;
}
await main().catch((error) => {
console.error(error instanceof Error ? error.message : String(error));
process.exitCode = 1;
});
tests/runner-e2e/live-fixtures.test.ts
+73
View File
@@ -0,0 +1,73 @@
import { describe, expect, it } from "vitest";
import type { RunnerApi } from "./api.js";
import { runnerMatrix } from "./catalog.js";
import { setupLiveFixtures } from "./live-fixtures.js";
describe("live runner fixtures", () => {
it("installs the Daytona provider through the public API before creating its environment", async () => {
const calls: string[] = [];
const api = {
async post(path: string, data?: Record<string, unknown>) {
calls.push(`POST ${path}`);
if (path === "/api/plugins/install") {
expect(data).toMatchObject({ isLocalPath: true });
expect(data?.packageName).toEqual(
expect.stringContaining(
"packages/plugins/sandbox-providers/daytona",
),
);
return {
id: "plugin-daytona",
pluginKey: "paperclip.daytona-sandbox-provider",
status: "ready",
};
}
if (path === "/api/companies") {
return { id: "company-1", name: "Runner E2E" };
}
if (path.endsWith("/environments")) {
expect(calls).toContain("POST /api/plugins/install");
return { id: "environment-1", driver: "sandbox" };
}
if (path.endsWith("/agents")) {
return { id: "agent-1", name: "Agent", companyId: "company-1" };
}
throw new Error(`Unexpected POST ${path}`);
},
async postSensitive(path: string, data?: Record<string, unknown>) {
calls.push(`POST ${path}`);
return { id: `secret-${String(data?.key).toLowerCase()}` };
},
async delete(path: string) {
calls.push(`DELETE ${path}`);
},
} as unknown as RunnerApi;
const execution = runnerMatrix.find(
(candidate) =>
candidate.id ===
"core-compatibility.legacy-codex.daytona.message-marker",
);
expect(execution).toBeDefined();
const fixtures = await setupLiveFixtures({
api,
execution: execution!,
executionNonce: "nonce",
workspacePath: "/tmp/workspace",
credentials: {
OPENAI_API_KEY: "openai-test-value",
DAYTONA_API_KEY: "daytona-test-value",
},
daytonaImage:
"ghcr.io/paperclip/image@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
});
expect(calls.indexOf("POST /api/plugins/install")).toBeLessThan(
calls.indexOf("POST /api/companies/company-1/environments"),
);
await fixtures.teardown();
expect(calls).toContain(
"DELETE /api/environments/environment-1?destroyReusableSandboxLeases=true",
);
});
});
tests/runner-e2e/live-fixtures.ts
+221
View File
@@ -0,0 +1,221 @@
import path from "node:path";
import { FixtureRegistry } from "./fixture-registry.js";
import type { RunnerApi } from "./api.js";
import type {
CredentialName,
MatrixExecution,
SecretReference,
SecretReferenceMap,
} from "./types.js";
interface CompanyRecord {
id: string;
issuePrefix?: string | null;
name: string;
}
interface SecretRecord {
id: string;
}
interface PluginRecord {
id: string;
pluginKey: string;
status: string;
}
interface EnvironmentRecord {
id: string;
driver: string;
config?: Record<string, unknown>;
}
interface AgentRecord {
id: string;
name: string;
companyId: string;
}
export interface LiveFixtureValues {
company: CompanyRecord;
secretRefs: SecretReferenceMap;
environment: EnvironmentRecord;
agent: AgentRecord;
teardown(): Promise<void>;
}
function value<T>(resolved: ReadonlyMap<string, unknown>, id: string): T {
const result = resolved.get(id);
if (!result) throw new Error(`Missing resolved fixture ${id}`);
return result as T;
}
async function deleteDaytonaEnvironment(api: RunnerApi, environmentId: string) {
const deadlineAt = Date.now() + 120_000;
let lastError: unknown;
while (Date.now() < deadlineAt) {
try {
await api.delete(
`/api/environments/${environmentId}?destroyReusableSandboxLeases=true`,
{
allowNotFound: true,
},
);
return;
} catch (error) {
lastError = error;
await new Promise((resolve) => setTimeout(resolve, 3_000));
}
}
throw new Error(
`Daytona lease cleanup failed: ${lastError instanceof Error ? lastError.message : String(lastError)}`,
);
}
export async function setupLiveFixtures(input: {
api: RunnerApi;
execution: MatrixExecution;
executionNonce: string;
workspacePath: string;
credentials: Partial<Record<CredentialName, string>>;
daytonaImage?: string;
}): Promise<LiveFixtureValues> {
const { api, execution } = input;
const registry = new FixtureRegistry();
if (execution.environment.id === "daytona") {
registry.register<PluginRecord>({
id: "sandbox-provider",
async setup() {
return api.post<PluginRecord>("/api/plugins/install", {
packageName: path.resolve(
import.meta.dirname,
"../../packages/plugins/sandbox-providers/daytona",
),
isLocalPath: true,
});
},
async teardown() {
// The plugin is installed only in the isolated instance/database. The
// launcher removes that complete instance after the environment lease
// has been destroyed, so no global uninstall mutation is necessary.
},
});
}
registry.register<CompanyRecord>({
id: "company",
async setup() {
return api.post<CompanyRecord>("/api/companies", {
name: `Runner E2E ${execution.id} ${input.executionNonce}`,
description: "Ephemeral paid full-stack runner acceptance fixture",
budgetMonthlyCents: 0,
});
},
async teardown() {
// The isolated instance/database is removed by the launcher. Do not call
// company deletion here: metered runs intentionally retain cost-event
// references until that instance-wide teardown.
},
});
registry.register<SecretReferenceMap>({
id: "secrets",
dependencies: ["company"],
async setup(resolved) {
const company = value<CompanyRecord>(resolved, "company");
const refs: SecretReferenceMap = {};
for (const credentialName of execution.requiredCredentials) {
const rawValue = input.credentials[credentialName];
if (!rawValue) throw new Error(`Missing credential ${credentialName}`);
const secret = await api.postSensitive<SecretRecord>(
`/api/companies/${company.id}/secrets`,
{
name: `Runner E2E ${credentialName} ${input.executionNonce}`,
key: credentialName,
value: rawValue,
description: `Ephemeral credential for ${execution.id}`,
},
);
refs[credentialName] = {
type: "secret_ref",
secretId: secret.id,
version: "latest",
} satisfies SecretReference;
}
return refs;
},
});
registry.register<EnvironmentRecord>({
id: "environment",
dependencies: [
"company",
"secrets",
...(execution.environment.id === "daytona" ? ["sandbox-provider"] : []),
],
async setup(resolved) {
const company = value<CompanyRecord>(resolved, "company");
const secretRefs = value<SecretReferenceMap>(resolved, "secrets");
if (execution.environment.id === "local") {
// Paperclip has one instance-managed local environment. The company
// creation API ensures it exists; creating a second local environment
// is intentionally rejected by the public API.
const environments = await api.get<EnvironmentRecord[]>(
`/api/companies/${company.id}/environments?driver=local`,
);
const local = environments.find(
(candidate) => candidate.driver === "local",
);
if (!local)
throw new Error(
"Isolated Paperclip instance did not create its managed local environment",
);
return local;
}
return api.post<EnvironmentRecord>(
`/api/companies/${company.id}/environments`,
execution.environment.buildEnvironment({
secretRefs,
daytonaImage: input.daytonaImage,
executionId: input.executionNonce,
}),
);
},
async teardown(environment) {
if (execution.environment.id === "daytona") {
await deleteDaytonaEnvironment(api, environment.id);
}
},
});
registry.register<AgentRecord>({
id: "agent",
dependencies: ["company", "secrets", "environment"],
async setup(resolved) {
const company = value<CompanyRecord>(resolved, "company");
const environment = value<EnvironmentRecord>(resolved, "environment");
const secretRefs = value<SecretReferenceMap>(resolved, "secrets");
return api.post<AgentRecord>(
`/api/companies/${company.id}/agents`,
execution.profile.buildAgent({
environmentId: environment.id,
environmentFixtureId: execution.environment.id,
workspacePath: input.workspacePath,
secretRefs,
executionId: input.executionNonce,
}),
);
},
async teardown() {
// Agent state is instance-local. Daytona environment teardown below is
// the only fixture cleanup that must reach an external provider.
},
});
const setup = await registry.setupAll();
return {
company: value<CompanyRecord>(setup.values, "company"),
secretRefs: value<SecretReferenceMap>(setup.values, "secrets"),
environment: value<EnvironmentRecord>(setup.values, "environment"),
agent: value<AgentRecord>(setup.values, "agent"),
teardown: setup.teardown,
};
}
tests/runner-e2e/matchers.ts
+149
View File
@@ -0,0 +1,149 @@
import { createRequire } from "node:module";
import { readFile } from "node:fs/promises";
import type { Matcher } from "./types.js";
interface JsonSchemaValidator {
(value: unknown): boolean;
errors?: unknown;
}
interface Ajv2020Instance {
compile(schema: Record<string, unknown>): JsonSchemaValidator;
}
const runnerRequire = createRequire(
new URL("../../packages/paperclip-runner/package.json", import.meta.url),
);
const Ajv2020 = runnerRequire("ajv/dist/2020.js").default as new (options: {
allErrors: boolean;
strict: boolean;
}) => Ajv2020Instance;
const jsonSchemaCompiler = new Ajv2020({ allErrors: true, strict: false });
export interface MatcherObservation {
message?: string;
issueStatus?: string;
runStatus?: string;
runtimeMode?: string;
environment?: string;
files?: Record<string, string>;
artifacts?: Array<{ name: string; mimeType?: string }>;
json?: unknown;
}
export interface MatcherResult {
matcher: Matcher;
passed: boolean;
detail: string;
}
function normalizeMessage(value: string | undefined) {
return (
(value ?? "")
.replace(/\r\n/g, "\n")
// Some provider renderers escape underscores in plain-text identifiers
// before persisting Markdown. Treat that presentation-only escape as the
// same visible marker for exact/contains/ordered message assertions.
.replace(/\\_/g, "_")
.replace(/[ \t]+/g, " ")
.trim()
);
}
function readJsonPath(value: unknown, path: string): unknown {
return path
.split(".")
.filter(Boolean)
.reduce<unknown>((current, segment) => {
if (!current || typeof current !== "object") return undefined;
return (current as Record<string, unknown>)[segment];
}, value);
}
export async function evaluateMatcher(
matcher: Matcher,
observation: MatcherObservation,
): Promise<MatcherResult> {
const message = normalizeMessage(observation.message);
let passed = false;
let actual: unknown;
if (matcher.kind === "message_exact") {
actual = message;
passed = message === normalizeMessage(matcher.expected);
} else if (matcher.kind === "message_contains") {
actual = message;
passed = message.includes(normalizeMessage(matcher.expected));
} else if (matcher.kind === "message_regex") {
actual = message;
passed = new RegExp(matcher.pattern, matcher.flags).test(message);
} else if (matcher.kind === "message_ordered") {
actual = message;
let cursor = 0;
passed = matcher.expected.every((expected) => {
const normalizedExpected = normalizeMessage(expected);
const index = message.indexOf(normalizedExpected, cursor);
if (index < 0) return false;
cursor = index + normalizedExpected.length;
return true;
});
} else if (matcher.kind === "issue_status") {
actual = observation.issueStatus;
passed = actual === matcher.expected;
} else if (matcher.kind === "run_status") {
actual = observation.runStatus;
passed = actual === matcher.expected;
} else if (matcher.kind === "runtime_mode") {
actual = observation.runtimeMode;
passed = actual === matcher.expected;
} else if (matcher.kind === "environment") {
actual = observation.environment;
passed = actual === matcher.expected;
} else if (
matcher.kind === "file_exists" ||
matcher.kind === "file_contains"
) {
try {
actual =
observation.files?.[matcher.path] ??
(await readFile(matcher.path, "utf8"));
passed =
matcher.kind === "file_exists" ||
String(actual).includes(matcher.expected);
} catch {
actual = undefined;
passed = false;
}
} else if (matcher.kind === "artifact_exists") {
actual = observation.artifacts ?? [];
passed = (observation.artifacts ?? []).some(
(artifact) =>
artifact.name === matcher.name &&
(!matcher.mimeType || artifact.mimeType === matcher.mimeType),
);
} else if (matcher.kind === "json_path") {
actual = readJsonPath(observation.json, matcher.path);
passed = JSON.stringify(actual) === JSON.stringify(matcher.expected);
} else {
const validate = jsonSchemaCompiler.compile(matcher.schema);
passed = validate(observation.json);
actual = passed
? observation.json
: { value: observation.json, errors: validate.errors ?? [] };
}
return {
matcher,
passed,
detail: passed
? "matched"
: `expected ${JSON.stringify(matcher)}; observed ${JSON.stringify(actual)}`,
};
}
export async function evaluateMatchers(
matchers: readonly Matcher[],
observation: MatcherObservation,
): Promise<MatcherResult[]> {
return Promise.all(
matchers.map((matcher) => evaluateMatcher(matcher, observation)),
);
}
tests/runner-e2e/merge.config.ts
+14
View File
@@ -0,0 +1,14 @@
import path from "node:path";
const output = path.resolve(
process.env.PAPERCLIP_RUNNER_E2E_MERGED_REPORT_DIR ??
"runner-e2e-merged-report",
);
export default {
testDir: ".",
reporter: [
["html", { open: "never", outputFolder: path.join(output, "html") }],
["junit", { outputFile: path.join(output, "playwright-junit.xml") }],
],
};
tests/runner-e2e/openrouter-models-update.ts
+59
View File
@@ -0,0 +1,59 @@
import { writeFile } from "node:fs/promises";
import {
rankingContentHash,
validateOpenRouterRankingSnapshot,
type OpenRouterRankedModel,
} from "./openrouter-ranking.js";
const sourceUrl =
"https://openrouter.ai/api/v1/models?sort=top-weekly&supported_parameters=tools";
interface OpenRouterModelResponse {
data?: Array<{
id?: string;
name?: string;
supported_parameters?: string[];
}>;
}
const response = await fetch(sourceUrl, {
headers: { Accept: "application/json" },
});
if (!response.ok) {
throw new Error(`OpenRouter ranking request failed with ${response.status}`);
}
const payload = (await response.json()) as OpenRouterModelResponse;
const models: OpenRouterRankedModel[] = (payload.data ?? [])
.filter(
(
model,
): model is Required<
NonNullable<OpenRouterModelResponse["data"]>[number]
> =>
Boolean(
model.id && model.name && model.supported_parameters?.includes("tools"),
),
)
.slice(0, 5)
.map((model, index) => ({
rank: index + 1,
id: model.id,
name: model.name,
supportedParameters: [...new Set(model.supported_parameters)].sort(),
}));
const capturedAt = new Date().toISOString();
const snapshot = validateOpenRouterRankingSnapshot({
schema: "paperclip.runner-e2e.openrouter-ranking/v1",
snapshotId: `top-weekly-tools-${capturedAt.slice(0, 10)}`,
ranking: "top-weekly",
requiredParameter: "tools",
sourceUrl,
capturedAt,
contentHash: rankingContentHash(models),
models,
});
const output = new URL("./openrouter-models.json", import.meta.url);
await writeFile(output, `${JSON.stringify(snapshot, null, 2)}\n`, "utf8");
console.log(
`Pinned ${snapshot.models.length} OpenRouter model(s) in ${output.pathname}`,
);
tests/runner-e2e/openrouter-models.json
+41
View File
@@ -0,0 +1,41 @@
{
"schema": "paperclip.runner-e2e.openrouter-ranking/v1",
"snapshotId": "top-weekly-tools-2026-08-28",
"ranking": "top-weekly",
"requiredParameter": "tools",
"sourceUrl": "https://openrouter.ai/api/v1/models?sort=top-weekly&supported_parameters=tools",
"capturedAt": "2026-08-28T00:00:00.000Z",
"contentHash": "c1b564582712dd2769e4855a79bc5604e860bf7a85fb98445fff6b3c4c100a3c",
"models": [
{
"rank": 1,
"id": "deepseek/deepseek-v4-flash-0731",
"name": "DeepSeek V4 Flash 0731",
"supportedParameters": ["tools"]
},
{
"rank": 2,
"id": "xiaomi/mimo-v2.5",
"name": "MiMo V2.5",
"supportedParameters": ["tools"]
},
{
"rank": 3,
"id": "tencent/hy3",
"name": "Tencent HY 3",
"supportedParameters": ["tools"]
},
{
"rank": 4,
"id": "nvidia/nemotron-3-ultra-550b-a55b:free",
"name": "Nemotron 3 Ultra 550B A55B (free)",
"supportedParameters": ["tools"]
},
{
"rank": 5,
"id": "openai/gpt-5.6-luna",
"name": "GPT-5.6 Luna",
"supportedParameters": ["tools"]
}
]
}
tests/runner-e2e/openrouter-ranking.ts
+78
View File
@@ -0,0 +1,78 @@
import { createHash } from "node:crypto";
import { readFileSync } from "node:fs";
export interface OpenRouterRankedModel {
rank: number;
id: string;
name: string;
supportedParameters: string[];
}
export interface OpenRouterRankingSnapshot {
schema: "paperclip.runner-e2e.openrouter-ranking/v1";
snapshotId: string;
ranking: "top-weekly";
requiredParameter: "tools";
sourceUrl: string;
capturedAt: string;
contentHash: string;
models: OpenRouterRankedModel[];
}
export function rankingContentHash(models: readonly OpenRouterRankedModel[]) {
return createHash("sha256").update(JSON.stringify(models)).digest("hex");
}
export function validateOpenRouterRankingSnapshot(
value: unknown,
): OpenRouterRankingSnapshot {
if (!value || typeof value !== "object" || Array.isArray(value)) {
throw new Error("OpenRouter ranking snapshot must be an object");
}
const snapshot = value as Partial<OpenRouterRankingSnapshot>;
if (snapshot.schema !== "paperclip.runner-e2e.openrouter-ranking/v1") {
throw new Error("OpenRouter ranking snapshot has an unknown schema");
}
if (
snapshot.ranking !== "top-weekly" ||
snapshot.requiredParameter !== "tools" ||
!snapshot.snapshotId?.trim() ||
!snapshot.sourceUrl?.startsWith("https://openrouter.ai/") ||
!snapshot.capturedAt ||
Number.isNaN(Date.parse(snapshot.capturedAt)) ||
!Array.isArray(snapshot.models) ||
snapshot.models.length !== 5
) {
throw new Error("OpenRouter ranking snapshot metadata is invalid");
}
const ids = new Set<string>();
for (const [index, model] of snapshot.models.entries()) {
if (
model.rank !== index + 1 ||
!/^[a-z0-9][a-z0-9._-]*\/[a-z0-9][a-z0-9._:-]*$/i.test(model.id) ||
!model.name?.trim() ||
!model.supportedParameters?.includes("tools") ||
ids.has(model.id)
) {
throw new Error(`OpenRouter ranked model ${index + 1} is invalid`);
}
ids.add(model.id);
}
if (snapshot.contentHash !== rankingContentHash(snapshot.models)) {
throw new Error("OpenRouter ranking snapshot content hash is invalid");
}
return snapshot as OpenRouterRankingSnapshot;
}
export const openRouterRankingSnapshot = validateOpenRouterRankingSnapshot(
JSON.parse(
readFileSync(new URL("./openrouter-models.json", import.meta.url), "utf8"),
),
);
export function openRouterProfileId(modelId: string) {
return `openrouter-${modelId}`
.toLowerCase()
.replace(/[^a-z0-9]+/g, "-")
.replace(/^-+|-+$/g, "");
}
tests/runner-e2e/playwright.config.ts
+69
View File
@@ -0,0 +1,69 @@
import path from "node:path";
import { defineConfig } from "@playwright/test";
import { runnerE2EWebServerCommand } from "./web-server-command.js";
function required(name: string) {
const value = process.env[name]?.trim();
if (!value) throw new Error(`${name} is required`);
return value;
}
const port = Number(required("PAPERCLIP_RUNNER_E2E_PORT"));
const temporaryRoot = required("PAPERCLIP_RUNNER_E2E_TEMP_ROOT");
const privateDir = required("PAPERCLIP_RUNNER_E2E_PRIVATE_DIR");
const paperclipHome = required("PAPERCLIP_HOME");
const configPath = required("PAPERCLIP_CONFIG");
const baseURL = `http://127.0.0.1:${port}`;
required("PAPERCLIP_INSTANCE_ID");
required("PAPERCLIP_AGENT_JWT_SECRET");
required("PAPERCLIP_DECISION_SIGNING_SECRET");
required("PAPERCLIP_TOOL_ACTION_SIGNING_SECRET");
required("BETTER_AUTH_SECRET");
if (
!paperclipHome.startsWith(`${temporaryRoot}${path.sep}`) ||
!configPath.startsWith(`${temporaryRoot}${path.sep}`)
) {
throw new Error("Paperclip server paths escape the isolated temporary root");
}
const repositoryRoot = path.resolve(import.meta.dirname, "../..");
export default defineConfig({
testDir: ".",
testMatch: "runner.spec.ts",
timeout: Number(process.env.PAPERCLIP_RUNNER_E2E_TEST_TIMEOUT_MS ?? 600_000),
expect: { timeout: 30_000 },
fullyParallel: false,
workers: 1,
retries: 0,
use: {
baseURL,
browserName: "chromium",
headless: true,
actionTimeout: 30_000,
navigationTimeout: 30_000,
screenshot: "only-on-failure",
trace: "retain-on-failure",
video: "retain-on-failure",
},
webServer: {
// Do not put an env object here: Playwright serializes webServer config in
// blob reports. The wrapper inherits the test process and strips provider
// keys before spawning the real Paperclip process.
command: runnerE2EWebServerCommand(repositoryRoot),
url: `${baseURL}/api/health`,
reuseExistingServer: false,
timeout: 180_000,
stdout: "pipe",
stderr: "pipe",
},
outputDir: path.join(privateDir, "playwright-output"),
reporter: [
["list"],
["blob", { outputDir: path.join(privateDir, "blob-report") }],
["junit", { outputFile: path.join(privateDir, "junit.xml") }],
[
"html",
{ open: "never", outputFolder: path.join(privateDir, "html-report") },
],
],
});
tests/runner-e2e/redaction.ts
+188
View File
@@ -0,0 +1,188 @@
import { createReadStream } from "node:fs";
import { readdir } from "node:fs/promises";
import path from "node:path";
import { redactDiagnosticText } from "../../packages/adapter-utils/src/command-redaction.js";
const SECRET_SHAPES = [
/\bsk-ant-[A-Za-z0-9_-]{16,}\b/g,
/\bsk-(?:proj-)?[A-Za-z0-9_-]{16,}\b/g,
/\b(?:openrouter|daytona)[-_]?(?:api)?[-_]?key["'=:\s]+[A-Za-z0-9._-]{12,}\b/gi,
] as const;
export function normalizedSecrets(values: readonly (string | undefined)[]) {
return [
...new Set(
values
.map((value) => value?.trim())
.filter((value): value is string => Boolean(value)),
),
].sort((left, right) => right.length - left.length);
}
export function isEphemeralCodexRuntimeAuthFile(
paperclipHome: string,
file: string,
) {
const relative = path.relative(paperclipHome, file).split(path.sep).join("/");
return (
/^instances\/[^/]+\/companies\/[^/]+\/agents\/[^/]+\/codex-home\/auth\.json$/.test(
relative,
) ||
/^instances\/[^/]+\/runtime\/paperclip-runner\/durable-sessions\/[^/]+\/codex-home\/auth\.json$/.test(
relative,
)
);
}
export function redactText(value: string, secrets: readonly string[]) {
let redacted = redactDiagnosticText(value, "[REDACTED]");
return redactKnownSecretsAndShapes(redacted, secrets);
}
function redactKnownSecretsAndShapes(
value: string,
secrets: readonly string[],
) {
let redacted = value;
for (const secret of normalizedSecrets(secrets))
redacted = redacted.split(secret).join("[REDACTED]");
for (const pattern of SECRET_SHAPES)
redacted = redacted.replace(pattern, "[REDACTED_SECRET_SHAPE]");
return redacted;
}
function redactStructuredText(value: string, secrets: readonly string[]) {
const knownSafe = redactKnownSecretsAndShapes(value, secrets);
if (
!/(?:Authorization\s*:\s*Bearer|(?:api[-_]?key|token|secret|password)\s*=)/i.test(
knownSafe,
)
) {
return knownSafe;
}
return redactKnownSecretsAndShapes(
redactDiagnosticText(knownSafe, "[REDACTED]"),
secrets,
);
}
export function findSecretLeak(
value: string | Buffer,
secrets: readonly string[],
options: { includeShapes?: boolean } = {},
): string | null {
const text = Buffer.isBuffer(value) ? value.toString("utf8") : value;
for (const secret of normalizedSecrets(secrets)) {
if (text.includes(secret)) return "exact secret value";
}
if (options.includeShapes !== false) {
for (const pattern of SECRET_SHAPES) {
pattern.lastIndex = 0;
if (pattern.test(text)) return "secret-shaped value";
}
}
return null;
}
/**
* Scan structured JSON one key/value string at a time. Scanning the serialized
* document for provider key shapes can join an object key, punctuation, and an
* unrelated value into a false positive that never existed in the payload.
*/
export function findSecretLeakInJsonValues(
value: unknown,
secrets: readonly string[],
options: { includeShapes?: boolean } = {},
): string | null {
if (typeof value === "string") return findSecretLeak(value, secrets, options);
if (Array.isArray(value)) {
for (const entry of value) {
const leak = findSecretLeakInJsonValues(entry, secrets, options);
if (leak) return leak;
}
return null;
}
if (value && typeof value === "object") {
for (const [key, entry] of Object.entries(value)) {
const keyLeak = findSecretLeak(key, secrets, options);
if (keyLeak) return keyLeak;
const valueLeak = findSecretLeakInJsonValues(entry, secrets, options);
if (valueLeak) return valueLeak;
}
}
return null;
}
export function sanitizeJson(
value: unknown,
secrets: readonly string[],
): unknown {
// Structured values are not shell diagnostics. Applying the command/JWT
// heuristic here would redact stable dotted identifiers such as our schema
// name. Exact loaded secrets and well-known provider key shapes are enough.
if (typeof value === "string") return redactStructuredText(value, secrets);
if (Array.isArray(value))
return value.map((entry) => sanitizeJson(entry, secrets));
if (value && typeof value === "object") {
return Object.fromEntries(
Object.entries(value).map(([key, entry]) => [
key,
sanitizeJson(entry, secrets),
]),
);
}
return value;
}
export function assertSecretFree(
value: string | Buffer,
secrets: readonly string[],
label: string,
options: { includeShapes?: boolean } = {},
) {
const leak = findSecretLeak(value, secrets, options);
if (leak) throw new Error(`Secret leak in ${label}: ${leak}`);
}
export async function findSecretLeakInDirectory(
root: string,
secrets: readonly string[],
options: {
includeShapes?: boolean;
ignoreFile?: (file: string) => boolean;
} = {},
): Promise<{ file: string; reason: string } | null> {
const overlap = Math.max(
256,
...normalizedSecrets(secrets).map((secret) => secret.length + 16),
);
const scan = async (
directory: string,
): Promise<{
file: string;
reason: string;
} | null> => {
const entries = await readdir(directory, { withFileTypes: true });
for (const entry of entries) {
const file = path.join(directory, entry.name);
if (entry.isDirectory()) {
const leak = await scan(file);
if (leak) return leak;
} else if (entry.isFile()) {
if (options.ignoreFile?.(file)) continue;
let carry = Buffer.alloc(0);
for await (const chunk of createReadStream(file)) {
const data = Buffer.concat([
carry,
Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk),
]);
const reason = findSecretLeak(data, secrets, options);
if (reason) return { file, reason };
carry = data.subarray(Math.max(0, data.length - overlap));
}
}
}
return null;
};
return scan(root);
}
tests/runner-e2e/report.test.ts
+358
View File
@@ -0,0 +1,358 @@
import { execFile } from "node:child_process";
import { promisify } from "node:util";
import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { afterEach, describe, expect, it } from "vitest";
import type { RunnerE2EResult } from "./types.js";
const execFileAsync = promisify(execFile);
const repositoryRoot = path.resolve(import.meta.dirname, "../..");
const cleanupDirectories: string[] = [];
afterEach(async () => {
await Promise.all(
cleanupDirectories
.splice(0)
.map((directory) => rm(directory, { recursive: true, force: true })),
);
});
describe("runner E2E report aggregation", () => {
it("selects the latest retry and enforces cleanup and pass evidence", async () => {
const root = await mkdtemp(
path.join(os.tmpdir(), "runner-e2e-report-test-"),
);
cleanupDirectories.push(root);
const executionId = "legacy-codex.local.message-marker";
const base: RunnerE2EResult = {
schema: "paperclip.runner-e2e.result/v1",
executionId,
attempt: 2,
status: "passed",
profileId: "legacy-codex",
environmentId: "local",
caseId: "message-marker",
provider: "codex",
model: "fixture-model",
runtimeMode: "legacy",
startedAt: "2026-08-26T00:00:00.000Z",
finishedAt: "2026-08-26T00:00:01.000Z",
durationMs: 1_000,
runIds: ["run-2"],
usage: {
inputTokens: 1_250,
outputTokens: 75,
cachedInputTokens: 500,
costUsd: 0.0125,
},
cleanup: "passed",
};
for (const attempt of [1, 2]) {
const directory = path.join(root, `attempt-${attempt}`);
await mkdir(directory, { recursive: true });
const result =
attempt === 1
? {
...base,
attempt,
status: "failed" as const,
failureClass: "transient_infrastructure" as const,
}
: {
...base,
matcherResults: [
{
matcher: {
kind: "message_contains" as const,
expected: "PAPERCLIP_E2E_OK",
},
passed: true,
detail: "matched",
},
],
screenshots: [
{
id: "final-state",
label: "Final visible task state",
file: "final-state.png",
},
],
};
await writeFile(
path.join(directory, "result.json"),
JSON.stringify(result),
);
if (attempt === 2) {
await writeFile(path.join(directory, "final-state.png"), "fake-png");
}
await writeFile(
path.join(directory, "evidence-manifest.json"),
JSON.stringify({
files: attempt === 2 ? ["final-state.png"] : [],
leaks: [],
missing: [],
}),
);
}
const staleDuplicate = path.join(root, "attempt-2-stale-duplicate");
await mkdir(staleDuplicate, { recursive: true });
await writeFile(
path.join(staleDuplicate, "result.json"),
JSON.stringify({
...base,
status: "failed",
failureClass: "candidate_failure",
finishedAt: "2026-08-26T00:00:00.500Z",
}),
);
await writeFile(
path.join(staleDuplicate, "evidence-manifest.json"),
JSON.stringify({ files: [], leaks: [], missing: [] }),
);
const output = path.join(root, "merged");
await execFileAsync(
process.execPath,
[
path.join(repositoryRoot, "cli/node_modules/tsx/dist/cli.mjs"),
path.join(repositoryRoot, "tests/runner-e2e/report.ts"),
],
{
cwd: repositoryRoot,
env: {
...process.env,
PAPERCLIP_RUNNER_E2E_REPORT_ROOT: root,
PAPERCLIP_RUNNER_E2E_REPORT_OUT: output,
PAPERCLIP_RUNNER_E2E_EXPECTED_IDS: JSON.stringify([executionId]),
},
},
);
const normalized = JSON.parse(
await readFile(path.join(output, "normalized-results.json"), "utf8"),
);
expect(normalized).toMatchObject({
schema: "paperclip.runner-e2e.campaign/v2",
selected: 1,
executed: 1,
passed: 1,
failed: 0,
retries: 1,
cleanupPassed: true,
});
expect(normalized.billing).toMatchObject({
reportedLlmCostUsd: 0.0125,
llm: {
inputTokens: 1_250,
outputTokens: 75,
runsWithReportedCost: 1,
},
});
expect(normalized.results[0]).toMatchObject({
attempt: 2,
evidenceValid: true,
});
const dashboard = await readFile(
path.join(output, "dashboard.html"),
"utf8",
);
expect(dashboard).toContain("Runner Full-Stack E2E");
expect(dashboard).toContain(executionId);
expect(dashboard).toContain("case-passed");
expect(dashboard).toContain(
"core-compatibility.runner-acpx-codex.daytona.message-marker",
);
expect(dashboard).toContain("case-not-selected");
expect(dashboard).toContain("<img");
expect(dashboard).toContain('class="brand-lockup"');
expect(dashboard).toContain("data-gallery-dialog");
expect(dashboard).toContain("data-gallery-previous");
expect(dashboard).toContain("data-gallery-next");
expect(dashboard).toContain("View gallery · 1");
expect(dashboard).toContain(
"Visual evidence is retained in the access-controlled workflow artifact",
);
expect(dashboard).toContain("Public history excludes visual evidence");
expect(dashboard).toContain("message_contains");
expect(dashboard).toContain("Matchers and test context");
expect(dashboard).toContain("Campaign billing summary");
expect(dashboard).toContain("LLM reported subtotal");
expect(dashboard).toContain("Agent execution time");
expect(dashboard).toContain("Daytona lease time");
expect(dashboard).toContain("1,250 in · 75 out");
expect(dashboard).toContain("$0.0125");
expect(dashboard).toContain("unpriced or unavailable runs are excluded");
expect(dashboard).toContain('class="profile-sticky"');
expect(dashboard).toContain('class="mobile-environment-header"');
expect(dashboard).toContain("data-gallery-profile=");
expect(dashboard).toContain("data-gallery-environment=");
expect(dashboard).toContain('aria-label="Previous"');
expect(dashboard).toContain('aria-label="Next"');
expect(dashboard).not.toContain("overflow: auto; max-height: calc(100vh");
expect(dashboard).toContain("@media (max-width: 1180px)");
expect(
await readFile(path.join(output, "assets", "favicon.svg"), "utf8"),
).toContain("<svg");
expect(
await readFile(path.join(output, "assets", "InterVariable.woff2")),
).not.toHaveLength(0);
expect(
await readFile(
path.join(
output,
"evidence",
`core-compatibility.${executionId}`,
"attempt-2",
"final-state.png",
),
"utf8",
),
).toBe("fake-png");
expect(await readFile(path.join(output, "index.html"), "utf8")).toBe(
dashboard,
);
});
it("prefers a valid rerun over a higher attempt number from an older campaign", async () => {
const root = await mkdtemp(
path.join(os.tmpdir(), "runner-e2e-report-rerun-test-"),
);
cleanupDirectories.push(root);
const executionId = "runner-opencode.local.ask-question";
const common: RunnerE2EResult = {
schema: "paperclip.runner-e2e.result/v1",
executionId,
attempt: 2,
status: "failed",
failureClass: "candidate_failure",
profileId: "runner-opencode",
environmentId: "local",
caseId: "ask-question",
provider: "opencode",
model: "fixture-model",
runtimeMode: "native",
startedAt: "2026-08-26T00:00:00.000Z",
finishedAt: "2026-08-26T00:00:01.000Z",
durationMs: 1_000,
cleanup: "not_started",
};
const failedDirectory = path.join(root, "old-campaign", "attempt-2");
const passedDirectory = path.join(root, "new-campaign", "attempt-1");
await mkdir(failedDirectory, { recursive: true });
await mkdir(passedDirectory, { recursive: true });
await writeFile(
path.join(failedDirectory, "result.json"),
JSON.stringify(common),
);
await writeFile(
path.join(failedDirectory, "evidence-manifest.json"),
JSON.stringify({ files: [], leaks: [], missing: [] }),
);
await writeFile(
path.join(passedDirectory, "result.json"),
JSON.stringify({
...common,
attempt: 1,
status: "passed",
failureClass: undefined,
finishedAt: "2026-08-26T00:00:02.000Z",
cleanup: "passed",
}),
);
await writeFile(
path.join(passedDirectory, "evidence-manifest.json"),
JSON.stringify({
files: ["final-state.png"],
leaks: [],
missing: [],
}),
);
await writeFile(path.join(passedDirectory, "final-state.png"), "fake-png");
const output = path.join(root, "merged");
await execFileAsync(
process.execPath,
[
path.join(repositoryRoot, "cli/node_modules/tsx/dist/cli.mjs"),
path.join(repositoryRoot, "tests/runner-e2e/report.ts"),
],
{
cwd: repositoryRoot,
env: {
...process.env,
PAPERCLIP_RUNNER_E2E_REPORT_ROOT: root,
PAPERCLIP_RUNNER_E2E_REPORT_OUT: output,
PAPERCLIP_RUNNER_E2E_EXPECTED_IDS: JSON.stringify([executionId]),
},
},
);
const normalized = JSON.parse(
await readFile(path.join(output, "normalized-results.json"), "utf8"),
);
expect(normalized).toMatchObject({ passed: 1, failed: 0 });
expect(normalized.results[0]).toMatchObject({
attempt: 1,
status: "passed",
evidenceValid: true,
});
});
it("constructs the public root JUnit from fixed markup and escaped fields", async () => {
const root = await mkdtemp(
path.join(os.tmpdir(), "runner-e2e-report-junit-test-"),
);
cleanupDirectories.push(root);
const executionId = "legacy-codex.local.message-marker";
const directory = path.join(root, "attempt-1");
await mkdir(directory, { recursive: true });
await writeFile(
path.join(directory, "result.json"),
JSON.stringify({
schema: "paperclip.runner-e2e.result/v1",
executionId,
attempt: 1,
status: "failed",
failureClass: "candidate_failure",
error: `provider said \"><script>alert(1)</script>&`,
profileId: "legacy-codex",
environmentId: "local",
caseId: "message-marker",
provider: "codex",
model: "fixture-model",
runtimeMode: "legacy",
startedAt: "2026-08-26T00:00:00.000Z",
finishedAt: "2026-08-26T00:00:01.000Z",
durationMs: 1_000,
cleanup: "passed",
} satisfies RunnerE2EResult),
);
await writeFile(
path.join(directory, "evidence-manifest.json"),
JSON.stringify({ files: [], leaks: [], missing: [] }),
);
const output = path.join(root, "merged");
await expect(
execFileAsync(
process.execPath,
[
path.join(repositoryRoot, "cli/node_modules/tsx/dist/cli.mjs"),
path.join(repositoryRoot, "tests/runner-e2e/report.ts"),
],
{
cwd: repositoryRoot,
env: {
...process.env,
PAPERCLIP_RUNNER_E2E_REPORT_ROOT: root,
PAPERCLIP_RUNNER_E2E_REPORT_OUT: output,
PAPERCLIP_RUNNER_E2E_EXPECTED_IDS: JSON.stringify([executionId]),
},
},
),
).rejects.toBeDefined();
const junit = await readFile(path.join(output, "junit.xml"), "utf8");
expect(junit).toContain(
`message="provider said &quot;&gt;&lt;script&gt;alert(1)&lt;/script&gt;&amp;"`,
);
expect(junit).not.toContain("<script>");
expect(junit).not.toContain("<?xml-stylesheet");
});
});
tests/runner-e2e/report.ts
+336
View File
@@ -0,0 +1,336 @@
import path from "node:path";
import {
copyFile,
mkdir,
readFile,
readdir,
writeFile,
} from "node:fs/promises";
import { runnerMatrix } from "./catalog.js";
import { summarizeExecutionBilling } from "./billing.js";
import { renderRunnerE2EDashboard } from "./dashboard.js";
import {
buildRunnerCampaign,
canonicalExecutionId,
upgradeRunnerResult,
} from "./history.js";
import type { RunnerE2EResult } from "./types.js";
const repositoryRoot = path.resolve(import.meta.dirname, "../..");
interface EvidenceManifest {
files?: string[];
leaks?: Array<{ file: string; reason: string }>;
missing?: string[];
}
interface AggregatedResult {
result: RunnerE2EResult;
evidence: EvidenceManifest | null;
directory: string;
valid: boolean;
errors: string[];
}
async function walk(root: string): Promise<string[]> {
const entries = await readdir(root, { withFileTypes: true }).catch(() => []);
const files: string[] = [];
for (const entry of entries) {
const full = path.join(root, entry.name);
if (entry.isDirectory()) files.push(...(await walk(full)));
else if (entry.isFile()) files.push(full);
}
return files;
}
function xml(value: unknown) {
return String(value ?? "")
.replaceAll("&", "&amp;")
.replaceAll("<", "&lt;")
.replaceAll(">", "&gt;")
.replaceAll('"', "&quot;")
.replaceAll("'", "&apos;");
}
function validate(result: RunnerE2EResult, evidence: EvidenceManifest | null) {
const errors: string[] = [];
if (result.status !== "passed")
errors.push(result.error ?? result.failureClass ?? "execution failed");
if (result.cleanup !== "passed") errors.push(`cleanup=${result.cleanup}`);
if (!evidence) errors.push("evidence manifest missing");
if (evidence?.leaks?.length)
errors.push(`secret leaks=${evidence.leaks.length}`);
if (evidence?.missing?.length)
errors.push(`missing evidence=${evidence.missing.join(",")}`);
if (
result.status === "passed" &&
!evidence?.files?.includes("final-state.png")
) {
errors.push("passing final-state screenshot missing");
}
return errors;
}
function safeEvidenceRelative(relative: string) {
if (path.isAbsolute(relative)) return null;
const segments = relative.split(/[\\/]/).filter(Boolean);
if (segments.length === 0 || segments.some((segment) => segment === ".."))
return null;
return segments;
}
async function stageDashboardEvidence(
selected: readonly AggregatedResult[],
output: string,
) {
const staged = new Map<string, { baseHref: string; files: string[] }>();
for (const entry of selected) {
const baseSegments = [
"evidence",
entry.result.executionId,
`attempt-${entry.result.attempt}`,
];
const copied: string[] = [];
for (const relative of entry.evidence?.files ?? []) {
const segments = safeEvidenceRelative(relative);
if (!segments) continue;
const source = path.join(entry.directory, ...segments);
const destination = path.join(output, ...baseSegments, ...segments);
const didCopy = await mkdir(path.dirname(destination), {
recursive: true,
})
.then(() => copyFile(source, destination))
.then(() => true)
.catch(() => false);
if (didCopy) copied.push(segments.join("/"));
}
staged.set(entry.result.executionId, {
baseHref: baseSegments.join("/"),
files: copied,
});
}
return staged;
}
async function stageDashboardBrandAssets(output: string) {
const assets = path.join(output, "assets");
await mkdir(assets, { recursive: true });
await Promise.all([
copyFile(
path.join(repositoryRoot, "ui/public/favicon.svg"),
path.join(assets, "favicon.svg"),
),
copyFile(
path.join(repositoryRoot, "ui/public/fonts/InterVariable.woff2"),
path.join(assets, "InterVariable.woff2"),
),
]);
}
async function main() {
const root = path.resolve(
process.env.PAPERCLIP_RUNNER_E2E_REPORT_ROOT ?? "tests/runner-e2e/results",
);
const output = path.resolve(
process.env.PAPERCLIP_RUNNER_E2E_REPORT_OUT ?? path.join(root, "merged"),
);
const expectedInput = JSON.parse(
process.env.PAPERCLIP_RUNNER_E2E_EXPECTED_IDS ?? "[]",
) as string[];
if (
!Array.isArray(expectedInput) ||
expectedInput.some((value) => typeof value !== "string")
) {
throw new Error(
"PAPERCLIP_RUNNER_E2E_EXPECTED_IDS must be a JSON string array",
);
}
const expected = expectedInput.map(canonicalExecutionId);
if (expected.length === 0 || new Set(expected).size !== expected.length) {
throw new Error(
"PAPERCLIP_RUNNER_E2E_EXPECTED_IDS must contain unique selected executions",
);
}
const resultFiles = (await walk(root)).filter(
(file) => path.basename(file) === "result.json",
);
const candidates = new Map<string, AggregatedResult[]>();
for (const resultFile of resultFiles) {
const parsed = JSON.parse(
await readFile(resultFile, "utf8"),
) as RunnerE2EResult;
if (
parsed.schema !== "paperclip.runner-e2e.result/v1" &&
parsed.schema !== "paperclip.runner-e2e.result/v2"
)
continue;
const result = upgradeRunnerResult(parsed);
const directory = path.dirname(resultFile);
const evidence = await readFile(
path.join(directory, "evidence-manifest.json"),
"utf8",
)
.then((value) => JSON.parse(value) as EvidenceManifest)
.catch(() => null);
const errors = validate(result, evidence);
const entry = {
result,
evidence,
directory,
valid: errors.length === 0,
errors,
};
candidates.set(result.executionId, [
...(candidates.get(result.executionId) ?? []),
entry,
]);
}
const selected: AggregatedResult[] = [];
for (const executionId of expected) {
const attempts = (candidates.get(executionId) ?? []).sort((left, right) => {
// A report root can contain artifacts from multiple local campaigns (or
// reruns downloaded from CI). Prefer retained evidence that actually
// satisfies the campaign contract, then the newest execution. Attempt
// numbers are only meaningful inside one campaign and must not let an
// older failed attempt shadow a later successful rerun.
if (left.valid !== right.valid) return left.valid ? -1 : 1;
return (
Date.parse(right.result.finishedAt) -
Date.parse(left.result.finishedAt) ||
right.result.attempt - left.result.attempt
);
});
if (attempts.length === 0) {
const now = new Date().toISOString();
const execution = runnerMatrix.find(
(candidate) => candidate.id === executionId,
);
const missing: RunnerE2EResult = {
schema: "paperclip.runner-e2e.result/v2",
executionId,
suiteId: execution?.suite.id ?? "core-compatibility",
suiteDefinitionHash: execution?.suiteDefinitionHash,
attempt: 0,
status: "failed",
failureClass: "permanent_infrastructure",
error: "No result artifact was uploaded",
profileId: execution?.profile.id ?? "unknown",
environmentId: execution?.environment.id ?? "local",
caseId: execution?.task.id ?? "unknown",
provider: execution?.profile.provider ?? "unknown",
model: execution?.profile.model ?? "unknown",
runtimeMode: execution?.profile.expectedRuntimeMode ?? "native",
startedAt: now,
finishedAt: now,
durationMs: 0,
cleanup: "not_started",
};
selected.push({
result: missing,
evidence: null,
directory: root,
valid: false,
errors: [missing.error!],
});
} else {
selected.push(attempts[0]);
}
}
await mkdir(output, { recursive: true });
const [stagedEvidence] = await Promise.all([
stageDashboardEvidence(selected, output),
stageDashboardBrandAssets(output),
]);
const resolvedResults = selected.map((entry) => ({
...entry.result,
status: entry.valid ? entry.result.status : ("failed" as const),
billing: summarizeExecutionBilling(entry.result),
}));
const generatedAt = new Date().toISOString();
const campaign = buildRunnerCampaign({
campaignId:
process.env.PAPERCLIP_E2E_CAMPAIGN_ID ??
(process.env.GITHUB_RUN_ID
? `gha-${process.env.GITHUB_RUN_ID}-${process.env.GITHUB_RUN_ATTEMPT ?? "1"}`
: `report-${generatedAt.replace(/[:.]/g, "-")}`),
generatedAt,
expected,
results: resolvedResults,
});
const billing = campaign.billing;
const normalized = {
...campaign,
results: selected.map((entry, index) => ({
...resolvedResults[index]!,
evidenceValid: entry.valid,
evidenceErrors: entry.errors,
})),
};
await writeFile(
path.join(output, "normalized-results.json"),
`${JSON.stringify(normalized, null, 2)}\n`,
);
const dashboard = renderRunnerE2EDashboard({
title: "Runner Full-Stack E2E",
generatedAt: normalized.generatedAt,
expected,
catalog: runnerMatrix,
entries: selected.map((entry, index) => ({
result: resolvedResults[index]!,
valid: entry.valid,
errors: entry.errors,
evidenceBaseHref: stagedEvidence.get(entry.result.executionId)?.baseHref,
evidenceFiles: stagedEvidence.get(entry.result.executionId)?.files,
})),
campaign,
});
await Promise.all([
writeFile(path.join(output, "dashboard.html"), dashboard, "utf8"),
writeFile(path.join(output, "index.html"), dashboard, "utf8"),
]);
const summaryLines = [
"# Runner Full-Stack E2E",
"",
`Passed: ${normalized.passed}/${selected.length}`,
"",
`Tokens: ${billing.llm.inputTokens} input / ${billing.llm.outputTokens} output / ${billing.llm.cachedInputTokens} cached`,
"",
`Provider-reported LLM cost: $${billing.reportedLlmCostUsd.toFixed(6)} (${billing.llm.runsWithReportedCost}/${billing.llm.runCount} runs priced)`,
"",
`Estimated Daytona list-price runtime cost: $${billing.estimatedRuntimeCostUsd.toFixed(6)}`,
"",
"| Cell | Attempt | Result | Runtime | Duration | Tokens (in/out) | LLM reported | Runtime estimate | Detail |",
"|---|---:|---|---|---:|---:|---:|---:|---|",
...selected.map((entry, index) => {
const detail = entry.errors.join("; ").replaceAll("|", "\\|") || "ok";
const resolved = resolvedResults[index]!;
const cellBilling = resolved.billing!;
const runtimeCost = cellBilling.runtime.estimatedListCostUsd;
return `| ${resolved.executionId} | ${resolved.attempt} | ${entry.valid ? "pass" : "fail"} | ${resolved.runtimeMode} | ${Math.round(resolved.durationMs / 1000)}s | ${cellBilling.llm.inputTokens}/${cellBilling.llm.outputTokens} | $${cellBilling.reportedCostUsd.toFixed(6)} (${cellBilling.llm.costStatus}) | ${runtimeCost === undefined ? cellBilling.runtime.costStatus : `$${runtimeCost.toFixed(6)} est.`} | ${detail} |`;
}),
"",
];
await writeFile(path.join(output, "summary.md"), summaryLines.join("\n"));
const failures = selected.filter((entry) => !entry.valid).length;
const cases = selected
.map((entry) => {
const failure = entry.valid
? ""
: `<failure message="${xml(entry.errors.join("; "))}"/>`;
return `<testcase classname="runner-full-stack-e2e" name="${xml(entry.result.executionId)}" time="${entry.result.durationMs / 1000}">${failure}</testcase>`;
})
.join("");
const junit = `<?xml version="1.0" encoding="UTF-8"?><testsuite name="Runner Full-Stack E2E" tests="${selected.length}" failures="${failures}">${cases}</testsuite>\n`;
await writeFile(path.join(output, "junit.xml"), junit);
console.log(summaryLines.join("\n"));
if (failures > 0) process.exitCode = 1;
}
await main().catch((error) => {
console.error(error instanceof Error ? error.message : String(error));
process.exitCode = 1;
});
tests/runner-e2e/run-observations.ts
+23
View File
@@ -0,0 +1,23 @@
export interface ObservableRunState {
status?: string | null;
errorCode?: string | null;
}
export function isNonExecutingReviewFenceRun(run: ObservableRunState) {
return (
run.status === "cancelled" &&
run.errorCode === "issue_continuation_waiting_on_review"
);
}
export function numberedPlanStepCount(body: string | null | undefined) {
return (body ?? "").split(/\r?\n/).filter((line) => {
const normalized = line
.replaceAll("**", "")
.replaceAll("__", "")
.replaceAll("`", "");
return /^\s*(?:#{1,6}\s*)?(?:[-*+]\s*)?(?:step\s+)?\d+(?:[.)]|\s*[-—:])(?:\s|$)/i.test(
normalized,
);
}).length;
}
tests/runner-e2e/runner.spec.ts
+1318
View File
File diff suppressed because it is too large. Load diff
tests/runner-e2e/selectors.ts
+203
View File
@@ -0,0 +1,203 @@
import { runnerMatrix } from "./catalog.js";
import type { MatrixExecution, MatrixJob } from "./types.js";
export interface RunnerSelectorOptions {
all: boolean;
list: boolean;
matrixJson: boolean;
ids: string[];
suites: string[];
groups: string[];
profiles: string[];
environments: string[];
cases: string[];
headed: boolean;
ui: boolean;
debug: boolean;
maxParallel: number;
}
export class RunnerSelectorError extends Error {}
function valueFor(args: string[], index: number, flag: string): string {
const value = args[index + 1];
if (!value || value.startsWith("--"))
throw new RunnerSelectorError(`${flag} requires a value`);
return value;
}
export function parseRunnerSelectors(
rawArgs: readonly string[],
): RunnerSelectorOptions {
const args = rawArgs.filter((value) => value !== "--");
const options: RunnerSelectorOptions = {
all: false,
list: false,
matrixJson: false,
ids: [],
suites: [],
groups: [],
profiles: [],
environments: [],
cases: [],
headed: false,
ui: false,
debug: false,
maxParallel: Number(process.env.PAPERCLIP_E2E_MAX_PARALLEL ?? "1"),
};
for (let index = 0; index < args.length; index += 1) {
const flag = args[index];
if (flag === "--all") options.all = true;
else if (flag === "--list") options.list = true;
else if (flag === "--matrix-json") options.matrixJson = true;
else if (flag === "--headed") options.headed = true;
else if (flag === "--ui") options.ui = true;
else if (flag === "--debug") options.debug = true;
else if (flag === "--max-parallel") {
const value = valueFor(args, index, flag);
index += 1;
options.maxParallel = Number(value);
} else if (
[
"--id",
"--suite",
"--group",
"--profile",
"--environment",
"--case",
].includes(flag)
) {
const value = valueFor(args, index, flag);
index += 1;
if (flag === "--id") options.ids.push(value);
else if (flag === "--suite") options.suites.push(value);
else if (flag === "--group") options.groups.push(value);
else if (flag === "--profile") options.profiles.push(value);
else if (flag === "--environment") options.environments.push(value);
else options.cases.push(value);
} else {
throw new RunnerSelectorError(`Unknown runner E2E argument: ${flag}`);
}
}
if (!Number.isInteger(options.maxParallel) || options.maxParallel < 1) {
throw new RunnerSelectorError("--max-parallel must be a positive integer");
}
const hasDimensions =
options.suites.length +
options.groups.length +
options.profiles.length +
options.environments.length +
options.cases.length >
0;
if (options.ids.length > 0 && (hasDimensions || options.all)) {
throw new RunnerSelectorError(
"--id is exclusive with --all and dimension filters",
);
}
if (options.all && hasDimensions) {
throw new RunnerSelectorError("--all is exclusive with dimension filters");
}
if (
!options.list &&
!options.matrixJson &&
!options.all &&
options.ids.length === 0 &&
!hasDimensions
) {
throw new RunnerSelectorError(
"Billable runner E2E runs require --all or an explicit selector",
);
}
return options;
}
function assertKnown(
label: string,
selected: readonly string[],
known: Set<string>,
) {
const unknown = selected.filter((value) => !known.has(value));
if (unknown.length > 0)
throw new RunnerSelectorError(`Unknown ${label}: ${unknown.join(", ")}`);
}
export function selectRunnerExecutions(
options: RunnerSelectorOptions,
matrix: readonly MatrixExecution[] = runnerMatrix,
): MatrixExecution[] {
const knownGroups = new Set(matrix.flatMap((execution) => execution.groups));
assertKnown(
"suite",
options.suites,
new Set(matrix.map((execution) => execution.suite.id)),
);
assertKnown("group", options.groups, knownGroups);
assertKnown(
"profile",
options.profiles,
new Set(matrix.map((execution) => execution.profile.id)),
);
assertKnown(
"environment",
options.environments,
new Set(matrix.map((execution) => execution.environment.id)),
);
assertKnown(
"case",
options.cases,
new Set(matrix.map((execution) => execution.task.id)),
);
assertKnown(
"execution id",
options.ids,
new Set(matrix.map((execution) => execution.id)),
);
const selected = matrix.filter((execution) => {
if (options.ids.length > 0) return options.ids.includes(execution.id);
if (
options.all ||
(options.list &&
options.groups.length === 0 &&
options.suites.length === 0 &&
options.profiles.length === 0 &&
options.environments.length === 0 &&
options.cases.length === 0)
)
return true;
return (
(options.suites.length === 0 ||
options.suites.includes(execution.suite.id)) &&
options.groups.every((group) => execution.groups.includes(group)) &&
(options.profiles.length === 0 ||
options.profiles.includes(execution.profile.id)) &&
(options.environments.length === 0 ||
options.environments.includes(execution.environment.id)) &&
(options.cases.length === 0 || options.cases.includes(execution.task.id))
);
});
if (selected.length === 0)
throw new RunnerSelectorError(
"Runner E2E selectors matched zero executions",
);
return selected;
}
export function buildMatrixJobs(
executions: readonly MatrixExecution[],
): MatrixJob[] {
return executions
.map((execution) => ({
executionId: execution.id,
suiteId: execution.suite.id,
profileId: execution.profile.id,
credentialName: execution.profile.credential,
environmentId: execution.environment.id,
caseId: execution.task.id,
timeoutMinutes: execution.environment.id === "daytona" ? 40 : 25,
needsDaytona: execution.environment.id === "daytona",
}))
.sort((left, right) => left.executionId.localeCompare(right.executionId));
}
tests/runner-e2e/server.ts
+104
View File
@@ -0,0 +1,104 @@
import { spawn } from "node:child_process";
import { createWriteStream } from "node:fs";
import { mkdir } from "node:fs/promises";
import path from "node:path";
import {
assertIsolatedServerEnvironment,
buildPaperclipServerEnvironment,
} from "./harness-env.js";
function required(name: string) {
const value = process.env[name]?.trim();
if (!value) throw new Error(`${name} is required`);
return value;
}
const logPath = required("PAPERCLIP_RUNNER_E2E_SERVER_LOG");
const temporaryRoot = required("PAPERCLIP_RUNNER_E2E_TEMP_ROOT");
const paperclipHome = required("PAPERCLIP_HOME");
const configPath = required("PAPERCLIP_CONFIG");
const repositoryRoot = path.resolve(import.meta.dirname, "../..");
const serverEnvironment = buildPaperclipServerEnvironment(process.env, {
NODE_ENV: "test",
PORT: required("PAPERCLIP_RUNNER_E2E_PORT"),
PAPERCLIP_HOME: paperclipHome,
PAPERCLIP_CONFIG: configPath,
PAPERCLIP_INSTANCE_ID: required("PAPERCLIP_INSTANCE_ID"),
PAPERCLIP_AGENT_JWT_SECRET: required("PAPERCLIP_AGENT_JWT_SECRET"),
PAPERCLIP_DECISION_SIGNING_SECRET: required(
"PAPERCLIP_DECISION_SIGNING_SECRET",
),
PAPERCLIP_TOOL_ACTION_SIGNING_SECRET: required(
"PAPERCLIP_TOOL_ACTION_SIGNING_SECRET",
),
BETTER_AUTH_SECRET: required("BETTER_AUTH_SECRET"),
PAPERCLIP_BIND: "loopback",
PAPERCLIP_BIND_HOST: "127.0.0.1",
PAPERCLIP_DEPLOYMENT_MODE: "local_trusted",
PAPERCLIP_DEPLOYMENT_EXPOSURE: "private",
SERVE_UI: "true",
PAPERCLIP_STORAGE_PROVIDER: "local_disk",
PAPERCLIP_STORAGE_LOCAL_DIR: path.join(temporaryRoot, "storage"),
PAPERCLIP_SECRETS_PROVIDER: "local_encrypted",
PAPERCLIP_SECRETS_STRICT_MODE: "true",
PAPERCLIP_DB_BACKUP_ENABLED: "false",
PAPERCLIP_DB_BACKUP_DIR: path.join(temporaryRoot, "backups"),
// Onboarding normally opens the app after listen. Browser ownership belongs
// to Playwright in this harness, so never create a developer desktop tab.
PAPERCLIP_OPEN_ON_LISTEN: "false",
});
assertIsolatedServerEnvironment(serverEnvironment, {
temporaryRoot,
paperclipHome,
configPath,
});
const definedServerEnvironment = Object.fromEntries(
Object.entries(serverEnvironment).filter(
(entry): entry is [string, string] => entry[1] !== undefined,
),
);
await mkdir(path.dirname(logPath), { recursive: true });
const log = createWriteStream(logPath, { flags: "a", mode: 0o600 });
const child = spawn("pnpm", ["paperclipai", "onboard", "--yes", "--run"], {
cwd: repositoryRoot,
env: definedServerEnvironment,
stdio: ["ignore", "pipe", "pipe"],
// Stay in the launcher-created process group. That lets the launcher stop
// Playwright, this wrapper, Paperclip, embedded Postgres, and runner children
// as one verified tree even if graceful web-server shutdown stalls.
detached: false,
});
child.stdout?.on("data", (chunk) => {
process.stdout.write(chunk);
log.write(chunk);
});
child.stderr?.on("data", (chunk) => {
process.stderr.write(chunk);
log.write(chunk);
});
function stop(signal: NodeJS.Signals) {
if (!child.pid) return;
try {
child.kill(signal);
} catch {
// The Paperclip process may already have exited.
}
}
for (const signal of ["SIGINT", "SIGTERM", "SIGHUP"] as const) {
process.on(signal, () => stop(signal));
}
const exitCode = await new Promise<number>((resolve) => {
child.once("error", (error) => {
log.end(`\nserver spawn failed: ${error.message}\n`);
resolve(1);
});
child.once("exit", (code, signal) => {
log.end(`\nserver exited code=${String(code)} signal=${String(signal)}\n`);
resolve(code ?? (signal ? 1 : 0));
});
});
process.exitCode = exitCode;
tests/runner-e2e/shared-memory.ts
+97
View File
@@ -0,0 +1,97 @@
import { execFileSync } from "node:child_process";
import os from "node:os";
export interface DarwinSharedMemorySegment {
id: string;
owner: string;
attachments: number;
creatorPid: number;
}
export function parseDarwinSharedMemory(
output: string,
): DarwinSharedMemorySegment[] {
return output.split(/\r?\n/).flatMap((line) => {
const fields = line.trim().split(/\s+/);
if (fields[0] !== "m" || fields.length < 11) return [];
const attachments = Number(fields[8]);
const creatorPid = Number(fields[10]);
if (!Number.isInteger(attachments) || !Number.isInteger(creatorPid))
return [];
return [
{
id: fields[1]!,
owner: fields[4]!,
attachments,
creatorPid,
},
];
});
}
export function snapshotDarwinSharedMemory(): ReadonlySet<string> {
if (process.platform !== "darwin") return new Set();
try {
return new Set(
parseDarwinSharedMemory(
execFileSync("ipcs", ["-m", "-a"], { encoding: "utf8" }),
).map((segment) => segment.id),
);
} catch {
return new Set();
}
}
function processIsAlive(pid: number) {
try {
process.kill(pid, 0);
return true;
} catch (error) {
return !(
error &&
typeof error === "object" &&
"code" in error &&
error.code === "ESRCH"
);
}
}
/**
* Embedded PostgreSQL can leave one 56-byte SysV segment on macOS when its
* supervising process tree is terminated. macOS defaults to only 32 segment
* IDs, so serial isolated cells eventually cannot run initdb. Reap only a
* segment that appeared during this attempt, belongs to this OS user, has no
* attachments, and whose creator has exited.
*/
export function reapNewDetachedDarwinSharedMemory(
baseline: ReadonlySet<string>,
): string[] {
if (process.platform !== "darwin") return [];
let current: DarwinSharedMemorySegment[];
try {
current = parseDarwinSharedMemory(
execFileSync("ipcs", ["-m", "-a"], { encoding: "utf8" }),
);
} catch {
return [];
}
const owner = os.userInfo().username;
const removed: string[] = [];
for (const segment of current) {
if (
baseline.has(segment.id) ||
segment.owner !== owner ||
segment.attachments !== 0 ||
processIsAlive(segment.creatorPid)
) {
continue;
}
try {
execFileSync("ipcrm", ["-m", segment.id], { stdio: "ignore" });
removed.push(segment.id);
} catch {
// Cleanup verification still catches a resulting bootstrap failure.
}
}
return removed;
}
tests/runner-e2e/support.test.ts
+684
View File
@@ -0,0 +1,684 @@
import { execFileSync } from "node:child_process";
import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { afterEach, describe, expect, it, vi } from "vitest";
import { packageEvidence } from "./evidence.js";
import { RunnerApi } from "./api.js";
import { FixtureRegistry } from "./fixture-registry.js";
import { classifyFailure, shouldRetryFailure } from "./failure-classifier.js";
import {
assertIsolatedServerEnvironment,
buildPaperclipServerEnvironment,
} from "./harness-env.js";
import { assertEmbeddedDatabaseIsolation } from "./instance-isolation.js";
import { evaluateMatchers } from "./matchers.js";
import {
assertSecretFree,
findSecretLeak,
findSecretLeakInJsonValues,
findSecretLeakInDirectory,
isEphemeralCodexRuntimeAuthFile,
redactText,
sanitizeJson,
} from "./redaction.js";
import { parseDarwinSharedMemory } from "./shared-memory.js";
import {
isNonExecutingReviewFenceRun,
numberedPlanStepCount,
} from "./run-observations.js";
import { runnerE2EWebServerCommand } from "./web-server-command.js";
const cleanupDirectories: string[] = [];
afterEach(async () => {
vi.unstubAllEnvs();
vi.unstubAllGlobals();
await Promise.all(
cleanupDirectories
.splice(0)
.map((directory) => rm(directory, { recursive: true, force: true })),
);
});
describe("runner E2E sensitive API boundary", () => {
it("keeps secret request bodies out of Playwright API tracing", async () => {
vi.stubEnv("PAPERCLIP_RUNNER_E2E_PORT", "43123");
const playwrightPost = vi.fn();
const fetchMock = vi.fn().mockResolvedValue(
new Response(JSON.stringify({ id: "secret-id" }), {
status: 201,
headers: { "content-type": "application/json" },
}),
);
vi.stubGlobal("fetch", fetchMock);
const api = new RunnerApi({ post: playwrightPost } as never);
await expect(
api.postSensitive("/api/companies/company-id/secrets", {
value: "fixture-secret-value",
}),
).resolves.toEqual({ id: "secret-id" });
expect(playwrightPost).not.toHaveBeenCalled();
expect(fetchMock).toHaveBeenCalledWith(
new URL("http://127.0.0.1:43123/api/companies/company-id/secrets"),
expect.objectContaining({ method: "POST" }),
);
});
});
describe("runner E2E structured evidence scanning", () => {
it("does not invent a secret shape across JSON syntax boundaries", () => {
const value = {
OPENROUTER_API_KEY: "generated-secret-reference-id",
};
expect(findSecretLeak(JSON.stringify(value), [])).toBe(
"secret-shaped value",
);
expect(findSecretLeakInJsonValues(value, [])).toBeNull();
});
it("still rejects exact and provider-shaped values in nested JSON", () => {
expect(
findSecretLeakInJsonValues(
{ nested: ["safe", "campaign-secret-value"] },
["campaign-secret-value"],
),
).toBe("exact secret value");
expect(
findSecretLeakInJsonValues({ nested: "sk-proj-abcdefghijklmnop" }, []),
).toBe("secret-shaped value");
});
});
describe("runner E2E fixture registry", () => {
it("sets up in dependency order and tears down in reverse", async () => {
const events: string[] = [];
const registry = new FixtureRegistry()
.register({
id: "company",
setup: async () => {
events.push("setup-company");
return "c";
},
teardown: async () => {
events.push("teardown-company");
},
})
.register({
id: "agent",
dependencies: ["company"],
setup: async () => {
events.push("setup-agent");
return "a";
},
teardown: async () => {
events.push("teardown-agent");
},
});
const active = await registry.setupAll();
await active.teardown();
expect(events).toEqual([
"setup-company",
"setup-agent",
"teardown-agent",
"teardown-company",
]);
});
it("tears down partial setup after a failure", async () => {
const cleanup = vi.fn();
const registry = new FixtureRegistry()
.register({ id: "company", setup: async () => "c", teardown: cleanup })
.register({
id: "agent",
dependencies: ["company"],
setup: async () => {
throw new Error("boom");
},
});
await expect(registry.setupAll()).rejects.toThrow("boom");
expect(cleanup).toHaveBeenCalledOnce();
});
it("preserves setup and partial-cleanup failures", async () => {
const registry = new FixtureRegistry()
.register({
id: "company",
setup: async () => "c",
teardown: async () => {
throw new Error("cleanup transport returned 503");
},
})
.register({
id: "agent",
dependencies: ["company"],
setup: async () => {
throw new Error("agent setup failed");
},
});
await expect(registry.setupAll()).rejects.toMatchObject({
name: "AggregateError",
message: expect.stringContaining("cleanup failed"),
errors: expect.arrayContaining([
expect.objectContaining({ message: "agent setup failed" }),
expect.objectContaining({ message: "Fixture teardown failed" }),
]),
});
});
});
describe("runner E2E matchers", () => {
it("normalizes message text and evaluates state invariants", async () => {
const results = await evaluateMatchers(
[
{ kind: "message_contains", expected: "PAPERCLIP_E2E_OK_nonce" },
{ kind: "issue_status", expected: "done" },
{ kind: "runtime_mode", expected: "native" },
],
{
message: " complete PAPERCLIP\\_E2E\\_OK\\_nonce ",
issueStatus: "done",
runtimeMode: "native",
},
);
expect(results.every((result) => result.passed)).toBe(true);
});
it("normalizes ordered fragments and evaluates nested JSON Schema", async () => {
const results = await evaluateMatchers(
[
{
kind: "message_ordered",
expected: ["first marker", "second marker"],
},
{
kind: "json_schema",
schema: {
type: "object",
required: ["run"],
additionalProperties: false,
properties: {
run: {
type: "object",
required: ["status"],
properties: { status: { const: "succeeded" } },
},
},
},
},
],
{
message: "first marker\nsecond marker",
json: { run: { status: "succeeded" } },
},
);
expect(results.every((result) => result.passed)).toBe(true);
});
});
describe("runner E2E run observations", () => {
it("counts provider-equivalent numbered Plan step formats", () => {
expect(numberedPlanStepCount("1. First\n2) Second")).toBe(2);
expect(
numberedPlanStepCount(
"# Plan\n\nStep 1 — First\n\nStep 2 — Second\n\nStep 3: Verify",
),
).toBe(3);
expect(
numberedPlanStepCount("## **Step 1** — First\n- **2.** Second"),
).toBe(2);
});
it("excludes only queued continuations fenced while awaiting review", () => {
expect(
isNonExecutingReviewFenceRun({
status: "cancelled",
errorCode: "issue_continuation_waiting_on_review",
}),
).toBe(true);
expect(
isNonExecutingReviewFenceRun({
status: "failed",
errorCode: "issue_continuation_waiting_on_review",
}),
).toBe(false);
expect(
isNonExecutingReviewFenceRun({
status: "cancelled",
errorCode: "provider_failure",
}),
).toBe(false);
});
});
describe("runner E2E failure policy", () => {
it("retries only transient infrastructure failures", () => {
expect(
classifyFailure(new Error("Daytona preview connection timed out")),
).toBe("transient_infrastructure");
expect(
classifyFailure(
new Error(
"Browser bootstrap failed before task creation: New Task button timed out after a Vite 504",
),
),
).toBe("transient_infrastructure");
expect(
shouldRetryFailure(
classifyFailure(new Error("Daytona preview connection timed out")),
),
).toBe(true);
expect(
shouldRetryFailure(classifyFailure(new Error("marker matcher failed"))),
).toBe(false);
expect(
classifyFailure(
new Error(
"Timed out waiting for issue abc and heartbeat run terminal state",
),
),
).toBe("candidate_failure");
expect(
classifyFailure(
new Error("Provider request timed out during generation"),
),
).toBe("transient_infrastructure");
expect(
classifyFailure(
new Error(
"runner_ingress_unavailable: paperclip-runnerd: cumulative ACK cannot move beyond the produced source cursor",
),
),
).toBe("transient_infrastructure");
expect(
shouldRetryFailure(classifyFailure(new Error("invalid API key"))),
).toBe(false);
expect(
classifyFailure(
new Error("Daytona lease cleanup failed: provider returned 503"),
),
).toBe("transient_infrastructure");
expect(classifyFailure(new Error("cleanup invariant failure"))).toBe(
"cleanup_failure",
);
});
});
describe("runner E2E server isolation", () => {
it("strips database and paid-provider credentials from the Paperclip process", () => {
const env = buildPaperclipServerEnvironment(
{
PATH: "/bin",
DATABASE_URL: "postgres://existing",
DATABASE_MIGRATION_URL: "postgres://migration",
OPENAI_API_KEY: "openai",
ANTHROPIC_API_KEY: "anthropic",
OPENROUTER_API_KEY: "openrouter",
DAYTONA_API_KEY: "daytona",
OPENAI_ORG_ID: "also-provider-sensitive",
PAPERCLIP_API_KEY: "ambient-board-key",
PAPERCLIP_AGENT_API_KEY: "ambient-agent-key",
PAPERCLIP_TASK_BRIDGE_TOKEN: "ambient-task-token",
PAPERCLIP_SETUP_TOKEN: "ambient-setup-token",
PAPERCLIP_SECRETS_MASTER_KEY: "ambient-master-key",
PAPERCLIP_SECRETS_MASTER_KEY_FILE: "/outside/master.key",
PAPERCLIP_STORAGE_S3_BUCKET: "production-bucket",
},
{
PAPERCLIP_HOME: "/tmp/cell/paperclip-home",
PAPERCLIP_CONFIG: "/tmp/cell/paperclip-home/instances/e2e/config.json",
PAPERCLIP_AGENT_JWT_SECRET: "generated-agent-jwt",
PAPERCLIP_DECISION_SIGNING_SECRET: "generated-decision-key",
PAPERCLIP_TOOL_ACTION_SIGNING_SECRET: "generated-tool-key",
BETTER_AUTH_SECRET: "generated-auth-key",
},
);
expect(env.PATH).toBe("/bin");
expect(env.DATABASE_URL).toBeUndefined();
expect(env.OPENAI_API_KEY).toBeUndefined();
expect(env.OPENAI_ORG_ID).toBeUndefined();
expect(env.PAPERCLIP_API_KEY).toBeUndefined();
expect(env.PAPERCLIP_AGENT_API_KEY).toBeUndefined();
expect(env.PAPERCLIP_AGENT_JWT_SECRET).toBe("generated-agent-jwt");
expect(env.PAPERCLIP_TASK_BRIDGE_TOKEN).toBeUndefined();
expect(env.PAPERCLIP_SETUP_TOKEN).toBeUndefined();
expect(env.PAPERCLIP_SECRETS_MASTER_KEY).toBeUndefined();
expect(env.PAPERCLIP_SECRETS_MASTER_KEY_FILE).toBeUndefined();
expect(env.PAPERCLIP_STORAGE_S3_BUCKET).toBeUndefined();
expect(() =>
assertIsolatedServerEnvironment(env, {
temporaryRoot: "/tmp/cell",
paperclipHome: "/tmp/cell/paperclip-home",
configPath: "/tmp/cell/paperclip-home/instances/e2e/config.json",
}),
).not.toThrow();
});
it("uses absolute repository paths for the Playwright web server", () => {
const command = runnerE2EWebServerCommand("/workspace/paperclip");
expect(command).toContain(
"'/workspace/paperclip/cli/node_modules/tsx/dist/cli.mjs'",
);
expect(command).toContain(
"'/workspace/paperclip/tests/runner-e2e/server.ts'",
);
});
it("accepts only embedded state paths beneath the temporary root", async () => {
const root = await mkdtemp(
path.join(os.tmpdir(), "runner-e2e-instance-isolation-test-"),
);
cleanupDirectories.push(root);
const database = path.join(root, "paperclip-home", "db");
const secretsKey = path.join(
root,
"paperclip-home",
"secrets",
"master.key",
);
const configPath = path.join(root, "paperclip-home", "config.json");
await mkdir(database, { recursive: true });
await mkdir(path.dirname(secretsKey), { recursive: true });
await writeFile(secretsKey, "generated-master-key");
const config = {
database: {
mode: "embedded-postgres",
embeddedPostgresDataDir: database,
backup: { dir: path.join(root, "backups") },
},
logging: { logDir: path.join(root, "logs") },
storage: {
provider: "local_disk",
localDisk: { baseDir: path.join(root, "storage") },
},
secrets: {
provider: "local_encrypted",
strictMode: true,
localEncrypted: { keyFilePath: secretsKey },
},
};
await writeFile(configPath, JSON.stringify(config));
await expect(
assertEmbeddedDatabaseIsolation(configPath, root),
).resolves.toBeUndefined();
await writeFile(
configPath,
JSON.stringify({
...config,
storage: {
...config.storage,
localDisk: { baseDir: "/outside/storage" },
},
}),
);
await expect(
assertEmbeddedDatabaseIsolation(configPath, root),
).rejects.toThrow("storage path escaped");
});
});
describe("runner E2E evidence redaction", () => {
const secret = "sk-proj-supersecretvalue123456";
it("redacts exact and shaped credentials recursively", () => {
expect(redactText(`token=${secret}`, [secret])).toBe("token=[REDACTED]");
expect(sanitizeJson({ nested: [secret] }, [secret])).toEqual({
nested: ["[REDACTED]"],
});
expect(sanitizeJson("paperclip.runner-e2e.evidence/v1", [secret])).toBe(
"paperclip.runner-e2e.evidence/v1",
);
});
it("detects leaks and accepts sanitized evidence", () => {
expect(findSecretLeak(Buffer.from(secret), [secret])).toBeTruthy();
expect(() => assertSecretFree("safe", [secret], "fixture")).not.toThrow();
expect(() =>
assertSecretFree(
"sk-proj-documentationfixture123456",
[secret],
"API source",
{ includeShapes: false },
),
).not.toThrow();
});
it("finds exact credentials across streamed persisted-state chunks", async () => {
const root = await mkdtemp(
path.join(os.tmpdir(), "runner-e2e-secret-scan-test-"),
);
cleanupDirectories.push(root);
await writeFile(
path.join(root, "database.bin"),
Buffer.concat([
Buffer.alloc(65_530, "x"),
Buffer.from(secret),
Buffer.alloc(32, "y"),
]),
);
await expect(
findSecretLeakInDirectory(root, [secret]),
).resolves.toMatchObject({ reason: "exact secret value" });
});
it("can ignore fake key shapes while scanning persisted package state", async () => {
const root = await mkdtemp(
path.join(os.tmpdir(), "runner-e2e-secret-shape-test-"),
);
cleanupDirectories.push(root);
await writeFile(
path.join(root, "provider-fixture.test.ts"),
'const fake = "sk-proj-documentationfixture123456";\n',
);
await expect(
findSecretLeakInDirectory(root, [secret], { includeShapes: false }),
).resolves.toBeNull();
});
it("can narrowly exclude a verified ephemeral runtime credential file", async () => {
const root = await mkdtemp(
path.join(os.tmpdir(), "runner-e2e-ephemeral-auth-test-"),
);
cleanupDirectories.push(root);
const runtimeAuth = path.join(root, "codex-home", "auth.json");
const forbiddenConfig = path.join(root, "config.json");
await mkdir(path.dirname(runtimeAuth), { recursive: true });
await writeFile(runtimeAuth, secret, { mode: 0o600 });
await writeFile(forbiddenConfig, secret);
await expect(
findSecretLeakInDirectory(root, [secret], {
includeShapes: false,
ignoreFile: (file) => file === runtimeAuth,
}),
).resolves.toMatchObject({ file: forbiddenConfig });
});
it("recognizes both managed and durable-session Codex runtime auth files", () => {
const root = path.join(os.tmpdir(), "paperclip-home");
expect(
isEphemeralCodexRuntimeAuthFile(
root,
path.join(
root,
"instances/instance-1/companies/company-1/agents/agent-1/codex-home/auth.json",
),
),
).toBe(true);
expect(
isEphemeralCodexRuntimeAuthFile(
root,
path.join(
root,
"instances/instance-1/runtime/paperclip-runner/durable-sessions/session-1/codex-home/auth.json",
),
),
).toBe(true);
expect(
isEphemeralCodexRuntimeAuthFile(
root,
path.join(root, "instances/instance-1/runtime/auth.json"),
),
).toBe(false);
expect(
isEphemeralCodexRuntimeAuthFile(
root,
path.join(
root,
"instances/instance-1/runtime/paperclip-runner/durable-sessions/session-1/codex-home/config.toml",
),
),
).toBe(false);
});
it("publishes only allowlisted sanitized files and reports source leaks", async () => {
const root = await mkdtemp(
path.join(os.tmpdir(), "runner-e2e-evidence-test-"),
);
cleanupDirectories.push(root);
const privateDir = path.join(root, "private");
const uploadDir = path.join(root, "upload");
await mkdir(privateDir, { recursive: true });
await writeFile(
path.join(privateDir, "result.json"),
JSON.stringify({ error: secret }),
);
await writeFile(path.join(privateDir, "database.sqlite"), secret);
const packaged = await packageEvidence({
privateDir,
uploadDir,
secrets: [secret],
expectPassScreenshot: false,
});
expect(packaged.leaks).toEqual([
{ file: "result.json", reason: "exact secret value" },
]);
expect(
await readFile(path.join(uploadDir, "result.json"), "utf8"),
).toContain("[REDACTED]");
await expect(
readFile(path.join(uploadDir, "database.sqlite")),
).rejects.toThrow();
});
it("keeps raster evidence private to CI and rejects active SVG content", async () => {
const root = await mkdtemp(
path.join(os.tmpdir(), "runner-e2e-visual-evidence-test-"),
);
cleanupDirectories.push(root);
const privateDir = path.join(root, "private");
const uploadDir = path.join(root, "upload");
const playwrightOutput = path.join(privateDir, "playwright-output");
await mkdir(playwrightOutput, { recursive: true });
await writeFile(path.join(privateDir, "final-state.png"), "png");
await writeFile(path.join(playwrightOutput, "failure.webm"), "webm");
await writeFile(
path.join(playwrightOutput, "active.svg"),
"<svg onload='alert(1)' />",
);
const packaged = await packageEvidence({
privateDir,
uploadDir,
secrets: [secret],
expectPassScreenshot: false,
});
expect(packaged.files).toEqual(
expect.arrayContaining([
"final-state.png",
path.join("playwright-output", "failure.webm"),
]),
);
expect(packaged.files).not.toContain(
path.join("playwright-output", "active.svg"),
);
await expect(
readFile(path.join(playwrightOutput, "active.svg"), "utf8"),
).resolves.toContain("onload");
await expect(
readFile(path.join(uploadDir, "playwright-output", "active.svg")),
).rejects.toThrow();
});
it("preserves valid JSON while redacting escaped command diagnostics", async () => {
const root = await mkdtemp(
path.join(os.tmpdir(), "runner-e2e-json-evidence-test-"),
);
cleanupDirectories.push(root);
const privateDir = path.join(root, "private");
const uploadDir = path.join(root, "upload");
await mkdir(path.join(privateDir, "snapshots"), { recursive: true });
await writeFile(
path.join(privateDir, "snapshots", "api-state.json"),
JSON.stringify({
log: String.raw`curl -H \"Authorization: Bearer temporary-run-token\" \\\n+ \"$PAPERCLIP_API_URL/api/issues\"`,
}),
);
await packageEvidence({
privateDir,
uploadDir,
secrets: [secret],
expectPassScreenshot: false,
});
const uploaded = await readFile(
path.join(uploadDir, "snapshots", "api-state.json"),
"utf8",
);
expect(() => JSON.parse(uploaded)).not.toThrow();
expect(uploaded).toContain("[REDACTED]");
});
it("streams large ZIP evidence and detects exact secrets", async () => {
const root = await mkdtemp(
path.join(os.tmpdir(), "runner-e2e-zip-evidence-test-"),
);
cleanupDirectories.push(root);
const privateDir = path.join(root, "private");
const uploadDir = path.join(root, "upload");
const blobDir = path.join(privateDir, "blob-report");
await mkdir(blobDir, { recursive: true });
await writeFile(
path.join(blobDir, "trace.txt"),
Buffer.concat([Buffer.alloc(2 * 1024 * 1024, "x"), Buffer.from(secret)]),
);
execFileSync("zip", ["-q", "report.zip", "trace.txt"], {
cwd: blobDir,
});
await rm(path.join(blobDir, "trace.txt"));
const packaged = await packageEvidence({
privateDir,
uploadDir,
secrets: [secret],
expectPassScreenshot: false,
});
expect(packaged.leaks).toEqual([
{
file: path.join("blob-report", "report.zip"),
reason: "exact secret value",
},
]);
await expect(
readFile(path.join(uploadDir, "blob-report", "report.zip")),
).rejects.toThrow();
});
});
describe("runner E2E macOS shared-memory cleanup", () => {
it("parses only shared-memory rows from ipcs output", () => {
expect(
parseDarwinSharedMemory(
`IPC status from <running system>\nT ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID\nm 327709 0x028ed0ab --rw------- dotta staff dotta staff 0 56 52172 52172\ns 123 0x0 --ra------- dotta staff\n`,
),
).toEqual([
{
id: "327709",
owner: "dotta",
attachments: 0,
creatorPid: 52172,
},
]);
});
});
tests/runner-e2e/tsconfig.json
+13
View File
@@ -0,0 +1,13 @@
{
"extends": "../../tsconfig.base.json",
"compilerOptions": {
"noEmit": true,
"rootDir": "../..",
"types": ["node"],
"typeRoots": [
"../../cli/node_modules/@types",
"../../server/node_modules/@types"
]
},
"include": ["./**/*.ts"]
}
tests/runner-e2e/types.ts
+371
View File
@@ -0,0 +1,371 @@
export const CREDENTIAL_NAMES = [
"OPENAI_API_KEY",
"ANTHROPIC_API_KEY",
"OPENROUTER_API_KEY",
"DAYTONA_API_KEY",
] as const;
export type CredentialName = (typeof CREDENTIAL_NAMES)[number];
export type RunnerGeneration = "legacy" | "native";
export type RunnerEnvironmentId = "local" | "daytona";
export type RunnerTaskWorkMode = "standard" | "planning" | "ask";
export type RunnerTaskFlow =
| "single_turn"
| "plan_revision_acceptance"
| "question_resume_completion"
| "plan_approval_completion";
export interface SecretReference {
type: "secret_ref";
secretId: string;
version: "latest";
}
export type SecretReferenceMap = Partial<
Record<CredentialName, SecretReference>
>;
export interface AgentFixtureBuildInput {
environmentId: string;
environmentFixtureId: RunnerEnvironmentId;
workspacePath: string;
secretRefs: SecretReferenceMap;
executionId: string;
}
export interface EnvironmentFixtureBuildInput {
secretRefs: SecretReferenceMap;
daytonaImage?: string;
executionId: string;
}
export interface RunnerProfileFixture {
id: string;
label: string;
generation: RunnerGeneration;
groups: readonly string[];
adapterType: string;
provider: string;
model: string;
modelQualification: {
source:
| "adapter_constant"
| "qualified_runner_profile"
| "openrouter_rankings_snapshot";
qualificationId: string;
};
ranking?: {
rank: number;
canonicalModelId: string;
snapshotId: string;
capturedAt: string;
sourceUrl: string;
};
credential: Exclude<CredentialName, "DAYTONA_API_KEY">;
supportedEnvironments: readonly RunnerEnvironmentId[];
expectedRuntimeMode: RunnerGeneration;
expectedRuntimeMetadata: {
adapterType: string;
provider: string;
};
buildAgent(input: AgentFixtureBuildInput): Record<string, unknown>;
}
export interface EnvironmentFixture {
id: RunnerEnvironmentId;
label: string;
groups: readonly string[];
driver: "local" | "sandbox";
provider: "local" | "daytona";
credential?: "DAYTONA_API_KEY";
lifecycle: {
setup: "instance_managed" | "create_via_api";
probe: "run_context_via_api";
cleanup: "instance_shutdown" | "delete_via_api_and_destroy_leases";
};
expectedExecutionTarget: {
kind: "local" | "remote";
transport?: "sandbox";
};
buildEnvironment(
input: EnvironmentFixtureBuildInput,
): Record<string, unknown>;
}
export type Matcher =
| { kind: "message_exact"; expected: string }
| { kind: "message_contains"; expected: string }
| { kind: "message_regex"; pattern: string; flags?: string }
| { kind: "message_ordered"; expected: readonly string[] }
| { kind: "issue_status"; expected: string }
| { kind: "run_status"; expected: string }
| { kind: "runtime_mode"; expected: RunnerGeneration }
| { kind: "environment"; expected: RunnerEnvironmentId }
| { kind: "file_exists"; path: string }
| { kind: "file_contains"; path: string; expected: string }
| { kind: "artifact_exists"; name: string; mimeType?: string }
| { kind: "json_path"; path: string; expected: unknown }
| { kind: "json_schema"; schema: Record<string, unknown> };
export interface RunnerTaskFixture {
id: string;
label: string;
groups: readonly string[];
workMode: RunnerTaskWorkMode;
flow: RunnerTaskFlow;
expectedRunCount: number;
attemptTimeoutMs: Readonly<Record<RunnerEnvironmentId, number>>;
expectedTerminalState: {
issue: "done";
run: "succeeded";
};
buildTitle(nonce: string): string;
buildPrompt(nonce: string): string;
buildVisibleMarker(nonce: string): string;
buildRevisionRequest?(nonce: string): string;
buildQuestionAnswer?(nonce: string): {
optionLabel: string;
expectedMarker: string;
};
buildPlanMarkers?(nonce: string): {
draft: string;
revised: string;
};
buildMatchers(nonce: string, execution: MatrixExecution): readonly Matcher[];
}
export interface MatrixExecution {
id: string;
suite: RunnerSuiteFixture;
suiteDefinitionHash: string;
profile: RunnerProfileFixture;
environment: EnvironmentFixture;
task: RunnerTaskFixture;
groups: readonly string[];
requiredCredentials: readonly CredentialName[];
}
export interface RunnerSuiteFixture {
id: string;
label: string;
description: string;
groups: readonly string[];
profiles: readonly RunnerProfileFixture[];
environments: readonly EnvironmentFixture[];
tasks: readonly RunnerTaskFixture[];
expectedMatrixSize: number;
definitionMetadata?: Readonly<Record<string, unknown>>;
}
export interface MatrixJob {
executionId: string;
suiteId: string;
profileId: string;
credentialName: Exclude<CredentialName, "DAYTONA_API_KEY">;
environmentId: RunnerEnvironmentId;
caseId: string;
timeoutMinutes: number;
needsDaytona: boolean;
}
export type FailureClass =
| "candidate_failure"
| "transient_infrastructure"
| "permanent_infrastructure"
| "secret_leak"
| "cleanup_failure";
export type RunnerE2ECostStatus =
| "reported"
| "estimated"
| "partial"
| "unpriced"
| "unavailable"
| "not_metered";
export interface RunnerE2ERuntimeUsage {
provider: RunnerEnvironmentId;
/** Sum of the selected Paperclip heartbeat-run spans. */
agentRunDurationMs: number;
/** Sum of provider lease windows when the environment exposes leases. */
leaseDurationMs: number | null;
leaseCount: number;
cpuCores?: number;
memoryGiB?: number;
diskGiB?: number;
estimatedListCostUsd?: number;
costStatus: "estimated" | "unavailable" | "not_metered";
costSource:
| "daytona_public_list_price"
| "provider_cost_unavailable"
| "local_not_metered";
pricingAsOf?: string;
pricingUrl?: string;
}
export interface RunnerE2EBillingSummary {
llm: {
runCount: number;
runsWithTokenUsage: number;
runsWithReportedCost: number;
inputTokens: number;
outputTokens: number;
cachedInputTokens: number;
totalTokens: number;
reportedCostUsd: number;
costStatus: Exclude<RunnerE2ECostStatus, "estimated" | "not_metered">;
};
runtime: RunnerE2ERuntimeUsage;
/** Provider-reported model spend only; never includes unknown/unpriced runs. */
reportedCostUsd: number;
/** Public-list-price estimate for metered execution infrastructure. */
estimatedRuntimeCostUsd: number;
/** Reported model subtotal plus the runtime list-price estimate. */
observedAndEstimatedCostUsd: number;
complete: boolean;
}
export interface RunnerE2EResult {
schema: "paperclip.runner-e2e.result/v1" | "paperclip.runner-e2e.result/v2";
executionId: string;
suiteId?: string;
suiteDefinitionHash?: string;
source?: {
sha: string | null;
ref: string | null;
workflowRunUrl: string | null;
};
rankingSnapshot?: {
snapshotId: string;
capturedAt: string;
sourceUrl: string;
rank: number;
canonicalModelId: string;
};
attempt: number;
status: "passed" | "failed";
failureClass?: FailureClass;
error?: string;
profileId: string;
environmentId: RunnerEnvironmentId;
caseId: string;
provider: string;
model: string;
runtimeMode: RunnerGeneration;
issueId?: string;
issueIdentifier?: string | null;
runIds?: string[];
startedAt: string;
finishedAt: string;
durationMs: number;
usage?: Record<string, unknown> | null;
runtimeUsage?: RunnerE2ERuntimeUsage;
billing?: RunnerE2EBillingSummary;
matcherResults?: Array<{
matcher: Matcher;
passed: boolean;
detail: string;
}>;
screenshots?: Array<{
id: string;
label: string;
file: string;
}>;
cleanup: "not_started" | "passed" | "failed";
}
export interface RunnerE2ESuiteSummary {
suiteId: string;
suiteDefinitionHash: string;
expected: number;
selected: number;
executed: number;
passed: number;
failed: number;
retries: number;
cleanupPassed: boolean;
complete: boolean;
durationMs: number;
billing: RunnerE2EAggregateBillingSummary;
}
export interface RunnerE2EAggregateBillingSummary {
testCount: number;
agentRunDurationMs: number;
leaseDurationMs: number;
llm: RunnerE2EBillingSummary["llm"];
reportedLlmCostUsd: number;
estimatedRuntimeCostUsd: number;
observedAndEstimatedCostUsd: number;
testsWithCompleteBilling: number;
}
export interface RunnerE2ECampaign {
schema: "paperclip.runner-e2e.campaign/v2";
campaignId: string;
generatedAt: string;
source: {
sha: string | null;
ref: string | null;
workflowRunUrl: string | null;
eventName: string | null;
};
expected: string[];
complete: boolean;
selected: number;
executed: number;
passed: number;
failed: number;
retries: number;
cleanupPassed: boolean;
rankingSnapshots: Array<{
snapshotId: string;
capturedAt: string;
sourceUrl: string;
}>;
billing: RunnerE2EAggregateBillingSummary;
suites: RunnerE2ESuiteSummary[];
results: RunnerE2EResult[];
}
export interface RunnerE2EHistoryExecution {
executionId: string;
suiteId: string;
profileId: string;
environmentId: RunnerEnvironmentId;
caseId: string;
provider: string;
model: string;
status: "passed" | "failed";
durationMs: number;
attempt: number;
cleanup: RunnerE2EResult["cleanup"];
billing: RunnerE2EBillingSummary;
}
export interface RunnerE2EHistoryCampaign {
campaignId: string;
generatedAt: string;
source: RunnerE2ECampaign["source"];
complete: boolean;
selected: number;
executed: number;
passed: number;
failed: number;
retries: number;
cleanupPassed: boolean;
publicUrl: string;
billing: RunnerE2EAggregateBillingSummary;
suites: RunnerE2ESuiteSummary[];
executions: RunnerE2EHistoryExecution[];
}
export interface RunnerE2EHistoryIndex {
schema: "paperclip.runner-e2e.history/v1";
updatedAt: string;
latestCampaignId: string | null;
latestGreenCampaignId: string | null;
latestBySuite: Record<string, string>;
latestGreenBySuite: Record<string, string>;
campaigns: RunnerE2EHistoryCampaign[];
}
tests/runner-e2e/vitest.config.ts
+10
View File
@@ -0,0 +1,10 @@
import { defineConfig } from "vitest/config";
export default defineConfig({
root: import.meta.dirname,
test: {
name: "runner-e2e-support",
include: ["**/*.test.ts"],
testTimeout: 30_000,
},
});
tests/runner-e2e/web-server-command.ts
+11
View File
@@ -0,0 +1,11 @@
import path from "node:path";
function shellQuote(value: string) {
return `'${value.replaceAll("'", `'\\''`)}'`;
}
export function runnerE2EWebServerCommand(repositoryRoot: string) {
const tsx = path.join(repositoryRoot, "cli/node_modules/tsx/dist/cli.mjs");
const server = path.join(repositoryRoot, "tests/runner-e2e/server.ts");
return `node ${shellQuote(tsx)} ${shellQuote(server)}`;
}
tests/runner-e2e/workflow-security.test.ts
+146
View File
@@ -0,0 +1,146 @@
import { readdir, readFile } from "node:fs/promises";
import path from "node:path";
import { describe, expect, it } from "vitest";
const repositoryRoot = path.resolve(import.meta.dirname, "../..");
describe("public repository paid workflow security", () => {
it("gates every provider-secret job with stable actor IDs", async () => {
const workflows = await Promise.all(
["runner-full-stack-e2e.yml", "runner-live-evals.yml", "e2e.yml"].map(
async (name) => ({
name,
contents: await readFile(
path.join(repositoryRoot, ".github/workflows", name),
"utf8",
),
}),
),
);
for (const { name, contents } of workflows) {
const authorize = contents.indexOf(" authorize:");
const reauthorize = contents.indexOf("Reauthorize");
const paidCheckout = contents.indexOf("actions/checkout@", reauthorize);
const providerAccess = contents.search(
/(?:OPENAI|ANTHROPIC|OPENROUTER|DAYTONA)_API_KEY:\s*\$\{\{\s*[^}]*secrets\./,
);
expect(authorize, `${name} must have an authorization job`).toBeGreaterThan(
0,
);
expect(
reauthorize,
`${name} must reauthorize partial job reruns`,
).toBeGreaterThan(authorize);
expect(
paidCheckout,
`${name} must authorize before checkout`,
).toBeGreaterThan(reauthorize);
expect(
providerAccess,
`${name} must authorize before provider access`,
).toBeGreaterThan(reauthorize);
expect(contents).toContain("RUNNER_E2E_ALLOWED_ACTOR_IDS");
expect(contents).toContain("github.actor_id");
expect(contents).toContain("github.triggering_actor");
expect(contents).toContain('refs/heads/$DEFAULT_BRANCH');
expect(contents).toContain("needs: authorize");
expect(contents).toContain("name: runner-e2e-paid");
expect(contents).not.toMatch(
/^\s*(?:pull_request|pull_request_target|push|workflow_call|workflow_run):/m,
);
const actionReferences = [
...contents.matchAll(/^\s*(?:-\s*)?uses:\s*([^\s#]+)/gm),
].map((match) => match[1]!);
expect(actionReferences.length).toBeGreaterThan(0);
for (const reference of actionReferences) {
expect(reference).toMatch(/^[^@]+@[0-9a-f]{40}$/);
}
}
const fullStack = workflows[0]!.contents;
for (const [secret, condition] of Object.entries({
OPENAI_API_KEY: "matrix.credentialName == 'OPENAI_API_KEY'",
ANTHROPIC_API_KEY: "matrix.credentialName == 'ANTHROPIC_API_KEY'",
OPENROUTER_API_KEY: "matrix.credentialName == 'OPENROUTER_API_KEY'",
DAYTONA_API_KEY: "matrix.environmentId == 'daytona'",
})) {
expect(fullStack).toContain(
`${secret}: \${{ ${condition} && secrets.${secret} || '' }}`,
);
}
});
it("keeps provider credentials inside explicitly gated paid workflows", async () => {
const workflowDirectory = path.join(repositoryRoot, ".github/workflows");
const allowedProviderWorkflows = new Set([
"e2e.yml",
"runner-full-stack-e2e.yml",
"runner-live-evals.yml",
]);
const names = (await readdir(workflowDirectory)).filter((name) =>
/\.ya?ml$/.test(name),
);
for (const name of names) {
const contents = await readFile(path.join(workflowDirectory, name), "utf8");
const providerSecretReferences = [
...contents.matchAll(
/secrets(?:\.(?:OPENAI_API_KEY|ANTHROPIC_API_KEY|OPENROUTER_API_KEY|DAYTONA_API_KEY)\b|\[['"](?:OPENAI_API_KEY|ANTHROPIC_API_KEY|OPENROUTER_API_KEY|DAYTONA_API_KEY)['"]\])/g,
),
];
if (providerSecretReferences.length > 0) {
expect(
allowedProviderWorkflows.has(name),
`${name} must not receive provider credentials`,
).toBe(true);
}
}
});
it("runs paid scheduled campaigns only on Sundays", async () => {
const workflows = await Promise.all(
["runner-full-stack-e2e.yml", "runner-live-evals.yml"].map((name) =>
readFile(
path.join(repositoryRoot, ".github/workflows", name),
"utf8",
),
),
);
for (const workflow of workflows) {
const crons = [...workflow.matchAll(/cron:\s*"([^"]+)"/g)].map(
(match) => match[1]!,
);
expect(crons).toHaveLength(1);
expect(crons[0]).toMatch(/^\d{1,2} \d{1,2} \* \* 0$/);
expect(workflow).toContain("workflow_dispatch:");
}
});
it("uses environment-scoped OIDC for a no-delete history publisher", async () => {
const workflow = await readFile(
path.join(repositoryRoot, ".github/workflows/runner-full-stack-e2e.yml"),
"utf8",
);
const publisher = workflow.slice(workflow.indexOf(" publish_history:"));
expect(publisher).toContain("id-token: write");
expect(publisher).toContain("name: runner-e2e-history");
expect(publisher).toContain("aws-actions/configure-aws-credentials@");
expect(publisher).toContain("RUNNER_E2E_HISTORY_AWS_ROLE_ARN");
expect(publisher).not.toContain("cache: pnpm");
expect(publisher).not.toMatch(/AWS_(?:ACCESS|SECRET)_KEY/);
expect(publisher).not.toMatch(/aws s3 (?:rm|sync .*--delete)/);
expect(workflow).toContain("history_source_ready");
expect(workflow).toContain(
"Verify history source report and private screenshot evidence",
);
expect(workflow).toContain("private_screenshot=");
expect(workflow).toContain("Publish pruned immutable history");
expect(workflow).toContain("Publish latest structured dashboard");
expect(workflow).not.toContain("dashboard_ready");
expect(workflow).not.toContain("Publish latest screenshot dashboard");
expect(
workflow.indexOf("pnpm test:e2e:runner:history:publish"),
).toBeLessThan(workflow.indexOf("actions/upload-pages-artifact@"));
});
});