Files
PaperClipAI/packages/paperclip-runner/package.json
T
Dotta af8439a70b feat(runner): restore direct live eval campaigns and reports (#12909)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The Runner executes agents through native and managed provider
drivers.
> - The direct live eval layer had drifted from the current Runner
contracts.
> - The old local workflow did not provide a complete parallel campaign
or durable report history.
> - The Runner also needed current native OpenCode and OpenRouter
qualification.
> - This pull request restores the direct campaign, corrects the runtime
gaps that the campaign found, and adds safe hosted Evalbook history.
> - The benefit is repeatable model comparison against an immutable
Runner and eval source revision.

## Linked Issues or Issue Description

Refs #11297
Refs #11634

**What existing behavior does this improve?**

This improves the direct live `paperclip-runner` eval workflow, provider
execution contract, and static Evalbook reporting path.

**Current behavior**

The direct evals do not have one maintained full campaign on current
`master`. OpenCode has no qualified multi-model OpenRouter roster.
Parallel provider bursts can compact committed events before the
transport observes them. Local reports do not have a separate safe S3
history index.

**Proposed behavior**

Run one immutable roster-plus-case matrix. Use the shared paid AWS
runner fleet. Keep raw artifacts access-controlled. Publish a sanitized
canonical Evalbook report under the separate `runner-protocol-evals` S3
prefix. Keep immutable campaign directories plus root history, latest,
and latest-green pointers.

**Reason and benefit**

Maintainers can compare native Codex, native OpenCode, ACPX, Claude
Managed, and AWS AgentCore behavior over time. They can inspect failures
without mixing this direct protocol layer with browser full-stack E2E.

**Breaking changes**

None. The new workflow and S3 prefix are additive. The existing Runner
full-stack E2E workflow and report remain separate.

## What Changed

- Added a trusted two-shard direct live workflow for up to 393
roster-plus-case cells.
- Reused the numeric actor allowlist, protected paid environment, and
RunsOn fleet controls from Runner full-stack E2E.
- Added immutable Runner and eval revision resolution, exact credential
boundaries, bounded retries, and cost ceilings.
- Added a public report projection that removes sessions, transcripts,
tool payloads, state, traces, raw failures, remote profile identities,
and credential-shaped values.
- Added additive S3 history under `runner-protocol-evals`, with
immutable campaigns and mutable root index pointers.
- Added native OpenCode model injection and current OpenRouter pricing
contracts.
- Fixed direct eval completion, workflow execution, semantic discovery,
warm-attach state reset, executable binding, and event-burst handling.
- Kept Runner browser full-stack E2E behavior and publication separate.
- Documented local and hosted direct eval operation.

## Verification

- `pnpm --filter @paperclipai/paperclip-runner
test:runner-protocol-eval-publish` — 15 passed.
- `pnpm --filter @paperclipai/paperclip-runner build:typescript` —
passed.
- `actionlint .github/workflows/runner-protocol-live-evals.yml
.github/workflows/runner-full-stack-e2e.yml` — passed.
- Local current matrix at the revision in
[paperclip-evals#17](https://github.com/paperclipai/paperclip-evals/pull/17)
— 323 cells across 10 enabled configurations completed.
- Final local current matrix — 269 passed, 11 behavior failures, and 43
expected macOS-only ACPX platform failures.
- Targeted Runner checks — 13/13 eval-session tests, 15/15
publisher/security tests, and package typecheck passed; complete PR CI
is green, including all browser E2E shards.

## Risks

- Paid live campaigns can consume provider budget. Actor authorization,
exact per-cell ceilings, protected environments, and explicit schedule
enablement bound this risk.
- Public reports can leak provider data. The workflow publishes only a
separately projected report and validates every file before upload.
- The new workflow cannot publish until it is present on the default
branch. This pull request does not change the existing
`runner-full-stack-e2e` publication path.
- The campaign is large. It uses two GitHub matrices and caps combined
concurrency at the shared fleet limit.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

OpenAI Codex on GPT-5. The exact deployment ID and context-window size
are not exposed. The model used reasoning, code editing, browser
inspection, repository tools, and live provider execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-09-05 20:18:11 -05:00

196 lines
15 KiB
JSON

{
"name": "@paperclipai/paperclip-runner",
"version": "0.0.0",
"private": true,
"license": "MIT",
"engines": {
"node": ">=24.11.0"
},
"type": "module",
"bin": {
"paperclip-runner-eval-session": "./dist/cli/eval-session.js",
"paperclip-runner-codex-proxy": "./dist/cli/codex-app-server-unix-proxy.js",
"paperclip-runner-opencode-proxy": "./dist/cli/opencode-app-server-proxy.js",
"paperclip-runner-acpx-sidecar": "./dist/cli/acpx-runtime-sidecar.js"
},
"exports": {
".": {
"types": "./dist/index.d.ts",
"import": "./dist/index.js"
},
"./testing": {
"types": "./dist/testing.d.ts",
"import": "./dist/testing.js"
},
"./evals": {
"types": "./dist/evals/index.d.ts",
"import": "./dist/evals/index.js"
},
"./live": {
"types": "./dist/live/index.d.ts",
"import": "./dist/live/index.js"
},
"./devtools": {
"types": "./dist/devtools/index.d.ts",
"import": "./dist/devtools/index.js"
},
"./browser": {
"types": "./dist/browser/index.d.ts",
"import": "./dist/browser/index.js"
},
"./react": {
"types": "./dist/react/index.d.ts",
"import": "./dist/react/index.js"
},
"./standalone": {
"types": "./dist/standalone/index.d.ts",
"import": "./dist/standalone/index.js"
},
"./styles.css": "./styles.css"
},
"files": [
"dist",
"dist-issue-thread",
"protocol",
"styles.css",
"README.md"
],
"sideEffects": false,
"scripts": {
"build": "pnpm run check:protocol-manifest && pnpm run build:typescript && pnpm run check:capability-contract && pnpm run check:capability-inventory && pnpm run check:protocol-coverage && pnpm run check:semantic-contracts && pnpm run check:runner-workflow-traceability && pnpm run build:binary && node scripts/generate-replay-goldens.mjs --check && node scripts/generate-semantic-action-catalog.mjs --check",
"build:typescript": "pnpm run ensure:eval-build-deps && pnpm run check:protocol-types && node ./node_modules/typescript/bin/tsc --version && node ./node_modules/typescript/bin/tsc -p tsconfig.json && node ./node_modules/typescript/bin/tsc -p tsconfig.surfaces.json && node scripts/build-verified-provider-entrypoints.mjs",
"build:rust": "cargo build --manifest-path runner/Cargo.toml --locked --workspace --bins",
"build:binary": "cargo build --release --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --bin paperclip-runnerd && node scripts/stage-runner-binary.mjs",
"build:provider-pack": "pnpm run build:typescript && node scripts/build-provider-pack.mjs",
"build:runner-binaries": "cargo build --manifest-path runner/Cargo.toml --locked --workspace --bins",
"build:browser": "vite build --config vite.config.ts",
"build:sdk": "vite build --config vite.sdk.config.ts",
"build:standalone": "vite build --config vite.standalone.config.ts",
"build:scenarios": "vite build --config vite.scenarios.config.ts && node scripts/check-capability-csp-bundle.mjs",
"build:issue-thread": "vite build --config vite.issue-thread.config.ts",
"typecheck": "pnpm run typecheck:typescript && pnpm run typecheck:rust",
"typecheck:typescript": "pnpm run ensure:eval-build-deps && node --check scripts/protocol-contract.mjs && node --check scripts/generate-protocol-manifest.mjs && node --check scripts/generate-protocol-schema-module.mjs && node --check scripts/generate-acpx-sidecar-contract.mjs && node --check scripts/generate-replay-goldens.mjs && node --check scripts/generate-semantic-action-catalog.mjs && pnpm run check:protocol-types && node ./node_modules/typescript/bin/tsc --version && node ./node_modules/typescript/bin/tsc -p tsconfig.json --noEmit && node ./node_modules/typescript/bin/tsc -p tsconfig.surfaces.json --noEmit",
"typecheck:rust": "cargo fmt --manifest-path runner/Cargo.toml --all -- --check && cargo check --manifest-path runner/Cargo.toml --locked --workspace",
"typecheck:browser": "tsc -p tsconfig.browser.json --noEmit",
"test": "pnpm run test:typescript && pnpm run test:rust",
"test:typescript": "pnpm run ensure:eval-build-deps && pnpm run build:rust && node --test test/protocol-contract.test.mjs test/acpx-sidecar-contract.test.mjs test/acpx-codex-package-contract.test.mjs scripts/aws-agentcore-provisioning.test.mjs scripts/build-verified-provider-entrypoints.test.mjs scripts/local-provider-smoke-environment.test.mjs scripts/materialize-opencode-binary.test.mjs && vitest run",
"test:rust": "cargo test --release --manifest-path runner/Cargo.toml --locked --workspace",
"test:codex": "cargo test --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --test codex_provider",
"test:durable": "cargo test --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core durable::",
"test:browser": "pnpm run build:typescript && pnpm run build:runner-binaries && pnpm run build:browser && playwright test --config devtools/browser/playwright.config.ts",
"test:sdk": "vitest run src/browser/client.test.ts src/browser/transcript-model.test.ts src/react/public-contract.test.tsx",
"test:scenarios": "vitest run src/scenarios src/live src/semantic-tools/semantic-tools.test.ts src/mock-core/capability-mock-control-plane-adapter.test.ts src/issue-thread devtools/issue-thread/src examples/scenario-explorer",
"test:browser:sdk": "pnpm run build:typescript && playwright test --config examples/playwright.config.ts",
"test:browser:scenarios": "playwright test --config examples/playwright.scenarios.config.ts && pnpm run test:browser:issue-thread",
"test:browser:issue-thread": "pnpm run build:typescript && pnpm run build:issue-thread && playwright test --config devtools/issue-thread/playwright.config.ts",
"record:sdk:codex": "pnpm run build:typescript && playwright test --config examples/playwright.real.config.ts",
"generate:protocol-manifest": "node scripts/generate-protocol-manifest.mjs",
"check:protocol-manifest": "node scripts/generate-protocol-manifest.mjs --check",
"generate:protocol-types": "node scripts/generate-protocol-schema-module.mjs && node scripts/generate-acpx-sidecar-contract.mjs",
"check:protocol-types": "node scripts/generate-protocol-schema-module.mjs --check && node scripts/generate-acpx-sidecar-contract.mjs --check",
"generate:capability-contract": "node scripts/generate-capability-contract.mjs",
"check:capability-contract": "node scripts/generate-capability-contract.mjs --check",
"generate:semantic-contracts": "pnpm run build:typescript && node scripts/generate-semantic-contracts.mjs",
"check:semantic-contracts": "pnpm run build:typescript && node scripts/generate-semantic-contracts.mjs --check",
"generate:semantic-action-catalog": "pnpm run build:typescript && node scripts/generate-semantic-action-catalog.mjs",
"check:semantic-action-catalog": "pnpm run build:typescript && node scripts/generate-semantic-action-catalog.mjs --check",
"check:protocol": "pnpm run typecheck:typescript && pnpm run check:protocol-manifest && pnpm run test:typescript && pnpm run check:replay-goldens",
"check:runner": "pnpm run typecheck:rust && pnpm run test:rust && pnpm run check:conformance-parity && pnpm run check:replay-parity",
"check:all": "pnpm run check:eval-kernel && pnpm run check:protocol && pnpm run check:runner",
"generate:replay-goldens": "pnpm run build:typescript && node scripts/generate-replay-goldens.mjs",
"check:replay-goldens": "pnpm run build:typescript && node scripts/generate-replay-goldens.mjs --check",
"check:browser-tokens": "node scripts/check-browser-tokens.mjs",
"check:forbidden-imports": "node scripts/check-forbidden-imports.mjs",
"check:tracked-imports": "node scripts/check-tracked-imports.mjs",
"check:numbered-milestones": "node scripts/check-numbered-milestones.mjs",
"check:package-boundaries": "node scripts/check-package-boundaries.mjs",
"check:clean-consumers": "node scripts/check-clean-consumers.mjs",
"check:eval-kernel": "pnpm --filter @paperclipai/paperclip-eval-kernel test",
"ensure:eval-build-deps": "pnpm --filter @paperclipai/paperclip-eval-kernel build",
"generate:capability-inventory": "node scripts/generate-capability-inventory.mjs",
"generate:protocol-coverage": "pnpm run build:typescript && node scripts/generate-protocol-coverage.mjs",
"check:protocol-coverage": "node scripts/generate-protocol-coverage.mjs --check",
"check:capability-inventory": "node scripts/check-capability-inventory.mjs",
"test:capability-inventory": "node scripts/check-capability-inventory.test.mjs",
"test:capability-evals": "vitest run src/conformance/capability-eval-suite.test.ts",
"test:eval-slice": "pnpm run ensure:eval-build-deps && vitest run src/eval",
"test:runner-workflow-evals": "pnpm run ensure:eval-build-deps && node --test scripts/render-runner-workflow-evalbook.test.mjs && vitest run src/eval/workflow-evals.test.ts src/eval/live-workflow-executor.test.ts",
"test:runner-protocol-eval-publish": "node --test scripts/runner-protocol-eval-campaign.test.mjs scripts/publish-runner-protocol-eval-history.test.mjs scripts/runner-protocol-eval-workflow-security.test.mjs",
"check:runner-workflow-traceability": "pnpm run build:typescript && node scripts/check-runner-workflow-traceability.mjs",
"report:capability-evals": "pnpm run build:typescript && node scripts/run-capability-eval-suite.mjs",
"report:capability-live-evals": "pnpm run build:typescript && cargo build --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --bin paperclip-runnerd && node scripts/run-capability-live-eval-matrix.mjs",
"report:runner-workflow-evals": "pnpm run build:typescript && node scripts/run-runner-workflow-evals.mjs",
"report:runner-live-evals": "pnpm run build:typescript && pnpm run build:runner-binaries && node scripts/run-runner-live-eval-schedule.mjs --mode nightly --execute",
"report:runner-protocol-eval:catalog": "node scripts/runner-protocol-eval-campaign.mjs catalog",
"report:runner-protocol-eval:publish": "node scripts/publish-runner-protocol-eval-history.mjs",
"report:runner-chaos-evals": "pnpm run build:typescript && node scripts/run-runner-live-eval-schedule.mjs --mode chaos",
"check:conformance-parity": "cargo test --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core runs_the_mock_core_path_with_stable_output",
"check:replay-parity": "cargo test --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core replay_fixture_parity",
"docs:validate": "node scripts/validate-doc-links.mjs",
"trace:conformance": "pnpm run trace:conformance:rust",
"trace:conformance:rust": "cargo run --quiet --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --bin conformance-tracer",
"trace:conformance:typescript": "pnpm run build:typescript && node dist/tracer/conformance.js",
"replay:fixture": "pnpm run build:typescript && node dist/cli/replay.js",
"trace:local-runner": "pnpm run build:typescript && pnpm run build:runner-binaries && node dist/cli/local-runner.js",
"trace:codex": "pnpm run build:typescript && node dist/cli/codex.js",
"trace:standalone": "pnpm run build:typescript && node dist/cli/paperclip-adapter.js",
"trace:live-runner": "pnpm run build:typescript && cargo build --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --bin paperclip-runnerd && node dist/cli/live-runner.js",
"demo:standalone": "pnpm run build:typescript && vite --config vite.standalone.config.ts --host 127.0.0.1 --port 4182",
"demo:scenarios": "pnpm run build:typescript && vite --config vite.scenarios.config.ts --host 127.0.0.1 --port 4183",
"console:issue-thread": "pnpm run build:typescript && vite --config vite.issue-thread.config.ts --host 127.0.0.1 --port 4184",
"aws-agentcore:provision": "bash scripts/aws-agentcore.sh provision",
"aws-agentcore:probe": "bash scripts/aws-agentcore.sh probe",
"aws-agentcore:lab": "bash scripts/aws-agentcore.sh lab",
"aws-agentcore:destroy": "bash scripts/aws-agentcore.sh destroy",
"test:aws-agentcore-provisioning": "node --test scripts/aws-agentcore-provisioning.test.mjs",
"smoke:capability:aws-agentcore": "pnpm run build:typescript && cargo build --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --bin paperclip-runnerd && node scripts/capability-aws-agentcore-smoke.mjs",
"demo:live-console": "pnpm run build:typescript && node scripts/live-console-demo-server.mjs",
"smoke:capability:ui": "pnpm run build:typescript && cargo build --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --bin paperclip-runnerd && node scripts/capability-issue-thread-smoke.mjs",
"smoke:capability:cleanroom": "pnpm run build:typescript && cargo build --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --bin paperclip-runnerd && node scripts/capability-clean-room-smoke.mjs",
"smoke:local-provider": "node scripts/run-local-provider-smoke.mjs",
"console:live-console": "pnpm run build:typescript && vite --config vite.config.ts --host 127.0.0.1 --port 4180",
"console:sdk": "pnpm run build:typescript && vite --config vite.sdk.config.ts --host 127.0.0.1 --port 4181",
"browser:dev": "pnpm run build:typescript && pnpm run build:runner-binaries && vite --config vite.config.ts",
"browser:preview": "vite preview --config vite.config.ts",
"verify": "pnpm run build && pnpm run typecheck && pnpm run check:replay-goldens && pnpm run test && pnpm run test:sdk && pnpm run test:scenarios && pnpm run check:browser-tokens && pnpm run test:browser && pnpm run test:browser:sdk && pnpm run test:browser:scenarios && pnpm run check:forbidden-imports && pnpm run check:tracked-imports && pnpm run check:numbered-milestones && pnpm run check:package-boundaries && pnpm run check:clean-consumers && pnpm run docs:validate && pnpm run check:conformance-parity && pnpm run check:replay-parity && pnpm run trace:conformance && pnpm run replay:fixture && pnpm run trace:local-runner -- --quiet",
"verify:rootless": "bash scripts/verify-rootless-linux.sh"
},
"dependencies": {
"@agentclientprotocol/claude-agent-acp": "0.70.0",
"@agentclientprotocol/codex-acp": "1.6.2",
"acpx": "0.13.1",
"ajv": "^8.20.0",
"json-schema-to-ts": "^3.1.1",
"opencode-ai": "1.18.17",
"react-markdown": "^10.1.0",
"remark-gfm": "^4.0.1"
},
"peerDependencies": {
"react": ">=18",
"react-dom": ">=18"
},
"peerDependenciesMeta": {
"react": {
"optional": true
},
"react-dom": {
"optional": true
}
},
"devDependencies": {
"@paperclipai/paperclip-eval-kernel": "workspace:*",
"@playwright/test": "^1.61.1",
"@types/node": "^24.0.0",
"@types/react": "^19.2.17",
"@types/react-dom": "^19.2.5",
"@vitejs/plugin-react": "^6.1.1",
"axe-core": "^4.12.1",
"react": "^19.2.7",
"react-dom": "^19.2.7",
"tsx": "^4.23.12",
"typescript": "^7.0.2",
"vite": "^6.1.0",
"vitest": "^4.1.11"
}
}