PaperClipAI/packages/paperclip-runner/docs/adr/0001-runner-testing-eval-package-boundaries.md

ADR 0001: Runner, Testing, and Eval Package Boundaries

• Status: Accepted
• Date: 2026-08-11
• Decision owners: Paperclip App

Context

The runner package previously exported production contracts, deterministic mocks, conformance fixtures, scenario/eval helpers, and demo servers from one root. A workspace checkout hid two packaging defects: consumers could use undeclared deep surfaces, and the public semantic dispatcher loaded ajv even though ajv was only a development dependency.

External conformance consumers must consume App contracts and test helpers without making the App runtime depend on scenario corpora, provider experiments, or a separate eval package.

Decision

Ownership is:

Owner	Stable responsibility
Paperclip App	PRP schemas and fixtures, canonical semantic catalog and dispatcher, runnerd/client interfaces, ControlPlanePort, the production binding, deterministic mock, and mock/real parity fixtures
Eval consumers	Scenario corpus, provider configuration, experiment reports, and provider-backed orchestration

The production binding remains App code at server/src/services/native-runtime/paperclip-control-plane-port.ts. It implements the App-owned ControlPlanePort; the runner package never imports the server, database, UI, or CLI.

Public runner exports are:

Export	Stability and purpose
@paperclipai/paperclip-runner	Runtime contracts, runner clients/backends, PRP validation/replay, canonical catalog/dispatcher, and compatibility preflight
@paperclipai/paperclip-runner/evals	Versioned native-attempt/build metadata, compatibility negotiation, and explicit runnerd artifact resolution
@paperclipai/paperclip-runner/testing	Deterministic mocks, PRP port conformance, and provider-neutral semantic conformance kit
./browser, ./react, ./standalone, ./styles.css	Existing explicitly named UI/standalone consumers

Mock adapters and conformance constants are no longer package-root exports. Tests and external conformance consumers must use ./testing. Scenario content, provider-backed matrices, reports, and provider configuration are not public App package exports.

The testkit remains an App ./testing subpath rather than a separate package. It shares the runner protocol/catalog release cadence, and no independent consumer or version cadence currently justifies another package. Split it only after an independent release requirement exists; a directory preference is not sufficient.

Generic credential-free matrix orchestration lives in the separately versioned, workspace-private @paperclipai/paperclip-eval-kernel package. It contains no runner imports, provider configuration, scenario corpus, scorer, or report renderer. The runner may use it only as a development dependency; runtime, optional, and peer dependency sets remain free of eval packages. Paid provider campaigns remain external.

The dependency graph is acyclic:

Paperclip App production binding
            |
            v
@paperclipai/paperclip-runner (runtime contracts)
            ^
            |
Eval consumers ----> @paperclipai/paperclip-runner/evals
       |             @paperclipai/paperclip-runner/testing
       +-----------> @paperclipai/paperclip-eval-kernel

No arrow points from App runtime to an external eval repository.

Compatibility and versioning

Package semver describes distribution compatibility. Independently versioned contracts are published in PAPERCLIP_RUNNER_COMPATIBILITY:

Component	Current contract	Compatibility rule
Canonical catalog	1	Operation removals, renames, placement changes, or incompatible schemas require a new contract version
PRP	1	Highest overlapping required protocol version; no overlap fails closed
Runner client	1	Breaking runnerd/client interface changes require a new version
runnerd artifact	2	Binary metadata/package disagreement or digest mismatch fails before launch
Harness driver	1	Breaking descriptor/config/session/conformance behavior requires a new version
Native execution	1	Breaking App attempt-bundle semantics require a new schema version and converter
Evals integration	1	Breaking package/binary/catalog/driver join behavior requires a new version
Control-plane adapter	1	Breaking ControlPlanePort or production-binding expectations require a new version
Testkit	1	Breaking mock seed, vector, observation, or conformance behavior requires a new version
Eval corpus	1	Runner declares a supported inclusive corpus-version range; out-of-range bundles fail before execution

assertPaperclipRunnerCompatibility performs a fail-closed preflight. It emits paperclip_runner_incompatible with stable issue codes for component mismatch, unsupported corpus versions, unknown catalog operations, missing provider capability declarations, and provider-operation gaps. Provider-specific runtime errors must not stand in for this preflight.

Clean-consumer proof

Run:

pnpm --filter @paperclipai/paperclip-runner check:package-boundaries
pnpm --filter @paperclipai/paperclip-runner check:clean-consumers

The second command builds and packs the runner, installs its tarball into a clean consumer, imports only the root, ./evals, and ./testing exports, executes deterministic PRP, harness-driver, and semantic conformance, and verifies the separately staged runnerd artifact digest. The consumer uses no workspace protocol, source-relative import, or deep package path. This is the packaging gate; workspace tests alone are not proof.

For installed dependencies without bundled dependencies, the gate packs a private copy outside pnpm's dependency tree, excluding only the package-root node_modules directory. Manifest bytes, file modes, symlinks, and npm file selection rules remain unchanged. Packages declaring bundled dependencies keep the original pack path so their dependency payload is preserved. Temporary pack inputs are removed after success or failure; installed sources are not modified.

Run the focused staging regression from the repository root:

node --test packages/paperclip-runner/scripts/installed-package-pack.test.mjs

This regression also runs in the Runner's test:typescript:prep gate.

Consequences

• Existing tests importing mock/conformance values from the package root must migrate to @paperclipai/paperclip-runner/testing.
• ajv is a runtime dependency because the public dispatcher imports it.
• The semantic conformance kit defines normalized comparison; real App service adapters and risk-weighted vectors may evolve behind its testkit version.
• Scenario corpus changes do not force an App runtime release unless their declared compatibility requirement changes.