mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-27 02:44:50 +01:00
feat(runner): publish eval integration contract
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
90caa73b95
commit
db349f7670
30 files changed
+2790
-12
No files matched your search
@@ -17,6 +17,9 @@ imports or starts Paperclip's server, UI, CLI, or production database.
|
||||
|
||||
- `@paperclipai/paperclip-runner` — production contracts, clients/backends,
|
||||
PRP validation/replay, canonical catalog/dispatcher, and compatibility check.
|
||||
- `@paperclipai/paperclip-runner/evals` — versioned native-attempt/build
|
||||
metadata, explicit digest-verified runnerd artifact resolution, and complete
|
||||
App/Evals compatibility negotiation.
|
||||
- `@paperclipai/paperclip-runner/testing` — deterministic mocks plus PRP and
|
||||
semantic conformance kits. Tests and Paperclip Evals import this explicitly.
|
||||
- `@paperclipai/paperclip-eval-kernel` — separately packed generic eval matrix
|
||||
@@ -164,6 +167,7 @@ and JSON content; see the protocol-server tutorial for direct `curl` examples.
|
||||
- [Capability live runnerd/Codex loop](docs/capability-live-runnerd-codex.md)
|
||||
- [Capability issue-thread UI](docs/capability-issue-thread-ui.md)
|
||||
- [PRP compatibility/versioning policy](docs/protocol-compatibility.md)
|
||||
- [Paperclip Evals integration contract](docs/evals-integration.md)
|
||||
- [PRP v1 expressiveness audit](spec/prp-v1-expressiveness-audit.md)
|
||||
- [Cumulative end-to-end tutorial](docs/tutorials/end-to-end.md)
|
||||
- [OKF knowledge bundle](knowledge/)
|
||||
|
||||
@@ -36,6 +36,7 @@ Public runner exports are:
|
||||
| Export | Stability and purpose |
|
||||
|---|---|
|
||||
| `@paperclipai/paperclip-runner` | Runtime contracts, runner clients/backends, PRP validation/replay, canonical catalog/dispatcher, and compatibility preflight |
|
||||
| `@paperclipai/paperclip-runner/evals` | Versioned native-attempt/build metadata, explicit runnerd artifact resolver, and complete App/Evals compatibility negotiation |
|
||||
| `@paperclipai/paperclip-runner/testing` | Deterministic mocks, PRP port conformance, and provider-neutral semantic conformance kit |
|
||||
| `./browser`, `./react`, `./standalone`, `./styles.css` | Existing explicitly named UI/standalone consumers |
|
||||
|
||||
@@ -85,6 +86,10 @@ contracts are published in `PAPERCLIP_RUNNER_COMPATIBILITY`:
|
||||
| Canonical catalog | 1 | Operation removals, renames, placement changes, or incompatible schemas require a new contract version |
|
||||
| PRP | 1 | Highest overlapping required protocol version; no overlap fails closed |
|
||||
| Runner client | 1 | Breaking runnerd/client interface changes require a new version |
|
||||
| runnerd artifact | 1 | Binary metadata/package disagreement or digest mismatch fails before launch |
|
||||
| Harness driver | 1 | Breaking descriptor/config/session/conformance behavior requires a new version |
|
||||
| Native execution | 1 | Breaking App attempt-bundle semantics require a new schema version and converter |
|
||||
| Evals integration | 1 | Breaking package/binary/catalog/driver join behavior requires a new version |
|
||||
| Control-plane adapter | 1 | Breaking `ControlPlanePort` or production-binding expectations require a new version |
|
||||
| Testkit | 1 | Breaking mock seed, vector, observation, or conformance behavior requires a new version |
|
||||
| Eval corpus | 1 | Runner declares a supported inclusive corpus-version range; out-of-range bundles fail before execution |
|
||||
@@ -108,9 +113,10 @@ pnpm --filter @paperclipai/paperclip-runner check:clean-consumers
|
||||
The second command builds and packs both packages, then installs tarballs into
|
||||
two newly created consumers:
|
||||
|
||||
1. An Evals-side consumer imports only declared runner root and `./testing`
|
||||
exports, executes the deterministic PRP conformance kit, and proves an
|
||||
unsupported provider operation fails explicitly.
|
||||
1. An Evals-side consumer imports only declared runner root, `./evals`, and
|
||||
`./testing` exports; validates the native bundle; executes deterministic PRP
|
||||
and harness-driver conformance; resolves a separately copied runnerd by
|
||||
explicit path/digest; and proves provider/driver mismatches fail explicitly.
|
||||
2. An App-side consumer installs the eval kernel as a development dependency,
|
||||
runs a generic matrix after runner compatibility preflight, and proves the
|
||||
runner manifest has no Evals runtime dependency.
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
byte-identical Conformance result
|
||||
|
||||
Paperclip App production binding --> implements ControlPlanePort
|
||||
Paperclip Evals --> consumes packed runtime + ./testing exports
|
||||
Paperclip Evals --> consumes packed runtime + ./evals + ./testing exports
|
||||
```
|
||||
|
||||
The dependency arrow always points from an implementation toward a contract.
|
||||
@@ -28,6 +28,9 @@ recorded in [ADR 0001](adr/0001-runner-testing-eval-package-boundaries.md).
|
||||
|
||||
- The package root is runtime-only: PRP, runner/client contracts, normalized
|
||||
backends, catalog/dispatcher, and compatibility preflight.
|
||||
- `./evals` is the stable integration join: native attempt schema/fixture,
|
||||
build metadata, complete package/binary/PRP/catalog/driver negotiation, and
|
||||
explicit digest-verified runnerd artifact resolution.
|
||||
- `./testing` contains deterministic mocks and conformance kits.
|
||||
- `@paperclipai/paperclip-eval-kernel` contains generic orchestration only and
|
||||
is permitted in App CI/tests as a development dependency, never at runtime.
|
||||
@@ -67,6 +70,11 @@ boundary and checks the same fixture summaries. Local runner adds the package-lo
|
||||
`paperclip-runnerd` and `fake-harness` binaries without changing that dependency
|
||||
direction.
|
||||
|
||||
Evals receives runnerd as a separate content-addressed artifact. The package
|
||||
does not discover or bundle a host-specific executable: the consumer supplies
|
||||
an explicit path and digest, then reads versioned metadata from that exact
|
||||
binary. See [the Evals integration contract](evals-integration.md).
|
||||
|
||||
## Language ownership
|
||||
|
||||
- Rust is the production direction for deterministic runner behavior,
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
# Paperclip Evals Integration Contract
|
||||
|
||||
## Stable consumer inputs
|
||||
|
||||
Paperclip Evals consumes two explicit App artifacts:
|
||||
|
||||
1. a packed/released `@paperclipai/paperclip-runner` package; and
|
||||
2. an explicit `paperclip-runnerd` executable path plus its
|
||||
`sha256:<lowercase hex>` digest.
|
||||
|
||||
The consumer must not import App source paths, search a workspace for a binary,
|
||||
or infer compatibility from a process failure. The package exposes three
|
||||
relevant entry points:
|
||||
|
||||
- package root: PRP, runner, semantic-tool, and control-plane runtime contracts;
|
||||
- `./evals`: build metadata, native-attempt schema/fixture, compatibility
|
||||
negotiation, semantic receipts/catalog, and explicit runnerd resolution;
|
||||
- `./testing`: deterministic driver/control-plane fakes and conformance kits.
|
||||
|
||||
`resolvePaperclipRunnerdArtifact` resolves only the supplied path, verifies its
|
||||
bytes against the supplied digest, and invokes that exact executable with
|
||||
`--build-metadata`. It never searches `PATH` or the App repository.
|
||||
|
||||
## Versioned join
|
||||
|
||||
`assertPaperclipRunnerEvalCompatibility` checks the complete join before any
|
||||
provider process starts:
|
||||
|
||||
| Dimension | V1 rule |
|
||||
| --- | --- |
|
||||
| Package | Exact package version matches loaded build metadata. |
|
||||
| Binary | runnerd package version and binary-artifact contract match the package. |
|
||||
| PRP | Package, runnerd, and consumer version ranges have a common version. |
|
||||
| Semantic catalog | Contract version and canonical content digest both match. |
|
||||
| Harness driver | Contract and negotiated PRP versions match and every required capability is explicitly true. |
|
||||
| Native output | Consumer and runnerd both select `paperclip-runner/native-execution/v1`. |
|
||||
|
||||
Failures use `paperclip_runner_eval_incompatible` and include all detected
|
||||
component issues with expected and received values. A catalog, driver, or PRP
|
||||
mismatch cannot degrade into an attempted run.
|
||||
|
||||
## Native attempt bundle
|
||||
|
||||
`paperclip-runner/native-execution/v1` is the App-owned raw attempt. Its checked
|
||||
schema is `protocol/schemas/native-execution.schema.json`; the no-spend seeded
|
||||
fixture is `protocol/fixtures/evals/native-execution-seeded.json`.
|
||||
|
||||
The bundle pins run/case/config/attempt identity, case and config digests,
|
||||
package/binary/catalog/driver versions and digests, ordered PRP events, semantic
|
||||
tool definitions/calls/results/denials, terminal state, check-ready
|
||||
observations, usage/cost/time/request totals, transcript completeness, and
|
||||
content-addressed artifacts. Unknown additive fields survive parsing.
|
||||
|
||||
The parser fails closed on an unknown native schema, malformed digest, event
|
||||
run mismatch, missing or conflicting terminal, semantic indexes that omit or
|
||||
reinterpret their PRP tool envelopes, denied results without matching denial
|
||||
receipts, inconsistent token totals, or ambiguous incomplete transcripts.
|
||||
Unknown additive fields are retained at nested contract objects as well as the
|
||||
bundle root. The seeded fixture deliberately preserves a rejected governed
|
||||
tool effect as `denied`; it never turns that effect into a successful mutation.
|
||||
|
||||
Evals owns conversion of this bundle into Evalbook ledger, scores, and reports.
|
||||
The App package does not implement an Evalbook environment loader, importer,
|
||||
grid, comparison, or report renderer.
|
||||
|
||||
## Deterministic conformance
|
||||
|
||||
`runHarnessDriverConformance` with `DeterministicHarnessDriver` covers, without
|
||||
network or provider credentials:
|
||||
|
||||
- capability and unsupported-feature description;
|
||||
- valid/invalid config handling;
|
||||
- session open, turn, snapshot, recovery, close, and cancel lifecycle;
|
||||
- paired provider-neutral semantic tool events;
|
||||
- PRP event validation;
|
||||
- interrupt and cancelled terminal behavior;
|
||||
- usage reporting; and
|
||||
- complete transcript accounting.
|
||||
|
||||
The input fixture is
|
||||
`protocol/fixtures/evals/harness-driver-conformance.json`. The packed
|
||||
clean-consumer gate builds and copies runnerd into an isolated artifact
|
||||
directory, packs the package, installs it offline, resolves only declared
|
||||
exports, validates the native fixture, runs driver conformance, verifies binary
|
||||
metadata/digest, and exercises both compatible and incompatible negotiation.
|
||||
|
||||
Run it with:
|
||||
|
||||
```sh
|
||||
pnpm --filter @paperclipai/paperclip-runner check:clean-consumers
|
||||
```
|
||||
@@ -23,6 +23,7 @@ Each tutorial explains the capability it exercises and the contract it proves.
|
||||
## Reference
|
||||
|
||||
- [Architecture and dependency boundary](architecture.md)
|
||||
- [Paperclip Evals integration contract](evals-integration.md)
|
||||
- [PRP compatibility and versioning policy](protocol-compatibility.md)
|
||||
- [PRP v1 expressiveness audit](../spec/prp-v1-expressiveness-audit.md)
|
||||
- [ADR 0001: runner, testing, and eval package boundaries](adr/0001-runner-testing-eval-package-boundaries.md)
|
||||
|
||||
@@ -155,3 +155,22 @@ tool error is not a compatibility negotiation mechanism.
|
||||
|
||||
See [ADR 0001](adr/0001-runner-testing-eval-package-boundaries.md) for the
|
||||
component rules and clean-consumer packaging gate.
|
||||
|
||||
## Evals integration negotiation
|
||||
|
||||
The packed `./evals` entry point adds a stricter execution preflight for the
|
||||
App/Evals join. `assertPaperclipRunnerEvalCompatibility` requires simultaneous
|
||||
agreement on package semver, runnerd build metadata, a common PRP version,
|
||||
semantic catalog version and SHA-256 digest, harness-driver contract and
|
||||
required capabilities, and the native-execution version. It reports
|
||||
`paperclip_runner_eval_incompatible` with expected/received values for every
|
||||
mismatch and must run before launching a provider.
|
||||
|
||||
runnerd itself reports `paperclip-runner/runnerd-build-metadata/v1` from
|
||||
`--build-metadata`. The consumer passes its path and expected content digest to
|
||||
`resolvePaperclipRunnerdArtifact`; implicit PATH or source-tree discovery is
|
||||
not part of the contract. Native attempt output is
|
||||
`paperclip-runner/native-execution/v1`, whose parser accepts unknown additive
|
||||
fields but rejects unknown required versions and inconsistent terminal,
|
||||
semantic-denial, usage, or transcript facts. Full fields and the deterministic
|
||||
gate are documented in [the Evals integration contract](evals-integration.md).
|
||||
@@ -13,6 +13,10 @@
|
||||
"types": "./dist/testing.d.ts",
|
||||
"import": "./dist/testing.js"
|
||||
},
|
||||
"./evals": {
|
||||
"types": "./dist/evals/index.d.ts",
|
||||
"import": "./dist/evals/index.js"
|
||||
},
|
||||
"./browser": {
|
||||
"types": "./dist/browser/index.d.ts",
|
||||
"import": "./dist/browser/index.js"
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"schema": "paperclip-runner/harness-driver-conformance-fixture/v1",
|
||||
"validConfig": {
|
||||
"schema": "paperclip-runner/deterministic-harness-driver-config/v1",
|
||||
"scenario": "semantic-tool-success"
|
||||
},
|
||||
"invalidConfig": {
|
||||
"schema": "paperclip-runner/deterministic-harness-driver-config/v1",
|
||||
"scenario": "unknown"
|
||||
},
|
||||
"completionMessage": "Run deterministic semantic-tool conformance.",
|
||||
"interruptMessage": "[conformance:interrupt] Keep the turn active until interrupted.",
|
||||
"requiredCapabilities": [
|
||||
"resume",
|
||||
"typedEvents",
|
||||
"interruption",
|
||||
"structuredResult",
|
||||
"read",
|
||||
"reconciliation",
|
||||
"usage",
|
||||
"dynamicTools"
|
||||
],
|
||||
"unsupportedFeatures": [
|
||||
"steering",
|
||||
"runtimeRequestResolution",
|
||||
"goals",
|
||||
"threadLineage"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,258 @@
|
||||
{
|
||||
"schema": "paperclip-runner/native-execution/v1",
|
||||
"provenance": "seeded",
|
||||
"identity": {
|
||||
"runId": "run_evals_seeded",
|
||||
"caseId": "case_semantic_denial",
|
||||
"configId": "config_deterministic_v1",
|
||||
"attemptId": "attempt_semantic_denial_0001"
|
||||
},
|
||||
"input": {
|
||||
"caseSha256": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
|
||||
"configSha256": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
|
||||
"deterministicSeed": "paperclip-evals-v1-seed"
|
||||
},
|
||||
"runner": {
|
||||
"package": {
|
||||
"name": "@paperclipai/paperclip-runner",
|
||||
"version": "0.1.2"
|
||||
},
|
||||
"binary": {
|
||||
"name": "paperclip-runnerd",
|
||||
"sha256": "sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc"
|
||||
},
|
||||
"prpVersion": 1,
|
||||
"nativeExecutionVersion": 1,
|
||||
"catalogVersion": 1,
|
||||
"catalogSha256": "sha256:b6c2723c0b2a041f7b23c00a6043bd81d69ef3d299f67b6dbd689afb7c36dc01",
|
||||
"driverContractVersion": 1,
|
||||
"driverKind": "paperclip-deterministic",
|
||||
"driverVersion": "1.0.0"
|
||||
},
|
||||
"events": [
|
||||
{
|
||||
"schema": "paperclip.prp.event.v1",
|
||||
"sourceEventId": "evals_seeded_01",
|
||||
"sourceSeq": 1,
|
||||
"sourceInstanceId": "runner_evals_seeded",
|
||||
"sourceKind": "runner",
|
||||
"runId": "run_evals_seeded",
|
||||
"normalizedSessionId": "session_evals_seeded",
|
||||
"turnId": "turn_evals_seeded",
|
||||
"itemId": "item_evals_seeded",
|
||||
"eventType": "mcp_app.tool_input",
|
||||
"schemaVersion": 1,
|
||||
"priority": 1,
|
||||
"emittedAt": "2026-08-11T12:00:00.000Z",
|
||||
"payload": {
|
||||
"semantic_tool": {
|
||||
"schema": "paperclip.prp.semantic_tool.v1",
|
||||
"schemaVersion": 1,
|
||||
"phase": "input",
|
||||
"operationId": "decide_approval",
|
||||
"callId": "call_evals_seeded",
|
||||
"correlation": {
|
||||
"runId": "run_evals_seeded",
|
||||
"normalizedSessionId": "session_evals_seeded",
|
||||
"turnId": "turn_evals_seeded",
|
||||
"itemId": "item_evals_seeded"
|
||||
},
|
||||
"idempotencyKey": "decide_evals_seeded",
|
||||
"content": {
|
||||
"digest": "sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd",
|
||||
"redactionDisposition": "redacted",
|
||||
"references": []
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"schema": "paperclip.prp.event.v1",
|
||||
"sourceEventId": "evals_seeded_02",
|
||||
"sourceSeq": 2,
|
||||
"sourceInstanceId": "runner_evals_seeded",
|
||||
"sourceKind": "runner",
|
||||
"runId": "run_evals_seeded",
|
||||
"normalizedSessionId": "session_evals_seeded",
|
||||
"turnId": "turn_evals_seeded",
|
||||
"itemId": "item_evals_seeded",
|
||||
"eventType": "mcp_app.tool_result",
|
||||
"schemaVersion": 1,
|
||||
"priority": 1,
|
||||
"emittedAt": "2026-08-11T12:00:00.010Z",
|
||||
"payload": {
|
||||
"semantic_tool": {
|
||||
"schema": "paperclip.prp.semantic_tool.v1",
|
||||
"schemaVersion": 1,
|
||||
"phase": "result",
|
||||
"operationId": "decide_approval",
|
||||
"callId": "call_evals_seeded",
|
||||
"correlation": {
|
||||
"runId": "run_evals_seeded",
|
||||
"normalizedSessionId": "session_evals_seeded",
|
||||
"turnId": "turn_evals_seeded",
|
||||
"itemId": "item_evals_seeded"
|
||||
},
|
||||
"idempotencyKey": "decide_evals_seeded",
|
||||
"content": {
|
||||
"digest": "sha256:eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee",
|
||||
"redactionDisposition": "redacted",
|
||||
"references": []
|
||||
},
|
||||
"outcome": "denied",
|
||||
"code": "required_claim_missing",
|
||||
"retryable": false,
|
||||
"authorizationBoundary": "grant",
|
||||
"operationReceiptId": "receipt_evals_seeded",
|
||||
"auditReceiptId": "audit_evals_seeded"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"schema": "paperclip.prp.event.v1",
|
||||
"sourceEventId": "evals_seeded_03",
|
||||
"sourceSeq": 3,
|
||||
"sourceInstanceId": "runner_evals_seeded",
|
||||
"sourceKind": "runner",
|
||||
"runId": "run_evals_seeded",
|
||||
"normalizedSessionId": "session_evals_seeded",
|
||||
"turnId": "turn_evals_seeded",
|
||||
"eventType": "run.result.proposed",
|
||||
"schemaVersion": 1,
|
||||
"priority": 0,
|
||||
"emittedAt": "2026-08-11T12:00:00.020Z",
|
||||
"payload": {
|
||||
"schema": "paperclip.run_result.v1",
|
||||
"reportedWorkDisposition": "needs_review",
|
||||
"summary": "The governed tool effect was rejected and preserved as a denial.",
|
||||
"completionClaim": {
|
||||
"contractRevision": "evals-native-execution-v1",
|
||||
"objectiveSatisfied": false,
|
||||
"criteria": [],
|
||||
"remainingWork": [
|
||||
{
|
||||
"description": "An authorized approver must decide.",
|
||||
"blocksCompletion": true
|
||||
}
|
||||
]
|
||||
},
|
||||
"evidence": [{ "receiptId": "receipt_evals_seeded" }],
|
||||
"verification": [
|
||||
{
|
||||
"commandOrCheck": "rejected tool effect stays denied",
|
||||
"status": "passed"
|
||||
}
|
||||
],
|
||||
"attentionRequests": [],
|
||||
"artifacts": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"schema": "paperclip.prp.event.v1",
|
||||
"sourceEventId": "evals_seeded_04",
|
||||
"sourceSeq": 4,
|
||||
"sourceInstanceId": "runner_evals_seeded",
|
||||
"sourceKind": "runner",
|
||||
"runId": "run_evals_seeded",
|
||||
"normalizedSessionId": "session_evals_seeded",
|
||||
"turnId": "turn_evals_seeded",
|
||||
"eventType": "run.terminal",
|
||||
"schemaVersion": 1,
|
||||
"priority": 0,
|
||||
"emittedAt": "2026-08-11T12:00:00.030Z",
|
||||
"payload": {
|
||||
"schema": "paperclip.prp.terminal.v1",
|
||||
"turnTerminalState": "completed",
|
||||
"runTerminalState": "succeeded",
|
||||
"reportedWorkDisposition": "needs_review",
|
||||
"workAssessmentId": "assessment_evals_seeded",
|
||||
"statusDecisionId": "decision_evals_seeded"
|
||||
}
|
||||
}
|
||||
],
|
||||
"semanticTools": {
|
||||
"definitions": [
|
||||
{
|
||||
"operationId": "decide_approval",
|
||||
"schemaVersion": 1,
|
||||
"availability": "denied"
|
||||
}
|
||||
],
|
||||
"calls": [
|
||||
{
|
||||
"callId": "call_evals_seeded",
|
||||
"operationId": "decide_approval",
|
||||
"eventId": "evals_seeded_01"
|
||||
}
|
||||
],
|
||||
"results": [
|
||||
{
|
||||
"callId": "call_evals_seeded",
|
||||
"operationId": "decide_approval",
|
||||
"eventId": "evals_seeded_02",
|
||||
"outcome": "denied"
|
||||
}
|
||||
],
|
||||
"denials": [
|
||||
{
|
||||
"callId": "call_evals_seeded",
|
||||
"code": "required_claim_missing",
|
||||
"authorizationBoundary": "grant",
|
||||
"retryable": false
|
||||
}
|
||||
]
|
||||
},
|
||||
"terminal": {
|
||||
"schema": "paperclip.prp.terminal.v1",
|
||||
"turnTerminalState": "completed",
|
||||
"runTerminalState": "succeeded",
|
||||
"reportedWorkDisposition": "needs_review",
|
||||
"workAssessmentId": "assessment_evals_seeded",
|
||||
"statusDecisionId": "decision_evals_seeded"
|
||||
},
|
||||
"observations": {
|
||||
"checks": [
|
||||
{
|
||||
"id": "tool-effect-denied",
|
||||
"passed": true,
|
||||
"receiptId": "receipt_evals_seeded"
|
||||
}
|
||||
]
|
||||
},
|
||||
"usage": {
|
||||
"inputTokens": 12,
|
||||
"outputTokens": 8,
|
||||
"totalTokens": 20,
|
||||
"requestCount": 1,
|
||||
"durationMs": 30,
|
||||
"cost": {
|
||||
"currency": "USD",
|
||||
"amountMicros": 0
|
||||
}
|
||||
},
|
||||
"transcript": {
|
||||
"uri": "artifact://native/attempt_semantic_denial_0001/transcript.jsonl",
|
||||
"sha256": "sha256:ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff",
|
||||
"byteSize": 2048,
|
||||
"mediaType": "application/x-ndjson",
|
||||
"complete": true,
|
||||
"eventCount": 4,
|
||||
"omissionReason": null
|
||||
},
|
||||
"artifactRoot": {
|
||||
"uri": "artifact://native/attempt_semantic_denial_0001/",
|
||||
"sha256": "sha256:1212121212121212121212121212121212121212121212121212121212121212",
|
||||
"byteSize": 4096,
|
||||
"mediaType": "application/vnd.paperclip.runner-artifact-root+json"
|
||||
},
|
||||
"artifacts": [
|
||||
{
|
||||
"uri": "artifact://native/attempt_semantic_denial_0001/denial-receipt.json",
|
||||
"sha256": "sha256:1313131313131313131313131313131313131313131313131313131313131313",
|
||||
"byteSize": 512,
|
||||
"mediaType": "application/json"
|
||||
}
|
||||
],
|
||||
"failure": null,
|
||||
"x_fixturePurpose": "Proves seeded provenance, denied effects, and additive field preservation."
|
||||
}
|
||||
@@ -0,0 +1,188 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://paperclip.dev/schemas/runner/native-execution/v1.json",
|
||||
"title": "Paperclip runner native execution bundle",
|
||||
"type": "object",
|
||||
"required": [
|
||||
"schema",
|
||||
"provenance",
|
||||
"identity",
|
||||
"input",
|
||||
"runner",
|
||||
"events",
|
||||
"semanticTools",
|
||||
"terminal",
|
||||
"observations",
|
||||
"usage",
|
||||
"transcript",
|
||||
"artifactRoot",
|
||||
"artifacts",
|
||||
"failure"
|
||||
],
|
||||
"properties": {
|
||||
"schema": { "const": "paperclip-runner/native-execution/v1" },
|
||||
"provenance": { "enum": ["seeded", "replay", "live"] },
|
||||
"identity": {
|
||||
"type": "object",
|
||||
"required": ["runId", "caseId", "configId", "attemptId"],
|
||||
"properties": {
|
||||
"runId": { "$ref": "#/$defs/nonEmpty" },
|
||||
"caseId": { "$ref": "#/$defs/nonEmpty" },
|
||||
"configId": { "$ref": "#/$defs/nonEmpty" },
|
||||
"attemptId": { "$ref": "#/$defs/nonEmpty" }
|
||||
},
|
||||
"additionalProperties": true
|
||||
},
|
||||
"input": {
|
||||
"type": "object",
|
||||
"required": ["caseSha256", "configSha256", "deterministicSeed"],
|
||||
"properties": {
|
||||
"caseSha256": { "$ref": "#/$defs/sha256" },
|
||||
"configSha256": { "$ref": "#/$defs/sha256" },
|
||||
"deterministicSeed": { "$ref": "#/$defs/nonEmpty" }
|
||||
},
|
||||
"additionalProperties": true
|
||||
},
|
||||
"runner": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"package",
|
||||
"binary",
|
||||
"prpVersion",
|
||||
"nativeExecutionVersion",
|
||||
"catalogVersion",
|
||||
"catalogSha256",
|
||||
"driverContractVersion",
|
||||
"driverKind",
|
||||
"driverVersion"
|
||||
],
|
||||
"properties": {
|
||||
"package": {
|
||||
"type": "object",
|
||||
"required": ["name", "version"],
|
||||
"properties": {
|
||||
"name": { "$ref": "#/$defs/nonEmpty" },
|
||||
"version": { "$ref": "#/$defs/nonEmpty" }
|
||||
},
|
||||
"additionalProperties": true
|
||||
},
|
||||
"binary": {
|
||||
"type": "object",
|
||||
"required": ["name", "sha256"],
|
||||
"properties": {
|
||||
"name": { "$ref": "#/$defs/nonEmpty" },
|
||||
"sha256": { "$ref": "#/$defs/sha256" }
|
||||
},
|
||||
"additionalProperties": true
|
||||
},
|
||||
"prpVersion": { "$ref": "#/$defs/version" },
|
||||
"nativeExecutionVersion": { "$ref": "#/$defs/version" },
|
||||
"catalogVersion": { "$ref": "#/$defs/version" },
|
||||
"catalogSha256": { "$ref": "#/$defs/sha256" },
|
||||
"driverContractVersion": { "$ref": "#/$defs/version" },
|
||||
"driverKind": { "$ref": "#/$defs/nonEmpty" },
|
||||
"driverVersion": { "$ref": "#/$defs/nonEmpty" }
|
||||
},
|
||||
"additionalProperties": true
|
||||
},
|
||||
"events": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"items": { "$ref": "https://paperclip.dev/schemas/prp/v1/event.schema.json" }
|
||||
},
|
||||
"semanticTools": {
|
||||
"type": "object",
|
||||
"required": ["definitions", "calls", "results", "denials"],
|
||||
"properties": {
|
||||
"definitions": { "type": "array", "items": { "type": "object" } },
|
||||
"calls": { "type": "array", "items": { "type": "object" } },
|
||||
"results": { "type": "array", "items": { "type": "object" } },
|
||||
"denials": { "type": "array", "items": { "type": "object" } }
|
||||
},
|
||||
"additionalProperties": true
|
||||
},
|
||||
"terminal": { "$ref": "https://paperclip.dev/schemas/prp/v1/terminal.schema.json" },
|
||||
"observations": { "type": "object", "additionalProperties": true },
|
||||
"usage": {
|
||||
"type": "object",
|
||||
"required": [
|
||||
"inputTokens",
|
||||
"outputTokens",
|
||||
"totalTokens",
|
||||
"requestCount",
|
||||
"durationMs",
|
||||
"cost"
|
||||
],
|
||||
"properties": {
|
||||
"inputTokens": { "type": "integer", "minimum": 0 },
|
||||
"outputTokens": { "type": "integer", "minimum": 0 },
|
||||
"totalTokens": { "type": "integer", "minimum": 0 },
|
||||
"requestCount": { "type": "integer", "minimum": 0 },
|
||||
"durationMs": { "type": "integer", "minimum": 0 },
|
||||
"cost": {
|
||||
"oneOf": [
|
||||
{ "type": "null" },
|
||||
{
|
||||
"type": "object",
|
||||
"required": ["currency", "amountMicros"],
|
||||
"properties": {
|
||||
"currency": { "$ref": "#/$defs/nonEmpty" },
|
||||
"amountMicros": { "type": "integer", "minimum": 0 }
|
||||
},
|
||||
"additionalProperties": true
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"additionalProperties": true
|
||||
},
|
||||
"transcript": {
|
||||
"allOf": [
|
||||
{ "$ref": "#/$defs/contentRef" },
|
||||
{
|
||||
"type": "object",
|
||||
"required": ["complete", "eventCount", "omissionReason"],
|
||||
"properties": {
|
||||
"complete": { "type": "boolean" },
|
||||
"eventCount": { "type": "integer", "minimum": 0 },
|
||||
"omissionReason": { "type": ["string", "null"] }
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"artifactRoot": { "$ref": "#/$defs/contentRef" },
|
||||
"artifacts": { "type": "array", "items": { "$ref": "#/$defs/contentRef" } },
|
||||
"failure": {
|
||||
"oneOf": [
|
||||
{ "type": "null" },
|
||||
{
|
||||
"type": "object",
|
||||
"required": ["classification", "code", "message"],
|
||||
"properties": {
|
||||
"classification": { "$ref": "#/$defs/nonEmpty" },
|
||||
"code": { "$ref": "#/$defs/nonEmpty" },
|
||||
"message": { "$ref": "#/$defs/nonEmpty" }
|
||||
},
|
||||
"additionalProperties": true
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"additionalProperties": true,
|
||||
"$defs": {
|
||||
"nonEmpty": { "type": "string", "minLength": 1 },
|
||||
"version": { "type": "integer", "minimum": 1 },
|
||||
"sha256": { "type": "string", "pattern": "^sha256:[a-f0-9]{64}$" },
|
||||
"contentRef": {
|
||||
"type": "object",
|
||||
"required": ["uri", "sha256", "byteSize", "mediaType"],
|
||||
"properties": {
|
||||
"uri": { "$ref": "#/$defs/nonEmpty" },
|
||||
"sha256": { "$ref": "#/$defs/sha256" },
|
||||
"byteSize": { "type": "integer", "minimum": 0 },
|
||||
"mediaType": { "$ref": "#/$defs/nonEmpty" }
|
||||
},
|
||||
"additionalProperties": true
|
||||
}
|
||||
}
|
||||
}
|
||||
+1
-1
@@ -177,7 +177,7 @@ checksum = "c08d65885ee38876c4f86fa503fb49d7b507c2b62552df7c70b2fce627e06381"
|
||||
|
||||
[[package]]
|
||||
name = "paperclip-runner-core"
|
||||
version = "0.0.0"
|
||||
version = "0.1.2"
|
||||
dependencies = [
|
||||
"aes-gcm",
|
||||
"hmac",
|
||||
|
||||
@@ -3,7 +3,7 @@ members = ["crates/runner-core"]
|
||||
resolver = "2"
|
||||
|
||||
[workspace.package]
|
||||
version = "0.0.0"
|
||||
version = "0.1.2"
|
||||
edition = "2021"
|
||||
license = "MIT"
|
||||
publish = false
|
||||
|
||||
@@ -6,6 +6,26 @@ use paperclip_runner_core::durable::{
|
||||
capture_bootstrap_ticket, run_durable_runner, BootstrapTicket, DurableRunnerConfig,
|
||||
};
|
||||
use paperclip_runner_core::local_runner::{run_local_runner, LocalRunnerError, RunnerConfig};
|
||||
use serde_json::json;
|
||||
|
||||
const RUNNERD_BUILD_METADATA_SCHEMA: &str = "paperclip-runner/runnerd-build-metadata/v1";
|
||||
|
||||
fn build_metadata() -> serde_json::Value {
|
||||
json!({
|
||||
"schema": RUNNERD_BUILD_METADATA_SCHEMA,
|
||||
"binaryName": "paperclip-runnerd",
|
||||
"packageName": "@paperclipai/paperclip-runner",
|
||||
"packageVersion": env!("CARGO_PKG_VERSION"),
|
||||
"binaryContractVersion": 1,
|
||||
"nativeExecutionVersion": 1,
|
||||
"harnessDriverVersion": 1,
|
||||
"prp": {
|
||||
"name": "paperclip.runner",
|
||||
"minimumVersion": 1,
|
||||
"maximumVersion": 1
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
fn value(args: &[String], name: &str) -> Result<String, LocalRunnerError> {
|
||||
let index = args
|
||||
@@ -162,6 +182,10 @@ fn run_durable_mode(
|
||||
|
||||
fn run(bootstrap_ticket: Option<BootstrapTicket>) -> Result<(), LocalRunnerError> {
|
||||
let args = std::env::args().skip(1).collect::<Vec<_>>();
|
||||
if args.as_slice() == ["--build-metadata"] {
|
||||
println!("{}", build_metadata());
|
||||
return Ok(());
|
||||
}
|
||||
if args
|
||||
.iter()
|
||||
.any(|argument| argument == "--codex-app-server-proxy")
|
||||
@@ -187,6 +211,23 @@ fn run(bootstrap_ticket: Option<BootstrapTicket>) -> Result<(), LocalRunnerError
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn build_metadata_names_the_versioned_eval_contracts() {
|
||||
let metadata = build_metadata();
|
||||
assert_eq!(metadata["schema"], RUNNERD_BUILD_METADATA_SCHEMA);
|
||||
assert_eq!(metadata["packageVersion"], "0.1.2");
|
||||
assert_eq!(metadata["binaryContractVersion"], 1);
|
||||
assert_eq!(metadata["nativeExecutionVersion"], 1);
|
||||
assert_eq!(metadata["harnessDriverVersion"], 1);
|
||||
assert_eq!(metadata["prp"]["minimumVersion"], 1);
|
||||
assert_eq!(metadata["prp"]["maximumVersion"], 1);
|
||||
}
|
||||
}
|
||||
|
||||
fn main() -> ExitCode {
|
||||
// Capture and remove the bootstrap capability before argument parsing or child work.
|
||||
let bootstrap_ticket = match capture_bootstrap_ticket() {
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { mkdir, mkdtemp, readFile, readdir, realpath, rm, writeFile } from "node:fs/promises";
|
||||
import { createHash } from "node:crypto";
|
||||
import { chmod, copyFile, mkdir, mkdtemp, readFile, readdir, realpath, rm, writeFile } from "node:fs/promises";
|
||||
import { dirname, join, resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { tmpdir } from "node:os";
|
||||
@@ -16,15 +17,18 @@ await mkdir(artifactsRoot, { recursive: true });
|
||||
|
||||
try {
|
||||
run("pnpm", ["run", "build:typescript"], runnerRoot);
|
||||
run("pnpm", ["run", "build:runner-binaries"], runnerRoot);
|
||||
run("pnpm", ["run", "build"], evalKernelRoot);
|
||||
const runnerTarball = await pack(runnerRoot, artifactsRoot);
|
||||
const evalKernelTarball = await pack(evalKernelRoot, artifactsRoot);
|
||||
const runtimeDependencyTarballs = await packRunnerRuntimeDependencies(artifactsRoot);
|
||||
const runnerdArtifact = await stageRunnerdArtifact(artifactsRoot);
|
||||
|
||||
await verifyEvalsConsumer(
|
||||
resolve(scratchRoot, "evals-consumer"),
|
||||
runnerTarball,
|
||||
runtimeDependencyTarballs,
|
||||
runnerdArtifact,
|
||||
);
|
||||
await verifyAppDevConsumer(
|
||||
resolve(scratchRoot, "app-dev-consumer"),
|
||||
@@ -42,7 +46,7 @@ try {
|
||||
if ("@paperclipai/paperclip-eval-kernel" in runtimeDependencies) {
|
||||
throw new Error("runner tarball has a runtime dependency on Paperclip Evals");
|
||||
}
|
||||
process.stdout.write("Clean-consumer pack/install checks passed for Evals -> App and App dev -> eval kernel.\n");
|
||||
process.stdout.write("Clean-consumer pack/install checks passed for Evals -> App package/runnerd and App dev -> eval kernel.\n");
|
||||
} finally {
|
||||
if (process.env.PAPERCLIP_KEEP_PACKAGE_CONSUMERS !== "1") {
|
||||
await rm(scratchRoot, { recursive: true, force: true });
|
||||
@@ -81,13 +85,28 @@ async function packRunnerRuntimeDependencies(destination) {
|
||||
return tarballs;
|
||||
}
|
||||
|
||||
async function stageRunnerdArtifact(destination) {
|
||||
const suffix = process.platform === "win32" ? ".exe" : "";
|
||||
const source = resolve(runnerRoot, `runner/target/debug/paperclip-runnerd${suffix}`);
|
||||
const executablePath = resolve(destination, `paperclip-runnerd${suffix}`);
|
||||
await copyFile(source, executablePath);
|
||||
if (process.platform !== "win32") await chmod(executablePath, 0o755);
|
||||
const sha256 = `sha256:${createHash("sha256").update(await readFile(executablePath)).digest("hex")}`;
|
||||
return { executablePath, sha256 };
|
||||
}
|
||||
|
||||
function localOverrides(tarballs) {
|
||||
return Object.fromEntries(
|
||||
Object.entries(tarballs).map(([packageName, tarball]) => [packageName, `file:${tarball}`]),
|
||||
);
|
||||
}
|
||||
|
||||
async function verifyEvalsConsumer(consumerRoot, runnerTarball, runtimeDependencyTarballs) {
|
||||
async function verifyEvalsConsumer(
|
||||
consumerRoot,
|
||||
runnerTarball,
|
||||
runtimeDependencyTarballs,
|
||||
runnerdArtifact,
|
||||
) {
|
||||
await mkdir(consumerRoot, { recursive: true });
|
||||
await writeFile(resolve(consumerRoot, "package.json"), `${JSON.stringify({
|
||||
name: "paperclip-evals-clean-consumer",
|
||||
@@ -101,6 +120,7 @@ async function verifyEvalsConsumer(consumerRoot, runnerTarball, runtimeDependenc
|
||||
}, null, 2)}\n`);
|
||||
await writeFile(resolve(consumerRoot, "verify.mjs"), `
|
||||
import * as runtime from "@paperclipai/paperclip-runner";
|
||||
import * as evals from "@paperclipai/paperclip-runner/evals";
|
||||
import * as testing from "@paperclipai/paperclip-runner/testing";
|
||||
|
||||
if ("MockControlPlaneAdapter" in runtime || "runControlPlanePortConformance" in runtime) {
|
||||
@@ -117,6 +137,40 @@ const report = await testing.runControlPlanePortConformance({
|
||||
});
|
||||
if (report.eventCount !== 3) throw new Error("packed conformance kit returned the wrong event count");
|
||||
|
||||
const nativeBundle = await evals.loadPaperclipNativeExecutionFixture();
|
||||
if (nativeBundle.schema !== "paperclip-runner/native-execution/v1") {
|
||||
throw new Error("packed native execution fixture is unavailable");
|
||||
}
|
||||
if (nativeBundle.semanticTools.results[0]?.outcome !== "denied") {
|
||||
throw new Error("native execution fixture lost its rejected tool effect");
|
||||
}
|
||||
const driverConformance = await testing.runHarnessDriverConformance({
|
||||
driver: new testing.DeterministicHarnessDriver(),
|
||||
});
|
||||
if (!driverConformance.checks.transcriptCompleteness || driverConformance.semanticToolCallCount !== 1) {
|
||||
throw new Error("packed harness-driver conformance did not cover transcript/tools");
|
||||
}
|
||||
const runnerd = await evals.resolvePaperclipRunnerdArtifact({
|
||||
executablePath: process.env.PAPERCLIP_RUNNERD_ARTIFACT,
|
||||
expectedSha256: process.env.PAPERCLIP_RUNNERD_SHA256,
|
||||
});
|
||||
const integration = evals.assertPaperclipRunnerEvalCompatibility({
|
||||
consumer: "paperclip-evals-clean-consumer",
|
||||
packageVersion: evals.PAPERCLIP_RUNNER_BUILD_METADATA.package.version,
|
||||
runnerd: runnerd.buildMetadata,
|
||||
nativeExecutionVersion: 1,
|
||||
prp: { minimumVersion: 1, maximumVersion: 1 },
|
||||
catalog: evals.PAPERCLIP_RUNNER_BUILD_METADATA.semanticCatalog,
|
||||
driver: {
|
||||
contractVersion: driverConformance.contractVersion,
|
||||
descriptor: driverConformance.descriptor,
|
||||
requiredCapabilities: ["typedEvents", "interruption", "usage", "dynamicTools"],
|
||||
},
|
||||
});
|
||||
if (integration.negotiatedPrpVersion !== 1) {
|
||||
throw new Error("package/binary PRP negotiation returned the wrong version");
|
||||
}
|
||||
|
||||
runtime.assertPaperclipRunnerCompatibility({
|
||||
consumer: "paperclip-evals-clean-consumer",
|
||||
components: { catalog: 1, protocol: 1, runnerClient: 1, controlPlaneAdapter: 1, testkit: 1 },
|
||||
@@ -137,6 +191,27 @@ try {
|
||||
}
|
||||
if (!failedExplicitly) throw new Error("provider incompatibility did not fail explicitly");
|
||||
|
||||
let driverMismatchFailed = false;
|
||||
try {
|
||||
evals.assertPaperclipRunnerEvalCompatibility({
|
||||
consumer: "incompatible-driver",
|
||||
packageVersion: evals.PAPERCLIP_RUNNER_BUILD_METADATA.package.version,
|
||||
runnerd: runnerd.buildMetadata,
|
||||
nativeExecutionVersion: 1,
|
||||
prp: { minimumVersion: 1, maximumVersion: 1 },
|
||||
catalog: evals.PAPERCLIP_RUNNER_BUILD_METADATA.semanticCatalog,
|
||||
driver: {
|
||||
contractVersion: 2,
|
||||
descriptor: driverConformance.descriptor,
|
||||
requiredCapabilities: [],
|
||||
},
|
||||
});
|
||||
} catch (error) {
|
||||
driverMismatchFailed = error?.code === "paperclip_runner_eval_incompatible"
|
||||
&& error.issues?.some((issue) => issue.code === "driver_contract_version_mismatch");
|
||||
}
|
||||
if (!driverMismatchFailed) throw new Error("driver mismatch did not fail explicitly");
|
||||
|
||||
const normalized = {
|
||||
authorization: { outcome: "allowed" },
|
||||
state: { status: "done" },
|
||||
@@ -151,7 +226,10 @@ await testing.runSemanticConformanceKit({
|
||||
],
|
||||
});
|
||||
`);
|
||||
installAndRun(consumerRoot);
|
||||
installAndRun(consumerRoot, {
|
||||
PAPERCLIP_RUNNERD_ARTIFACT: runnerdArtifact.executablePath,
|
||||
PAPERCLIP_RUNNERD_SHA256: runnerdArtifact.sha256,
|
||||
});
|
||||
}
|
||||
|
||||
async function verifyAppDevConsumer(
|
||||
@@ -209,7 +287,7 @@ if (results.length !== 1 || results[0].score.passed !== true) {
|
||||
installAndRun(consumerRoot);
|
||||
}
|
||||
|
||||
function installAndRun(consumerRoot) {
|
||||
function installAndRun(consumerRoot, extraEnv = {}) {
|
||||
run("pnpm", [
|
||||
"install",
|
||||
"--offline",
|
||||
@@ -220,7 +298,7 @@ function installAndRun(consumerRoot) {
|
||||
"--config.auto-install-peers=false",
|
||||
"--reporter=append-only",
|
||||
], consumerRoot, { env: { NODE_ENV: "development" } });
|
||||
run(process.execPath, ["verify.mjs"], consumerRoot);
|
||||
run(process.execPath, ["verify.mjs"], consumerRoot, { env: extraEnv });
|
||||
}
|
||||
|
||||
function run(command, args, cwd, { quiet = false, env = {} } = {}) {
|
||||
|
||||
@@ -12,6 +12,9 @@ const violations = [];
|
||||
if (runnerManifest.exports?.["./testing"] === undefined) {
|
||||
violations.push("runner package must declare the ./testing export");
|
||||
}
|
||||
if (runnerManifest.exports?.["./evals"] === undefined) {
|
||||
violations.push("runner package must declare the ./evals integration export");
|
||||
}
|
||||
for (const privatePath of ["mock-core", "conformance", "scenarios", "src/tools", "./tools/"]) {
|
||||
if (runtimeIndex.includes(privatePath)) {
|
||||
violations.push(`runtime root must not export test/eval path: ${privatePath}`);
|
||||
|
||||
@@ -88,6 +88,7 @@ function violationReason({ file, packageRoot, specifier }) {
|
||||
"@paperclipai/paperclip-runner/react",
|
||||
"@paperclipai/paperclip-runner/standalone",
|
||||
"@paperclipai/paperclip-runner/testing",
|
||||
"@paperclipai/paperclip-runner/evals",
|
||||
"@paperclipai/paperclip-runner/styles.css",
|
||||
]);
|
||||
if (
|
||||
|
||||
@@ -7,7 +7,12 @@ export const PAPERCLIP_RUNNER_COMPATIBILITY = Object.freeze({
|
||||
components: Object.freeze({
|
||||
catalog: 1,
|
||||
protocol: 1,
|
||||
prp: 1,
|
||||
runnerClient: 1,
|
||||
runnerdBinary: 1,
|
||||
harnessDriver: 1,
|
||||
nativeExecution: 1,
|
||||
evalIntegration: 1,
|
||||
controlPlaneAdapter: 1,
|
||||
testkit: 1,
|
||||
}),
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { DeterministicHarnessDriver } from "../mock-core/deterministic-harness-driver.js";
|
||||
import {
|
||||
loadHarnessDriverConformanceFixture,
|
||||
runHarnessDriverConformance,
|
||||
} from "./harness-driver.js";
|
||||
|
||||
describe("harness-driver conformance V1", () => {
|
||||
it("ships a deterministic fixture covering the complete Evals hook set", async () => {
|
||||
const fixture = await loadHarnessDriverConformanceFixture();
|
||||
expect(fixture.requiredCapabilities).toContain("dynamicTools");
|
||||
expect(fixture.unsupportedFeatures).toContain("steering");
|
||||
|
||||
const report = await runHarnessDriverConformance({
|
||||
driver: new DeterministicHarnessDriver(),
|
||||
fixture,
|
||||
});
|
||||
expect(report).toMatchObject({
|
||||
schema: "paperclip-runner/harness-driver-conformance-report/v1",
|
||||
contractVersion: 1,
|
||||
eventCount: 7,
|
||||
semanticToolCallCount: 1,
|
||||
checks: {
|
||||
capabilityDescription: true,
|
||||
configValidation: true,
|
||||
sessionLifecycle: true,
|
||||
sessionRecovery: true,
|
||||
semanticTools: true,
|
||||
eventValidation: true,
|
||||
interruptAndCancel: true,
|
||||
usage: true,
|
||||
transcriptCompleteness: true,
|
||||
unsupportedFeatures: true,
|
||||
},
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,239 @@
|
||||
import { readFile } from "node:fs/promises";
|
||||
|
||||
import {
|
||||
HARNESS_DRIVER_CONTRACT_VERSION,
|
||||
type HarnessDriver,
|
||||
type HarnessDriverDescriptor,
|
||||
} from "../contracts/harness-driver.js";
|
||||
import { validatePrpEvent, type PrpEvent } from "../protocol/replay-contract.js";
|
||||
|
||||
export const HARNESS_DRIVER_CONFORMANCE_FIXTURE_SCHEMA =
|
||||
"paperclip-runner/harness-driver-conformance-fixture/v1" as const;
|
||||
export const HARNESS_DRIVER_CONFORMANCE_REPORT_SCHEMA =
|
||||
"paperclip-runner/harness-driver-conformance-report/v1" as const;
|
||||
|
||||
export const harnessDriverConformanceFixtureUrl = new URL(
|
||||
"../../protocol/fixtures/evals/harness-driver-conformance.json",
|
||||
import.meta.url,
|
||||
);
|
||||
|
||||
export interface HarnessDriverConformanceFixture {
|
||||
schema: typeof HARNESS_DRIVER_CONFORMANCE_FIXTURE_SCHEMA;
|
||||
validConfig: Record<string, unknown>;
|
||||
invalidConfig: Record<string, unknown>;
|
||||
completionMessage: string;
|
||||
interruptMessage: string;
|
||||
requiredCapabilities: string[];
|
||||
unsupportedFeatures: string[];
|
||||
}
|
||||
|
||||
export interface HarnessDriverConformanceReport {
|
||||
schema: typeof HARNESS_DRIVER_CONFORMANCE_REPORT_SCHEMA;
|
||||
contractVersion: typeof HARNESS_DRIVER_CONTRACT_VERSION;
|
||||
descriptor: HarnessDriverDescriptor;
|
||||
checks: {
|
||||
capabilityDescription: true;
|
||||
configValidation: true;
|
||||
sessionLifecycle: true;
|
||||
sessionRecovery: true;
|
||||
semanticTools: true;
|
||||
eventValidation: true;
|
||||
interruptAndCancel: true;
|
||||
usage: true;
|
||||
transcriptCompleteness: true;
|
||||
unsupportedFeatures: true;
|
||||
};
|
||||
eventCount: number;
|
||||
semanticToolCallCount: number;
|
||||
usage: Record<string, unknown>;
|
||||
}
|
||||
|
||||
function record(value: unknown, path: string): Record<string, unknown> {
|
||||
if (typeof value !== "object" || value === null || Array.isArray(value)) {
|
||||
throw new Error(`${path} must be an object`);
|
||||
}
|
||||
return value as Record<string, unknown>;
|
||||
}
|
||||
|
||||
function text(value: unknown, path: string): string {
|
||||
if (typeof value !== "string" || value.trim().length === 0) {
|
||||
throw new Error(`${path} must be a non-empty string`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function strings(value: unknown, path: string): string[] {
|
||||
if (!Array.isArray(value) || value.some((entry) => typeof entry !== "string")) {
|
||||
throw new Error(`${path} must be an array of strings`);
|
||||
}
|
||||
return [...value] as string[];
|
||||
}
|
||||
|
||||
export function parseHarnessDriverConformanceFixture(
|
||||
value: unknown,
|
||||
): HarnessDriverConformanceFixture {
|
||||
const fixture = record(value, "fixture");
|
||||
if (fixture.schema !== HARNESS_DRIVER_CONFORMANCE_FIXTURE_SCHEMA) {
|
||||
throw new Error(
|
||||
`fixture.schema must be ${HARNESS_DRIVER_CONFORMANCE_FIXTURE_SCHEMA}`,
|
||||
);
|
||||
}
|
||||
return {
|
||||
schema: HARNESS_DRIVER_CONFORMANCE_FIXTURE_SCHEMA,
|
||||
validConfig: structuredClone(record(fixture.validConfig, "fixture.validConfig")),
|
||||
invalidConfig: structuredClone(record(fixture.invalidConfig, "fixture.invalidConfig")),
|
||||
completionMessage: text(fixture.completionMessage, "fixture.completionMessage"),
|
||||
interruptMessage: text(fixture.interruptMessage, "fixture.interruptMessage"),
|
||||
requiredCapabilities: strings(fixture.requiredCapabilities, "fixture.requiredCapabilities"),
|
||||
unsupportedFeatures: strings(fixture.unsupportedFeatures, "fixture.unsupportedFeatures"),
|
||||
};
|
||||
}
|
||||
|
||||
export async function loadHarnessDriverConformanceFixture(
|
||||
url: URL = harnessDriverConformanceFixtureUrl,
|
||||
): Promise<HarnessDriverConformanceFixture> {
|
||||
return parseHarnessDriverConformanceFixture(
|
||||
JSON.parse(await readFile(url, "utf8")) as unknown,
|
||||
);
|
||||
}
|
||||
|
||||
async function collectEvents(events: AsyncIterable<PrpEvent>): Promise<PrpEvent[]> {
|
||||
const collected: PrpEvent[] = [];
|
||||
for await (const event of events) {
|
||||
const validation = validatePrpEvent(event);
|
||||
if (!validation.ok) {
|
||||
throw new Error(
|
||||
`driver conformance received invalid event: ${validation.issues[0]?.message ?? "unknown issue"}`,
|
||||
);
|
||||
}
|
||||
collected.push(structuredClone(validation.event));
|
||||
}
|
||||
return collected;
|
||||
}
|
||||
|
||||
/** Run the deterministic, provider-free harness-driver V1 contract. */
|
||||
export async function runHarnessDriverConformance(input: {
|
||||
driver: HarnessDriver;
|
||||
fixture?: HarnessDriverConformanceFixture;
|
||||
}): Promise<HarnessDriverConformanceReport> {
|
||||
const fixture = input.fixture ?? await loadHarnessDriverConformanceFixture();
|
||||
const descriptor = await input.driver.descriptor();
|
||||
if (!descriptor.kind || !descriptor.displayName || !descriptor.version) {
|
||||
throw new Error("driver descriptor is missing stable identity fields");
|
||||
}
|
||||
for (const capability of fixture.requiredCapabilities) {
|
||||
if (descriptor.capabilities[capability as keyof typeof descriptor.capabilities] !== true) {
|
||||
throw new Error(`driver capability ${capability} is required by conformance V1`);
|
||||
}
|
||||
}
|
||||
for (const unsupported of fixture.unsupportedFeatures) {
|
||||
if (!descriptor.capabilities.unsupported?.includes(unsupported)) {
|
||||
throw new Error(`driver must explicitly describe unsupported feature ${unsupported}`);
|
||||
}
|
||||
}
|
||||
|
||||
if (input.driver.validateConfig === undefined) {
|
||||
throw new Error("driver does not expose deterministic config validation");
|
||||
}
|
||||
const valid = await input.driver.validateConfig(fixture.validConfig);
|
||||
const invalid = await input.driver.validateConfig(fixture.invalidConfig);
|
||||
if (!valid.ok || invalid.ok || invalid.issues.length === 0) {
|
||||
throw new Error("driver config validation did not accept valid and reject invalid input");
|
||||
}
|
||||
|
||||
const completeSession = await input.driver.openSession({
|
||||
runId: "run_driver_conformance_complete",
|
||||
normalizedSessionId: "session_driver_conformance_complete",
|
||||
workingDirectory: "/deterministic/conformance",
|
||||
});
|
||||
await completeSession.startTurn({ message: { role: "user", text: fixture.completionMessage } });
|
||||
const completeEvents = await collectEvents(completeSession.events());
|
||||
const semanticInputs = completeEvents.filter((event) => event.eventType === "mcp_app.tool_input");
|
||||
const semanticResults = completeEvents.filter((event) => event.eventType === "mcp_app.tool_result");
|
||||
if (semanticInputs.length === 0 || semanticInputs.length !== semanticResults.length) {
|
||||
throw new Error("driver conformance requires paired semantic tool input/result events");
|
||||
}
|
||||
const snapshot = await completeSession.snapshot();
|
||||
if (snapshot.semanticResult === undefined || snapshot.semanticResult === null) {
|
||||
throw new Error("completed driver session did not persist a semantic result");
|
||||
}
|
||||
const usage = await completeSession.usage?.();
|
||||
if (usage === undefined || usage === null) {
|
||||
throw new Error("driver conformance requires a deterministic usage snapshot");
|
||||
}
|
||||
const transcript = await completeSession.transcript?.();
|
||||
if (
|
||||
transcript === undefined
|
||||
|| !transcript.complete
|
||||
|| transcript.eventCount !== completeEvents.length
|
||||
|| transcript.events.length !== completeEvents.length
|
||||
|| transcript.omissionReason !== null
|
||||
) {
|
||||
throw new Error("driver conformance transcript is incomplete or inconsistent");
|
||||
}
|
||||
if (input.driver.recoverSession === undefined) {
|
||||
throw new Error("driver declares resume but does not expose session recovery");
|
||||
}
|
||||
const recovery = await input.driver.recoverSession(snapshot);
|
||||
if (
|
||||
!recovery.recovered
|
||||
|| recovery.session === undefined
|
||||
|| recovery.session.ids().driverSessionId !== snapshot.driverSessionId
|
||||
) {
|
||||
throw new Error("driver did not recover the persisted conformance session identity");
|
||||
}
|
||||
const recoveredSnapshot = await recovery.session.snapshot();
|
||||
if (
|
||||
recoveredSnapshot.runId !== snapshot.runId
|
||||
|| recoveredSnapshot.normalizedSessionId !== snapshot.normalizedSessionId
|
||||
|| recoveredSnapshot.semanticResult?.fingerprint !== snapshot.semanticResult?.fingerprint
|
||||
) {
|
||||
throw new Error("driver recovery did not preserve the persisted conformance state");
|
||||
}
|
||||
await recovery.session.close({ reason: "conformance_recovery_complete" });
|
||||
await completeSession.close({ reason: "conformance_complete" });
|
||||
|
||||
const interruptedSession = await input.driver.openSession({
|
||||
runId: "run_driver_conformance_interrupt",
|
||||
normalizedSessionId: "session_driver_conformance_interrupt",
|
||||
workingDirectory: "/deterministic/conformance",
|
||||
});
|
||||
const { turnId } = await interruptedSession.startTurn({
|
||||
message: { role: "user", text: fixture.interruptMessage },
|
||||
});
|
||||
if (interruptedSession.interrupt === undefined) {
|
||||
throw new Error("driver conformance requires interrupt support");
|
||||
}
|
||||
await interruptedSession.interrupt({ turnId, reason: "conformance_interrupt" });
|
||||
const interruptedEvents = await collectEvents(interruptedSession.events());
|
||||
if (
|
||||
!interruptedEvents.some((event) => event.eventType === "turn.interrupted")
|
||||
|| !interruptedEvents.some((event) =>
|
||||
event.eventType === "run.terminal"
|
||||
&& record(event.payload, "terminal payload").runTerminalState === "cancelled")
|
||||
) {
|
||||
throw new Error("driver interrupt did not produce an explicit cancelled terminal");
|
||||
}
|
||||
await interruptedSession.close({ reason: "conformance_cancel", force: true });
|
||||
|
||||
return {
|
||||
schema: HARNESS_DRIVER_CONFORMANCE_REPORT_SCHEMA,
|
||||
contractVersion: HARNESS_DRIVER_CONTRACT_VERSION,
|
||||
descriptor: structuredClone(descriptor),
|
||||
checks: {
|
||||
capabilityDescription: true,
|
||||
configValidation: true,
|
||||
sessionLifecycle: true,
|
||||
sessionRecovery: true,
|
||||
semanticTools: true,
|
||||
eventValidation: true,
|
||||
interruptAndCancel: true,
|
||||
usage: true,
|
||||
transcriptCompleteness: true,
|
||||
unsupportedFeatures: true,
|
||||
},
|
||||
eventCount: completeEvents.length,
|
||||
semanticToolCallCount: semanticInputs.length,
|
||||
usage: structuredClone(usage),
|
||||
};
|
||||
}
|
||||
@@ -4,6 +4,26 @@ import type {
|
||||
} from "../protocol/replay-contract.js";
|
||||
import type { NativeSessionCapabilities, NativeUserMessage } from "./types.js";
|
||||
|
||||
export const HARNESS_DRIVER_CONTRACT_VERSION = 1 as const;
|
||||
|
||||
export interface HarnessDriverConfigIssue {
|
||||
path: string;
|
||||
code: string;
|
||||
message: string;
|
||||
}
|
||||
|
||||
export type HarnessDriverConfigValidation =
|
||||
| { ok: true; config: Record<string, unknown>; issues: [] }
|
||||
| { ok: false; config: null; issues: HarnessDriverConfigIssue[] };
|
||||
|
||||
export interface HarnessTranscriptSnapshot {
|
||||
schema: "paperclip-runner/harness-transcript/v1";
|
||||
complete: boolean;
|
||||
eventCount: number;
|
||||
events: PrpEvent[];
|
||||
omissionReason: string | null;
|
||||
}
|
||||
|
||||
export interface HarnessDriverDescriptor {
|
||||
kind: string;
|
||||
displayName: string;
|
||||
@@ -322,6 +342,7 @@ export interface HarnessSession {
|
||||
read?(): Promise<Record<string, unknown>>;
|
||||
reconcile?(): Promise<Record<string, unknown>>;
|
||||
usage?(): Promise<Record<string, unknown> | null>;
|
||||
transcript?(): Promise<HarnessTranscriptSnapshot>;
|
||||
snapshot(): Promise<PersistedHarnessSession>;
|
||||
close(input: { reason: string; force?: boolean }): Promise<void>;
|
||||
}
|
||||
@@ -329,6 +350,7 @@ export interface HarnessSession {
|
||||
/** A local harness implementation hidden behind the runner daemon boundary. */
|
||||
export interface HarnessDriver {
|
||||
descriptor(): Promise<HarnessDriverDescriptor>;
|
||||
validateConfig?(config: unknown): Promise<HarnessDriverConfigValidation>;
|
||||
openSession(input: OpenHarnessSessionInput): Promise<HarnessSession>;
|
||||
recoverSession?(
|
||||
snapshot: PersistedHarnessSession,
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
import { createHash } from "node:crypto";
|
||||
|
||||
import { PAPERCLIP_RUNNER_COMPATIBILITY } from "../compatibility.js";
|
||||
import { PRP_PROTOCOL_NAME, PRP_PROTOCOL_VERSION } from "../protocol/replay-contract.js";
|
||||
import { canonicalCapabilitySemanticCatalog } from "../semantic-tools/catalog.js";
|
||||
|
||||
export const PAPERCLIP_RUNNER_BUILD_METADATA_SCHEMA =
|
||||
"paperclip-runner/build-metadata/v1" as const;
|
||||
export const PAPERCLIP_RUNNER_NATIVE_EXECUTION_SCHEMA =
|
||||
"paperclip-runner/native-execution/v1" as const;
|
||||
export const PAPERCLIP_RUNNER_EVAL_INTEGRATION_SCHEMA =
|
||||
"paperclip-runner/evals-integration/v1" as const;
|
||||
export const PAPERCLIP_RUNNERD_BUILD_METADATA_SCHEMA =
|
||||
"paperclip-runner/runnerd-build-metadata/v1" as const;
|
||||
|
||||
export const PAPERCLIP_RUNNER_SEMANTIC_CATALOG_SHA256 =
|
||||
`sha256:${createHash("sha256")
|
||||
.update(canonicalCapabilitySemanticCatalog())
|
||||
.digest("hex")}` as const;
|
||||
|
||||
/**
|
||||
* App-owned release metadata that Evals pins beside every native attempt.
|
||||
* Contract versions are independent from package semver so consumers can give
|
||||
* a precise mismatch instead of guessing from a package version.
|
||||
*/
|
||||
export const PAPERCLIP_RUNNER_BUILD_METADATA = Object.freeze({
|
||||
schema: PAPERCLIP_RUNNER_BUILD_METADATA_SCHEMA,
|
||||
package: Object.freeze({
|
||||
name: PAPERCLIP_RUNNER_COMPATIBILITY.packageName,
|
||||
version: PAPERCLIP_RUNNER_COMPATIBILITY.packageVersion,
|
||||
}),
|
||||
contracts: Object.freeze({
|
||||
evalIntegration: PAPERCLIP_RUNNER_COMPATIBILITY.components.evalIntegration,
|
||||
nativeExecution: PAPERCLIP_RUNNER_COMPATIBILITY.components.nativeExecution,
|
||||
runnerdArtifact: PAPERCLIP_RUNNER_COMPATIBILITY.components.runnerdBinary,
|
||||
prp: PRP_PROTOCOL_VERSION,
|
||||
semanticCatalog: PAPERCLIP_RUNNER_COMPATIBILITY.components.catalog,
|
||||
harnessDriver: PAPERCLIP_RUNNER_COMPATIBILITY.components.harnessDriver,
|
||||
controlPlaneAdapter: PAPERCLIP_RUNNER_COMPATIBILITY.components.controlPlaneAdapter,
|
||||
testkit: PAPERCLIP_RUNNER_COMPATIBILITY.components.testkit,
|
||||
}),
|
||||
prp: Object.freeze({
|
||||
name: PRP_PROTOCOL_NAME,
|
||||
minimumVersion: PRP_PROTOCOL_VERSION,
|
||||
maximumVersion: PRP_PROTOCOL_VERSION,
|
||||
}),
|
||||
semanticCatalog: Object.freeze({
|
||||
version: PAPERCLIP_RUNNER_COMPATIBILITY.components.catalog,
|
||||
sha256: PAPERCLIP_RUNNER_SEMANTIC_CATALOG_SHA256,
|
||||
}),
|
||||
runnerd: Object.freeze({
|
||||
binaryName: "paperclip-runnerd" as const,
|
||||
metadataSchema: PAPERCLIP_RUNNERD_BUILD_METADATA_SCHEMA,
|
||||
digestAlgorithm: "sha256" as const,
|
||||
}),
|
||||
});
|
||||
|
||||
export type PaperclipRunnerBuildMetadata = typeof PAPERCLIP_RUNNER_BUILD_METADATA;
|
||||
@@ -0,0 +1,120 @@
|
||||
import { readFileSync } from "node:fs";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import {
|
||||
PAPERCLIP_RUNNER_BUILD_METADATA,
|
||||
PAPERCLIP_RUNNERD_BUILD_METADATA_SCHEMA,
|
||||
} from "./build-metadata.js";
|
||||
import {
|
||||
PaperclipRunnerEvalCompatibilityError,
|
||||
assertPaperclipRunnerEvalCompatibility,
|
||||
type PaperclipRunnerEvalCompatibilityRequirement,
|
||||
} from "./compatibility.js";
|
||||
|
||||
function compatible(): PaperclipRunnerEvalCompatibilityRequirement {
|
||||
return {
|
||||
consumer: "paperclip-evals",
|
||||
packageVersion: PAPERCLIP_RUNNER_BUILD_METADATA.package.version,
|
||||
runnerd: {
|
||||
schema: PAPERCLIP_RUNNERD_BUILD_METADATA_SCHEMA,
|
||||
binaryName: "paperclip-runnerd",
|
||||
packageName: "@paperclipai/paperclip-runner",
|
||||
packageVersion: PAPERCLIP_RUNNER_BUILD_METADATA.package.version,
|
||||
binaryContractVersion: 1,
|
||||
nativeExecutionVersion: 1,
|
||||
harnessDriverVersion: 1,
|
||||
prp: { name: "paperclip.runner", minimumVersion: 1, maximumVersion: 1 },
|
||||
},
|
||||
nativeExecutionVersion: 1,
|
||||
prp: { minimumVersion: 1, maximumVersion: 1 },
|
||||
catalog: {
|
||||
version: PAPERCLIP_RUNNER_BUILD_METADATA.semanticCatalog.version,
|
||||
sha256: PAPERCLIP_RUNNER_BUILD_METADATA.semanticCatalog.sha256,
|
||||
},
|
||||
driver: {
|
||||
contractVersion: 1,
|
||||
requiredCapabilities: ["typedEvents", "interruption", "usage", "dynamicTools"],
|
||||
descriptor: {
|
||||
kind: "paperclip-deterministic",
|
||||
displayName: "Deterministic",
|
||||
version: "1.0.0",
|
||||
protocolVersion: "prp.v1",
|
||||
capabilities: {
|
||||
resume: true,
|
||||
typedEvents: true,
|
||||
steering: false,
|
||||
interruption: true,
|
||||
structuredResult: true,
|
||||
usage: true,
|
||||
dynamicTools: true,
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
describe("Paperclip Evals integration compatibility", () => {
|
||||
it("keeps build metadata synchronized with package semver", () => {
|
||||
const packageJson = JSON.parse(readFileSync(
|
||||
fileURLToPath(new URL("../../package.json", import.meta.url)),
|
||||
"utf8",
|
||||
));
|
||||
expect(PAPERCLIP_RUNNER_BUILD_METADATA.package.version).toBe(packageJson.version);
|
||||
});
|
||||
|
||||
it("negotiates package, binary, PRP, catalog, and driver V1", () => {
|
||||
expect(assertPaperclipRunnerEvalCompatibility(compatible())).toMatchObject({
|
||||
schema: "paperclip-runner/evals-integration/v1",
|
||||
consumer: "paperclip-evals",
|
||||
negotiatedPrpVersion: 1,
|
||||
driverKind: "paperclip-deterministic",
|
||||
});
|
||||
});
|
||||
|
||||
it("fails closed with actionable codes for every independently versioned input", () => {
|
||||
const requirement = compatible();
|
||||
requirement.packageVersion = "9.0.0";
|
||||
requirement.runnerd.packageVersion = "8.0.0";
|
||||
requirement.runnerd.binaryContractVersion = 2;
|
||||
requirement.runnerd.nativeExecutionVersion = 2;
|
||||
requirement.runnerd.harnessDriverVersion = 2;
|
||||
requirement.nativeExecutionVersion = 2;
|
||||
requirement.prp = { minimumVersion: 2, maximumVersion: 2 };
|
||||
requirement.runnerd.prp = { name: "paperclip.runner", minimumVersion: 2, maximumVersion: 2 };
|
||||
requirement.catalog = { version: 2, sha256: `sha256:${"0".repeat(64)}` };
|
||||
requirement.driver.contractVersion = 2;
|
||||
requirement.driver.descriptor.capabilities.dynamicTools = false;
|
||||
|
||||
try {
|
||||
assertPaperclipRunnerEvalCompatibility(requirement);
|
||||
throw new Error("expected compatibility failure");
|
||||
} catch (error) {
|
||||
expect(error).toBeInstanceOf(PaperclipRunnerEvalCompatibilityError);
|
||||
expect((error as PaperclipRunnerEvalCompatibilityError).issues.map((item) => item.code))
|
||||
.toEqual([
|
||||
"package_version_mismatch",
|
||||
"binary_package_version_mismatch",
|
||||
"binary_contract_version_mismatch",
|
||||
"native_execution_version_mismatch",
|
||||
"prp_version_no_overlap",
|
||||
"catalog_version_mismatch",
|
||||
"catalog_digest_mismatch",
|
||||
"driver_contract_version_mismatch",
|
||||
"driver_capability_unsupported",
|
||||
]);
|
||||
expect((error as Error).message).toContain("runnerd was built for package 8.0.0");
|
||||
}
|
||||
});
|
||||
|
||||
it("rejects a driver that does not speak the negotiated PRP version", () => {
|
||||
const requirement = compatible();
|
||||
requirement.driver.descriptor.protocolVersion = "prp.v2";
|
||||
|
||||
expect(() => assertPaperclipRunnerEvalCompatibility(requirement)).toThrow(
|
||||
expect.objectContaining({
|
||||
issues: [expect.objectContaining({ code: "driver_protocol_version_mismatch" })],
|
||||
}),
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,244 @@
|
||||
import type { HarnessDriverDescriptor } from "../contracts/harness-driver.js";
|
||||
import type { NativeSessionCapabilities } from "../contracts/types.js";
|
||||
import { negotiateProtocolVersion } from "../protocol/replay-contract.js";
|
||||
import {
|
||||
PAPERCLIP_RUNNER_BUILD_METADATA,
|
||||
PAPERCLIP_RUNNER_EVAL_INTEGRATION_SCHEMA,
|
||||
} from "./build-metadata.js";
|
||||
import type { PaperclipRunnerdBuildMetadata } from "./runnerd-artifact.js";
|
||||
|
||||
export type RequiredHarnessDriverCapability = Exclude<
|
||||
keyof NativeSessionCapabilities,
|
||||
"unsupported"
|
||||
>;
|
||||
|
||||
export type PaperclipRunnerEvalCompatibilityIssueCode =
|
||||
| "package_version_mismatch"
|
||||
| "binary_package_version_mismatch"
|
||||
| "binary_contract_version_mismatch"
|
||||
| "native_execution_version_mismatch"
|
||||
| "prp_name_mismatch"
|
||||
| "prp_version_no_overlap"
|
||||
| "catalog_version_mismatch"
|
||||
| "catalog_digest_mismatch"
|
||||
| "driver_contract_version_mismatch"
|
||||
| "driver_protocol_version_mismatch"
|
||||
| "driver_capability_unsupported";
|
||||
|
||||
export interface PaperclipRunnerEvalCompatibilityIssue {
|
||||
code: PaperclipRunnerEvalCompatibilityIssueCode;
|
||||
component: "package" | "binary" | "nativeExecution" | "prp" | "catalog" | "driver";
|
||||
expected: string;
|
||||
received: string;
|
||||
message: string;
|
||||
}
|
||||
|
||||
export interface PaperclipRunnerEvalCompatibilityRequirement {
|
||||
consumer: string;
|
||||
packageVersion: string;
|
||||
runnerd: PaperclipRunnerdBuildMetadata;
|
||||
nativeExecutionVersion: number;
|
||||
prp: { minimumVersion: number; maximumVersion: number };
|
||||
catalog: { version: number; sha256: string };
|
||||
driver: {
|
||||
contractVersion: number;
|
||||
descriptor: HarnessDriverDescriptor;
|
||||
requiredCapabilities: readonly RequiredHarnessDriverCapability[];
|
||||
};
|
||||
}
|
||||
|
||||
export interface PaperclipRunnerEvalCompatibilityReceipt {
|
||||
schema: typeof PAPERCLIP_RUNNER_EVAL_INTEGRATION_SCHEMA;
|
||||
consumer: string;
|
||||
packageVersion: string;
|
||||
runnerdPackageVersion: string;
|
||||
negotiatedPrpVersion: number;
|
||||
catalogSha256: string;
|
||||
driverKind: string;
|
||||
driverVersion: string;
|
||||
}
|
||||
|
||||
export class PaperclipRunnerEvalCompatibilityError extends Error {
|
||||
readonly code = "paperclip_runner_eval_incompatible" as const;
|
||||
|
||||
constructor(
|
||||
readonly consumer: string,
|
||||
readonly issues: readonly PaperclipRunnerEvalCompatibilityIssue[],
|
||||
) {
|
||||
super(
|
||||
`Paperclip runner eval compatibility check failed for ${consumer}: ${issues
|
||||
.map((issue) => `${issue.code}: ${issue.message}`)
|
||||
.join("; ")}`,
|
||||
);
|
||||
this.name = "PaperclipRunnerEvalCompatibilityError";
|
||||
this.issues = Object.freeze(issues.map((issue) => Object.freeze({ ...issue })));
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Negotiate the complete App/Evals join before starting a provider process.
|
||||
* Every independently versioned input is checked and all mismatches are
|
||||
* returned together so remediation is actionable.
|
||||
*/
|
||||
export function assertPaperclipRunnerEvalCompatibility(
|
||||
requirement: PaperclipRunnerEvalCompatibilityRequirement,
|
||||
): PaperclipRunnerEvalCompatibilityReceipt {
|
||||
const expected = PAPERCLIP_RUNNER_BUILD_METADATA;
|
||||
const issues: PaperclipRunnerEvalCompatibilityIssue[] = [];
|
||||
const issue = (
|
||||
code: PaperclipRunnerEvalCompatibilityIssueCode,
|
||||
component: PaperclipRunnerEvalCompatibilityIssue["component"],
|
||||
expectedValue: string | number,
|
||||
receivedValue: string | number,
|
||||
message: string,
|
||||
): void => {
|
||||
issues.push({
|
||||
code,
|
||||
component,
|
||||
expected: String(expectedValue),
|
||||
received: String(receivedValue),
|
||||
message,
|
||||
});
|
||||
};
|
||||
|
||||
if (requirement.packageVersion !== expected.package.version) {
|
||||
issue(
|
||||
"package_version_mismatch",
|
||||
"package",
|
||||
expected.package.version,
|
||||
requirement.packageVersion,
|
||||
`package ${requirement.packageVersion} does not match the loaded contract package ${expected.package.version}`,
|
||||
);
|
||||
}
|
||||
if (requirement.runnerd.packageVersion !== expected.package.version) {
|
||||
issue(
|
||||
"binary_package_version_mismatch",
|
||||
"binary",
|
||||
expected.package.version,
|
||||
requirement.runnerd.packageVersion,
|
||||
`runnerd was built for package ${requirement.runnerd.packageVersion}, not ${expected.package.version}`,
|
||||
);
|
||||
}
|
||||
if (requirement.runnerd.binaryContractVersion !== expected.contracts.runnerdArtifact) {
|
||||
issue(
|
||||
"binary_contract_version_mismatch",
|
||||
"binary",
|
||||
expected.contracts.runnerdArtifact,
|
||||
requirement.runnerd.binaryContractVersion,
|
||||
"runnerd artifact contract is incompatible with this package",
|
||||
);
|
||||
}
|
||||
if (
|
||||
requirement.nativeExecutionVersion !== expected.contracts.nativeExecution
|
||||
|| requirement.runnerd.nativeExecutionVersion !== expected.contracts.nativeExecution
|
||||
) {
|
||||
issue(
|
||||
"native_execution_version_mismatch",
|
||||
"nativeExecution",
|
||||
expected.contracts.nativeExecution,
|
||||
`${requirement.nativeExecutionVersion}/${requirement.runnerd.nativeExecutionVersion}`,
|
||||
"consumer and runnerd must both emit paperclip-runner/native-execution/v1",
|
||||
);
|
||||
}
|
||||
|
||||
if (requirement.runnerd.prp.name !== expected.prp.name) {
|
||||
issue(
|
||||
"prp_name_mismatch",
|
||||
"prp",
|
||||
expected.prp.name,
|
||||
requirement.runnerd.prp.name,
|
||||
"runnerd speaks a different protocol family",
|
||||
);
|
||||
}
|
||||
const packageAndConsumerPrp = negotiateProtocolVersion(
|
||||
{ min: expected.prp.minimumVersion, max: expected.prp.maximumVersion },
|
||||
{ min: requirement.prp.minimumVersion, max: requirement.prp.maximumVersion },
|
||||
);
|
||||
const negotiatedPrpVersion = packageAndConsumerPrp === null
|
||||
? null
|
||||
: negotiateProtocolVersion(
|
||||
{ min: packageAndConsumerPrp, max: packageAndConsumerPrp },
|
||||
{
|
||||
min: requirement.runnerd.prp.minimumVersion,
|
||||
max: requirement.runnerd.prp.maximumVersion,
|
||||
},
|
||||
);
|
||||
if (negotiatedPrpVersion === null) {
|
||||
issue(
|
||||
"prp_version_no_overlap",
|
||||
"prp",
|
||||
`${expected.prp.minimumVersion}..${expected.prp.maximumVersion}`,
|
||||
`consumer ${requirement.prp.minimumVersion}..${requirement.prp.maximumVersion}; runnerd ${requirement.runnerd.prp.minimumVersion}..${requirement.runnerd.prp.maximumVersion}`,
|
||||
"package, runnerd, and consumer have no common PRP version",
|
||||
);
|
||||
}
|
||||
|
||||
if (requirement.catalog.version !== expected.semanticCatalog.version) {
|
||||
issue(
|
||||
"catalog_version_mismatch",
|
||||
"catalog",
|
||||
expected.semanticCatalog.version,
|
||||
requirement.catalog.version,
|
||||
"semantic catalog contract version is incompatible",
|
||||
);
|
||||
}
|
||||
if (requirement.catalog.sha256 !== expected.semanticCatalog.sha256) {
|
||||
issue(
|
||||
"catalog_digest_mismatch",
|
||||
"catalog",
|
||||
expected.semanticCatalog.sha256,
|
||||
requirement.catalog.sha256,
|
||||
"semantic catalog content digest does not match the loaded package",
|
||||
);
|
||||
}
|
||||
if (
|
||||
requirement.driver.contractVersion !== expected.contracts.harnessDriver
|
||||
|| requirement.runnerd.harnessDriverVersion !== expected.contracts.harnessDriver
|
||||
) {
|
||||
issue(
|
||||
"driver_contract_version_mismatch",
|
||||
"driver",
|
||||
expected.contracts.harnessDriver,
|
||||
`${requirement.driver.contractVersion}/${requirement.runnerd.harnessDriverVersion}`,
|
||||
"consumer and runnerd harness-driver contract versions must match the package",
|
||||
);
|
||||
}
|
||||
if (
|
||||
negotiatedPrpVersion !== null
|
||||
&& requirement.driver.descriptor.protocolVersion !== `prp.v${negotiatedPrpVersion}`
|
||||
) {
|
||||
issue(
|
||||
"driver_protocol_version_mismatch",
|
||||
"driver",
|
||||
`prp.v${negotiatedPrpVersion}`,
|
||||
String(requirement.driver.descriptor.protocolVersion),
|
||||
`driver ${requirement.driver.descriptor.kind}@${requirement.driver.descriptor.version} does not speak the negotiated PRP version`,
|
||||
);
|
||||
}
|
||||
for (const capability of [...new Set(requirement.driver.requiredCapabilities)].sort()) {
|
||||
if (requirement.driver.descriptor.capabilities[capability] !== true) {
|
||||
issue(
|
||||
"driver_capability_unsupported",
|
||||
"driver",
|
||||
`${capability}=true`,
|
||||
`${capability}=${String(requirement.driver.descriptor.capabilities[capability])}`,
|
||||
`driver ${requirement.driver.descriptor.kind}@${requirement.driver.descriptor.version} does not support required capability ${capability}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
if (issues.length > 0) {
|
||||
throw new PaperclipRunnerEvalCompatibilityError(requirement.consumer, issues);
|
||||
}
|
||||
|
||||
return {
|
||||
schema: PAPERCLIP_RUNNER_EVAL_INTEGRATION_SCHEMA,
|
||||
consumer: requirement.consumer,
|
||||
packageVersion: expected.package.version,
|
||||
runnerdPackageVersion: requirement.runnerd.packageVersion,
|
||||
negotiatedPrpVersion: negotiatedPrpVersion!,
|
||||
catalogSha256: expected.semanticCatalog.sha256,
|
||||
driverKind: requirement.driver.descriptor.kind,
|
||||
driverVersion: requirement.driver.descriptor.version,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
/** Stable App-owned integration surface for Paperclip Evals. */
|
||||
export * from "./build-metadata.js";
|
||||
export * from "./compatibility.js";
|
||||
export * from "./native-execution.js";
|
||||
export * from "./runnerd-artifact.js";
|
||||
|
||||
export {
|
||||
PRP_PROTOCOL_NAME,
|
||||
PRP_PROTOCOL_VERSION,
|
||||
negotiateProtocolVersion,
|
||||
validatePrpEvent,
|
||||
validatePrpFixture,
|
||||
validatePrpStructuredRunResult,
|
||||
type PrpCapabilities,
|
||||
type PrpEvent,
|
||||
type PrpSemanticToolEnvelope,
|
||||
type PrpStructuredRunResult,
|
||||
type PrpTerminalState,
|
||||
} from "../protocol/replay-contract.js";
|
||||
export * from "../protocol/semantic-tool-receipts.js";
|
||||
export {
|
||||
CAPABILITY_SEMANTIC_TOOL_CATALOG,
|
||||
canonicalCapabilitySemanticCatalog,
|
||||
capabilitySemanticToolDescriptor,
|
||||
} from "../semantic-tools/catalog.js";
|
||||
export type {
|
||||
HarnessDriver,
|
||||
HarnessDriverDescriptor,
|
||||
HarnessSession,
|
||||
OpenHarnessSessionInput,
|
||||
PersistedHarnessSession,
|
||||
} from "../contracts/harness-driver.js";
|
||||
@@ -0,0 +1,132 @@
|
||||
import { readFile } from "node:fs/promises";
|
||||
import Ajv2020 from "ajv/dist/2020.js";
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { prpSchemaBundle } from "../protocol/generated/schema-bundle.js";
|
||||
import {
|
||||
loadPaperclipNativeExecutionFixture,
|
||||
paperclipNativeExecutionFixtureUrl,
|
||||
paperclipNativeExecutionSchemaUrl,
|
||||
parsePaperclipNativeExecution,
|
||||
} from "./native-execution.js";
|
||||
import { PAPERCLIP_RUNNER_BUILD_METADATA } from "./build-metadata.js";
|
||||
|
||||
describe("paperclip-runner/native-execution/v1", () => {
|
||||
it("keeps the shipped seeded fixture valid against the published JSON Schema", async () => {
|
||||
const ajv = new Ajv2020({ allErrors: true, strict: false });
|
||||
for (const schema of Object.values(prpSchemaBundle)) ajv.addSchema(schema);
|
||||
const schema = JSON.parse(
|
||||
await readFile(paperclipNativeExecutionSchemaUrl, "utf8"),
|
||||
) as Record<string, unknown>;
|
||||
const validate = ajv.compile(schema);
|
||||
const fixture = JSON.parse(
|
||||
await readFile(paperclipNativeExecutionFixtureUrl, "utf8"),
|
||||
) as unknown;
|
||||
|
||||
expect(validate(fixture), JSON.stringify(validate.errors)).toBe(true);
|
||||
});
|
||||
|
||||
it("loads the shipped seeded attempt and preserves nested additive fields", async () => {
|
||||
const bundle = await loadPaperclipNativeExecutionFixture();
|
||||
expect(bundle.provenance).toBe("seeded");
|
||||
expect(bundle.semanticTools.results).toEqual([
|
||||
expect.objectContaining({ outcome: "denied", callId: "call_evals_seeded" }),
|
||||
]);
|
||||
expect(bundle.semanticTools.denials).toHaveLength(1);
|
||||
expect(bundle.transcript).toMatchObject({ complete: true, eventCount: 4 });
|
||||
expect(bundle.usage.cost).toEqual({ currency: "USD", amountMicros: 0 });
|
||||
expect(bundle.runner.package).toEqual(PAPERCLIP_RUNNER_BUILD_METADATA.package);
|
||||
expect(bundle.runner.catalogSha256).toBe(
|
||||
PAPERCLIP_RUNNER_BUILD_METADATA.semanticCatalog.sha256,
|
||||
);
|
||||
expect(bundle.x_fixturePurpose).toContain("additive field preservation");
|
||||
|
||||
const extended = structuredClone(bundle);
|
||||
extended.identity.x_identity = "identity-extension";
|
||||
extended.input.x_input = "input-extension";
|
||||
extended.runner.x_runner = "runner-extension";
|
||||
extended.runner.package.x_package = "package-extension";
|
||||
extended.runner.binary.x_binary = "binary-extension";
|
||||
extended.semanticTools.x_semanticTools = "semantic-extension";
|
||||
extended.usage.x_usage = "usage-extension";
|
||||
extended.usage.cost!.x_cost = "cost-extension";
|
||||
|
||||
expect(parsePaperclipNativeExecution(extended)).toMatchObject({
|
||||
identity: { x_identity: "identity-extension" },
|
||||
input: { x_input: "input-extension" },
|
||||
runner: {
|
||||
x_runner: "runner-extension",
|
||||
package: { x_package: "package-extension" },
|
||||
binary: { x_binary: "binary-extension" },
|
||||
},
|
||||
semanticTools: { x_semanticTools: "semantic-extension" },
|
||||
usage: { x_usage: "usage-extension", cost: { x_cost: "cost-extension" } },
|
||||
});
|
||||
});
|
||||
|
||||
it("rejects unknown native schema versions", async () => {
|
||||
const bundle = await loadPaperclipNativeExecutionFixture();
|
||||
expect(() => parsePaperclipNativeExecution({ ...bundle, schema: "paperclip-runner/native-execution/v2" }))
|
||||
.toThrow(/unsupported schema/);
|
||||
});
|
||||
|
||||
it("rejects a terminal that conflicts with the event stream", async () => {
|
||||
const bundle = await loadPaperclipNativeExecutionFixture();
|
||||
expect(() => parsePaperclipNativeExecution({
|
||||
...bundle,
|
||||
terminal: { ...bundle.terminal, runTerminalState: "failed" },
|
||||
})).toThrow(/must equal the run\.terminal event payload/);
|
||||
});
|
||||
|
||||
it("rejects missing denial receipts and incomplete transcript ambiguity", async () => {
|
||||
const bundle = await loadPaperclipNativeExecutionFixture();
|
||||
expect(() => parsePaperclipNativeExecution({
|
||||
...bundle,
|
||||
semanticTools: { ...bundle.semanticTools, denials: [] },
|
||||
})).toThrow(/needs a denial receipt/);
|
||||
expect(() => parsePaperclipNativeExecution({
|
||||
...bundle,
|
||||
transcript: { ...bundle.transcript, complete: false, omissionReason: null },
|
||||
})).toThrow(/omissionReason.*required/);
|
||||
});
|
||||
|
||||
it("rejects semantic indexes that reinterpret or omit PRP tool events", async () => {
|
||||
const bundle = await loadPaperclipNativeExecutionFixture();
|
||||
const mismatched = structuredClone(bundle);
|
||||
const resultPayload = mismatched.events[1]!.payload as {
|
||||
semantic_tool: { outcome: string };
|
||||
};
|
||||
resultPayload.semantic_tool.outcome = "succeeded";
|
||||
expect(() => parsePaperclipNativeExecution(mismatched))
|
||||
.toThrow(/does not match semantic envelope/);
|
||||
|
||||
const mismatchedCorrelation = structuredClone(bundle);
|
||||
const inputPayload = mismatchedCorrelation.events[0]!.payload as {
|
||||
semantic_tool: { correlation: { runId: string } };
|
||||
};
|
||||
inputPayload.semantic_tool.correlation.runId = "other_run";
|
||||
expect(() => parsePaperclipNativeExecution(mismatchedCorrelation))
|
||||
.toThrow(/correlation\.runId.*enclosing event/);
|
||||
|
||||
expect(() => parsePaperclipNativeExecution({
|
||||
...bundle,
|
||||
semanticTools: { ...bundle.semanticTools, calls: [] },
|
||||
})).toThrow(/no matching call|tool-input event .* is not indexed/);
|
||||
});
|
||||
|
||||
it("rejects unordered native events", async () => {
|
||||
const bundle = await loadPaperclipNativeExecutionFixture();
|
||||
const events = structuredClone(bundle.events);
|
||||
events[1] = { ...events[1]!, sourceSeq: 1 };
|
||||
expect(() => parsePaperclipNativeExecution({ ...bundle, events }))
|
||||
.toThrow(/must be ordered after 1/);
|
||||
|
||||
const duplicateIds = structuredClone(bundle.events);
|
||||
duplicateIds[1] = {
|
||||
...duplicateIds[1]!,
|
||||
sourceEventId: duplicateIds[0]!.sourceEventId,
|
||||
};
|
||||
expect(() => parsePaperclipNativeExecution({ ...bundle, events: duplicateIds }))
|
||||
.toThrow(/sourceEventId.*unique/);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,538 @@
|
||||
import { readFile } from "node:fs/promises";
|
||||
|
||||
import {
|
||||
validatePrpEvent,
|
||||
type PrpEvent,
|
||||
type PrpTerminalState,
|
||||
} from "../protocol/replay-contract.js";
|
||||
import { PAPERCLIP_RUNNER_NATIVE_EXECUTION_SCHEMA } from "./build-metadata.js";
|
||||
|
||||
const SHA256_PATTERN = /^sha256:[a-f0-9]{64}$/;
|
||||
const TOOL_OUTCOMES = new Set([
|
||||
"succeeded",
|
||||
"denied",
|
||||
"conflict",
|
||||
"duplicate",
|
||||
"unavailable",
|
||||
"failed",
|
||||
]);
|
||||
|
||||
export const paperclipNativeExecutionSchemaUrl = new URL(
|
||||
"../../protocol/schemas/native-execution.schema.json",
|
||||
import.meta.url,
|
||||
);
|
||||
export const paperclipNativeExecutionFixtureUrl = new URL(
|
||||
"../../protocol/fixtures/evals/native-execution-seeded.json",
|
||||
import.meta.url,
|
||||
);
|
||||
|
||||
export interface NativeExecutionContentRef {
|
||||
uri: string;
|
||||
sha256: string;
|
||||
byteSize: number;
|
||||
mediaType: string;
|
||||
[key: string]: unknown;
|
||||
}
|
||||
|
||||
export interface PaperclipNativeExecutionV1 {
|
||||
schema: typeof PAPERCLIP_RUNNER_NATIVE_EXECUTION_SCHEMA;
|
||||
provenance: "seeded" | "replay" | "live";
|
||||
identity: {
|
||||
runId: string;
|
||||
caseId: string;
|
||||
configId: string;
|
||||
attemptId: string;
|
||||
[key: string]: unknown;
|
||||
};
|
||||
input: {
|
||||
caseSha256: string;
|
||||
configSha256: string;
|
||||
deterministicSeed: string;
|
||||
[key: string]: unknown;
|
||||
};
|
||||
runner: {
|
||||
package: { name: string; version: string; [key: string]: unknown };
|
||||
binary: { name: string; sha256: string; [key: string]: unknown };
|
||||
prpVersion: number;
|
||||
nativeExecutionVersion: number;
|
||||
catalogVersion: number;
|
||||
catalogSha256: string;
|
||||
driverContractVersion: number;
|
||||
driverKind: string;
|
||||
driverVersion: string;
|
||||
[key: string]: unknown;
|
||||
};
|
||||
events: PrpEvent[];
|
||||
semanticTools: {
|
||||
definitions: Array<Record<string, unknown>>;
|
||||
calls: Array<{ callId: string; operationId: string; eventId: string; [key: string]: unknown }>;
|
||||
results: Array<{
|
||||
callId: string;
|
||||
operationId: string;
|
||||
eventId: string;
|
||||
outcome: string;
|
||||
[key: string]: unknown;
|
||||
}>;
|
||||
denials: Array<{
|
||||
callId: string;
|
||||
code: string;
|
||||
authorizationBoundary: string;
|
||||
retryable: boolean;
|
||||
[key: string]: unknown;
|
||||
}>;
|
||||
[key: string]: unknown;
|
||||
};
|
||||
terminal: PrpTerminalState;
|
||||
observations: Record<string, unknown>;
|
||||
usage: {
|
||||
inputTokens: number;
|
||||
outputTokens: number;
|
||||
totalTokens: number;
|
||||
requestCount: number;
|
||||
durationMs: number;
|
||||
cost: { currency: string; amountMicros: number; [key: string]: unknown } | null;
|
||||
[key: string]: unknown;
|
||||
};
|
||||
transcript: NativeExecutionContentRef & {
|
||||
complete: boolean;
|
||||
eventCount: number;
|
||||
omissionReason: string | null;
|
||||
};
|
||||
artifactRoot: NativeExecutionContentRef;
|
||||
artifacts: NativeExecutionContentRef[];
|
||||
failure: {
|
||||
classification: string;
|
||||
code: string;
|
||||
message: string;
|
||||
[key: string]: unknown;
|
||||
} | null;
|
||||
[key: string]: unknown;
|
||||
}
|
||||
|
||||
export class PaperclipNativeExecutionError extends Error {
|
||||
readonly code = "paperclip_native_execution_invalid" as const;
|
||||
|
||||
constructor(
|
||||
message: string,
|
||||
readonly path: string,
|
||||
) {
|
||||
super(`${path}: ${message}`);
|
||||
this.name = "PaperclipNativeExecutionError";
|
||||
}
|
||||
}
|
||||
|
||||
function fail(path: string, message: string): never {
|
||||
throw new PaperclipNativeExecutionError(message, path);
|
||||
}
|
||||
|
||||
function record(value: unknown, path: string): Record<string, unknown> {
|
||||
if (typeof value !== "object" || value === null || Array.isArray(value)) {
|
||||
fail(path, "must be an object");
|
||||
}
|
||||
return value as Record<string, unknown>;
|
||||
}
|
||||
|
||||
function array(value: unknown, path: string): unknown[] {
|
||||
if (!Array.isArray(value)) fail(path, "must be an array");
|
||||
return value;
|
||||
}
|
||||
|
||||
function text(value: unknown, path: string): string {
|
||||
if (typeof value !== "string" || value.trim().length === 0) {
|
||||
fail(path, "must be a non-empty string");
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function integer(value: unknown, path: string, minimum = 0): number {
|
||||
if (!Number.isSafeInteger(value) || (value as number) < minimum) {
|
||||
fail(path, `must be an integer >= ${minimum}`);
|
||||
}
|
||||
return value as number;
|
||||
}
|
||||
|
||||
function boolean(value: unknown, path: string): boolean {
|
||||
if (typeof value !== "boolean") fail(path, "must be a boolean");
|
||||
return value;
|
||||
}
|
||||
|
||||
function digest(value: unknown, path: string): string {
|
||||
const parsed = text(value, path);
|
||||
if (!SHA256_PATTERN.test(parsed)) fail(path, "must use sha256:<64 lowercase hex>");
|
||||
return parsed;
|
||||
}
|
||||
|
||||
function contentRef(value: unknown, path: string): NativeExecutionContentRef {
|
||||
const ref = record(value, path);
|
||||
return {
|
||||
...structuredClone(ref),
|
||||
uri: text(ref.uri, `${path}.uri`),
|
||||
sha256: digest(ref.sha256, `${path}.sha256`),
|
||||
byteSize: integer(ref.byteSize, `${path}.byteSize`),
|
||||
mediaType: text(ref.mediaType, `${path}.mediaType`),
|
||||
} as NativeExecutionContentRef;
|
||||
}
|
||||
|
||||
function canonicalJson(value: unknown): string {
|
||||
if (Array.isArray(value)) return `[${value.map(canonicalJson).join(",")}]`;
|
||||
if (typeof value === "object" && value !== null) {
|
||||
const object = value as Record<string, unknown>;
|
||||
return `{${Object.keys(object)
|
||||
.sort()
|
||||
.map((key) => `${JSON.stringify(key)}:${canonicalJson(object[key])}`)
|
||||
.join(",")}}`;
|
||||
}
|
||||
return JSON.stringify(value) ?? "null";
|
||||
}
|
||||
|
||||
function semanticEntry(
|
||||
value: unknown,
|
||||
path: string,
|
||||
fields: readonly string[],
|
||||
): Record<string, unknown> {
|
||||
const entry = record(value, path);
|
||||
for (const field of fields) text(entry[field], `${path}.${field}`);
|
||||
return structuredClone(entry);
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate the App-native attempt bundle while preserving unknown additive
|
||||
* fields. Required identities, digests, event bindings, terminal consistency,
|
||||
* semantic-tool outcomes, usage, and transcript completeness fail closed.
|
||||
*/
|
||||
export function parsePaperclipNativeExecution(
|
||||
value: unknown,
|
||||
): PaperclipNativeExecutionV1 {
|
||||
const bundle = record(value, "bundle");
|
||||
if (bundle.schema !== PAPERCLIP_RUNNER_NATIVE_EXECUTION_SCHEMA) {
|
||||
fail(
|
||||
"bundle.schema",
|
||||
`unsupported schema ${String(bundle.schema)}; expected ${PAPERCLIP_RUNNER_NATIVE_EXECUTION_SCHEMA}`,
|
||||
);
|
||||
}
|
||||
if (bundle.provenance !== "seeded" && bundle.provenance !== "replay" && bundle.provenance !== "live") {
|
||||
fail("bundle.provenance", "must be seeded, replay, or live");
|
||||
}
|
||||
|
||||
const identity = record(bundle.identity, "bundle.identity");
|
||||
const parsedIdentity = {
|
||||
...structuredClone(identity),
|
||||
runId: text(identity.runId, "bundle.identity.runId"),
|
||||
caseId: text(identity.caseId, "bundle.identity.caseId"),
|
||||
configId: text(identity.configId, "bundle.identity.configId"),
|
||||
attemptId: text(identity.attemptId, "bundle.identity.attemptId"),
|
||||
};
|
||||
const input = record(bundle.input, "bundle.input");
|
||||
const parsedInput = {
|
||||
...structuredClone(input),
|
||||
caseSha256: digest(input.caseSha256, "bundle.input.caseSha256"),
|
||||
configSha256: digest(input.configSha256, "bundle.input.configSha256"),
|
||||
deterministicSeed: text(input.deterministicSeed, "bundle.input.deterministicSeed"),
|
||||
};
|
||||
|
||||
const runner = record(bundle.runner, "bundle.runner");
|
||||
const runnerPackage = record(runner.package, "bundle.runner.package");
|
||||
const runnerBinary = record(runner.binary, "bundle.runner.binary");
|
||||
const parsedRunner = {
|
||||
...structuredClone(runner),
|
||||
package: {
|
||||
...structuredClone(runnerPackage),
|
||||
name: text(runnerPackage.name, "bundle.runner.package.name"),
|
||||
version: text(runnerPackage.version, "bundle.runner.package.version"),
|
||||
},
|
||||
binary: {
|
||||
...structuredClone(runnerBinary),
|
||||
name: text(runnerBinary.name, "bundle.runner.binary.name"),
|
||||
sha256: digest(runnerBinary.sha256, "bundle.runner.binary.sha256"),
|
||||
},
|
||||
prpVersion: integer(runner.prpVersion, "bundle.runner.prpVersion", 1),
|
||||
nativeExecutionVersion: integer(
|
||||
runner.nativeExecutionVersion,
|
||||
"bundle.runner.nativeExecutionVersion",
|
||||
1,
|
||||
),
|
||||
catalogVersion: integer(runner.catalogVersion, "bundle.runner.catalogVersion", 1),
|
||||
catalogSha256: digest(runner.catalogSha256, "bundle.runner.catalogSha256"),
|
||||
driverContractVersion: integer(
|
||||
runner.driverContractVersion,
|
||||
"bundle.runner.driverContractVersion",
|
||||
1,
|
||||
),
|
||||
driverKind: text(runner.driverKind, "bundle.runner.driverKind"),
|
||||
driverVersion: text(runner.driverVersion, "bundle.runner.driverVersion"),
|
||||
};
|
||||
|
||||
const sourceCursors = new Map<string, number>();
|
||||
const sourceEventIds = new Set<string>();
|
||||
const events = array(bundle.events, "bundle.events").map((event, index) => {
|
||||
const validation = validatePrpEvent(event);
|
||||
if (!validation.ok) {
|
||||
fail(
|
||||
`bundle.events[${index}]${validation.issues[0]?.path ?? ""}`,
|
||||
validation.issues[0]?.message ?? "invalid PRP event",
|
||||
);
|
||||
}
|
||||
if (validation.event.runId !== parsedIdentity.runId) {
|
||||
fail(`bundle.events[${index}].runId`, "must match bundle.identity.runId");
|
||||
}
|
||||
if (sourceEventIds.has(validation.event.sourceEventId)) {
|
||||
fail(`bundle.events[${index}].sourceEventId`, "must be unique within the bundle");
|
||||
}
|
||||
sourceEventIds.add(validation.event.sourceEventId);
|
||||
const previousSourceSeq = sourceCursors.get(validation.event.sourceInstanceId) ?? 0;
|
||||
if (validation.event.sourceSeq <= previousSourceSeq) {
|
||||
fail(
|
||||
`bundle.events[${index}].sourceSeq`,
|
||||
`must be ordered after ${previousSourceSeq} for source ${validation.event.sourceInstanceId}`,
|
||||
);
|
||||
}
|
||||
sourceCursors.set(validation.event.sourceInstanceId, validation.event.sourceSeq);
|
||||
return structuredClone(validation.event);
|
||||
});
|
||||
if (events.length === 0) fail("bundle.events", "must contain native events");
|
||||
|
||||
const terminal = record(bundle.terminal, "bundle.terminal") as unknown as PrpTerminalState;
|
||||
if (terminal.schema !== "paperclip.prp.terminal.v1") {
|
||||
fail("bundle.terminal.schema", "must be paperclip.prp.terminal.v1");
|
||||
}
|
||||
const terminalEvents = events.filter((event) => event.eventType === "run.terminal");
|
||||
if (terminalEvents.length !== 1) {
|
||||
fail("bundle.events", "must contain exactly one run.terminal event");
|
||||
}
|
||||
if (canonicalJson(terminalEvents[0]?.payload) !== canonicalJson(terminal)) {
|
||||
fail("bundle.terminal", "must equal the run.terminal event payload");
|
||||
}
|
||||
|
||||
const semanticTools = record(bundle.semanticTools, "bundle.semanticTools");
|
||||
const definitions = array(semanticTools.definitions, "bundle.semanticTools.definitions")
|
||||
.map((entry, index) => semanticEntry(entry, `bundle.semanticTools.definitions[${index}]`, ["operationId"]));
|
||||
const calls = array(semanticTools.calls, "bundle.semanticTools.calls").map((entry, index) => {
|
||||
const parsed = semanticEntry(entry, `bundle.semanticTools.calls[${index}]`, [
|
||||
"callId",
|
||||
"operationId",
|
||||
"eventId",
|
||||
]);
|
||||
return parsed as PaperclipNativeExecutionV1["semanticTools"]["calls"][number];
|
||||
});
|
||||
const results = array(semanticTools.results, "bundle.semanticTools.results").map((entry, index) => {
|
||||
const path = `bundle.semanticTools.results[${index}]`;
|
||||
const parsed = semanticEntry(entry, path, ["callId", "operationId", "eventId", "outcome"]);
|
||||
if (!TOOL_OUTCOMES.has(parsed.outcome as string)) fail(`${path}.outcome`, "is unsupported");
|
||||
return parsed as PaperclipNativeExecutionV1["semanticTools"]["results"][number];
|
||||
});
|
||||
const denials = array(semanticTools.denials, "bundle.semanticTools.denials").map((entry, index) => {
|
||||
const path = `bundle.semanticTools.denials[${index}]`;
|
||||
const parsed = semanticEntry(entry, path, ["callId", "code", "authorizationBoundary"]);
|
||||
parsed.retryable = boolean(parsed.retryable, `${path}.retryable`);
|
||||
return parsed as PaperclipNativeExecutionV1["semanticTools"]["denials"][number];
|
||||
});
|
||||
const callById = new Map(calls.map((call) => [call.callId, call]));
|
||||
if (callById.size !== calls.length) {
|
||||
fail("bundle.semanticTools.calls", "callId values must be unique");
|
||||
}
|
||||
const resultByCallId = new Map(results.map((result) => [result.callId, result]));
|
||||
if (resultByCallId.size !== results.length) {
|
||||
fail("bundle.semanticTools.results", "callId values must be unique");
|
||||
}
|
||||
const eventById = new Map(events.map((event) => [event.sourceEventId, event]));
|
||||
const referencedCallEventIds = new Set<string>();
|
||||
for (const call of calls) {
|
||||
const event = eventById.get(call.eventId);
|
||||
if (event?.eventType !== "mcp_app.tool_input") {
|
||||
fail("bundle.semanticTools.calls", `call ${call.callId} does not reference a tool-input event`);
|
||||
}
|
||||
const envelope = semanticToolEnvelope(event, "input", `event ${call.eventId}`);
|
||||
if (envelope.callId !== call.callId || envelope.operationId !== call.operationId) {
|
||||
fail(
|
||||
"bundle.semanticTools.calls",
|
||||
`call ${call.callId} does not match semantic envelope ${String(envelope.callId)}/${String(envelope.operationId)}`,
|
||||
);
|
||||
}
|
||||
if (referencedCallEventIds.has(call.eventId)) {
|
||||
fail("bundle.semanticTools.calls", `event ${call.eventId} is referenced more than once`);
|
||||
}
|
||||
referencedCallEventIds.add(call.eventId);
|
||||
}
|
||||
const referencedResultEventIds = new Set<string>();
|
||||
for (const result of results) {
|
||||
const call = callById.get(result.callId);
|
||||
if (call === undefined || call.operationId !== result.operationId) {
|
||||
fail(
|
||||
"bundle.semanticTools.results",
|
||||
`result ${result.callId} has no matching call with operation ${result.operationId}`,
|
||||
);
|
||||
}
|
||||
const event = eventById.get(result.eventId);
|
||||
if (event?.eventType !== "mcp_app.tool_result") {
|
||||
fail("bundle.semanticTools.results", `result ${result.callId} does not reference a tool-result event`);
|
||||
}
|
||||
const envelope = semanticToolEnvelope(event, "result", `event ${result.eventId}`);
|
||||
if (
|
||||
envelope.callId !== result.callId
|
||||
|| envelope.operationId !== result.operationId
|
||||
|| envelope.outcome !== result.outcome
|
||||
) {
|
||||
fail(
|
||||
"bundle.semanticTools.results",
|
||||
`result ${result.callId} does not match semantic envelope ${String(envelope.callId)}/${String(envelope.operationId)}/${String(envelope.outcome)}`,
|
||||
);
|
||||
}
|
||||
if (referencedResultEventIds.has(result.eventId)) {
|
||||
fail("bundle.semanticTools.results", `event ${result.eventId} is referenced more than once`);
|
||||
}
|
||||
referencedResultEventIds.add(result.eventId);
|
||||
if (result.outcome === "denied" && !denials.some((denial) => denial.callId === result.callId)) {
|
||||
fail("bundle.semanticTools.denials", `denied result ${result.callId} needs a denial receipt`);
|
||||
}
|
||||
}
|
||||
for (const event of events) {
|
||||
if (event.eventType === "mcp_app.tool_input" && !referencedCallEventIds.has(event.sourceEventId)) {
|
||||
fail("bundle.semanticTools.calls", `tool-input event ${event.sourceEventId} is not indexed`);
|
||||
}
|
||||
if (event.eventType === "mcp_app.tool_result" && !referencedResultEventIds.has(event.sourceEventId)) {
|
||||
fail("bundle.semanticTools.results", `tool-result event ${event.sourceEventId} is not indexed`);
|
||||
}
|
||||
}
|
||||
const denialCallIds = new Set<string>();
|
||||
for (const denial of denials) {
|
||||
if (denialCallIds.has(denial.callId)) {
|
||||
fail("bundle.semanticTools.denials", `call ${denial.callId} has duplicate denial receipts`);
|
||||
}
|
||||
denialCallIds.add(denial.callId);
|
||||
const result = resultByCallId.get(denial.callId);
|
||||
if (result?.outcome !== "denied") {
|
||||
fail("bundle.semanticTools.denials", `denial ${denial.callId} has no denied result`);
|
||||
}
|
||||
const resultEvent = eventById.get(result.eventId)!;
|
||||
const envelope = semanticToolEnvelope(
|
||||
resultEvent,
|
||||
"result",
|
||||
`event ${result.eventId}`,
|
||||
);
|
||||
if (
|
||||
envelope.code !== denial.code
|
||||
|| envelope.authorizationBoundary !== denial.authorizationBoundary
|
||||
|| envelope.retryable !== denial.retryable
|
||||
) {
|
||||
fail(
|
||||
"bundle.semanticTools.denials",
|
||||
`denial ${denial.callId} does not match its semantic result receipt`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
const usage = record(bundle.usage, "bundle.usage");
|
||||
const inputTokens = integer(usage.inputTokens, "bundle.usage.inputTokens");
|
||||
const outputTokens = integer(usage.outputTokens, "bundle.usage.outputTokens");
|
||||
const totalTokens = integer(usage.totalTokens, "bundle.usage.totalTokens");
|
||||
if (totalTokens !== inputTokens + outputTokens) {
|
||||
fail("bundle.usage.totalTokens", "must equal inputTokens + outputTokens");
|
||||
}
|
||||
const costRecord = usage.cost === null ? null : record(usage.cost, "bundle.usage.cost");
|
||||
const parsedUsage = {
|
||||
...structuredClone(usage),
|
||||
inputTokens,
|
||||
outputTokens,
|
||||
totalTokens,
|
||||
requestCount: integer(usage.requestCount, "bundle.usage.requestCount"),
|
||||
durationMs: integer(usage.durationMs, "bundle.usage.durationMs"),
|
||||
cost: costRecord === null
|
||||
? null
|
||||
: {
|
||||
...structuredClone(costRecord),
|
||||
currency: text(costRecord.currency, "bundle.usage.cost.currency"),
|
||||
amountMicros: integer(costRecord.amountMicros, "bundle.usage.cost.amountMicros"),
|
||||
},
|
||||
};
|
||||
|
||||
const transcriptRecord = record(bundle.transcript, "bundle.transcript");
|
||||
const transcript = {
|
||||
...contentRef(transcriptRecord, "bundle.transcript"),
|
||||
complete: boolean(transcriptRecord.complete, "bundle.transcript.complete"),
|
||||
eventCount: integer(transcriptRecord.eventCount, "bundle.transcript.eventCount"),
|
||||
omissionReason: transcriptRecord.omissionReason === null
|
||||
? null
|
||||
: text(transcriptRecord.omissionReason, "bundle.transcript.omissionReason"),
|
||||
};
|
||||
if (transcript.complete && transcript.eventCount !== events.length) {
|
||||
fail("bundle.transcript.eventCount", "must equal events.length for a complete transcript");
|
||||
}
|
||||
if (!transcript.complete && transcript.omissionReason === null) {
|
||||
fail("bundle.transcript.omissionReason", "is required when the transcript is incomplete");
|
||||
}
|
||||
|
||||
const observations = structuredClone(record(bundle.observations, "bundle.observations"));
|
||||
const artifactRoot = contentRef(bundle.artifactRoot, "bundle.artifactRoot");
|
||||
const artifacts = array(bundle.artifacts, "bundle.artifacts")
|
||||
.map((entry, index) => contentRef(entry, `bundle.artifacts[${index}]`));
|
||||
const failure = bundle.failure === null
|
||||
? null
|
||||
: (() => {
|
||||
const parsed = record(bundle.failure, "bundle.failure");
|
||||
return {
|
||||
...structuredClone(parsed),
|
||||
classification: text(parsed.classification, "bundle.failure.classification"),
|
||||
code: text(parsed.code, "bundle.failure.code"),
|
||||
message: text(parsed.message, "bundle.failure.message"),
|
||||
};
|
||||
})();
|
||||
|
||||
return {
|
||||
...structuredClone(bundle),
|
||||
schema: PAPERCLIP_RUNNER_NATIVE_EXECUTION_SCHEMA,
|
||||
provenance: bundle.provenance,
|
||||
identity: parsedIdentity,
|
||||
input: parsedInput,
|
||||
runner: parsedRunner,
|
||||
events,
|
||||
semanticTools: { ...structuredClone(semanticTools), definitions, calls, results, denials },
|
||||
terminal: structuredClone(terminal),
|
||||
observations,
|
||||
usage: parsedUsage,
|
||||
transcript,
|
||||
artifactRoot,
|
||||
artifacts,
|
||||
failure,
|
||||
} as PaperclipNativeExecutionV1;
|
||||
}
|
||||
|
||||
function semanticToolEnvelope(
|
||||
event: PrpEvent,
|
||||
phase: "input" | "result",
|
||||
path: string,
|
||||
): Record<string, unknown> {
|
||||
const payload = record(event.payload, `${path}.payload`);
|
||||
const envelope = record(payload.semantic_tool, `${path}.payload.semantic_tool`);
|
||||
if (
|
||||
envelope.schema !== "paperclip.prp.semantic_tool.v1"
|
||||
|| envelope.schemaVersion !== 1
|
||||
|| envelope.phase !== phase
|
||||
) {
|
||||
fail(
|
||||
`${path}.payload.semantic_tool`,
|
||||
`must be a paperclip.prp.semantic_tool.v1 ${phase} envelope`,
|
||||
);
|
||||
}
|
||||
const correlation = record(envelope.correlation, `${path}.payload.semantic_tool.correlation`);
|
||||
for (const [field, eventValue] of [
|
||||
["runId", event.runId],
|
||||
["normalizedSessionId", event.normalizedSessionId],
|
||||
["turnId", event.turnId],
|
||||
["itemId", event.itemId],
|
||||
] as const) {
|
||||
if (correlation[field] !== eventValue) {
|
||||
fail(
|
||||
`${path}.payload.semantic_tool.correlation.${field}`,
|
||||
`must match the enclosing event ${field}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
return envelope;
|
||||
}
|
||||
|
||||
export async function loadPaperclipNativeExecutionFixture(
|
||||
url: URL = paperclipNativeExecutionFixtureUrl,
|
||||
): Promise<PaperclipNativeExecutionV1> {
|
||||
const value = JSON.parse(await readFile(url, "utf8")) as unknown;
|
||||
return parsePaperclipNativeExecution(value);
|
||||
}
|
||||
@@ -0,0 +1,57 @@
|
||||
import { mkdtemp, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { PAPERCLIP_RUNNERD_BUILD_METADATA_SCHEMA } from "./build-metadata.js";
|
||||
import {
|
||||
PaperclipRunnerdArtifactError,
|
||||
parsePaperclipRunnerdBuildMetadata,
|
||||
resolvePaperclipRunnerdArtifact,
|
||||
} from "./runnerd-artifact.js";
|
||||
|
||||
const valid = {
|
||||
schema: PAPERCLIP_RUNNERD_BUILD_METADATA_SCHEMA,
|
||||
binaryName: "paperclip-runnerd",
|
||||
packageName: "@paperclipai/paperclip-runner",
|
||||
packageVersion: "0.1.2",
|
||||
binaryContractVersion: 1,
|
||||
nativeExecutionVersion: 1,
|
||||
harnessDriverVersion: 1,
|
||||
prp: { name: "paperclip.runner", minimumVersion: 1, maximumVersion: 1 },
|
||||
};
|
||||
|
||||
describe("runnerd artifact metadata", () => {
|
||||
it("parses the exact runnerd identity", () => {
|
||||
expect(parsePaperclipRunnerdBuildMetadata(valid)).toEqual(valid);
|
||||
});
|
||||
|
||||
it("rejects an unknown binary metadata schema", () => {
|
||||
expect(() => parsePaperclipRunnerdBuildMetadata({ ...valid, schema: "runnerd/v2" }))
|
||||
.toThrow(PaperclipRunnerdArtifactError);
|
||||
expect(() => parsePaperclipRunnerdBuildMetadata({ ...valid, schema: "runnerd/v2" }))
|
||||
.toThrow(/unsupported/);
|
||||
});
|
||||
|
||||
it("rejects an invalid or mismatched explicit artifact digest before execution", async () => {
|
||||
await expect(resolvePaperclipRunnerdArtifact({
|
||||
executablePath: "/does/not/matter",
|
||||
expectedSha256: "not-a-digest",
|
||||
})).rejects.toMatchObject({ issue: "digest_invalid" });
|
||||
|
||||
const root = await mkdtemp(join(tmpdir(), "paperclip-runnerd-artifact-"));
|
||||
const executablePath = join(root, "paperclip-runnerd");
|
||||
try {
|
||||
await writeFile(executablePath, "not the expected binary");
|
||||
await expect(resolvePaperclipRunnerdArtifact({
|
||||
executablePath,
|
||||
expectedSha256: `sha256:${"0".repeat(64)}`,
|
||||
})).rejects.toMatchObject({
|
||||
issue: "digest_mismatch",
|
||||
message: expect.stringContaining("observed sha256:"),
|
||||
});
|
||||
} finally {
|
||||
await rm(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,207 @@
|
||||
import { execFile } from "node:child_process";
|
||||
import { createHash } from "node:crypto";
|
||||
import { readFile, realpath, stat } from "node:fs/promises";
|
||||
import { promisify } from "node:util";
|
||||
|
||||
import {
|
||||
PAPERCLIP_RUNNERD_BUILD_METADATA_SCHEMA,
|
||||
PAPERCLIP_RUNNER_BUILD_METADATA,
|
||||
} from "./build-metadata.js";
|
||||
|
||||
const execFileAsync = promisify(execFile);
|
||||
const SHA256_PATTERN = /^sha256:[a-f0-9]{64}$/;
|
||||
|
||||
export interface PaperclipRunnerdBuildMetadata {
|
||||
schema: typeof PAPERCLIP_RUNNERD_BUILD_METADATA_SCHEMA;
|
||||
binaryName: "paperclip-runnerd";
|
||||
packageName: "@paperclipai/paperclip-runner";
|
||||
packageVersion: string;
|
||||
binaryContractVersion: number;
|
||||
nativeExecutionVersion: number;
|
||||
harnessDriverVersion: number;
|
||||
prp: {
|
||||
name: string;
|
||||
minimumVersion: number;
|
||||
maximumVersion: number;
|
||||
};
|
||||
}
|
||||
|
||||
export interface PaperclipRunnerdArtifact {
|
||||
executablePath: string;
|
||||
sha256: string;
|
||||
byteSize: number;
|
||||
buildMetadata: PaperclipRunnerdBuildMetadata;
|
||||
}
|
||||
|
||||
export class PaperclipRunnerdArtifactError extends Error {
|
||||
readonly code = "paperclip_runnerd_artifact_invalid" as const;
|
||||
|
||||
constructor(
|
||||
message: string,
|
||||
readonly issue:
|
||||
| "path_invalid"
|
||||
| "digest_invalid"
|
||||
| "digest_mismatch"
|
||||
| "metadata_unavailable"
|
||||
| "metadata_invalid",
|
||||
) {
|
||||
super(message);
|
||||
this.name = "PaperclipRunnerdArtifactError";
|
||||
}
|
||||
}
|
||||
|
||||
function record(value: unknown, path: string): Record<string, unknown> {
|
||||
if (typeof value !== "object" || value === null || Array.isArray(value)) {
|
||||
throw new PaperclipRunnerdArtifactError(`${path} must be an object`, "metadata_invalid");
|
||||
}
|
||||
return value as Record<string, unknown>;
|
||||
}
|
||||
|
||||
function text(value: unknown, path: string): string {
|
||||
if (typeof value !== "string" || value.trim().length === 0) {
|
||||
throw new PaperclipRunnerdArtifactError(`${path} must be a non-empty string`, "metadata_invalid");
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function version(value: unknown, path: string): number {
|
||||
if (!Number.isSafeInteger(value) || (value as number) < 1) {
|
||||
throw new PaperclipRunnerdArtifactError(`${path} must be a positive integer`, "metadata_invalid");
|
||||
}
|
||||
return value as number;
|
||||
}
|
||||
|
||||
/** Parse the runnerd response without accepting a look-alike binary. */
|
||||
export function parsePaperclipRunnerdBuildMetadata(
|
||||
value: unknown,
|
||||
): PaperclipRunnerdBuildMetadata {
|
||||
const metadata = record(value, "runnerd build metadata");
|
||||
if (metadata.schema !== PAPERCLIP_RUNNERD_BUILD_METADATA_SCHEMA) {
|
||||
throw new PaperclipRunnerdArtifactError(
|
||||
`runnerd metadata schema ${String(metadata.schema)} is unsupported; expected ${PAPERCLIP_RUNNERD_BUILD_METADATA_SCHEMA}`,
|
||||
"metadata_invalid",
|
||||
);
|
||||
}
|
||||
if (metadata.binaryName !== "paperclip-runnerd") {
|
||||
throw new PaperclipRunnerdArtifactError(
|
||||
`runnerd metadata names unexpected binary ${String(metadata.binaryName)}`,
|
||||
"metadata_invalid",
|
||||
);
|
||||
}
|
||||
if (metadata.packageName !== PAPERCLIP_RUNNER_BUILD_METADATA.package.name) {
|
||||
throw new PaperclipRunnerdArtifactError(
|
||||
`runnerd metadata names unexpected package ${String(metadata.packageName)}`,
|
||||
"metadata_invalid",
|
||||
);
|
||||
}
|
||||
const prp = record(metadata.prp, "runnerd build metadata.prp");
|
||||
const minimumVersion = version(
|
||||
prp.minimumVersion,
|
||||
"runnerd build metadata.prp.minimumVersion",
|
||||
);
|
||||
const maximumVersion = version(
|
||||
prp.maximumVersion,
|
||||
"runnerd build metadata.prp.maximumVersion",
|
||||
);
|
||||
if (minimumVersion > maximumVersion) {
|
||||
throw new PaperclipRunnerdArtifactError(
|
||||
"runnerd build metadata.prp minimumVersion must not exceed maximumVersion",
|
||||
"metadata_invalid",
|
||||
);
|
||||
}
|
||||
return {
|
||||
schema: PAPERCLIP_RUNNERD_BUILD_METADATA_SCHEMA,
|
||||
binaryName: "paperclip-runnerd",
|
||||
packageName: PAPERCLIP_RUNNER_BUILD_METADATA.package.name,
|
||||
packageVersion: text(metadata.packageVersion, "runnerd build metadata.packageVersion"),
|
||||
binaryContractVersion: version(
|
||||
metadata.binaryContractVersion,
|
||||
"runnerd build metadata.binaryContractVersion",
|
||||
),
|
||||
nativeExecutionVersion: version(
|
||||
metadata.nativeExecutionVersion,
|
||||
"runnerd build metadata.nativeExecutionVersion",
|
||||
),
|
||||
harnessDriverVersion: version(
|
||||
metadata.harnessDriverVersion,
|
||||
"runnerd build metadata.harnessDriverVersion",
|
||||
),
|
||||
prp: {
|
||||
name: text(prp.name, "runnerd build metadata.prp.name"),
|
||||
minimumVersion,
|
||||
maximumVersion,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve one explicitly supplied runnerd artifact, verify its content digest,
|
||||
* and read version metadata from that exact executable. This never searches
|
||||
* PATH or the App source tree.
|
||||
*/
|
||||
export async function resolvePaperclipRunnerdArtifact(input: {
|
||||
executablePath: string;
|
||||
expectedSha256: string;
|
||||
metadataTimeoutMs?: number;
|
||||
}): Promise<PaperclipRunnerdArtifact> {
|
||||
if (!SHA256_PATTERN.test(input.expectedSha256)) {
|
||||
throw new PaperclipRunnerdArtifactError(
|
||||
"expectedSha256 must use the sha256:<64 lowercase hex> form",
|
||||
"digest_invalid",
|
||||
);
|
||||
}
|
||||
|
||||
let executablePath: string;
|
||||
let fileStat: Awaited<ReturnType<typeof stat>>;
|
||||
let bytes: Buffer;
|
||||
try {
|
||||
executablePath = await realpath(input.executablePath);
|
||||
fileStat = await stat(executablePath);
|
||||
if (!fileStat.isFile()) throw new Error("path is not a file");
|
||||
bytes = await readFile(executablePath);
|
||||
} catch (error) {
|
||||
throw new PaperclipRunnerdArtifactError(
|
||||
`runnerd artifact ${input.executablePath} is unavailable: ${error instanceof Error ? error.message : String(error)}`,
|
||||
"path_invalid",
|
||||
);
|
||||
}
|
||||
|
||||
const observedSha256 = `sha256:${createHash("sha256").update(bytes).digest("hex")}`;
|
||||
if (observedSha256 !== input.expectedSha256) {
|
||||
throw new PaperclipRunnerdArtifactError(
|
||||
`runnerd artifact digest mismatch: expected ${input.expectedSha256}, observed ${observedSha256}`,
|
||||
"digest_mismatch",
|
||||
);
|
||||
}
|
||||
|
||||
let stdout: string;
|
||||
try {
|
||||
({ stdout } = await execFileAsync(executablePath, ["--build-metadata"], {
|
||||
encoding: "utf8",
|
||||
timeout: input.metadataTimeoutMs ?? 5_000,
|
||||
maxBuffer: 64 * 1024,
|
||||
}));
|
||||
} catch (error) {
|
||||
throw new PaperclipRunnerdArtifactError(
|
||||
`runnerd artifact did not return build metadata: ${error instanceof Error ? error.message : String(error)}`,
|
||||
"metadata_unavailable",
|
||||
);
|
||||
}
|
||||
|
||||
let parsed: unknown;
|
||||
try {
|
||||
parsed = JSON.parse(stdout) as unknown;
|
||||
} catch (error) {
|
||||
throw new PaperclipRunnerdArtifactError(
|
||||
`runnerd build metadata is not JSON: ${error instanceof Error ? error.message : String(error)}`,
|
||||
"metadata_invalid",
|
||||
);
|
||||
}
|
||||
|
||||
return {
|
||||
executablePath,
|
||||
sha256: observedSha256,
|
||||
byteSize: fileStat.size,
|
||||
buildMetadata: parsePaperclipRunnerdBuildMetadata(parsed),
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,353 @@
|
||||
import {
|
||||
type HarnessDriver,
|
||||
type HarnessDriverConfigValidation,
|
||||
type HarnessDriverDescriptor,
|
||||
type HarnessSession,
|
||||
type HarnessSessionRecoveryResult,
|
||||
type HarnessTranscriptSnapshot,
|
||||
type OpenHarnessSessionInput,
|
||||
type PersistedHarnessSession,
|
||||
} from "../contracts/harness-driver.js";
|
||||
import {
|
||||
createPrpSemanticToolInputEnvelope,
|
||||
createPrpSemanticToolResultEnvelope,
|
||||
} from "../protocol/semantic-tool-receipts.js";
|
||||
import type {
|
||||
PrpEvent,
|
||||
PrpStructuredRunResult,
|
||||
PrpTerminalState,
|
||||
} from "../protocol/replay-contract.js";
|
||||
|
||||
export const DETERMINISTIC_HARNESS_DRIVER_CONFIG_SCHEMA =
|
||||
"paperclip-runner/deterministic-harness-driver-config/v1" as const;
|
||||
|
||||
export interface DeterministicHarnessDriverConfig {
|
||||
schema: typeof DETERMINISTIC_HARNESS_DRIVER_CONFIG_SCHEMA;
|
||||
scenario: "semantic-tool-success";
|
||||
}
|
||||
|
||||
const DEFAULT_CONFIG: DeterministicHarnessDriverConfig = {
|
||||
schema: DETERMINISTIC_HARNESS_DRIVER_CONFIG_SCHEMA,
|
||||
scenario: "semantic-tool-success",
|
||||
};
|
||||
|
||||
function parseConfig(value: unknown): HarnessDriverConfigValidation {
|
||||
if (typeof value !== "object" || value === null || Array.isArray(value)) {
|
||||
return {
|
||||
ok: false,
|
||||
config: null,
|
||||
issues: [{ path: "/", code: "type", message: "config must be an object" }],
|
||||
};
|
||||
}
|
||||
const config = value as Record<string, unknown>;
|
||||
if (config.schema !== DETERMINISTIC_HARNESS_DRIVER_CONFIG_SCHEMA) {
|
||||
return {
|
||||
ok: false,
|
||||
config: null,
|
||||
issues: [{
|
||||
path: "/schema",
|
||||
code: "unsupported_schema",
|
||||
message: `schema must be ${DETERMINISTIC_HARNESS_DRIVER_CONFIG_SCHEMA}`,
|
||||
}],
|
||||
};
|
||||
}
|
||||
if (config.scenario !== "semantic-tool-success") {
|
||||
return {
|
||||
ok: false,
|
||||
config: null,
|
||||
issues: [{
|
||||
path: "/scenario",
|
||||
code: "unsupported_scenario",
|
||||
message: "scenario must be semantic-tool-success",
|
||||
}],
|
||||
};
|
||||
}
|
||||
return { ok: true, config: structuredClone(config), issues: [] };
|
||||
}
|
||||
|
||||
/** Provider-free fake used by packed consumers and App/Evals conformance. */
|
||||
export class DeterministicHarnessDriver implements HarnessDriver {
|
||||
readonly #config: DeterministicHarnessDriverConfig;
|
||||
|
||||
constructor(config: DeterministicHarnessDriverConfig = DEFAULT_CONFIG) {
|
||||
const validation = parseConfig(config);
|
||||
if (!validation.ok) throw new Error(validation.issues[0]?.message ?? "invalid config");
|
||||
this.#config = structuredClone(config);
|
||||
}
|
||||
|
||||
async descriptor(): Promise<HarnessDriverDescriptor> {
|
||||
return {
|
||||
kind: "paperclip-deterministic",
|
||||
displayName: "Paperclip deterministic harness driver",
|
||||
version: "1.0.0",
|
||||
protocolVersion: "prp.v1",
|
||||
capabilities: {
|
||||
resume: true,
|
||||
typedEvents: true,
|
||||
steering: false,
|
||||
interruption: true,
|
||||
structuredResult: true,
|
||||
read: true,
|
||||
reconciliation: true,
|
||||
usage: true,
|
||||
dynamicTools: true,
|
||||
runtimeRequestResolution: false,
|
||||
goals: false,
|
||||
threadLineage: false,
|
||||
unsupported: ["steering", "runtimeRequestResolution", "goals", "threadLineage"],
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async validateConfig(config: unknown): Promise<HarnessDriverConfigValidation> {
|
||||
return parseConfig(config);
|
||||
}
|
||||
|
||||
async openSession(input: OpenHarnessSessionInput): Promise<HarnessSession> {
|
||||
return new DeterministicHarnessSession(input, this.#config);
|
||||
}
|
||||
|
||||
async recoverSession(
|
||||
snapshot: PersistedHarnessSession,
|
||||
): Promise<HarnessSessionRecoveryResult> {
|
||||
if (
|
||||
snapshot.driverKind !== "paperclip-deterministic"
|
||||
|| snapshot.runId === undefined
|
||||
|| snapshot.normalizedSessionId === undefined
|
||||
) {
|
||||
return { recovered: false, reason: "snapshot does not belong to the deterministic driver" };
|
||||
}
|
||||
return {
|
||||
recovered: true,
|
||||
session: new DeterministicHarnessSession({
|
||||
runId: snapshot.runId,
|
||||
normalizedSessionId: snapshot.normalizedSessionId,
|
||||
workingDirectory: "/deterministic/recovered",
|
||||
}, this.#config, snapshot),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
class DeterministicHarnessSession implements HarnessSession {
|
||||
readonly #input: OpenHarnessSessionInput;
|
||||
readonly #config: DeterministicHarnessDriverConfig;
|
||||
#events: PrpEvent[] = [];
|
||||
#semanticResult: PrpStructuredRunResult | null = null;
|
||||
#turnId: string | null = null;
|
||||
#closed = false;
|
||||
#active = false;
|
||||
#nextSourceSeq = 1;
|
||||
|
||||
constructor(
|
||||
input: OpenHarnessSessionInput,
|
||||
config: DeterministicHarnessDriverConfig,
|
||||
snapshot?: PersistedHarnessSession,
|
||||
) {
|
||||
this.#input = structuredClone(input);
|
||||
this.#config = structuredClone(config);
|
||||
this.#turnId = snapshot?.activeTurnId ?? snapshot?.semanticResult?.turnId ?? null;
|
||||
this.#semanticResult = snapshot?.semanticResult?.result ?? null;
|
||||
}
|
||||
|
||||
ids() {
|
||||
return {
|
||||
driverSessionId: `driver_${this.#input.normalizedSessionId}`,
|
||||
providerSessionId: null,
|
||||
displayId: this.#input.normalizedSessionId,
|
||||
};
|
||||
}
|
||||
|
||||
async *events(): AsyncIterable<PrpEvent> {
|
||||
for (const event of this.#events) yield structuredClone(event);
|
||||
}
|
||||
|
||||
async startTurn(input: { message: { role: "user"; text: string } }): Promise<{ turnId: string }> {
|
||||
this.#assertOpen();
|
||||
if (this.#turnId !== null) throw new Error("deterministic session accepts one turn");
|
||||
this.#turnId = `turn_${this.#input.runId}`;
|
||||
this.#active = true;
|
||||
this.#events.push(
|
||||
this.#event("session.started", {}, { turn: false }),
|
||||
this.#event("turn.started", { message: input.message.text }),
|
||||
);
|
||||
if (input.message.text.includes("[conformance:interrupt]")) {
|
||||
return { turnId: this.#turnId };
|
||||
}
|
||||
|
||||
const itemId = `item_${this.#input.runId}`;
|
||||
const callId = `call_${this.#input.runId}`;
|
||||
const correlation = {
|
||||
runId: this.#input.runId,
|
||||
normalizedSessionId: this.#input.normalizedSessionId,
|
||||
turnId: this.#turnId,
|
||||
itemId,
|
||||
};
|
||||
const inputEnvelope = createPrpSemanticToolInputEnvelope({
|
||||
operationId: "finish_task",
|
||||
callId,
|
||||
correlation,
|
||||
idempotencyKey: `finish_${this.#input.runId}`,
|
||||
content: { summary: "Deterministic conformance complete." },
|
||||
});
|
||||
const resultEnvelope = createPrpSemanticToolResultEnvelope({
|
||||
operationId: "finish_task",
|
||||
callId,
|
||||
correlation,
|
||||
idempotencyKey: `finish_${this.#input.runId}`,
|
||||
content: { disposition: "applied", status: "done" },
|
||||
outcome: "succeeded",
|
||||
code: "applied",
|
||||
retryable: false,
|
||||
authorizationBoundary: "active_task",
|
||||
auditReceiptId: `audit_${this.#input.runId}`,
|
||||
});
|
||||
this.#events.push(
|
||||
this.#event("mcp_app.tool_input", { semantic_tool: inputEnvelope }, { itemId }),
|
||||
this.#event("mcp_app.tool_result", { semantic_tool: resultEnvelope }, { itemId }),
|
||||
);
|
||||
this.#semanticResult = completedResult();
|
||||
const terminal = completedTerminal();
|
||||
this.#events.push(
|
||||
this.#event("run.result.proposed", this.#semanticResult),
|
||||
this.#event("turn.completed", {}),
|
||||
this.#event("run.terminal", terminal, { turn: false }),
|
||||
);
|
||||
this.#active = false;
|
||||
return { turnId: this.#turnId };
|
||||
}
|
||||
|
||||
async interrupt(input: { turnId?: string; reason?: string }): Promise<void> {
|
||||
this.#assertOpen();
|
||||
if (!this.#active || this.#turnId === null) throw new Error("no active turn to interrupt");
|
||||
if (input.turnId !== undefined && input.turnId !== this.#turnId) {
|
||||
throw new Error(`turn ${input.turnId} is not active`);
|
||||
}
|
||||
this.#events.push(
|
||||
this.#event("turn.interrupted", { reason: input.reason ?? "interrupted" }),
|
||||
this.#event("run.terminal", cancelledTerminal(), { turn: false }),
|
||||
);
|
||||
this.#active = false;
|
||||
}
|
||||
|
||||
async read(): Promise<Record<string, unknown>> {
|
||||
return { scenario: this.#config.scenario, eventCount: this.#events.length };
|
||||
}
|
||||
|
||||
async reconcile(): Promise<Record<string, unknown>> {
|
||||
return { active: this.#active, closed: this.#closed, eventCount: this.#events.length };
|
||||
}
|
||||
|
||||
async usage(): Promise<Record<string, unknown>> {
|
||||
return {
|
||||
inputTokens: this.#turnId === null ? 0 : 12,
|
||||
outputTokens: this.#semanticResult === null ? 0 : 8,
|
||||
totalTokens: this.#turnId === null ? 0 : this.#semanticResult === null ? 12 : 20,
|
||||
requestCount: this.#turnId === null ? 0 : 1,
|
||||
cost: { currency: "USD", amountMicros: 0 },
|
||||
};
|
||||
}
|
||||
|
||||
async transcript(): Promise<HarnessTranscriptSnapshot> {
|
||||
return {
|
||||
schema: "paperclip-runner/harness-transcript/v1",
|
||||
complete: true,
|
||||
eventCount: this.#events.length,
|
||||
events: structuredClone(this.#events),
|
||||
omissionReason: null,
|
||||
};
|
||||
}
|
||||
|
||||
async snapshot(): Promise<PersistedHarnessSession> {
|
||||
return {
|
||||
driverKind: "paperclip-deterministic",
|
||||
driverSessionId: `driver_${this.#input.normalizedSessionId}`,
|
||||
providerSessionId: null,
|
||||
runId: this.#input.runId,
|
||||
normalizedSessionId: this.#input.normalizedSessionId,
|
||||
activeTurnId: this.#active ? this.#turnId : null,
|
||||
lastSourceSequence: this.#events.length,
|
||||
semanticResult: this.#semanticResult === null || this.#turnId === null
|
||||
? null
|
||||
: {
|
||||
result: structuredClone(this.#semanticResult),
|
||||
fingerprint: JSON.stringify(this.#semanticResult),
|
||||
turnId: this.#turnId,
|
||||
},
|
||||
terminalTurns: this.#events.some((event) => event.eventType === "run.terminal") && this.#turnId !== null
|
||||
? [{ turnId: this.#turnId, fingerprint: `terminal_${this.#input.runId}` }]
|
||||
: [],
|
||||
};
|
||||
}
|
||||
|
||||
async close(input: { reason: string; force?: boolean }): Promise<void> {
|
||||
if (this.#closed) return;
|
||||
if (this.#active) {
|
||||
await this.interrupt({ reason: input.reason });
|
||||
}
|
||||
this.#closed = true;
|
||||
}
|
||||
|
||||
#event(
|
||||
eventType: PrpEvent["eventType"],
|
||||
payload: Record<string, unknown> | PrpStructuredRunResult | PrpTerminalState,
|
||||
options: { itemId?: string; turn?: boolean } = {},
|
||||
): PrpEvent {
|
||||
const sourceSeq = this.#nextSourceSeq++;
|
||||
return {
|
||||
schema: "paperclip.prp.event.v1",
|
||||
sourceEventId: `event_${this.#input.runId}_${sourceSeq}`,
|
||||
sourceSeq,
|
||||
sourceInstanceId: `runner_${this.#input.runId}`,
|
||||
sourceKind: "runner",
|
||||
runId: this.#input.runId,
|
||||
normalizedSessionId: this.#input.normalizedSessionId,
|
||||
...(options.turn === false || this.#turnId === null ? {} : { turnId: this.#turnId }),
|
||||
...(options.itemId === undefined ? {} : { itemId: options.itemId }),
|
||||
eventType,
|
||||
schemaVersion: 1,
|
||||
priority: eventType === "run.terminal" || eventType === "run.result.proposed" ? 0 : 1,
|
||||
emittedAt: new Date(Date.UTC(2026, 7, 11, 12, 0, sourceSeq)).toISOString(),
|
||||
payload: structuredClone(payload),
|
||||
} as PrpEvent;
|
||||
}
|
||||
|
||||
#assertOpen(): void {
|
||||
if (this.#closed) throw new Error("deterministic session is closed");
|
||||
}
|
||||
}
|
||||
|
||||
function completedResult(): PrpStructuredRunResult {
|
||||
return {
|
||||
schema: "paperclip.run_result.v1",
|
||||
reportedWorkDisposition: "done",
|
||||
summary: "Deterministic harness-driver conformance completed.",
|
||||
completionClaim: {
|
||||
contractRevision: "harness-driver-conformance-v1",
|
||||
objectiveSatisfied: true,
|
||||
criteria: [],
|
||||
remainingWork: [],
|
||||
},
|
||||
evidence: [],
|
||||
verification: [{ commandOrCheck: "deterministic harness flow", status: "passed" }],
|
||||
attentionRequests: [],
|
||||
artifacts: [],
|
||||
};
|
||||
}
|
||||
|
||||
function completedTerminal(): PrpTerminalState {
|
||||
return {
|
||||
schema: "paperclip.prp.terminal.v1",
|
||||
turnTerminalState: "completed",
|
||||
runTerminalState: "succeeded",
|
||||
reportedWorkDisposition: "done",
|
||||
};
|
||||
}
|
||||
|
||||
function cancelledTerminal(): PrpTerminalState {
|
||||
return {
|
||||
schema: "paperclip.prp.terminal.v1",
|
||||
turnTerminalState: "interrupted",
|
||||
runTerminalState: "cancelled",
|
||||
reportedWorkDisposition: "yielded",
|
||||
};
|
||||
}
|
||||
@@ -9,7 +9,9 @@ export * from "./index.js";
|
||||
export * from "./conformance/control-plane-port.js";
|
||||
export * from "./conformance/semantic-conformance.js";
|
||||
export * from "./conformance/capability-semantic-conformance.js";
|
||||
export * from "./conformance/harness-driver.js";
|
||||
export * from "./mock-core/mock-control-plane-adapter.js";
|
||||
export * from "./mock-core/deterministic-harness-driver.js";
|
||||
export * from "./mock-core/capability-control-plane-types.js";
|
||||
export * from "./mock-core/capability-mock-control-plane-adapter.js";
|
||||
export * from "./protocol/conformance-fixture.js";
|
||||
|
||||
Reference in new issue
Block a user