mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-11 23:36:51 +02:00
DottaandPaperclip
57e977be72
## Thinking Path > - Paperclip manages AI agents and their work. > - The experimental Runner owns provider processes and durable sessions. > - Pi needs working task execution and human controls. > - The five-PR stack must preserve changes already on master. > - Each layer now carries the complete integrated source for a safe sequential fallback. > - This PR belongs to native GitHub stack #15602, ending at #14956. ## Linked Issues or Issue Description Refs #14436, #14631, #14743 and #14956. Ship Pi 1.0 through the experimental Paperclip Runner. The five PRs are #14921, #14922, #14923, #14924 and #14956. The user authorized the complete merge after checks pass. Existing `pi_local` execution is unchanged. Accounting and wider provider/platform qualification remain deferred. ## What Changed - Recover missing final replies after workspace finalization changes owners, using accepted-turn evidence without rerunning work or granting external-chat publication. - Preserve the admitted Pi instruction root across warm runs, while retaining changed-root rejection. - Give Pi a bounded 15-second default shutdown grace so stop, drain acknowledgement and durable suspension can complete. Explicit deadlines and other providers retain their existing behavior. - Integrate the Pi 1.0 runtime and master contracts. - Use Pi profile 22. Preserve explicit caller-selected models and exact native thinking levels. Keep Pi's wrapper, helper, extension and question/control behavior unchanged from the qualified profile-19 runtime. - Preserve master's Dot lifecycle and consent fields, configured task environment, status guards and current Codex/Claude dependency versions. Cursor stays qualified. Copilot stays pending; profile 17 binds the changed shared protocol validation sources. - Exclude general AWS IAM credentials from Pi static/custom provider bindings and selected task projections; preserve the provider-scoped Bedrock bearer key. Profile 21 is retained as historical provenance. Rust and cloud install probes use the current declaration. - Patch bundled brace-expansion 5.0.9 to the exact official 5.0.12 payload. Pin the patch and complete runtime closures. Include the patch in normal installed setup tooling. Keep the upstream Pi shrinkwrap as provenance and permit only this exact security correction. - Include current attestation files in the Docker build context. Keep the repository lockfile unchanged from master. CI and private image builds resolve manifest changes before their frozen installation. ## Verification - Full local `pnpm -r typecheck` passes, including Runner Rust, server and UI. Focused integration checks pass: 194 Runner admission/environment tests, 63 profile/credential tests with one expected skip, 152 Dot/UI configuration tests, and Pi transcript/notice tests. - Full local `pnpm build` passes on the final source. - Fresh final-source checks pass: all 698 Rust workspace tests (32 binaries), 156 credential/profile/controller tests with one expected skip, Runner TypeScript typecheck, and 20 package/setup/sandbox tests. - The profile-21 Pi materializer passes on the native host with the official pinned Node 24.21.0 and its npm. It verifies all 150 locked packages, the patched dependency and the exact closure. Setup/package bundle tests and UI token gates pass. - The old hashes were reproduced for all three supported targets before calculating the patched graph. New closure hashes are darwin-arm64 `282022db10150c6632b3444df421342e7d534bdf5d5fb1097a2e79d0625a2bcf`, darwin-x64 `64e251e19009f755c0b04f73ce2138246faab71a961b0f13d75ebfcc34bef12e`, and linux-x64 `713b1fdff42fb56a1518bdc084f181d70bee8ebadc3e4b1d76321ed9108c8410`. Independent native platform execution is separate from graph identity reproduction. - Historical cloud qualification remains unchanged: all seven core cases pass on shipping source `10dc43c9ec65d88c2f782d62afb296d09494f215`, harness `1a4408a48cfb5a1f094a311141c257c92cd7a893`, image `sha256:5b3a775b383591bda1b0c1889e509acc70ce7f37c53f09733c81d59037f02280`, and accepted Sonnet 4.6/low fixture. All 215 canonical files and all seven cleanup checks pass independent verification. These are profile-19 results and are not relabeled as fresh profile-22 runs. - Current Pi digest: `sha256:e92078bee3c23bec4100aa589013a44613d054cd686826534025d8019e9f39a9`. [The readiness plan](https://github.com/paperclipai/paperclip/blob/codex/pi-production-readiness/doc/plans/2026-10-02-pi-production-readiness.md) preserves campaign and failed-attempt provenance. - Merge only after every PR's current-head CI and fresh review pass. Linux CI covers the full suites, build and browser tests. The local embedded Postgres API-authority suite cannot start on this macOS/Node 26 host, so Linux CI must confirm that suite. ### Fresh profile-22 core qualification — 2026-10-08 All seven accepted core cases pass canonically on Pi profile 22, with `openrouter/anthropic/claude-sonnet-4.6` and native-confirmed low thinking. This model is a fixture; production accepts the caller's explicit Pi provider/model. Runtime/install source: `3241a992f2a7703e59e97ed0fd3e5d6405de4401`. Frozen accepted harness: `1a4408a48cfb5a1f094a311141c257c92cd7a893`. Immutable cloud image: `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:506f22db7edd78f37c0c40bec1cc084af1850455026dbf467194bfbb8fcef141`. Pi digest: `sha256:e92078bee3c23bec4100aa589013a44613d054cd686826534025d8019e9f39a9`. [Hosted Linux image and clean-install verification](https://github.com/paperclipai/paperclip/actions/runs/37868328023) passes, including all 20 source-bound archives, normal CLI/Pi setup, companion import and the production pack reader. This exact installation source includes the latest master integration and the corrected Pi warm instruction-root fence. Full local typecheck/build and current-head hosted CI verify the final stack. All 13 focused real-root regressions pass. The full local executor suite passed 662 tests; 15 database tests could not start the Mac embedded PostgreSQL service. Hosted Linux CI passes the full required verification and E2E checks. These fresh results keep their own source identity; profile-19 results remain historical. | Core path | Canonical campaign | Retained archive SHA-256 | | --- | --- | --- | | File edit, validation, download and Done | `pi-core22-replyfix-0-1791511228` | 23 files; `a473e8603a3dd4737863291f8d3d1e392391f0b16d433c3e0e0e9d8baf7a97b0` | | Pending question and controller restart | `pi-core22-replyfix-1-1791511376` | 33 files; `6b829c4eb74e1f32a89c692a4ae7130dbfc1c6d3cf13915effe2103d9e242c8e` | | Three-turn session/process/workspace continuity | `pi-core22-replyfix-2-1791511587` | 23 files; `7a87021f8f9a3fdd3c58bb4467f8d82c635e3ea4795d6e75f144d9aa14818df8` | | Four typed questions and browser reconnects | `pi-core22-replyfix-3-1791511881` | 42 files; `9e31755252be1f4f9cb0626c984c142d4d1ae5f5bee3a7af08444db8d12c280a` | | Plan approval and completion | `pi-core22-replyfix-4-1791512031` | 22 files; `a0383ce1aab38e7b5a25ce0e9dd3bebea5c037ebd96ae6b29dae19015da2ae2c` | | Same-turn steering and permission denial | `pi-core22-replyfix-5-1791512261` | 39 files; `c929b8c7070f0b66aedc17e65ca46e6beab1e363926ac9f7e2a75fb250f05949` | | Stop during pending permission | `pi-core22-replyfix-6-1791512390` | 33 files; `7f0a58ae0f4d5bfc76149435f4e322537089c5bd16e7ffe9b5ad71f10a621a07` | All 215 canonical files (28714587 bytes) are independently hash-verified. All seven cleanup grades pass, with no owned runtime process or temporary root after each case. Automatic retries are zero. The owned cloud host stopped normally after retention. The prior profile-22 warm attempt remains failed and separately retained: archive SHA-256 `1e54eba5ec72b50cee1534b23d1d1d4f21a090006b8a64501ba70db972abfde5`. Its original canonical classification is preserved. Diagnosis reproduced a product bug comparing an agent-files root against an unset checkpoint-only field. The fix stores the admitted physical root separately from the adopted per-run collection capability. The real-root regression fails before the fix and passes afterward, including rejection of a changed physical root. Fixture, grader, model and all seven accepted case IDs are unchanged; this fresh campaign tests final-reply publication after file registration first. The intermediate restart attempt also remains failed and retained: archive SHA-256 `5dcaefdf1d17cf4cd54fd4cf810f45e736667392339b8ce7caf08bb4e225277f`. Its original canonical classification is preserved. Pi resumed, wrote the verified answer and completed its task; exact runner suspension was proven, but idle stop consumed about 5.2s and left under 3s for the drain acknowledgement. The Pi-only default shutdown grace is now 15s, preserving a full 5s drain round trip and a finite suspension reserve. Explicit caller deadlines, other provider defaults, literal drain receipts and exact suspension identity checks remain unchanged. The timing regression fails before this correction and passes afterward; all 18 focused settlement tests and Runner typecheck pass. The final-source file attempt is also preserved as failed (`candidate_failure`), archive SHA-256 `db6767b6773ea618997927ac77bdb005a5ac81492c7b9c0ffbc900449f829bc9`. Native edit, validation, exact downloadable artifact and Done/succeeded all passed, and the exact final reply was durably recorded. A workspace recovery owner completed before the live heartbeat reached presentation, leaving that reply absent from task chat. Recovery now materializes only a completed final reply from the accepted turn of an ordinary internal Done task, preserving issue/run/contract binding, suppression, external-chat authorization and same-run deduplication. The database regression covers the generated file-preparation receipt, suppression, unapproved external continuation and replay. Server typecheck and all 49 response-selection tests pass; hosted Linux verifies the database regression because embedded PostgreSQL cannot start on this Mac. The delayed-final-answer database regression passes on [the final root-source Linux server shard](https://github.com/paperclipai/paperclip/actions/runs/37868262553/job/113628594152), alongside 1,108 passing tests. The first root Runner shard had one unchanged durable-resume test exceed its 5-second timeout; the identical top-source shard and the isolated exact test passed. One rerun of that failed job and its required aggregate passed without source or test changes. The original failed job log and the single-rerun receipt remain retained. ### October 9 merge verification Current merge head: `5a8fe63512a7166aaef5cf50065a25008aa8b44b`. All current-head checks pass, including `ci / verify` and `ci / e2e`; exact-head Greptile review is 5/5 with no unresolved threads. Current master conflicts are resolved. The user authorized the maintainer override of the code-owner review gate after these checks. The seven retained live core cases remain bound to source `3241a992f2a7703e59e97ed0fd3e5d6405de4401` and its recorded cloud image. ## Risks - The security correction changes the dependency closure and profile identity. Old sessions must reopen on the new profile. Exact identities and credential bindings fail closed. - The runner remains experimental and requires explicit selection. Legacy Pi Local is unchanged. Caller model IDs pass through; the E2E model is a fixture. - Accounting and the broad platform/provider matrix remain deferred. This merge does not publish a release or deploy a service. ## Model Used OpenAI GPT-6 through Codex assisted with reasoning, repository inspection, editing and tool use. The exact serving ID and context window are not exposed in this session. Final live qualification uses Pi 1.0.0 with `openrouter/anthropic/claude-sonnet-4.6` and native-confirmed low thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
455 lines
27 KiB
Markdown
455 lines
27 KiB
Markdown
# Paperclip evaluation guide
|
|
|
|
The [Slack connector probe catalog](../server/src/services/connectors/slack/evals/README.md)
|
|
organizes eleven manual model acceptance probes and a selector for existing
|
|
deterministic regressions (`pnpm test:slack-connector`). It is not a registered
|
|
model campaign; transport fixtures do not prove that an agent chooses a tool
|
|
or that a real Slack interaction completes.
|
|
|
|
The explicit-only [live provider connection suite](../tests/runner-e2e/PROVIDER-CONNECTIONS.md)
|
|
is a Product E2E workflow for fresh subscription/API-key/gateway connections,
|
|
with attended login and independent artifact checks against local or staging targets.
|
|
|
|
Paperclip has two live eval families with different questions, owners, and
|
|
evidence. Choose the family before selecting a model, profile, or case.
|
|
|
|
The explicit-only [native instruction consolidation comparison](plans/2026-10-03-native-completion-consolidation.md)
|
|
uses six Product E2E cells per source variant. It measures the completion
|
|
constraint reduction separately from the earlier native tool-description
|
|
trial. Provider-free start/resume payload capture is a byte measurement;
|
|
behavioral qualification requires the original paired live outcomes and
|
|
retained content. Neither source admission nor a scripted pass proves model
|
|
behavior.
|
|
|
|
- **Runner Evals:** real Runner/provider behavior against a seeded mock control
|
|
plane. Definitions live in `paperclip-evals/evals/paperclip-runner`; see the
|
|
[direct live protocol evals](../packages/paperclip-runner/docs/runner-protocol-live-evals.md).
|
|
- **Product E2E Evals:** real browser, Paperclip server, database, Runner,
|
|
provider, and (where selected) Daytona, using an isolated instance and
|
|
grading oracle. See [`tests/runner-e2e`](../tests/runner-e2e/README.md) and
|
|
[Everyday Workflows](../tests/runner-e2e/EVERYDAY-WORKFLOWS.md).
|
|
|
|
Runner Evals answer whether a real runner/provider can perform a bounded
|
|
protocol operation against the expected control-plane contract. Product E2E
|
|
Evals answer whether a person can complete a product workflow through the real
|
|
Paperclip surfaces and whether the resulting artifact and state are usable.
|
|
The names describe the system under test; “headless” is an execution option,
|
|
not an eval category.
|
|
|
|
The explicit Product E2E `completion-updates` suite compares onboarding and
|
|
idle, busy, multiple-task, and restart Agent Chat handoffs on native Claude/Codex. It separates mechanical
|
|
completion delivery/result access from semantic review of the retained answer;
|
|
see the [probe contract](../tests/runner-e2e/README.md#completion-update-probes-explicit-only).
|
|
|
|
The explicit-only [task-titles suite](../tests/runner-e2e/README.md#automatic-task-titles)
|
|
checks that production guidance causes a real native agent to name prompt-only
|
|
standard/Ask tasks early, while preserving user-supplied titles. Its oracle
|
|
correlates browser creation, native tool receipts, durable titles, audit ownership,
|
|
and the reloaded task UI; fixture prompts contain no naming instructions.
|
|
|
|
The explicit-only [native connection guidance suite](../tests/runner-e2e/README.md#native-connection-guidance-explicit-only)
|
|
adds neutral decline prompts, same-task run-attributed explanations, and measured
|
|
no-use controls across three native local profiles. Its fifteen configured cells
|
|
are preparation for future matched instruction comparisons, not a live result.
|
|
Historical Everyday cases and production prompts are preserved.
|
|
|
|
## Selecting a family
|
|
|
|
Pi's explicit-only `pi-controls` Product suite tests Stop while native permission
|
|
is unanswered and browser-originated same-turn steering, locally and on Daytona.
|
|
Its [fixture contract](../tests/runner-e2e/FIXTURES.md#pi-active-controls) separates
|
|
control acknowledgment, actual message consumption and owned process retirement.
|
|
The current Pi matrix has 26 explicit cells (13 local and 13 Daytona), including
|
|
these four controls, pending-native-question controller restart, and a Daytona-only
|
|
exact-Pi-child provider-death journey. Provider death must expire the original
|
|
unanswered card and reject stale replies without a replacement run; any durable
|
|
fallback is distinct from restoration. File editing
|
|
also requires native edit/validation events and a registered artifact download.
|
|
All remain unqualified until measured on the exact candidate runtime and harness.
|
|
|
|
Use **Runner Evals** for a runner protocol, adapter, transport, native session,
|
|
tool grant, or one-turn provider qualification question. The workflow checks
|
|
out an exact `paperclip-evals` revision, builds the Runner and viewer, runs a
|
|
live roster, and renders the canonical Evalbook report. The control plane is a
|
|
seeded test authority, so a passing result does not prove browser UX, production
|
|
server behavior, database persistence, Daytona behavior, or a real third-party
|
|
mutation.
|
|
|
|
Use **Product E2E Evals** for browser interaction, issue/task lifecycle,
|
|
approval and clarification UI, project/repository selection, persistence over a
|
|
controller restart, artifact delivery, billing/evidence behavior, or runner
|
|
continuity in local or Daytona environments. The harness creates a fresh
|
|
Paperclip instance per cell and uses public APIs and the production browser
|
|
surface. The suite's [Everyday Workflows](../tests/runner-e2e/EVERYDAY-WORKFLOWS.md)
|
|
are Product E2E even when their results are imported into Evalbook.
|
|
|
|
Do not combine a partial Runner campaign and a partial Product E2E campaign into
|
|
one score. A campaign is comparable when its definition/grader, model/profile,
|
|
environment, and contract match. The evaluated Paperclip revision may
|
|
intentionally differ for a before/after fix comparison; record it as a
|
|
comparison axis.
|
|
|
|
## Ownership and codepaths
|
|
|
|
Runner Evals are owned by the Runner/evals maintainers. Definitions, rosters,
|
|
case prompts, and the report program live in the sibling private repository
|
|
`paperclipai/paperclip-evals`; Runner integration, viewer, aggregation, and
|
|
publication code live under `packages/paperclip-runner` and the
|
|
`runner-protocol-live-evals.yml` workflow. The public-facing report uses the
|
|
same Evalbook renderer and Runner Lab viewer as the trusted report after
|
|
sanitization.
|
|
|
|
Product E2E Evals are owned by the runner E2E maintainers. The catalog and
|
|
harness are under `tests/runner-e2e`; the package scripts are `test:e2e:runner`,
|
|
`test:e2e:runner:unit`, `test:e2e:runner:typecheck`, and
|
|
`test:e2e:runner:report`. `README.md`, `FIXTURES.md`, `SECURITY.md`, and
|
|
`EVERYDAY-WORKFLOWS.md` are the detailed sources of truth. The harness starts
|
|
the server and embedded database, creates the company/agent/task through the
|
|
real APIs, drives Chromium, and invokes the selected local or Daytona runner.
|
|
|
|
The explicit-only `agent-chat-hardening` Product E2E suite covers native chat
|
|
recovery, hiring, status evidence, and review handoff on local and selected warm
|
|
Daytona paths. Its [fixture contract](../tests/runner-e2e/README.md) distinguishes
|
|
startup cancellation from active response cancellation and HTTP send replay
|
|
from ambiguous provider action recovery. Select it explicitly; `--all` excludes it.
|
|
|
|
The explicit-only [production hiring templates suite](../tests/runner-e2e/README.md#production-hiring-templates)
|
|
adds two local native Codex/Claude cells. It exercises API-created production
|
|
CEO defaults, an explicitly requested hiring skill/reference read, a permanent
|
|
coder hire, independently computed saved JSON fixtures and worker reuse.
|
|
Each cell requires five work turns and admits at most two strictly attributed
|
|
server task-completion turns. Every actual run remains counted; unknown or
|
|
extra-work turns fail. Source/read coverage and workflow outcome are
|
|
separate: missing read provenance leaves the candidate/baseline pair
|
|
uncomparable even if work succeeds. Baseline bundles and coder examples derive
|
|
from their own source revision, without requiring candidate wording or length.
|
|
|
|
The explicit-only `context-integrity` Product E2E suite covers ordered public
|
|
comment continuation and explicit invocation of an assigned pinned skill across
|
|
the seven selected legacy/native local profiles. Select it by suite or exact
|
|
execution ID because `--all` excludes explicit-only suites. Each cell applies a
|
|
1,000-cent company and agent budget hard stop before task creation and records
|
|
both limits in its evidence.
|
|
|
|
The explicit-only [stock-harness suite](../tests/runner-e2e/STOCK-HARNESS.md)
|
|
reuses skill, ordered-continuation, and chat-restart journeys across eight local
|
|
legacy/native profiles with production-default hires. It closes the custom QA
|
|
manual coverage gap. Its required credential-free prerequisite maps vendor
|
|
instruction layering, the tiny hire bundle, and shared startup/resume reductions
|
|
to executable checks. The 24 live cells are configured; no live qualification is
|
|
claimed from their setup or unit calibration.
|
|
|
|
The explicit-only `agent-chat-stories` suite covers the experimental settings
|
|
lifecycle for a configured native agent and follow-ups during active work. Its
|
|
fixture-driven file wait and persisted-plan oracle are documented in the
|
|
[Product E2E guide](../tests/runner-e2e/README.md). It does not qualify the native
|
|
onboarding wizard or change the native API-tool rollout defaults.
|
|
|
|
The explicit-only `grok-qualification` and `grok-subscription-qualification`
|
|
Product suites exercise Grok Build with API and company subscription
|
|
authentication respectively. Keep their results separate; the subscription
|
|
fixture seeds an explicitly supplied login and does not qualify interactive
|
|
login. See the [Grok fixture contract](../tests/runner-e2e/README.md#grok-build-qualification).
|
|
|
|
The explicit [Direct blocker guidance suite](../tests/runner-e2e/README.md#direct-blocker-guidance)
|
|
checks the legacy coordination skill against human authority, missing hiring
|
|
permission, and requester scope decisions through saved browser interactions.
|
|
|
|
## Validation ladder
|
|
|
|
The explicit-only [public MCP suite](../tests/runner-e2e/PUBLIC-MCP.md) evaluates
|
|
paid assistant delegation, later retrieval, feedback, review, uncertain retries
|
|
and permission boundaries. It uses the Product E2E fixtures, launcher, evidence
|
|
packaging and dashboard, with separate external-assistant and team-worker billing.
|
|
The [2026-10-01 results](plans/2026-10-01-public-mcp-paid-eval-results.md) retain
|
|
two complete model matrices, provenance, costs and the earlier failure history.
|
|
|
|
Start with credential-free checks and a catalog listing. For Product E2E:
|
|
|
|
```sh
|
|
pnpm test:e2e:runner:typecheck
|
|
pnpm test:e2e:runner:unit
|
|
pnpm test:e2e:runner -- --list
|
|
```
|
|
|
|
For one explicitly selected local cell, configure only the credentials named
|
|
by that cell in `.env.runner-e2e.local`, then run a narrow ID:
|
|
|
|
```sh
|
|
pnpm test:e2e:runner -- --id core-compatibility.runner-codex.local.message-marker
|
|
```
|
|
|
|
Use the selectors documented in the [runner E2E README](../tests/runner-e2e/README.md)
|
|
for a suite, profile, case, group, or environment. Daytona needs the immutable
|
|
image digest and `DAYTONA_API_KEY`; follow the README and fixture security guide.
|
|
`--all` excludes manual suites such as `everyday-workflows`. Select that suite
|
|
explicitly; use a narrow selector while developing a fixture.
|
|
|
|
For Runner Evals, the narrowest useful local validation is the report program's
|
|
help/validation path and the deterministic Runner checks documented in
|
|
[`runner-workflow-evals.md`](../packages/paperclip-runner/docs/runner-workflow-evals.md).
|
|
Hosted direct live runs must use the default-branch workflow, an exact 40
|
|
character `evals_sha`, an explicitly selected roster (or the maintained
|
|
enabled `all` campaign), and the protected paid environment. The complete
|
|
hosted command is intentionally kept in the workflow and
|
|
[direct live protocol guide](../packages/paperclip-runner/docs/runner-protocol-live-evals.md).
|
|
Live provider runs can spend money; use the existing workflow authorization and
|
|
the user's stated scope when selecting them.
|
|
|
|
## Failure taxonomy
|
|
|
|
Record the primary failure class and preserve the evidence that supports it.
|
|
|
|
- **Product failure:** evidence shows Paperclip or Runner behavior violates the
|
|
authored case or a hard invariant, such as wrong task state, missing approval
|
|
gate, lost persistence, bad artifact, or incorrect protocol operation.
|
|
- **Model/provider behavior failure:** the provider turn completed with usable
|
|
evidence but the model gave the wrong answer, ignored an interaction, failed
|
|
to complete the authored operation, or violated a semantic assertion. It is
|
|
scored as behavior, not silently retried as infrastructure.
|
|
- **Grading/evidence failure:** the case or matcher cannot establish its claim,
|
|
a required recording/screenshot/result is malformed, or the report contract
|
|
is invalid. Fix the harness or grader before interpreting the score.
|
|
- **Infrastructure failure:** the evidence points to provider/profile
|
|
unavailability, transport admission failure, service startup failure, a
|
|
missing credential/image, or inability to produce usable evidence. Startup,
|
|
transport, and timeout symptoms can instead be product defects when evidence
|
|
implicates Paperclip or Runner; classify from the observed failure and
|
|
supported cause, rather than the symptom name alone. Preserve the artifact.
|
|
|
|
Missing usage or price data means unknown, not free. Keep provider-reported
|
|
costs separate from estimates, and include retry costs when available.
|
|
Latency, cleanup, billing coverage, and unpriced usage are dimensions of the
|
|
result and should remain visible alongside the primary class. A timeout after
|
|
successful product state reads can be a product behavior failure; a failed
|
|
server-health read may be infrastructure, but inspect its cause. Use the
|
|
family-specific classifier and read the attempt evidence before changing an
|
|
analytical label.
|
|
|
|
## Evidence, provenance, and history
|
|
|
|
Retained result snapshots and dated measurement reports belong in
|
|
`paperclip-evals`; application tests, Product E2E fixtures/graders, and executable
|
|
scenario inventories remain in this repository. Keep a compact results index
|
|
with immutable archive links and public report links, as in the
|
|
[lifecycle baseline](../tests/lifecycle-baseline/README.md#recorded-results-moved-to-paperclip-evals).
|
|
The private archive is not a dependency of app test execution. Keep large logs,
|
|
traces, and videos in the existing campaign artifact storage.
|
|
|
|
An Evalbook report is a presentation of immutable attempt records, not the
|
|
source of truth. Keep the campaign ID, Paperclip commit, `paperclip-evals`
|
|
commit, catalog/roster or definition fingerprint, model/profile, environment,
|
|
grader version, selected cells, retries, and provider/runtime usage with the
|
|
report. Public projections follow each family's reviewed allowlist and may
|
|
include sanitized fixture conversation, named tool outcomes, screenshots, and
|
|
structured evidence intended for public history. Credentials, secrets, private
|
|
data, raw unredacted records, and hidden reasoning stay out of public
|
|
projections.
|
|
|
|
Distinguish a complete campaign from a partial campaign. A narrow selector,
|
|
manual diagnostic, missing cell, or infrastructure retry can be useful evidence
|
|
without being a qualification run. History should retain both, with explicit
|
|
coverage and completeness, while trend and latest-green views compare only
|
|
compatible complete campaigns. Refreshing an existing report from retained
|
|
evidence has zero provider calls and is a new presentation of the old
|
|
measurement, not a new model run.
|
|
|
|
Existing public histories are available at
|
|
[Runner protocol history](https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/index.html)
|
|
and [Runner Product E2E history](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/).
|
|
The consolidated eval hub is at
|
|
[pages.paperclip.ing/evals](https://pages.paperclip.ing/evals/).
|
|
|
|
For a repeatable workflow, use the matching skill: [paperclip-evals](../.agents/skills/paperclip-evals/SKILL.md),
|
|
[add-runner-eval](../.agents/skills/add-runner-eval/SKILL.md), or
|
|
[add-product-e2e-eval](../.agents/skills/add-product-e2e-eval/SKILL.md).
|
|
|
|
## Diagnose failures before buying another campaign
|
|
|
|
Use this loop to turn eval failures into product improvements. The unit of work
|
|
is a broken user outcome or invariant, not an individual red cell.
|
|
|
|
1. **Freeze the evidence.** Record the inspected application revision and each
|
|
campaign's evaluated revision, definition/grader fingerprint, model/profile,
|
|
environment, exact selected IDs, attempts, and usage coverage. Read the
|
|
history feed and retained attempt records before launching models. A report
|
|
refresh, skipped workflow, passing unit suite, or old-definition green cell
|
|
is not a new live measurement. Inventory explicit-only suites separately
|
|
from `--all`.
|
|
2. **Reconstruct the failed boundary.** Read durable task state, interactions,
|
|
event chronology, source/worker identity, delivered output, and the failing
|
|
assertion. State whether the test reached the boundary it claims to test.
|
|
Separate observed failure, machine class, analytical cause, and confidence.
|
|
A cancelled queued wake does not by itself prove lost work; a failed decline
|
|
assertion does not prove unauthorized execution. A saved artifact does not
|
|
prove the user received a correct completion update.
|
|
3. **Group by cause and product contract.** Join cells only when their evidence
|
|
supports the same mechanism. Check for already-merged fixes and definition
|
|
corrections before proposing new work. Keep product defects, provider/model
|
|
behavior, grading defects, infrastructure, and unexercised boundaries
|
|
distinct. Preserve the original grades when attribution changes.
|
|
4. **Design the smallest general correction.** Name the desired user behavior,
|
|
the authoritative state/transaction or provider boundary that owns it, the
|
|
affected callers, and the existing guarantees that must survive. Check it
|
|
against `PRODUCT.md` and `SPEC-implementation.md`. A change to an intentional
|
|
product rule is a contract change, not an excuse to delete its guard. Avoid
|
|
case-name branches, phrase-specific prompts, unconditional retries, or
|
|
weakening approval, ownership, cancellation, and budget gates to get green.
|
|
5. **Prove the mechanism cheaply.** Calibrate a grader against correct and
|
|
plausible wrong retained evidence. Reproduce a product race with scripted
|
|
providers or service tests. Pair every proposed fix with a regression that
|
|
exercises the opposite boundary (for example, eligible delivery versus
|
|
revoked access, benign obsolete wake versus interrupted active work).
|
|
Presentation-only changes use the existing report refresh path and make no
|
|
provider calls; do not claim replay can prove changed runtime behavior.
|
|
6. **Select a bounded live confirmation.** Write the exact failed representative
|
|
IDs and only the passing controls affected by the change. Pin the source,
|
|
definitions, models, and remote image. Record an attempt cap, provider and
|
|
compute budget, wall-clock deadline, and concurrency before dispatch under
|
|
the existing live-run authorization. Missing pricing means unknown, not free.
|
|
Do not rerun unaffected green cells during diagnosis or retry usable behavior
|
|
failures until they happen to pass. Preserve every attempt. Expand to a
|
|
compatible qualification campaign only after the causal fix passes.
|
|
7. **Close with evidence and remaining scope.** Report original versus new
|
|
measurements, exact selected coverage, regression results, cost coverage,
|
|
and unresolved boundaries. A partial verification may close one defect; it
|
|
cannot turn the full catalog green or establish reliability from one attempt.
|
|
|
|
Keep one triage record per cause with: affected cell IDs; evidence links;
|
|
observed failure; supported cause and confidence; existing fix/revision; proposed
|
|
product contract; invariant regressions; next exact live selection; budget and
|
|
stop condition; owner; and disposition. Useful dispositions include confirmed
|
|
product defect, model behavior, grader correction, infrastructure repair,
|
|
fixed-but-not-remeasured, historical pass, and unqualified coverage.
|
|
|
|
Parallelize independent artifact inventory, deterministic checks, and isolated
|
|
cells. Keep a cell's dependent turns ordered. When delegating, use inexpensive
|
|
agents for bounded extraction, catalog reconciliation, and test execution; keep
|
|
causal attribution, product design, and final review with the lead. Begin local
|
|
browser campaigns at the documented conservative concurrency and increase only
|
|
with measured host headroom. Hosted fanout must respect the workflow's provider
|
|
and fleet caps; more simultaneous timeouts do not improve wall-clock efficiency.
|
|
|
|
Current tools support exact-ID selection and retained-evidence report refresh,
|
|
but not an automatic cause-aware "rerun unresolved failures" planner. Build an
|
|
explicit selection manifest rather than treating `--all` as that planner. Review
|
|
the launcher's automatic retry policy when budgeting; Product E2E may create one
|
|
fresh attempt for a retryable failure.
|
|
|
|
## Install the authoring skills
|
|
|
|
The reviewable sources live in this repository's `.agents/skills`. For a
|
|
multi-repository workspace, install the three skills at
|
|
`~/paperclipai/.agents/skills` (not `~/paperclipai/skills`). From the Paperclip
|
|
checkout, run:
|
|
|
|
```sh
|
|
for skill in paperclip-evals add-runner-eval add-product-e2e-eval; do
|
|
install -d "$HOME/paperclipai/.agents/skills/$skill"
|
|
install -m 644 ".agents/skills/$skill/SKILL.md" \
|
|
"$HOME/paperclipai/.agents/skills/$skill/SKILL.md"
|
|
done
|
|
```
|
|
|
|
This replaces only the three named skill entrypoints. Run it again after
|
|
updating their tracked sources. Each skill locates the repository independently
|
|
of its installation directory.
|
|
|
|
## Maintain the public hub
|
|
|
|
The hub is a static directory with two links to the existing history systems.
|
|
It displays a dated snapshot, not a live scoreboard. It does not run models,
|
|
create another result archive, or change the existing campaign URLs.
|
|
|
|
Build from the public history feeds and check its summary logic:
|
|
|
|
```sh
|
|
python3 -m unittest discover -s scripts/evals-hub -p 'test_*.py'
|
|
python3 scripts/evals-hub/build.py --output .paperclip/evals-hub
|
|
```
|
|
|
|
The hub checks need Python 3 and do not call model providers.
|
|
|
|
For offline checks, pass `--history-dir <directory>` containing
|
|
`runner-protocol-evals-history.json` and `runner-e2e-history.json`.
|
|
For a pre-merge preview, pass `--docs-ref <branch-or-sha>` to link the guide
|
|
at that revision. The default guide link uses `master`.
|
|
|
|
Publish with the [Paperclip page helper](../.agents/skills/paperclip-page/SKILL.md)
|
|
and the configured page-uploader credentials. Use Bash 4 or newer; macOS's
|
|
system Bash 3 cannot run this helper. On macOS with Homebrew Bash installed,
|
|
put `$(brew --prefix bash)/bin` first in `PATH` before these commands:
|
|
|
|
```sh
|
|
export PAPERCLIP_PAGE_BUCKET=pages.paperclip.ing
|
|
export PAPERCLIP_PAGE_BASE_URL=https://pages.paperclip.ing
|
|
export AWS_REGION=us-east-1
|
|
bash .agents/skills/paperclip-page/scripts/publish.sh .paperclip/evals-hub --slug evals --dry-run
|
|
bash .agents/skills/paperclip-page/scripts/publish.sh .paperclip/evals-hub --slug evals
|
|
```
|
|
|
|
For later refreshes, rebuild in the same output directory and publish with
|
|
`--update`. Keep its ignored `.paperclip-page/state.json` ownership record;
|
|
without that record, the helper will refuse to overwrite an existing prefix.
|
|
Verify the public page and its links after publication. This manual refresh
|
|
does not add a scheduled workflow. Preserve the measurement date when choosing
|
|
a newer rendering of the same campaign.
|
|
|
|
Remaining native chat boundaries are in the explicit-only
|
|
`agent-chat-qualification` suite: active task reassignment, user Retry after
|
|
verified worker process loss, and multi-turn answers grounded in actual task
|
|
records. See the [workflow and qualification limits](../tests/runner-e2e/README.md#remaining-native-agent-chat-qualification).
|
|
The 26 native `first-task` cells exercise onboarding before native selection
|
|
becomes the UI default. Live results and semantic answer reviews must accompany
|
|
any qualification claim; catalog presence alone is not a pass.
|
|
|
|
The explicit-only [native question/resume qualification](../tests/runner-e2e/README.md#task-continuation)
|
|
separates a completed two-answer user journey from semantic-tool documentation
|
|
qualification. It verifies the exact provider-pause or semantic-response-wake
|
|
binding for each answer and preserves prior grades when the definition changes.
|
|
|
|
## Lifecycle behavior baseline
|
|
|
|
The credential-free [lifecycle baseline](../tests/lifecycle-baseline/README.md)
|
|
joins unit, scripted-runner, and database integration assertions to a scenario
|
|
inventory before changing narrative-based lifecycle policy. Run
|
|
`pnpm test:lifecycle-baseline` to retain current passes and failures. Its Product
|
|
E2E matcher calibration is separate from live execution; unrun live coverage
|
|
remains explicitly unmeasured.
|
|
|
|
The separate [live lifecycle baseline](../tests/runner-e2e/LIFECYCLE-BASELINE.md)
|
|
defines 46 real-provider Product E2E cells, including paired narrative probes and
|
|
named existing controls on legacy and native Codex. Discover it with
|
|
`pnpm test:e2e:runner -- --list --suite lifecycle-baseline`. Historical execution
|
|
results and follow-up coverage are recorded in that suite's guide.
|
|
|
|
Continuation accounting has an explicit-only eight-cell Product E2E [baseline suite](../tests/runner-e2e/CONTINUATION-ACCOUNTING.md), complementing the deterministic lifecycle inventory.
|
|
|
|
The explicit Product E2E `instruction-persistence` suite verifies private file
|
|
edits, nested and binary agent files, stopped-provider directory saves, server restart, and a fresh task's
|
|
downloaded proof on local native/legacy Codex and native Daytona. See the
|
|
[Product E2E runbook](../tests/runner-e2e/README.md).
|
|
|
|
The explicit-only Product E2E `api-response-reading` suite verifies retrieval of
|
|
large saved API responses on local and Daytona native Codex runs. See the
|
|
[Runner E2E guide](../tests/runner-e2e/README.md#bounded-api-response-reading).
|
|
|
|
The explicit-only Product E2E `extended-harnesses` suite covers pending Cursor,
|
|
Copilot and Pi ACP profiles on local and Daytona. See the
|
|
[fixture admission, credentials and budget contract](../tests/runner-e2e/README.md#extended-acp-harnesses-explicit-only).
|
|
The private Runner Evals campaign of the same name provides complementary
|
|
semantic protocol cases; catalog membership is not live qualification.
|
|
|
|
The explicit local [Copilot protection fixtures](../tests/runner-e2e/FIXTURES.md#copilot-native-protection)
|
|
exercise native permission denial with operator cancellation and bounded attached
|
|
command settlement. Their registration remains separate from paid qualification;
|
|
source/pack provenance and complete persisted evidence are required.
|
|
The explicit-only Product E2E `confirmation-replies` suite tests conversational
|
|
approval and rejection, persisted message provenance, approval before execution,
|
|
ambiguous proposals, and the existing card-click path with native Claude/Codex.
|
|
See the [suite contract](../tests/runner-e2e/README.md#conversational-confirmation-replies-explicit-only).
|
|
|
|
Hiring notification accounting now also requires exact completed action attribution. Missing native/provider ID mapping is uncomparable evidence; it must not be reported as a model task regression or waived through name/order matching. The fixture waits for both known completion callbacks and settled bracketed observations, including the gap before pending outbox work becomes a wake. Strict action replay and original machine verdicts are retained separately.
|
|
|
|
The explicit-only [planning guidance utility comparison](../tests/runner-e2e/PLAN-TASK-GUIDANCE.md) measures task decomposition and handoffs with current, short, and disabled skills.
|