DottaandPaperclip
a386a59998
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native agents receive task constraints and completion tools from Paperclip. > - Completion tools already define the procedure for reporting a result. > - Repeated procedure text adds instructions to each full task turn. > - The final reply must still explain a blocker and link a saved document. > - This pull request removes repeated procedure text and keeps these visible outcome requirements explicit. > - A document receipt supplies the exact link, and stricter evals check the persisted reply and browser navigation. ## Linked Issues or Issue Description Refs: #14961. Related: #14948 and #15007. **What happened?** Native task envelopes repeat completion procedure text. A reduced envelope needs explicit final-reply requirements. The `write_document` receipt also lacks a canonical document link. **Expected behavior** Keep the completion tools as the source of procedure details. Require one accepted completion result before the final reply. A blocked reply must explain the reason, owner and unblock action. A document reply must contain a working link to the saved document. **Steps to reproduce** 1. Run the native assigned-skill document case and native blocker case. 2. Inspect the run-attributed provider final and its persisted comment. 3. Check the blocker explanation or open the final reply's document link. ## What Changed - Remove repeated completion procedure text from the native task constraints and backend instructions. - Keep explicit blocker and document-link requirements in full task turns. - Return a company/task-scoped `documentHref` from `write_document`. Preserve the link in the idempotent mutation receipt. - Repeat canonical links for this run's current saved revisions in accepted completion feedback. Give blocked providers final-response guidance for the cause, owner and unblock action. - Keep internal document/comment anchors when Markdown issue links load cached issue details. - Add a manual six-cell comparison suite with strict source, build, default-instruction and budget admission. - Capture eighteen shared runnerd RPC projections and six direct OpenCode HTTP projections across start, resume and continuation phases, using scripted local transports and no provider execution. - Apply v3 checks only to the manual instruction comparison; preserve v2 checks for the existing native completion suite. Check the actual persisted blocker reason and exact saved-document link. Click the rendered document link and check the original content marker in the classic document card or the new document tab. - Forward exact OpenCode finishing calls through the controller. Wait for acceptance, keep accepted feedback and concrete rejection text, and reject malformed responses. Preserve ordinary dynamic-tool response handling. - Settle the completion decision and tool response before mapping a racing idle/error/abort event or handling explicit close/interruption. Reject a concurrent finishing call before controller admission. - Add a provider-free regression through real runnerd, the OpenCode proxy and a fake provider. Reject the first completion, accept the corrected report in the same turn, and propose one result. - Keep all original verdicts unchanged. Treat replay under new checks as separate diagnostics. ## Verification - `pnpm -r typecheck` and `pnpm build` pass locally. - Native document-authority tests pass, including company/run authorization and idempotent replay. - Native runtime-context, backend and measurement tests pass. - Final-answer calibration, protocol scoring, source-admission and catalog tests pass. Wrong reasons, absent links and wrong link targets fail. - `pnpm test:e2e:runner:typecheck` passes. Discovery lists exactly six single-attempt local cells with the declared models. - Exported `prepareNativeInstructionPreflight` then `verifyNativeInstructionPreflight` pass on this clean committed source. They build locally and make zero provider calls. - Corrective live confirmation is incomplete. Source3a7349dpassed both Claude and both Codex cases. OpenCode saved the correct document but omitted its final link; its blocker case was canceled before paid execution. Preserve this failure. Thee171282confirmation was stopped during build after fresh review found a completion-settlement race; it executed zero providers. Sourcec3e0cb303fixes that race. Two affected OpenCode cases await fresh review and one bounded confirmation; earlier results remain attributed to their original source. - OpenCode proxy parsing, driver, factory and input tests: 81 pass across retained focused runs, including six settlement races. Evaluator/scoring/admission checks: 122 pass. The real proxy regression passes. Fresh local prepare then verify passes with 18 shared and 6 direct scripted captures, fresh SDK/Rust builds and zero providers. - The full local suite recorded two failures: a webhook timeout and a Git-scan load count mismatch. Both files pass in isolation with unchanged assertions/time budgets; preserve the original failure log. Freshc3e0cb303CI and review are pending. This PR remains draft. ## Risks - Final-answer wording can vary by provider. The checks cover the declared release-access blocker and saved document fixture, not general answer quality. - A single trial does not establish general equivalence, cause, speed, cost or live resume behavior. - `documentHref` is an additive receipt field. It points to the current saved document, not an immutable historical revision. Replaying an older receipt does not fabricate a new link. - The correction adds four production paths for document receipts, accepted completion feedback and UI navigation, plus four OpenCode controller/proxy paths, beyond the original three instruction paths. Completion rejection must remain repairable; the production-boundary regression covers it. - Preserve the frozen comparison context for live measurement. A merge-tree check against current master is clean. Do not relabel earlier live results as results from a later source tree. ## Model Used - OpenAI Codex, GPT-6 family. The exact serving model ID and context window are unavailable in this session. Capabilities used: reasoning, code editing, shell execution, test authoring and evidence review. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
24 KiB
Paperclip evaluation guide
Paperclip has two live eval families with different questions, owners, and evidence. Choose the family before selecting a model, profile, or case.
The explicit-only native instruction consolidation comparison uses six Product E2E cells per source variant. It measures the completion constraint reduction separately from the earlier native tool-description trial. Provider-free start/resume payload capture is a byte measurement; behavioral qualification requires the original paired live outcomes and retained content. Neither source admission nor a scripted pass proves model behavior.
- Runner Evals: real Runner/provider behavior against a seeded mock control
plane. Definitions live in
paperclip-evals/evals/paperclip-runner; see the direct live protocol evals. - Product E2E Evals: real browser, Paperclip server, database, Runner,
provider, and (where selected) Daytona, using an isolated instance and
grading oracle. See
tests/runner-e2eand Everyday Workflows.
Runner Evals answer whether a real runner/provider can perform a bounded protocol operation against the expected control-plane contract. Product E2E Evals answer whether a person can complete a product workflow through the real Paperclip surfaces and whether the resulting artifact and state are usable. The names describe the system under test; “headless” is an execution option, not an eval category.
The explicit Product E2E completion-updates suite compares onboarding and
idle, busy, multiple-task, and restart Agent Chat handoffs on native Claude/Codex. It separates mechanical
completion delivery/result access from semantic review of the retained answer;
see the probe contract.
The explicit-only task-titles suite checks that production guidance causes a real native agent to name prompt-only standard/Ask tasks early, while preserving user-supplied titles. Its oracle correlates browser creation, native tool receipts, durable titles, audit ownership, and the reloaded task UI; fixture prompts contain no naming instructions.
Selecting a family
Use Runner Evals for a runner protocol, adapter, transport, native session,
tool grant, or one-turn provider qualification question. The workflow checks
out an exact paperclip-evals revision, builds the Runner and viewer, runs a
live roster, and renders the canonical Evalbook report. The control plane is a
seeded test authority, so a passing result does not prove browser UX, production
server behavior, database persistence, Daytona behavior, or a real third-party
mutation.
Use Product E2E Evals for browser interaction, issue/task lifecycle, approval and clarification UI, project/repository selection, persistence over a controller restart, artifact delivery, billing/evidence behavior, or runner continuity in local or Daytona environments. The harness creates a fresh Paperclip instance per cell and uses public APIs and the production browser surface. The suite's Everyday Workflows are Product E2E even when their results are imported into Evalbook.
Do not combine a partial Runner campaign and a partial Product E2E campaign into one score. A campaign is comparable when its definition/grader, model/profile, environment, and contract match. The evaluated Paperclip revision may intentionally differ for a before/after fix comparison; record it as a comparison axis.
Ownership and codepaths
Runner Evals are owned by the Runner/evals maintainers. Definitions, rosters,
case prompts, and the report program live in the sibling private repository
paperclipai/paperclip-evals; Runner integration, viewer, aggregation, and
publication code live under packages/paperclip-runner and the
runner-protocol-live-evals.yml workflow. The public-facing report uses the
same Evalbook renderer and Runner Lab viewer as the trusted report after
sanitization.
Product E2E Evals are owned by the runner E2E maintainers. The catalog and
harness are under tests/runner-e2e; the package scripts are test:e2e:runner,
test:e2e:runner:unit, test:e2e:runner:typecheck, and
test:e2e:runner:report. README.md, FIXTURES.md, SECURITY.md, and
EVERYDAY-WORKFLOWS.md are the detailed sources of truth. The harness starts
the server and embedded database, creates the company/agent/task through the
real APIs, drives Chromium, and invokes the selected local or Daytona runner.
The explicit-only agent-chat-hardening Product E2E suite covers native chat
recovery, hiring, status evidence, and review handoff on local and selected warm
Daytona paths. Its fixture contract distinguishes
startup cancellation from active response cancellation and HTTP send replay
from ambiguous provider action recovery. Select it explicitly; --all excludes it.
The explicit-only production hiring templates suite adds two local native Codex/Claude cells. It exercises API-created production CEO defaults, an explicitly requested hiring skill/reference read, a permanent coder hire, independently computed saved JSON fixtures and worker reuse. Each cell requires five work turns and admits at most two strictly attributed server task-completion turns. Every actual run remains counted; unknown or extra-work turns fail. Source/read coverage and workflow outcome are separate: missing read provenance leaves the candidate/baseline pair uncomparable even if work succeeds. Baseline bundles and coder examples derive from their own source revision, without requiring candidate wording or length.
The explicit-only context-integrity Product E2E suite covers ordered public
comment continuation and explicit invocation of an assigned pinned skill across
the seven selected legacy/native local profiles. Select it by suite or exact
execution ID because --all excludes explicit-only suites. Each cell applies a
1,000-cent company and agent budget hard stop before task creation and records
both limits in its evidence.
The explicit-only stock-harness suite reuses skill, ordered-continuation, and chat-restart journeys across eight local legacy/native profiles with production-default hires. It closes the custom QA manual coverage gap. Its required credential-free prerequisite maps vendor instruction layering, the tiny hire bundle, and shared startup/resume reductions to executable checks. The 24 live cells are configured; no live qualification is claimed from their setup or unit calibration.
The explicit-only agent-chat-stories suite covers the experimental settings
lifecycle for a configured native agent and follow-ups during active work. Its
fixture-driven file wait and persisted-plan oracle are documented in the
Product E2E guide. It does not qualify the native
onboarding wizard or change the native API-tool rollout defaults.
The explicit-only grok-qualification and grok-subscription-qualification
Product suites exercise Grok Build with API and company subscription
authentication respectively. Keep their results separate; the subscription
fixture seeds an explicitly supplied login and does not qualify interactive
login. See the Grok fixture contract.
The explicit Direct blocker guidance suite checks the legacy coordination skill against human authority, missing hiring permission, and requester scope decisions through saved browser interactions.
Validation ladder
Start with credential-free checks and a catalog listing. For Product E2E:
pnpm test:e2e:runner:typecheck
pnpm test:e2e:runner:unit
pnpm test:e2e:runner -- --list
For one explicitly selected local cell, configure only the credentials named
by that cell in .env.runner-e2e.local, then run a narrow ID:
pnpm test:e2e:runner -- --id core-compatibility.runner-codex.local.message-marker
Use the selectors documented in the runner E2E README
for a suite, profile, case, group, or environment. Daytona needs the immutable
image digest and DAYTONA_API_KEY; follow the README and fixture security guide.
--all excludes manual suites such as everyday-workflows. Select that suite
explicitly; use a narrow selector while developing a fixture.
For Runner Evals, the narrowest useful local validation is the report program's
help/validation path and the deterministic Runner checks documented in
runner-workflow-evals.md.
Hosted direct live runs must use the default-branch workflow, an exact 40
character evals_sha, an explicitly selected roster (or the maintained
enabled all campaign), and the protected paid environment. The complete
hosted command is intentionally kept in the workflow and
direct live protocol guide.
Live provider runs can spend money; use the existing workflow authorization and
the user's stated scope when selecting them.
Failure taxonomy
Record the primary failure class and preserve the evidence that supports it.
- Product failure: evidence shows Paperclip or Runner behavior violates the authored case or a hard invariant, such as wrong task state, missing approval gate, lost persistence, bad artifact, or incorrect protocol operation.
- Model/provider behavior failure: the provider turn completed with usable evidence but the model gave the wrong answer, ignored an interaction, failed to complete the authored operation, or violated a semantic assertion. It is scored as behavior, not silently retried as infrastructure.
- Grading/evidence failure: the case or matcher cannot establish its claim, a required recording/screenshot/result is malformed, or the report contract is invalid. Fix the harness or grader before interpreting the score.
- Infrastructure failure: the evidence points to provider/profile unavailability, transport admission failure, service startup failure, a missing credential/image, or inability to produce usable evidence. Startup, transport, and timeout symptoms can instead be product defects when evidence implicates Paperclip or Runner; classify from the observed failure and supported cause, rather than the symptom name alone. Preserve the artifact.
Missing usage or price data means unknown, not free. Keep provider-reported costs separate from estimates, and include retry costs when available. Latency, cleanup, billing coverage, and unpriced usage are dimensions of the result and should remain visible alongside the primary class. A timeout after successful product state reads can be a product behavior failure; a failed server-health read may be infrastructure, but inspect its cause. Use the family-specific classifier and read the attempt evidence before changing an analytical label.
Evidence, provenance, and history
Retained result snapshots and dated measurement reports belong in
paperclip-evals; application tests, Product E2E fixtures/graders, and executable
scenario inventories remain in this repository. Keep a compact results index
with immutable archive links and public report links, as in the
lifecycle baseline.
The private archive is not a dependency of app test execution. Keep large logs,
traces, and videos in the existing campaign artifact storage.
An Evalbook report is a presentation of immutable attempt records, not the
source of truth. Keep the campaign ID, Paperclip commit, paperclip-evals
commit, catalog/roster or definition fingerprint, model/profile, environment,
grader version, selected cells, retries, and provider/runtime usage with the
report. Public projections follow each family's reviewed allowlist and may
include sanitized fixture conversation, named tool outcomes, screenshots, and
structured evidence intended for public history. Credentials, secrets, private
data, raw unredacted records, and hidden reasoning stay out of public
projections.
Distinguish a complete campaign from a partial campaign. A narrow selector, manual diagnostic, missing cell, or infrastructure retry can be useful evidence without being a qualification run. History should retain both, with explicit coverage and completeness, while trend and latest-green views compare only compatible complete campaigns. Refreshing an existing report from retained evidence has zero provider calls and is a new presentation of the old measurement, not a new model run.
Existing public histories are available at Runner protocol history and Runner Product E2E history. The consolidated eval hub is at pages.paperclip.ing/evals.
For a repeatable workflow, use the matching skill: paperclip-evals, add-runner-eval, or add-product-e2e-eval.
Diagnose failures before buying another campaign
Use this loop to turn eval failures into product improvements. The unit of work is a broken user outcome or invariant, not an individual red cell.
- Freeze the evidence. Record the inspected application revision and each
campaign's evaluated revision, definition/grader fingerprint, model/profile,
environment, exact selected IDs, attempts, and usage coverage. Read the
history feed and retained attempt records before launching models. A report
refresh, skipped workflow, passing unit suite, or old-definition green cell
is not a new live measurement. Inventory explicit-only suites separately
from
--all. - Reconstruct the failed boundary. Read durable task state, interactions, event chronology, source/worker identity, delivered output, and the failing assertion. State whether the test reached the boundary it claims to test. Separate observed failure, machine class, analytical cause, and confidence. A cancelled queued wake does not by itself prove lost work; a failed decline assertion does not prove unauthorized execution. A saved artifact does not prove the user received a correct completion update.
- Group by cause and product contract. Join cells only when their evidence supports the same mechanism. Check for already-merged fixes and definition corrections before proposing new work. Keep product defects, provider/model behavior, grading defects, infrastructure, and unexercised boundaries distinct. Preserve the original grades when attribution changes.
- Design the smallest general correction. Name the desired user behavior,
the authoritative state/transaction or provider boundary that owns it, the
affected callers, and the existing guarantees that must survive. Check it
against
PRODUCT.mdandSPEC-implementation.md. A change to an intentional product rule is a contract change, not an excuse to delete its guard. Avoid case-name branches, phrase-specific prompts, unconditional retries, or weakening approval, ownership, cancellation, and budget gates to get green. - Prove the mechanism cheaply. Calibrate a grader against correct and plausible wrong retained evidence. Reproduce a product race with scripted providers or service tests. Pair every proposed fix with a regression that exercises the opposite boundary (for example, eligible delivery versus revoked access, benign obsolete wake versus interrupted active work). Presentation-only changes use the existing report refresh path and make no provider calls; do not claim replay can prove changed runtime behavior.
- Select a bounded live confirmation. Write the exact failed representative IDs and only the passing controls affected by the change. Pin the source, definitions, models, and remote image. Record an attempt cap, provider and compute budget, wall-clock deadline, and concurrency before dispatch under the existing live-run authorization. Missing pricing means unknown, not free. Do not rerun unaffected green cells during diagnosis or retry usable behavior failures until they happen to pass. Preserve every attempt. Expand to a compatible qualification campaign only after the causal fix passes.
- Close with evidence and remaining scope. Report original versus new measurements, exact selected coverage, regression results, cost coverage, and unresolved boundaries. A partial verification may close one defect; it cannot turn the full catalog green or establish reliability from one attempt.
Keep one triage record per cause with: affected cell IDs; evidence links; observed failure; supported cause and confidence; existing fix/revision; proposed product contract; invariant regressions; next exact live selection; budget and stop condition; owner; and disposition. Useful dispositions include confirmed product defect, model behavior, grader correction, infrastructure repair, fixed-but-not-remeasured, historical pass, and unqualified coverage.
Parallelize independent artifact inventory, deterministic checks, and isolated cells. Keep a cell's dependent turns ordered. When delegating, use inexpensive agents for bounded extraction, catalog reconciliation, and test execution; keep causal attribution, product design, and final review with the lead. Begin local browser campaigns at the documented conservative concurrency and increase only with measured host headroom. Hosted fanout must respect the workflow's provider and fleet caps; more simultaneous timeouts do not improve wall-clock efficiency.
Current tools support exact-ID selection and retained-evidence report refresh,
but not an automatic cause-aware "rerun unresolved failures" planner. Build an
explicit selection manifest rather than treating --all as that planner. Review
the launcher's automatic retry policy when budgeting; Product E2E may create one
fresh attempt for a retryable failure.
Install the authoring skills
The reviewable sources live in this repository's .agents/skills. For a
multi-repository workspace, install the three skills at
~/paperclipai/.agents/skills (not ~/paperclipai/skills). From the Paperclip
checkout, run:
for skill in paperclip-evals add-runner-eval add-product-e2e-eval; do
install -d "$HOME/paperclipai/.agents/skills/$skill"
install -m 644 ".agents/skills/$skill/SKILL.md" \
"$HOME/paperclipai/.agents/skills/$skill/SKILL.md"
done
This replaces only the three named skill entrypoints. Run it again after updating their tracked sources. Each skill locates the repository independently of its installation directory.
Maintain the public hub
The hub is a static directory with two links to the existing history systems. It displays a dated snapshot, not a live scoreboard. It does not run models, create another result archive, or change the existing campaign URLs.
Build from the public history feeds and check its summary logic:
python3 -m unittest discover -s scripts/evals-hub -p 'test_*.py'
python3 scripts/evals-hub/build.py --output .paperclip/evals-hub
The hub checks need Python 3 and do not call model providers.
For offline checks, pass --history-dir <directory> containing
runner-protocol-evals-history.json and runner-e2e-history.json.
For a pre-merge preview, pass --docs-ref <branch-or-sha> to link the guide
at that revision. The default guide link uses master.
Publish with the Paperclip page helper
and the configured page-uploader credentials. Use Bash 4 or newer; macOS's
system Bash 3 cannot run this helper. On macOS with Homebrew Bash installed,
put $(brew --prefix bash)/bin first in PATH before these commands:
export PAPERCLIP_PAGE_BUCKET=pages.paperclip.ing
export PAPERCLIP_PAGE_BASE_URL=https://pages.paperclip.ing
export AWS_REGION=us-east-1
bash .agents/skills/paperclip-page/scripts/publish.sh .paperclip/evals-hub --slug evals --dry-run
bash .agents/skills/paperclip-page/scripts/publish.sh .paperclip/evals-hub --slug evals
For later refreshes, rebuild in the same output directory and publish with
--update. Keep its ignored .paperclip-page/state.json ownership record;
without that record, the helper will refuse to overwrite an existing prefix.
Verify the public page and its links after publication. This manual refresh
does not add a scheduled workflow. Preserve the measurement date when choosing
a newer rendering of the same campaign.
Remaining native chat boundaries are in the explicit-only
agent-chat-qualification suite: active task reassignment, user Retry after
verified worker process loss, and multi-turn answers grounded in actual task
records. See the workflow and qualification limits.
The 26 native first-task cells exercise onboarding before native selection
becomes the UI default. Live results and semantic answer reviews must accompany
any qualification claim; catalog presence alone is not a pass.
Lifecycle behavior baseline
The credential-free lifecycle baseline
joins unit, scripted-runner, and database integration assertions to a scenario
inventory before changing narrative-based lifecycle policy. Run
pnpm test:lifecycle-baseline to retain current passes and failures. Its Product
E2E matcher calibration is separate from live execution; unrun live coverage
remains explicitly unmeasured.
The separate live lifecycle baseline
defines 46 real-provider Product E2E cells, including paired narrative probes and
named existing controls on legacy and native Codex. Discover it with
pnpm test:e2e:runner -- --list --suite lifecycle-baseline. Historical execution
results and follow-up coverage are recorded in that suite's guide.
Continuation accounting has an explicit-only eight-cell Product E2E baseline suite, complementing the deterministic lifecycle inventory.
The explicit Product E2E instruction-persistence suite verifies private file
edits, nested and binary agent files, stopped-provider directory saves, server restart, and a fresh task's
downloaded proof on local native/legacy Codex and native Daytona. See the
Product E2E runbook.
The explicit-only Product E2E api-response-reading suite verifies retrieval of
large saved API responses on local and Daytona native Codex runs. See the
Runner E2E guide.
The explicit-only Product E2E extended-harnesses suite covers pending Cursor,
Copilot and Pi ACP profiles on local and Daytona. See the
fixture admission, credentials and budget contract.
The private Runner Evals campaign of the same name provides complementary
semantic protocol cases; catalog membership is not live qualification.
The explicit-only Product E2E confirmation-replies suite tests conversational
approval and rejection, persisted message provenance, approval before execution,
ambiguous proposals, and the existing card-click path with native Claude/Codex.
See the suite contract.
Hiring notification accounting now also requires exact completed action attribution. Missing native/provider ID mapping is uncomparable evidence; it must not be reported as a model task regression or waived through name/order matching. The fixture waits for both known completion callbacks and settled bracketed observations, including the gap before pending outbox work becomes a wake. Strict action replay and original machine verdicts are retained separately.