DottaandPaperclip
0884e73b46
## Thinking Path > - Paperclip lets people manage agent work and reusable company skills. > - A saved assigned skill must affect a task's actual execution. > - Existing QA coverage creates and edits skills. > - The daily target needs coverage for ordinary saved-skill use. > - The task receives its receipt data only through one saved skill. > - The admitted immutable version, native read and saved document establish the outcome. > - This PR adds one bounded journey for that gap. > - A shared lease test also needs a stable clock to assert its exact timeout. ## Linked Issues or Issue Description Refs: #18305. Supporting repair #18885 merged as `057898530c702ec14c1e14d163e96dfb045cd8c0`. It closes the recovery-conflict prerequisite. This branch does not duplicate that production patch. **What existing behavior does this improve?** Product E2E coverage for ordinary assigned company skill use on an existing deployment target. **Current behavior** The deployment journey catalog can create and edit a company skill. It has no separate case that requires a native task to load its assigned saved skill and use data available only inside that source. **Proposed behavior** Create one synthetic managed company skill through the public API. Assign its canonical key with the normal Latest selection. Create the task through the production browser composer. Require one native work run, the exact created immutable version in its admitted skill-version map, a successful native source read or load, and an agent-authored saved receipt. Reload and verify the result, revision and original source. ## What Changed - Add `saved-skill-use` with one native work run and seven required checks. The first check is `saved-skill-assignment-bound`. - Use ordinary `desiredSkills: [canonicalKey]` assignment. Require the returned stored assignment to have `versionId: null`. This works with Beta skills off and reads or changes no experimental setting. - Bind the creation receipt, pre-dispatch source read and version file inventory. Require a managed skill with the exact company key. Require the run's authoritative `skillSelectionSnapshot: true` and admitted version map to match the created immutable version. - Keep the random receipt reference and numeric inputs only in the saved source. The task prompt names the skill without supplying its answer. - Require exact company, task, agent, run, turn and runner bindings for a complete successful native source receipt. Accept the deployed OpenCode read wrapper only when both the saved assignment and native execution input bind its runtime directory to the created immutable skill version. Also accept the first-party assigned-skill read with a provider-independent canonical key/version binding from both the saved API assignment and admitted native input. Require the exact full source, checksum, byte count and complete page. Keep the materialized directory-name guard specific to the OpenCode filesystem path. Reject claims, shell output, foreign or incomplete reads and wrong skill loads. - Add a narrow Codex reader for the retained controller `semantic_tool.input` and `semantic_tool.result` pair. Require the actual native provider, runner, session, call, turn and item; the admitted immutable key/version; the read-only `read_assigned_skill` operation; and its successful active-task receipt. Recompute the input digest and the exact complete seven-field read-result digest from the saved source. Raw dynamic or MCP names grant no authority. Keep the builtin OpenCode/filesystem reader body unchanged. - Require authoritative native commit before qualifying the saved receipt. Independently check its value and same-run document revision authorship. Reload and verify the result and unchanged source. Retain those existing readback receipts in the private checkpoint for independent verification. - Keep the exact raw campaign ID in the journal, API name, frontmatter and saved name. A lost successful creation response remains recoverable by name. - Register discovery, source fingerprints, closed checks and existing feature associations. Update the case and fixture docs. Feature-map files are unchanged. - Stabilize one existing shared sandbox lease regression with a scoped `Date.now` spy and `finally` restoration. Keep its exact timeout, no-destroy, no-replacement and recorded lease assertions. Change no production service code. ## Verification - **Genuine QA staging qualification passed on this exact `2fb654f7be0bac3bfeb4ec6b7a2eb7a06b8de7e4` head**: [workflow41763421118](https://github.com/paperclipai/paperclip-cloud/actions/runs/41763421118), [TestOps launch2257/result4034](https://paperclip.testops.cloud/launch/2257/tree/4034?treeId=0), [sanitized report](https://pages.paperclip.ing/evals-qa-smoke-41763421118-1/). All seven required checks and all five phases passed, with exactly one configured OpenCode/DeepSeek native run succeeded and committed. Owned cleanup closed. A separate source-pinned replay verified25 original-evidence controls and16 rejection controls without altering originals. Automatic #testops PASS was independently read back. Both earlier failed QA grades remain unchanged. - Current-head CI `41758727042` attempt 2 passed with 48 successful jobs and one skip. The coordinator authorized one unchanged-source failed-jobs diagnostic. Start and end timestamps prove three fresh executions (Runner package, server shard 7 and aggregate verification), 45 retained successful executions and one skipped non-execution. The prepared-context test passed in 2,534 ms. All 28 repository rehome tests passed. The original failed attempt remains sealed. Its timeout causes are still unknown. No source, timeout or assertion changed, and no reliability fix is claimed. - Fresh exact-head Greptile review is 5/5 with zero unresolved threads. All 58 current check-runs and the status rollup are terminal and clean. The native Paperclip review ended neutral and incomplete, so it is not a validated assessment. The PR remains mergeable after the dependency case merged in #18901. The source remains frozen. A coordinator-only qualification draft is prepared with approval false. A genuine QA staging pass is still required before daily enrollment. - Current head `2fb654f7be0bac3bfeb4ec6b7a2eb7a06b8de7e4`: 269 focused tests across eight files passed. These include 42 Codex producer, converter and false-positive controls. The canonical E2E typecheck passed with its own declaration prerequisites. Credential-free discovery lists 24 journeys. Fresh exact-head Greptile review passed at 5/5 with zero unresolved threads. CI `41758727042` attempt 1 failed. It completed 45 successful jobs, one skip and three failed jobs. The primary failures were one existing OpenCode prepared-context test at its unchanged 5-second timeout and three existing repository rehome tests at their unchanged 15-second timeouts. The third failed job aggregates those results. The timeout causes are unknown. The original attempt remains sealed. A direct passing twin ran the prepared-context test in 2,533 ms. All package, script, CLI, UI, workflow and root inputs match between the actual tested merges. Three server files differ, including `task-retry.ts` inside the broader rehome import graph; no full-server equality or causal fix is claimed. The later single diagnostic pass is recorded above. The native Paperclip review ended neutral with an incomplete execution, so it is not a validated assessment. No live QA pass is claimed. - The valid review on `e1a474` found that Codex displays its first-party tool as dynamic. Source inspection confirmed that its raw name has no authority. The retained controller semantic input/result pair supplies the real operation authority. Core `09162fe893b464dd2ccdafd14bd3ef1611202a86` and current source have identical relevant envelope, digest, result-unwrapping and immutable-read implementation regions. - Three real-producer positive controls failed against the old grader. All 42 now pass. The tests call the actual immutable-file reader, production protocol digest builders, Codex display converter and API redaction. They validate the event schema and cover omitted defaults, the exact runtime alias and Unicode bytes. Missing, duplicate, reordered, failed, foreign, MCP, unbound, partial-page, wrong-digest, wrong-key/version and nonce-only evidence fails. Generic dynamic output fails even with copied full text and invented read-only markers. - The original builtin reader body and source-only fixture function are byte-identical to `e1a474`. All seven checks, the one-run limit, 500-cent guard, ordinary Latest assignment and Beta-off behavior remain unchanged. The independent verifier is pinned to this new head. Its synthetic semantic receipt passes 24 negative controls and its filesystem receipt passes 16. Older helpers and both original failed QA seals remain unchanged. - Earlier integrated head `e1a47438b1c7e51a5bb2896a4ec9b6ed6560318b`: 376 focused checks across eleven files passed, including Saved, settings and Chat overlap controls. All seven canonical server type layers and exact-head E2E types passed. All four lease controls passed. Credential-free discovery lists 24 catalog journeys. No new staging run has occurred. - Normal integrations consume actual master `fbb7055b9020e505cd1bf903b1b1c301ba8f3eef` and actual Chat squash `d7283e875772ca8c684dc86e471e6b48d543b7b1`. Preserve all five Saved case/parser files byte-for-byte and retain settings and Chat registrations. The only manual resolutions were two additive handler/type anchors. All seven checks, one-run limit and 500-cent guard remain unchanged. The independent verifier is pinned to this exact head. - Earlier dbdf CI passed after one authorized failed-jobs recheck of a confirmed Spot-terminated Build runner. Timestamp comparison proves two fresh executions (Build and verify), 46 retained successful executions and one skip. The original failed attempt remains sealed. Its native Paperclip Review completed neutral with an incomplete execution and is not a validated assessment. - New master entries made the earlier dbdf PR conflict. The frozen qualification guard correctly blocked dispatch before any pin or receipt. Preserve its unapproved draft as history; it did not authorize a QA run. This normal integration resolves the observed conflict. Fresh current-head CI/review and one genuine staging pass remain required. - Earlier merged head `dbdf1f3cbeaacc7cf9a2a457618035c945bd09f0`: 227 focused checks passed across seven files. All seven canonical server type layers passed. The E2E typecheck passed using the declarations built in this same checkout on this exact head. All four existing clock and lease controls passed. Credential-free discovery lists 22 catalog journeys, including both saved-skill use and task lifecycle. These are local checks, not a live staging qualification. - Normal merge of master `8771606fd87f69ed4057779e3dce3a9bca8088e6` retains the reviewed task-lifecycle case and its Cursor fixture repair from #18782. Resolve six shared journey registrations by preserving both cases. The five saved-skill case files remain byte-for-byte equal to reviewed `ff3814`. All seven required checks, one-run limit, 500-cent bound and ordinary Beta-off Latest assignment remain unchanged. No feature-map delta was added. - The independent sealed-run verifier is now pinned to this exact merged head. Synthetic semantic evidence passed with 24 negative controls. Synthetic filesystem evidence passed with 16 negative controls. Foreign document authorship and missing reopen/source readback still fail. The old `ff3814` verifier and both original failed QA seals remain unchanged. - Common generated outputs were copied only after all 4,225 package, script and root inputs matched, installed lock/TypeScript hashes matched, and all 4,955 source output paths and hashes passed before and after copy. No server declarations or editable source/module links were copied. The server declarations were rebuilt locally. A full frozen workspace install disabled lifecycle scripts and used the same locked packages. No Rust build ran locally. - Previous `ff3814` CI `41733864320` exposed a real wrapped PostgreSQL uniqueness error in a recovery server test. That production defect is independently fixed and tested in #18885. The later supporting PR merged separately. This branch does not duplicate its production patch or regrade the original failure. Preserve the original failed result and the separate Spot interruption. New full CI and a fresh exact-head 5/5 review remain required. - Previously, **225 tests passed across six files** in the focused evidence, flow, contract, catalog and report set after the semantic-read correction. The prior 181-test proof and all original failed attempts remain retained. This correction changes only the saved-skill evidence helper, flow argument, their tests and case documentation. The shared clock repair remains unchanged. - E2E TypeScript check passed on Node **24.22.2**. - Earlier credential-free discovery listed 21 cases and exposes the ordinary saved-skill journey without a Beta prerequisite. These local checks make no provider or model calls. - Source inspection of Core `09162fe893b464dd2ccdafd14bd3ef1611202a86` confirms key-only selection becomes a stored null Latest selection. Managed execution resolves its current immutable version and records the admitted map in `contextSnapshot`. The public run API preserves that map independently of optional result presentation data. - Seven invalid admitted-version fixtures wrongly qualified without the new guard. The regressions now reject them before the read or document can qualify. Beta-off and Beta-on ordinary request controls use the production sync validator. Wrong assignments, missing null selection, source drift, creation-version mismatch and another skill's complete native output cannot pass. - Existing native commit, null/compact result data, read provenance, nonce, saved-result, reload and lost-creation-response controls remain required. Their original red results are retained. - The earlier experimental version-pin variant remains unqualified. Its missing-prerequisite regression and the denied target-setting attempt remain separate failure evidence. This case qualifies ordinary assignment and the version actually admitted by the server. - Normal merge of master `5f1b59ba2c9adfb40a681bf419a8b7e7837d1125` includes the reviewed inbox snapshot fix #18798 and App/Shell access-fixture fix #18799. The merge has no conflicts. All 17 journey source, helper, registration and documentation files remain byte-identical to reviewed head `eab260955ff43eeacfdafc9b75dfc9b116f0eb79`. - Previous CI `41672686614` failed two UI test shards with obsolete user-ID fixtures; its aggregate verify job also failed. The merged access-fixture repair addresses those failures. This is a new source CI run; the failed run is retained. - Dependency output reuse follows exact path and byte checks for 10,977 tracked build inputs and 3,411 generated outputs. The first copy guard detected 12 bundles regenerated after its initial manifest and stopped before any destination mutation. The stable replacement manifest passed all checks before copying. Final focused tests, E2E typecheck and discovery passed after the verified copy. No editable source or module link was shared. - An earlier integration found stale generated Cursor Cloud declarations. Canonical dependency builds refreshed them, then typecheck passed. The original typecheck failure remains recorded. - The initial report support failure came from a blocked `tsx` IPC pipe before report execution. The unchanged test passed with normal local IPC permission. The original failure and diagnostic remain recorded. - `git diff --check` passes. Full workspace recursive typecheck, unit run and build remain assigned to hosted CI and coordinator verification. Full CI `41679720041` passed on the prior `f3f288a54177f2b9a7b225b1d1ab35440dfc597c` source. A clean 5/5 review passed on `0f445cf1c622afb25a1af92398784209df542696`. Fresh CI and review are required after the shared clock fixture repair. - The first real QA staging attempt on `f3f288a54177f2b9a7b225b1d1ab35440dfc597c` failed `saved-skill-native-read-attributed`. Assignment, immutable admitted version and cleanup passed. The test stopped before document, reload and source-after observations. The original failed attempt and its complete evidence remain sealed. - The original successful builtin `read` event contained the full saved procedure. OpenCode returned its file wrapper with an unknown normalized operation, null target and a hashed runtime skill name. The old parser rejected that shape. A synthetic regression failed before this correction. The corrected parser recognizes that exact retained read in memory, with no change to the original grade. Wrong paths, copied shell text, missing starts, mismatched calls, failed/truncated reads and wrong version/name bindings still fail. A later staging attempt is described below. Neither failed original was regraded. - Original CI `41703535552` failed one shared server test and its dependent verification job. The lease-resume test expected an RPC timeout of 31,234 ms, but the runtime correctly supplied 31,233 ms after one elapsed millisecond. The test and runtime service were byte-identical to the previous passing source. The failed run remains recorded. - A scoped advancing clock reproduced that exact 31,233-versus-31,234 failure locally. A stable clock around only the acquisition call now preserves the exact original assertion. The repaired test and three related sandbox lifecycle controls pass on disposable embedded Postgres. No production timeout, lease preservation, cleanup or replacement behavior changed. - The first local reproduction stopped before test execution because six same-package native Postgres library aliases were missing. Their exact existing versioned libraries matched the canonical dependency. Restoring only those internal relative aliases allowed the separate deterministic red and green measurements. This prerequisite failure remains recorded. - The second real staging attempt on `02582aadd23fc128ae29daa69482113cf0f95ad1` also failed `saved-skill-native-read-attributed`. Authentication, setup, assignment, source-only input, collection and cleanup passed. It completed one native run. Later document, reload and source-after checks were not executed. The complete original remains sealed as failed. - That attempt used the ordinary first-party `paperclip_read_assigned_skill` tool. Its complete native receipt contained the exact canonical skill key, created version, `SKILL.md` bytes, SHA-256, byte count and null next-page offset. Core verifies the pinned manifest and file checksum before returning that receipt. The old grader recognized only filesystem and native skill-load output. An actual-shape regression failed before this correction. The corrected parser recognizes the retained read in memory. Wrong keys, versions, source bytes, hashes, partial pages, semantic failures, copied shell output and foreign or unpaired native calls still fail. No model prompt, product source or original grade changed. A live pass on this corrected source remains required. - CI `41708636059` on the previous `02582` source passed at attempt 3. The first two interrupted attempts remain recorded. AWS evidence identified Spot capacity termination for the second Rust runner. The targeted retry executed Rust and aggregate verification; 46 successful jobs retained their exact prior execution timestamps and one job remained skipped. New metadata IDs do not establish new executions. Fresh full CI and review remain required for this source. - Earlier CI `41730743279` passed on `1ee86838ea528edd8c60fb9ddd858faf3c83e783`. A later review finding correctly showed that the semantic read still depended on an OpenCode-only directory-name helper. That finding remains recorded. Three local Codex, ACPX Cursor and Cursor Cloud flow controls reproduced the rejection before this correction. - The semantic path now binds the canonical key and immutable version through the common native runtime skill selection contract. Its checksum, complete source, page and native call attribution checks stay unchanged. Missing, duplicate, wrong-key and foreign-version API/native assignment controls still fail for all three provider configurations. The OpenCode filesystem wrapper still requires its exact provider and materialized directory name. These local controls test the existing canonical receipt shape; they do not claim live provider qualification or add a guessed provider encoding. Fresh CI and review are required after this correction. - This public PR does not enroll the case in the daily plan or change protected QA configuration. ## Risks The target must supply a disposable native agent with normal company skill-write access and cleanup. It must preserve the authoritative admitted version map and provide a complete source read or named native skill-load receipt. Missing evidence fails the case even when the receipt document has the right value. The OpenCode wrapper path requires matching assignment and native input records for the same immutable version; it does not accept a generic targetless read. The builtin first-party path requires the same version binding and exact full-source receipt. Codex must provide its authoritative controller pair and a digest that commits to that exact complete result. Partial pages, unknown operations and semantic failures do not qualify. Local flow controls simulate the target API and browser boundaries. They do not establish live Cloud qualification. Experimental assignment pins need separate qualification. The journey changes only its owned skill, assignment and task. It changes no shared agent, setting, model, connection or budget. ## Model Used OpenAI GPT-6 through Codex with reasoning and code execution. The runtime does not expose a more specific model identifier or context-window size. Local checks made no product-model calls. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
42 KiB
Paperclip evaluation guide
The explicit deployment-targeted saved agent instructions journey checks an owned agent’s ordinary UI save, persisted instruction bytes, one current-run output, and restoration. The task request omits its instruction-only reference. It reuses the existing Product E2E target and report path; local controls do not establish a staging pass.
The explicit-only verified API artifact transfer cell checks a native Codex agent on Daytona installing an authorized saved API response and delivering unchanged bytes. The oracle binds the source capture, tool completion and independent attachment download, including the complete 388131-byte synthetic description. Registration and oracle calibration are separate from live provider qualification.
The explicit-only Kimi Code qualification suite registers twenty candidate cells across local and Daytona. It reuses the browser, semantic interaction, file/artifact, Stop and continuation oracles with scoped candidate admission and an encrypted Code credential. Registration does not establish a live pass, native question/plan support or usable cost accounting.
The public directory calls full-product testing E2E tests. Suites describe what is tested; deployment targets describe which app instance was tested. QA smoke is a named selection of existing journeys against QA, and appears in the same campaign history as other E2E runs. Agent execution (local or Daytona), provider, model and profile are independent filters. Browser-only journeys need no agent or model. Historical records without target metadata remain explicitly "Not recorded" rather than being assigned an inferred environment.
The history sidebar includes the current suite catalog and retained historical
suites. Each suite has its latest matching run and latest full run, with full
coverage determined from the recorded suite definition, selection and execution
counts. An unexecuted full selection remains unfinished, and a subset
can pass without qualifying as a full suite run; failed full selections remain
visible. Changing a model filter does not redefine the suite's recorded coverage.
Existing runner-e2e URLs, test IDs, commands, and measurement timestamps remain
stable. Runner protocol evals remain a separate category because they test
the runner/provider against a mock control plane.
Sidebar suite links open that suite's campaign history with other filters cleared. The Suite dropdown combines with the current filters instead. Explicit URLs retain their selected filters; when a suite is absent from a preset, the empty state explains the combination and links to the suite's runs. Active filters have blue borders and tinted fields. The filter toolbar counts active values, including a count beside More filters for collapsed controls. Reset all filters clears every campaign filter and the sidebar suite search.
node --import ./cli/node_modules/tsx/dist/loader.mjs tests/runner-e2e/history-browser.browser.mjs
exercises the history UI using synthetic metadata and Chromium, with no model calls. It checks suite coverage, correlated
suite/target/model/execution filters, saved QA selections, URL history and mobile
navigation. Install the existing Playwright Chromium dependency before running.
For presentation-only refreshes,
node --import ./cli/node_modules/tsx/dist/loader.mjs tests/runner-e2e/history-refresh.ts --history <history.json> --output <directory>
renders an existing history file. Its optional --public-campaign, --public-url and
--preset arguments import a reviewed public measurement without replacing
existing campaigns. Private Cloud bundles are rejected. The output contains
only index HTML and history/pointer JSON; it does not upload evidence or change
retained campaign bundles.
The explicit-only Cursor Cloud native cells cover task/artifact delivery, same-agent follow-up, active-run UI Stop and semantic Plan revision/approval from a local controller. They require explicit model/repository/key and a callback-only public HTTPS proxy. Controlled tests and discovery do not qualify hosted work. These cells remain private because raw evidence can contain provider session IDs.
The explicit-only chat feature evals promote two natural-language task scenarios into local native Codex subscription and OpenCode Product E2E cells. They verify saving an unstarted task and delegating a saved announcement back to chat, using ordinary production instructions, real browser messages and independent state/provenance checks. A separately selected native OpenCode/DeepSeek1017 backlog cell supplied historical TestOps integration acceptance with the same oracle and original bounds. Its former smoke plan and the three-case completion pilot are retired; their cases and measurements remain in history. The full-catalog TestOps lane maps all 244 YAML scenarios into a separate candidate-scoped plan. Its app and fixtures use a preserved immutable catalog revision; trusted workflow helpers run from master. Registration, known gaps and private prerequisites remain distinct from measured results. This lane does not qualify the current master app or reset earlier campaign accounting.
The explicit-only Daytona restart continuity journey extends the existing native restart oracle to a 300-second controller outage and another replacement after completion, using the recorded Claude and Codex models in disposable development fixtures. It requires same-process adoption, saved files and finalization without replay; local regression checks and catalog discovery are separate from live acceptance.
The explicit-only GitHub review bot journey converts the Ponytail walkthrough into attended App setup, scoped installation, automatic PR review, and independent two-service checks. It supports attached self-hosted/Cloud targets and separately selected local/Daytona agents. Account and credential bindings stay in the private eval repo; registration and oracle tests do not establish a completed live pass or unattended CI qualification.
The deployment-targeted question-answer-resume journey reuses the classic
two-question scenario and native continuation oracle on an attached target.
It checks the actual board answers, one to three bound work runs, an agent-authored
task document and browser readback after reload. A typed reference may use a text
field or the native card's explicit written-answer control; a predefined option
cannot replace the exact typed answer. It accepts provider questions and
durable questions proven by an applied semantic receipt or an immutable completed
native API mutation receipt. It does not qualify named-tool selection. Its provider owns target setup,
usage and cleanup. Registration and local calibration do not establish a fresh
Cloud pass or inclusion in a recurring selection.
The explicit-only necessary free-text question journey checks one essential input, the rendered prompt and context, the real answer, one bound continuation, and an exact saved confirmation. Its four development cells use the recorded Codex and Claude models. Registration and calibration do not establish Slack, staging authentication, warm lifecycle, or live qualification.
The explicit followup-while-running Product journey
checks a saved message during independently observed native work, one attributed
reply with the late requirement, and persistence after reload. Its attached and
managed targets share the same story and permit one steered run or one normal
successor. Registration and calibration are separate from a live staging pass.
The explicit dependency continuation journey checks a finite blocked wait and one automatic native continuation after a browser-completed blocker. Local evidence controls are separate from a deployed QA pass.
Reusable Product journeys add explicit
deployment targets and starting-state recipes. The shared task-properties
journey and task-document create/edit/reopen journey use no model; Cloud
onboarding supplies reusable verified state. The document journey checks exact
saved content and both revision records through independent API reads. The explicit chat-continuity selection verifies browser reload/reopen, prompt-hidden recall,
ordinary /new reset, fresh native context and retained history against an attached target.
It requires a fresh owned agent with no prior work runs and uses three provider turns plus one synthetic reset row; it does not restart a shared stack.
Cloud fleet operations and trusted CI remain private. Local eval gateway qualification
does not qualify Cloud routes, and these selections stay outside --all.
The deployment-targeted saved skill journey checks one company skill assigned by key, its exact admitted immutable version, a bound native source read/load, and a run-authored saved result from skill-only inputs. It uses the reusable Product journey target and remains unqualified until a real target run passes.
The explicit-only case handoff workflow
contains arithmetic and receipt smoke cells for a same-task round trip among three native agents using real case/task
APIs, saved documents, handoff receipts, and browser readback. Receipt smoke has
an explicit terminal handoff to done on its fourth run, proving case closure
through normal task completion without a fifth provider run. It also has
an initial task-owned brief supplied at kickoff through initialDocuments,
ordinary native task document reads/writes, and case-page rollup readback.
Earlier case-document campaign results remain preserved and do not qualify
this subsequent document ownership change. The suite has
separately selectable local Codex API-key and subscription profiles with the
same model and limits. The API profile uses an encrypted company-secret
reference; the subscription profile receives only an explicitly supplied private
login fixture. The original arithmetic cell remains subscription-only.
The receipt cell declares exactly four task work runs, 90-second work-run limits,
a five-minute attempt, and 100-cent accounted-work budgets. Initial agent setup
also performs production hello probes outside heartbeat counts, budget accounting
and the work-run usage report. Up to three initial probes are expected; their
detailed tokens and cost are unavailable, and a timeout need not cancel an
in-flight probe immediately. These limits do not establish an absolute provider
inference cap, and unknown setup usage is not free. The original arithmetic cell
retains its own limits.
Service tests and workflow discovery remain distinct from a completed provider run.
The Slack connector probe catalog
organizes eleven manual model acceptance probes and a selector for existing
deterministic regressions (pnpm test:slack-connector). It is not a registered
model campaign; transport fixtures do not prove that an agent chooses a tool
or that a real Slack interaction completes.
The explicit-only live provider connection suite is a Product E2E workflow for fresh subscription/API-key/gateway connections, with attended login and independent artifact checks against local or staging targets.
The explicit-only Antigravity additional-agent journey converts the saved-subscription setup and two-turn file walkthrough into one local native cell. It measures agent creation through the ordinary UI, instruction-dependent downloaded bytes, same native conversation, durable Done lineage and owned runtime cleanup. Current support requires an attached self-hosted loopback macOS ARM64 candidate; fresh OAuth, Cloud/remote execution and global release qualification are separate gates.
The bounded personal Muse receiver probe uses ordinary task APIs and retained native receipts to measure unattended receiver behavior. It is a separate manual qualification probe, not a registered Product E2E campaign. Synthetic receiver tests cannot qualify personal Muse; real setup, clarification and fresh-conversation collaboration remain separate release gates.
Paperclip has two live eval families with different questions, owners, and evidence. Choose the family before selecting a model, profile, or case.
The explicit-only native instruction consolidation comparison uses six Product E2E cells per source variant. It measures the completion constraint reduction separately from the earlier native tool-description trial. Provider-free start/resume payload capture is a byte measurement; behavioral qualification requires the original paired live outcomes and retained content. Neither source admission nor a scripted pass proves model behavior.
- Runner Evals: real Runner/provider behavior against a seeded mock control
plane. Definitions live in
paperclip-evals/evals/paperclip-runner; see the direct live protocol evals. - Product E2E Evals: real browser, Paperclip server, database, Runner,
provider, and (where selected) Daytona, using an isolated instance and
grading oracle. See
tests/runner-e2eand Everyday Workflows.
The chat scenario catalog is a planning inventory within Product E2E: natural user requests, assumed context and expected outcomes linked to the feature and agent-access maps. It includes conditional setup, permission, approval and recovery journeys. Its YAML files are not registered runnable cells and contribute no passing coverage or scores.
Runner Evals answer whether a real runner/provider can perform a bounded protocol operation against the expected control-plane contract. Product E2E Evals answer whether a person can complete a product workflow through the real Paperclip surfaces and whether the resulting artifact and state are usable. The names describe the system under test; “headless” is an execution option, not an eval category.
The explicit Product E2E completion-updates suite compares onboarding and
idle, busy, multiple-task, and restart Agent Chat handoffs on native Claude/Codex. It separates mechanical
completion delivery/result access from semantic review of the retained answer;
see the probe contract.
The explicit-only task-titles suite checks that production guidance causes a real native agent to name prompt-only standard/Ask tasks early, while preserving user-supplied titles. Its oracle correlates browser creation, native tool receipts, durable titles, audit ownership, and the reloaded task UI; fixture prompts contain no naming instructions.
The explicit-only native connection guidance suite adds neutral decline prompts, same-task run-attributed explanations, and measured no-use controls across three native local profiles. Its fifteen configured cells are preparation for future matched instruction comparisons, not a live result. Historical Everyday cases and production prompts are preserved.
Selecting a family
Pi's explicit-only pi-controls Product suite tests Stop while native permission
is unanswered and browser-originated same-turn steering, locally and on Daytona.
Its fixture contract separates
control acknowledgment, actual message consumption and owned process retirement.
The current Pi matrix has 26 explicit cells (13 local and 13 Daytona), including
these four controls, pending-native-question controller restart, and a Daytona-only
exact-Pi-child provider-death journey. Provider death must expire the original
unanswered card and reject stale replies without a replacement run; any durable
fallback is distinct from restoration. File editing
also requires native edit/validation events and a registered artifact download.
All remain unqualified until measured on the exact candidate runtime and harness.
Use Runner Evals for a runner protocol, adapter, transport, native session,
tool grant, or one-turn provider qualification question. The workflow checks
out an exact paperclip-evals revision, builds the Runner and viewer, runs a
live roster, and renders the canonical Evalbook report. The control plane is a
seeded test authority, so a passing result does not prove browser UX, production
server behavior, database persistence, Daytona behavior, or a real third-party
mutation.
Use Product E2E Evals for browser interaction, issue/task lifecycle, approval and clarification UI, project/repository selection, persistence over a controller restart, artifact delivery, billing/evidence behavior, or runner continuity in local or Daytona environments. The harness creates a fresh Paperclip instance per cell and uses public APIs and the production browser surface. The suite's Everyday Workflows are Product E2E even when their results are imported into Evalbook.
Do not combine a partial Runner campaign and a partial Product E2E campaign into one score. A campaign is comparable when its definition/grader, model/profile, environment, and contract match. The evaluated Paperclip revision may intentionally differ for a before/after fix comparison; record it as a comparison axis.
Ownership and codepaths
Runner Evals are owned by the Runner/evals maintainers. Definitions, rosters,
case prompts, and the report program live in the sibling private repository
paperclipai/paperclip-evals; Runner integration, viewer, aggregation, and
publication code live under packages/paperclip-runner and the
runner-protocol-live-evals.yml workflow. The public-facing report uses the
same Evalbook renderer and Runner Lab viewer as the trusted report after
sanitization.
Product E2E Evals are owned by the runner E2E maintainers. The catalog and
harness are under tests/runner-e2e; the package scripts are test:e2e:runner,
test:e2e:runner:unit, test:e2e:runner:typecheck, and
test:e2e:runner:report. README.md, FIXTURES.md, SECURITY.md, and
EVERYDAY-WORKFLOWS.md are the detailed sources of truth. The harness starts
the server and embedded database, creates the company/agent/task through the
real APIs, drives Chromium, and invokes the selected local or Daytona runner.
The explicit-only agent-chat-hardening Product E2E suite covers native chat
recovery, hiring, status evidence, and review handoff on local and selected warm
Daytona paths. Its fixture contract distinguishes
startup cancellation from active response cancellation and HTTP send replay
from ambiguous provider action recovery. Select it explicitly; --all excludes it.
The explicit-only production hiring templates suite adds two local native Codex/Claude cells. It exercises API-created production CEO defaults, an explicitly requested hiring skill/reference read, a permanent coder hire, independently computed saved JSON fixtures and worker reuse. Each cell requires five work turns and admits at most two strictly attributed server task-completion turns. Every actual run remains counted; unknown or extra-work turns fail. Source/read coverage and workflow outcome are separate: missing read provenance leaves the candidate/baseline pair uncomparable even if work succeeds. Baseline bundles and coder examples derive from their own source revision, without requiring candidate wording or length.
The explicit-only context-integrity Product E2E suite covers ordered public
comment continuation and explicit invocation of an assigned pinned skill across
the seven selected legacy/native local profiles. Select it by suite or exact
execution ID because --all excludes explicit-only suites. Each cell applies a
1,000-cent company and agent budget hard stop before task creation and records
both limits in its evidence.
The explicit-only stock-harness suite reuses skill, ordered-continuation, and chat-restart journeys across eight local legacy/native profiles with production-default hires. It closes the custom QA manual coverage gap. Its required credential-free prerequisite maps vendor instruction layering, the tiny hire bundle, and shared startup/resume reductions to executable checks. The 24 live cells are configured; no live qualification is claimed from their setup or unit calibration.
The explicit-only agent-chat-stories suite covers the experimental settings
lifecycle for a configured native agent and follow-ups during active work. Its
fixture-driven file wait and persisted-plan oracle are documented in the
Product E2E guide. It does not qualify the native
onboarding wizard or change the native API-tool rollout defaults.
The explicit-only grok-qualification and grok-subscription-qualification
Product suites exercise Grok Build with API and company subscription
authentication respectively. Keep their results separate; the subscription
fixture seeds an explicitly supplied login and does not qualify interactive
login. See the Grok fixture contract.
The explicit Direct blocker guidance suite checks the legacy coordination skill against human authority, missing hiring permission, and requester scope decisions through saved browser interactions.
Validation ladder
The native Codex Daytona recover-controller cell in Everyday Workflows
requires active process and lease continuity, native receipt continuity and
workspace finalization. See the restart evidence contract.
Catalog discovery and deterministic calibration do not establish live recovery.
The explicit-only public MCP suite evaluates paid assistant delegation, later retrieval, feedback, review, uncertain retries and permission boundaries. It uses the Product E2E fixtures, launcher, evidence packaging and dashboard, with separate external-assistant and team-worker billing. The 2026-10-01 results retain two complete model matrices, provenance, costs and the earlier failure history.
Start with credential-free checks and a catalog listing. For Product E2E:
pnpm test:e2e:runner:typecheck
pnpm test:e2e:runner:unit
pnpm test:e2e:runner -- --list
For one explicitly selected local cell, configure only the credentials named
by that cell in .env.runner-e2e.local, then run a narrow ID:
pnpm test:e2e:runner -- --id core-compatibility.runner-codex.local.message-marker
Use the selectors documented in the runner E2E README
for a suite, profile, case, group, or environment. Daytona needs the immutable
image digest and DAYTONA_API_KEY; follow the README and fixture security guide.
--all excludes manual suites such as everyday-workflows. Select that suite
explicitly; use a narrow selector while developing a fixture.
For Runner Evals, the narrowest useful local validation is the report program's
help/validation path and the deterministic Runner checks documented in
runner-workflow-evals.md.
Hosted direct live runs must use the default-branch workflow, an exact 40
character evals_sha, an explicitly selected roster (or the maintained
enabled all campaign), and the protected paid environment. The complete
hosted command is intentionally kept in the workflow and
direct live protocol guide.
Live provider runs can spend money; use the existing workflow authorization and
the user's stated scope when selecting them.
Failure taxonomy
Record the primary failure class and preserve the evidence that supports it.
- Product failure: evidence shows Paperclip or Runner behavior violates the authored case or a hard invariant, such as wrong task state, missing approval gate, lost persistence, bad artifact, or incorrect protocol operation.
- Model/provider behavior failure: the provider turn completed with usable evidence but the model gave the wrong answer, ignored an interaction, failed to complete the authored operation, or violated a semantic assertion. It is scored as behavior, not silently retried as infrastructure.
- Grading/evidence failure: the case or matcher cannot establish its claim, a required recording/screenshot/result is malformed, or the report contract is invalid. Fix the harness or grader before interpreting the score.
- Infrastructure failure: the evidence points to provider/profile unavailability, transport admission failure, service startup failure, a missing credential/image, or inability to produce usable evidence. Startup, transport, and timeout symptoms can instead be product defects when evidence implicates Paperclip or Runner; classify from the observed failure and supported cause, rather than the symptom name alone. Preserve the artifact.
Missing usage or price data means unknown, not free. Keep provider-reported costs separate from estimates, and include retry costs when available. Latency, cleanup, billing coverage, and unpriced usage are dimensions of the result and should remain visible alongside the primary class. A timeout after successful product state reads can be a product behavior failure; a failed server-health read may be infrastructure, but inspect its cause. Use the family-specific classifier and read the attempt evidence before changing an analytical label.
Evidence, provenance, and history
Polling latency is an interval rather than an exact first-detection time. The Product E2E timing contract keeps last-absent/first-present bounds, resolution, strict or inclusive targets, and pass/fail/inconclusive timing verdicts alongside the unchanged workflow grade.
Retained result snapshots and dated measurement reports belong in
paperclip-evals; application tests, Product E2E fixtures/graders, and executable
scenario inventories remain in this repository. Keep a compact results index
with immutable archive links and public report links, as in the
lifecycle baseline.
The private archive is not a dependency of app test execution. Keep large logs,
traces, and videos in the existing campaign artifact storage.
An Evalbook report is a presentation of immutable attempt records, not the
source of truth. Keep the campaign ID, Paperclip commit, paperclip-evals
commit, catalog/roster or definition fingerprint, model/profile, environment,
grader version, selected cells, retries, and provider/runtime usage with the
report. Public projections follow each family's reviewed allowlist and may
include sanitized fixture conversation, named tool outcomes, screenshots, and
structured evidence intended for public history. Credentials, secrets, private
data, raw unredacted records, and hidden reasoning stay out of public
projections.
Distinguish a complete campaign from a partial campaign. A narrow selector, manual diagnostic, missing cell, or infrastructure retry can be useful evidence without being a qualification run. History should retain both, with explicit coverage and completeness, while trend and latest-green views compare only compatible complete campaigns. Refreshing an existing report from retained evidence has zero provider calls and is a new presentation of the old measurement, not a new model run.
Product E2E test details and screenshot galleries show the recorded application
URL when it is available. The worker environment, such as Daytona, does not
imply an application URL. Public projections can retain the reviewed URL in
targetUrl. Test details put recorded checks first and keep raw usage, timing,
and startup evidence under Diagnostics. Optional presentation metadata in a
retained campaign supplies its title, description, notes, and evidence links
inside the shared report shell when the dashboard is regenerated.
Standard runner Product E2E attempts also retain a sanitized taskChat recording
for Test details: task briefs, complete saved messages, requests and answers,
and native agent progress. Related task threads are separate; incomplete capture
is labeled. This automatic capture runs before teardown on success and failure.
See the Product E2E evidence contract.
Existing public histories are available at Runner protocol history and Runner Product E2E history. The consolidated eval hub is at pages.paperclip.ing/evals.
For a repeatable workflow, use the matching skill: paperclip-evals, add-runner-eval, or add-product-e2e-eval.
Diagnose failures before buying another campaign
Use this loop to turn eval failures into product improvements. The unit of work is a broken user outcome or invariant, not an individual red cell.
- Freeze the evidence. Record the inspected application revision and each
campaign's evaluated revision, definition/grader fingerprint, model/profile,
environment, exact selected IDs, attempts, and usage coverage. Read the
history feed and retained attempt records before launching models. A report
refresh, skipped workflow, passing unit suite, or old-definition green cell
is not a new live measurement. Inventory explicit-only suites separately
from
--all. - Reconstruct the failed boundary. Read durable task state, interactions, event chronology, source/worker identity, delivered output, and the failing assertion. State whether the test reached the boundary it claims to test. Separate observed failure, machine class, analytical cause, and confidence. A cancelled queued wake does not by itself prove lost work; a failed decline assertion does not prove unauthorized execution. A saved artifact does not prove the user received a correct completion update.
- Group by cause and product contract. Join cells only when their evidence supports the same mechanism. Check for already-merged fixes and definition corrections before proposing new work. Keep product defects, provider/model behavior, grading defects, infrastructure, and unexercised boundaries distinct. Preserve the original grades when attribution changes.
- Design the smallest general correction. Name the desired user behavior,
the authoritative state/transaction or provider boundary that owns it, the
affected callers, and the existing guarantees that must survive. Check it
against
PRODUCT.mdandSPEC-implementation.md. A change to an intentional product rule is a contract change, not an excuse to delete its guard. Avoid case-name branches, phrase-specific prompts, unconditional retries, or weakening approval, ownership, cancellation, and budget gates to get green. - Prove the mechanism cheaply. Calibrate a grader against correct and plausible wrong retained evidence. Reproduce a product race with scripted providers or service tests. Pair every proposed fix with a regression that exercises the opposite boundary (for example, eligible delivery versus revoked access, benign obsolete wake versus interrupted active work). Presentation-only changes use the existing report refresh path and make no provider calls; do not claim replay can prove changed runtime behavior.
- Select a bounded live confirmation. Write the exact failed representative IDs and only the passing controls affected by the change. Pin the source, definitions, models, and remote image. Record an attempt cap, provider and compute budget, wall-clock deadline, and concurrency before dispatch under the existing live-run authorization. Missing pricing means unknown, not free. Do not rerun unaffected green cells during diagnosis or retry usable behavior failures until they happen to pass. Preserve every attempt. Expand to a compatible qualification campaign only after the causal fix passes.
- Close with evidence and remaining scope. Report original versus new measurements, exact selected coverage, regression results, cost coverage, and unresolved boundaries. A partial verification may close one defect; it cannot turn the full catalog green or establish reliability from one attempt.
Keep one triage record per cause with: affected cell IDs; evidence links; observed failure; supported cause and confidence; existing fix/revision; proposed product contract; invariant regressions; next exact live selection; budget and stop condition; owner; and disposition. Useful dispositions include confirmed product defect, model behavior, grader correction, infrastructure repair, fixed-but-not-remeasured, historical pass, and unqualified coverage.
Parallelize independent artifact inventory, deterministic checks, and isolated cells. Keep a cell's dependent turns ordered. When delegating, use inexpensive agents for bounded extraction, catalog reconciliation, and test execution; keep causal attribution, product design, and final review with the lead. Begin local browser campaigns at the documented conservative concurrency and increase only with measured host headroom. Hosted fanout must respect the workflow's provider and fleet caps; more simultaneous timeouts do not improve wall-clock efficiency.
Current tools support exact-ID selection and retained-evidence report refresh,
but not an automatic cause-aware "rerun unresolved failures" planner. Build an
explicit selection manifest rather than treating --all as that planner. Review
the launcher's automatic retry policy when budgeting; Product E2E may create one
fresh attempt for a retryable failure.
Install the authoring skills
The reviewable sources live in this repository's .agents/skills. For a
multi-repository workspace, install the three skills at
~/paperclipai/.agents/skills (not ~/paperclipai/skills). From the Paperclip
checkout, run:
for skill in paperclip-evals add-runner-eval add-product-e2e-eval; do
install -d "$HOME/paperclipai/.agents/skills/$skill"
install -m 644 ".agents/skills/$skill/SKILL.md" \
"$HOME/paperclipai/.agents/skills/$skill/SKILL.md"
done
This replaces only the three named skill entrypoints. Run it again after updating their tracked sources. Each skill locates the repository independently of its installation directory.
Maintain the public hub
The hub is a static directory with two links to the existing history systems. It displays a dated snapshot, not a live scoreboard. It does not run models, create another result archive, or change the existing campaign URLs.
Build from the public history feeds and check its summary logic:
python3 -m unittest discover -s scripts/evals-hub -p 'test_*.py'
python3 scripts/evals-hub/build.py --output .paperclip/evals-hub
The hub checks need Python 3 and do not call model providers.
For offline checks, pass --history-dir <directory> containing
runner-protocol-evals-history.json and runner-e2e-history.json.
For a pre-merge preview, pass --docs-ref <branch-or-sha> to link the guide
at that revision. The default guide link uses master.
Publish with the Paperclip page helper
and the configured page-uploader credentials. Use Bash 4 or newer; macOS's
system Bash 3 cannot run this helper. On macOS with Homebrew Bash installed,
put $(brew --prefix bash)/bin first in PATH before these commands:
export PAPERCLIP_PAGE_BUCKET=pages.paperclip.ing
export PAPERCLIP_PAGE_BASE_URL=https://pages.paperclip.ing
export AWS_REGION=us-east-1
bash .agents/skills/paperclip-page/scripts/publish.sh .paperclip/evals-hub --slug evals --dry-run
bash .agents/skills/paperclip-page/scripts/publish.sh .paperclip/evals-hub --slug evals
For later refreshes, rebuild in the same output directory and publish with
--update. Keep its ignored .paperclip-page/state.json ownership record;
without that record, the helper will refuse to overwrite an existing prefix.
Verify the public page and its links after publication. This manual refresh
does not add a scheduled workflow. Preserve the measurement date when choosing
a newer rendering of the same campaign.
Remaining native chat boundaries are in the explicit-only
agent-chat-qualification suite: active task reassignment, user Retry after
verified worker process loss, and multi-turn answers grounded in actual task
records. See the workflow and qualification limits.
The 26 native first-task cells inspect the production wizard's saved native
default on qualified targets, without an API runner switch. The 26 Legacy cells
select Legacy explicitly in Advanced. Live results and semantic answer reviews must accompany
any qualification claim; catalog presence alone is not a pass.
The explicit-only native question/resume qualification separates a completed two-answer user journey from semantic-tool documentation qualification. It verifies the exact provider-pause or native response-wake binding for each answer and preserves prior grades when the definition changes.
Lifecycle behavior baseline
The credential-free lifecycle baseline
joins unit, scripted-runner, and database integration assertions to a scenario
inventory before changing narrative-based lifecycle policy. Run
pnpm test:lifecycle-baseline to retain current passes and failures. Its Product
E2E matcher calibration is separate from live execution; unrun live coverage
remains explicitly unmeasured.
The separate live lifecycle baseline
defines 46 real-provider Product E2E cells, including paired narrative probes and
named existing controls on legacy and native Codex. Discover it with
pnpm test:e2e:runner -- --list --suite lifecycle-baseline. Historical execution
results and follow-up coverage are recorded in that suite's guide.
Continuation accounting has an explicit-only eight-cell Product E2E baseline suite, complementing the deterministic lifecycle inventory.
The explicit Product E2E instruction-persistence suite verifies private file
edits, nested and binary agent files, stopped-provider directory saves, server restart, and a fresh task's
downloaded proof on local native/legacy Codex and native Daytona. See the
Product E2E runbook.
The explicit-only Product E2E api-response-reading suite verifies retrieval of
large saved API responses on local and Daytona native Codex runs. See the
Runner E2E guide.
The explicit-only Product E2E extended-harnesses suite covers pending Cursor,
Copilot and Pi ACP profiles on local and Daytona. See the
fixture admission, credentials and budget contract.
The private Runner Evals campaign of the same name provides complementary
semantic protocol cases; catalog membership is not live qualification.
The explicit local Copilot protection fixtures
exercise native permission denial with operator cancellation and bounded attached
command settlement. Their registration remains separate from paid qualification;
source/pack provenance and complete persisted evidence are required.
The explicit-only Product E2E confirmation-replies suite tests conversational
approval and rejection, persisted message provenance, approval before execution,
ambiguous proposals, and the existing card-click path with native Claude/Codex.
See the suite contract.
Hiring notification accounting now also requires exact completed action attribution. Missing native/provider ID mapping is uncomparable evidence; it must not be reported as a model task regression or waived through name/order matching. The fixture waits for both known completion callbacks and settled bracketed observations, including the gap before pending outbox work becomes a wake. Strict action replay and original machine verdicts are retained separately.
The explicit-only planning guidance utility comparison measures task decomposition and handoffs with current, short, and disabled skills.
The explicit-only task-workspaces Product E2E suite covers projectless task files, deferred repository acquisition, controller restart and nested Git/dirty-file persistence on legacy local, native local and native Daytona OpenCode. Its three configured cells use independent filesystem, public API and download oracles; no live qualification is implied by fixture registration.
The explicit-only feedback persistence case checks real native complaint/suggestion submissions and retained bodies with run-attributed tool receipts on local and Daytona execution. Its four native Claude/Codex cells are explicit-only. Remote registrations remain unqualified until retained live attempts prove them. It uses the normal Product E2E report and keeps synthetic feedback distinct from actionable product findings.