Files
PaperClipAI/tests/runner-e2e/chat-scenarios
DottaandClaude Opus 5.5 2eda54162a fix(evals): grade recovery on fresh continuation, accept in-turn queued-comment reads, verify the saved service model (#21210)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog checks that agents in chat do the right thing,
graded on saved product state and rendered links
> - Full cohort 8 (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master 7747487ee7)
failed six cases after two deliberate product changes and one harness
race
> - #21135 continues an unfinished run in a fresh run from saved work
instead of leaving an exhausted, board-owned recovery action. Three
recovery cases still waited for that action.
> - #21175 counts a `get_task_history` read of a queued comment's
current revision as delivery. Two steering cases still required a
continuation run.
> - `companies.prompt-injection-guard` read the settings back before its
own save finished
> - This pull request grades the recovery and steering cases on the new
contracts and makes the settings harness verify the saved model before
it continues
> - The benefit is that the evals accept the intended product behavior
and still fail a broken product

## Linked Issues or Issue Description

Refs #21135
Refs #21175

**What happened?**
Cohort 8 failed `recovery.explain-stall`, `recovery.retry-exhausted` and
`recovery.source-action`: the harness waited 30 s for an "exhausted
source recovery action with no automatic wake". #21135 now logs
"Unfinished run continues in a fresh run from saved work" (`trigger:
recovery_refused`, `delivery: recorded`) instead.
`steering.edit-queued-message` and `agent-chat.multiple-handoffs` failed
their run counts: the running worker turn read the queued comment
(current revision) and applied it, so no continuation run was queued.
`companies.prompt-injection-guard` read `openai/gpt-oss-safeguard-20b`:
the server log shows the harness GET finished before the settings PUT.

**Expected behavior**
Infrastructure loss continues automatically in one fresh run, the chat
explains it, and the continuation finishes the task. A queued comment is
consumed once, in the running turn or in one continuation. The settings
harness tests only the model it saved.

**Steps to reproduce**
Run the six `chat-features.runner-opencode.local.*` cases on master
after #21135 and #21175.

## What Changed

- `runtime-infrastructure.ts` (`recovery.explain-stall`,
`recovery.source-action`, oracle `infrastructure-recovery.v6`): after
three real Runner losses, the setup waits for one system-owned
fresh-continuation record of the failed run (`?inspectSourceRunId`),
resolved as `restored`, `replay: "none"`, the failed run marked
superseded, and one run with `retryOfRunId` = the failed run that waits
for its brief in its own workspace. The chat question is asked while it
runs. The harness then releases the brief, and the grade requires the
continuation to succeed, save the brief's reference as the latest
revision and complete the original task. No person-owned hold and no
blocked task at the reply. Three runs: failed 1, succeeded 2. The reply
must still come from a run that read the recovery record (now also the
continuation run) and link the original task. Setup timing is saved to
`runtime-fresh-continuation.json`.
- `runtime-infrastructure-retry.ts` (`recovery.retry-exhausted`, oracle
`retry-exhausted.v6`): an infrastructure loss no longer offers Retry.
The case uses a failure only the person can resolve
(`native_provider_model_rejected`, a `PERSON_OWNED_FAILURE_CODES`
member) and keeps the Retry-under-budget-hold journey unchanged. The
setup proves the task waits blocked with no fresh continuation and no
successor run for 45 s.
- `runtime-busy-worker.ts`: the Retry listener helpers move here; a
listener counts only when its `.waiting` file is newer than the run's
start.
- `runtime-queue.ts`: `historyReadRevisions`, `readCurrentRevision`,
`runHistoryReads` read `task_history.read` events; `settledQueue` needs
a 30 s quiet settled state.
- `runtime-queue-controls.ts` (`steering.edit-queued-message`, oracle
`structural.v5`) and `runtime-multiple-handoffs.ts` (oracle `v5`):
accept in-turn consumption only with a worker `task_history.read` of
every queued comment's current revision, the change saved as the latest
revision (blue; "Launch sooner"), an empty queue, a done task and zero
continuations. The continuation path stays: exactly one continuation
with the unread comments in order, when the turn did not read the
current revision. Run bounds 2–3 and 4–5. The worker brief no longer
tells the agent to leave follow-ups for a continuation.
- `identity-company-models.ts` (`prompt-injection-guard` oracle
`prompt-injection-guard.v2`, `fast-responses` `v4`): before saving,
check the model field holds the requested model; wait for the PUT;
require 200, the sent model, the saved model, then a matching read-back.
- `recovery.yaml`: context and expected outcome for the three recovery
cases. Recorded in `chat-full-catalog-testops-manifest.json`
`changedFields` and pinned in its test. Descriptors and the inventory
digest are updated. `maps/feature-map/recovery.md` exhaustion line
corrected.

## Verification

- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog tests/runner-e2e/chat-catalog-domains`:
283 files, 3895 tests pass. New calibration tests cover valid,
reworded-valid and broken cases: no continuation, a person-owned hold,
two continuation records or runs, a replayed turn, an unfinished
continuation, a missing saved result; an old-revision-only read, a
missing outcome, a double application, two continuations; a model field
that lost the model, a different sent model, a wrong saved model.
- `pnpm test:e2e:runner:typecheck`: pass.
- `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10/10
pass.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <head sha>
--check`: `changed: false`.
- `node scripts/check-maps.mjs`: aligned.
- Paid run: Runner Full-Stack E2E 43321773136 at 9de2bd5a6f, six cases:
prompt-injection-guard passed. The two fresh-continuation cases found a
product failure (the continuation never started), fixed in #21217.
retry-exhausted showed that a model-rejected failure offers no Retry
control. multiple-handoffs failed correctly: the turn read the
instruction and did not apply it. edit-queued-message did not reach the
new grade (6-minute CEO turn). Details are in a PR comment.

## Risks

- The recovery cases depend on the fresh continuation starting within 5
minutes. The paid run showed it did not start; #21217 fixes that. Until
#21217 merges, those cases fail on master.
- `recovery.retry-exhausted` expects a Retry control that the product
does not show for a model-rejected failure. It fails until a product
decision is made.
- `recovery.retry-exhausted` no longer covers infrastructure exhaustion;
the two other recovery cases cover the new automatic path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 17:26:50 -06:00
..
…

Chat feature scenarios

This is the planning catalog for what happens when a person asks an agent to use Paperclip. It belongs to Product E2E, alongside the existing chat and Everyday Workflows suites. These files contain human requests, assumed context and loosely specified outcomes. Entries alone do not run models, register tests, or contribute passing results to a report. Two task entries now have explicit executable Product E2E cases: tasks.create-backlog and tasks.delegate-announcement. Their results remain in the existing E2E reports.

The feature map answers what the product lets a person do. The agent access map records which APIs and actual agent entry points exist and their boundaries. This catalog asks whether a person can accomplish that work through a conversation, including discovering a prerequisite, completing a human step and returning to work. All three reuse the same capability IDs.

Start with these requests

  • Hire a Claude Code engineer: agents.hire-claude-code, with missing-account, approval-required, denied-permission and ambiguous-wording cases.
  • Create a website task: tasks.website-task, with missing-project, attached-brief, ambiguous-task, privacy and ownership-conflict cases.
  • Archive completed inbox items: inbox-search.archive-done, with missing authority, partial failure and undo cases.
  • Read Notion: connection-setup.notion-connected and connection-setup.notion-missing deliberately use the same sentence. The second requires setup in the conversation and continuation after usable access exists. Other cases cover decline, wrong account, unassigned agent and expired access.

Reading a scenario

Each area file uses catalog.schema.json and implicitly inherits defaults.yaml. Scenario context overrides those defaults. A scenario is independent: mentions of previous turns or existing resources are fixture requirements, not dependencies on running an earlier catalog entry.

Field Meaning
schema_version Version of this planning format; currently 1.
feature Primary feature-map filename without .md.
status Always planned; execution results belong to the existing E2E system.
scenarios[].id Stable, globally unique <feature>.<scenario> identifier.
capabilities Full feature-map IDs exercised by this request; may span areas.
context Existing records, current identity, authority, accounts, gates and significant starting conditions, plus any later human action or fault needed for the branch.
query The natural-language message a person would send, including an answer or follow-up where that is the subject of the case.
expected_outcome A paragraph describing the visible conversation, resulting durable state, necessary human interaction and meaningful boundaries.

For example, two fixtures can send “Read my Notion and tell me the latest on the launch.” In the connected fixture, the answer comes from current pages and links back to them. In the unconnected fixture, the agent requests setup, the addressed person completes it, and the original work continues. A third fixture declines setup and should receive a truthful explanation without any provider read.

Context and expected outcomes are evaluator material. Do not paste them into an agent's system prompt or teach the model which API to choose. A future fixture should expose the actual product state, ordinary production instructions, the human query and any genuine attachments or previous conversation. The model must discover the available operations normally.

What coverage means here

Every current feature-map capability has at least one associated scenario. This is inventory coverage, not complete operation coverage, source verification or live acceptance. Broad capabilities can still need more CRUD, actor, transport, provider and lifecycle variations. The maps' Policy and Gaps sections remain the source for access assumptions; recheck them against implementation when promoting an entry. The YAML does not grant permissions or promise an unimplemented UI.

The default starting point is a person talking to their CEO in Agent Chat with legitimate agent permissions. CEO is not board identity. A valid expected outcome can therefore be direct action, a grounded answer, a necessary clarification, a connection or approval request, a precise human handoff, or a truthful boundary. An unsupported operation must not be “passed” because the agent says it succeeded. A vague “you can do this in settings” is also insufficient when a supported specific control or request card exists.

Some capabilities cannot originate inside an already working chat: first server installation, login, an unfinished first-agent wizard, browser preferences and developer-only labs. Their entries explicitly describe the relevant boundary or companion journey. External MCP and channel cases also name their different surface. Preserve those labels in future selection and reporting; a successful operator, browser-only or external-client step cannot certify a direct Agent Chat action. Where only guidance is currently possible, qualifying automated chat execution requires a separate product change and updated access contract.

Conditional coverage to preserve

Condition Representative scenarios
Account ready / missing / wrong / expired / not assigned connection-setup.notion-*
Allowed / human approval / denied / wrong resolver agents.hire-*, questions-and-approvals.wrong-reviewer
Unambiguous / missing required detail / ambiguous reference tasks.website-task, questions-and-approvals.necessary-question, tasks.ambiguous-task
Complete list / partial mutation / undo tasks.organize-open-work, inbox-search.archive-partial-failure, inbox-search.restore-archive
Same company / other company / private resource access.other-company, access.private-resource, tasks.private-task
Current decision / stale decision / duplicate answer questions-and-approvals.ambiguous-yes, questions-and-approvals.review-plan, questions-and-approvals.duplicate-answer
Idle / busy / steering / stop / saved continuation steering.*, recovery.post-stop-message
Known result / uncertain side effect / failed accounting recovery.unknown-email-outcome, budgets-costs.repair-cost-recording
Feature enabled / disabled / different surface cases.cases-disabled, agent-chat.chat-disabled, developer-labs.gated-lab
Saved record / actually ready / independently usable output runs-adapters.validate-hire, workspaces.preview-service, documents-artifacts.produce-download

Different starting conditions should usually be separate scenarios with the same query, so one favorable fixture cannot satisfy all branches. Outcomes can mention invariant guards without turning each paragraph into a complete executable test. When implementing a branch that currently contains several human choices, split those choices into explicit cells while preserving the planning scenario's ID as traceability metadata.

Promoting a scenario later

  1. Read the matching product recipe and access companion. Confirm the operation, actual admitted agent surface and permission rules; keep current gaps visible. Decide whether this is an Agent Chat journey or a named companion boundary.
  2. Reuse or extend the existing Product E2E fixture, catalog and report paths. Everyday Workflows already cover hiring/reuse, delegation, connection decisions and delivered work. Existing chat-hardening, chat-qualification, completion-update and confirmation suites are also documented in the E2E guide. Overlap is a reuse opportunity, not evidence that this exact query and context have passed.
  3. Create isolated synthetic state through supported production setup surfaces. Pin the clock, identities, policy, runtime and provider profile; record feature gates and owned test connections. Keep actual secret material out of this catalog. Follow fixture and security rules.
  4. Send the query through the real conversation UI. Drive each specified human answer, connection callback, navigation, reconnect or fault at its actual boundary. Preserve the source chat and linked tasks across those transitions.
  5. Turn the outcome into independent assertions: rendered response and correct links; saved records and attribution; exact target and audience; actual file bytes or provider receipt; absence of unintended mutations; correct return to work. Validate the oracle against a plausible wrong outcome. Success prose, a tool declaration or an accepted HTTP response alone cannot pass a journey.
  6. Declare the ordinary E2E execution selection, environment, turn/time/cost limits, cleanup and evidence contract. Retain failures, retries, source and definition revisions, actual model/profile, usage and missing evidence in the existing report. A changed setup or model is a separate measurement, and a missing observer is not a pass. No live or paid runs are authorized by these planning files themselves.

The first useful runnable slice is the user's three core jobs plus the paired Notion branches above. Extend from that slice across this catalog without building a second runner, scheduler, scoring system or evidence store.

Maintaining the catalog

Keep area filenames aligned with the feature map and IDs stable when wording changes. Add a capability reference only when the query and outcome exercise it; linking every scenario to every related subsystem would inflate coverage. Write specific context and an observable paragraph rather than mechanically repeating a feature description. Use cross-feature references for real combined journeys.

Before handing off edits, parse every YAML file, validate area files against the schema, check unique IDs and nonempty prose, and reconcile capability references with the Sub-features keys in maps/feature-map/*.md. Check both missing and unknown IDs. The schema alone cannot validate references or uniqueness across files. defaults.yaml is shared context and is not an area-schema document.

chat-scenario-catalog.test.ts performs these structural checks in pnpm test:e2e:runner:unit, including the 35 areas, 244 scenario IDs and 191 mapped capabilities. Update the inventory counts intentionally when the catalog grows.

After editing map links, also run node scripts/check-maps.mjs. That existing check validates the maps, not this YAML catalog and not live chat behavior. Keep structural validation and any future execution evidence separate. Do not label a scenario passed, register a paid cell or alter default suite selection merely because its planning text has been reviewed.

Browse by feature

The feature recipe and same-named access companion define each area's behavior and authority. Each YAML file can also reference capabilities from other areas.

Scenarios Product recipe Agent access
access Feature Access
activity Feature Access
agent-chat Feature Access
agents Feature Access
app-permissions Feature Access
assistant-connections Feature Access
budgets-costs Feature Access
cases Feature Access
chat-channels Feature Access
cli-operations Feature Access
companies Feature Access
connection-setup Feature Access
developer-labs Feature Access
documents-artifacts Feature Access
execution-environments Feature Access
gateways-profiles Feature Access
goals Feature Access
inbox-search Feature Access
instance-operations Feature Access
navigation-preferences Feature Access
onboarding Feature Access
plugins Feature Access
projects Feature Access
questions-and-approvals Feature Access
recovery Feature Access
routines Feature Access
runs-adapters Feature Access
secrets Feature Access
skills Feature Access
status-cards Feature Access
steering Feature Access
task-coordination Feature Access
tasks Feature Access
teams Feature Access
workspaces Feature Access