Files
PaperClipAI/maps/feature-map
DottaandClaude Opus 5.5 2eda54162a fix(evals): grade recovery on fresh continuation, accept in-turn queued-comment reads, verify the saved service model (#21210)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog checks that agents in chat do the right thing,
graded on saved product state and rendered links
> - Full cohort 8 (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master 7747487ee7)
failed six cases after two deliberate product changes and one harness
race
> - #21135 continues an unfinished run in a fresh run from saved work
instead of leaving an exhausted, board-owned recovery action. Three
recovery cases still waited for that action.
> - #21175 counts a `get_task_history` read of a queued comment's
current revision as delivery. Two steering cases still required a
continuation run.
> - `companies.prompt-injection-guard` read the settings back before its
own save finished
> - This pull request grades the recovery and steering cases on the new
contracts and makes the settings harness verify the saved model before
it continues
> - The benefit is that the evals accept the intended product behavior
and still fail a broken product

## Linked Issues or Issue Description

Refs #21135
Refs #21175

**What happened?**
Cohort 8 failed `recovery.explain-stall`, `recovery.retry-exhausted` and
`recovery.source-action`: the harness waited 30 s for an "exhausted
source recovery action with no automatic wake". #21135 now logs
"Unfinished run continues in a fresh run from saved work" (`trigger:
recovery_refused`, `delivery: recorded`) instead.
`steering.edit-queued-message` and `agent-chat.multiple-handoffs` failed
their run counts: the running worker turn read the queued comment
(current revision) and applied it, so no continuation run was queued.
`companies.prompt-injection-guard` read `openai/gpt-oss-safeguard-20b`:
the server log shows the harness GET finished before the settings PUT.

**Expected behavior**
Infrastructure loss continues automatically in one fresh run, the chat
explains it, and the continuation finishes the task. A queued comment is
consumed once, in the running turn or in one continuation. The settings
harness tests only the model it saved.

**Steps to reproduce**
Run the six `chat-features.runner-opencode.local.*` cases on master
after #21135 and #21175.

## What Changed

- `runtime-infrastructure.ts` (`recovery.explain-stall`,
`recovery.source-action`, oracle `infrastructure-recovery.v6`): after
three real Runner losses, the setup waits for one system-owned
fresh-continuation record of the failed run (`?inspectSourceRunId`),
resolved as `restored`, `replay: "none"`, the failed run marked
superseded, and one run with `retryOfRunId` = the failed run that waits
for its brief in its own workspace. The chat question is asked while it
runs. The harness then releases the brief, and the grade requires the
continuation to succeed, save the brief's reference as the latest
revision and complete the original task. No person-owned hold and no
blocked task at the reply. Three runs: failed 1, succeeded 2. The reply
must still come from a run that read the recovery record (now also the
continuation run) and link the original task. Setup timing is saved to
`runtime-fresh-continuation.json`.
- `runtime-infrastructure-retry.ts` (`recovery.retry-exhausted`, oracle
`retry-exhausted.v6`): an infrastructure loss no longer offers Retry.
The case uses a failure only the person can resolve
(`native_provider_model_rejected`, a `PERSON_OWNED_FAILURE_CODES`
member) and keeps the Retry-under-budget-hold journey unchanged. The
setup proves the task waits blocked with no fresh continuation and no
successor run for 45 s.
- `runtime-busy-worker.ts`: the Retry listener helpers move here; a
listener counts only when its `.waiting` file is newer than the run's
start.
- `runtime-queue.ts`: `historyReadRevisions`, `readCurrentRevision`,
`runHistoryReads` read `task_history.read` events; `settledQueue` needs
a 30 s quiet settled state.
- `runtime-queue-controls.ts` (`steering.edit-queued-message`, oracle
`structural.v5`) and `runtime-multiple-handoffs.ts` (oracle `v5`):
accept in-turn consumption only with a worker `task_history.read` of
every queued comment's current revision, the change saved as the latest
revision (blue; "Launch sooner"), an empty queue, a done task and zero
continuations. The continuation path stays: exactly one continuation
with the unread comments in order, when the turn did not read the
current revision. Run bounds 2–3 and 4–5. The worker brief no longer
tells the agent to leave follow-ups for a continuation.
- `identity-company-models.ts` (`prompt-injection-guard` oracle
`prompt-injection-guard.v2`, `fast-responses` `v4`): before saving,
check the model field holds the requested model; wait for the PUT;
require 200, the sent model, the saved model, then a matching read-back.
- `recovery.yaml`: context and expected outcome for the three recovery
cases. Recorded in `chat-full-catalog-testops-manifest.json`
`changedFields` and pinned in its test. Descriptors and the inventory
digest are updated. `maps/feature-map/recovery.md` exhaustion line
corrected.

## Verification

- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog tests/runner-e2e/chat-catalog-domains`:
283 files, 3895 tests pass. New calibration tests cover valid,
reworded-valid and broken cases: no continuation, a person-owned hold,
two continuation records or runs, a replayed turn, an unfinished
continuation, a missing saved result; an old-revision-only read, a
missing outcome, a double application, two continuations; a model field
that lost the model, a different sent model, a wrong saved model.
- `pnpm test:e2e:runner:typecheck`: pass.
- `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10/10
pass.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <head sha>
--check`: `changed: false`.
- `node scripts/check-maps.mjs`: aligned.
- Paid run: Runner Full-Stack E2E 43321773136 at 9de2bd5a6f, six cases:
prompt-injection-guard passed. The two fresh-continuation cases found a
product failure (the continuation never started), fixed in #21217.
retry-exhausted showed that a model-rejected failure offers no Retry
control. multiple-handoffs failed correctly: the turn read the
instruction and did not apply it. edit-queued-message did not reach the
new grade (6-minute CEO turn). Details are in a PR comment.

## Risks

- The recovery cases depend on the fresh continuation starting within 5
minutes. The paid run showed it did not start; #21217 fixes that. Until
#21217 merges, those cases fail on master.
- `recovery.retry-exhausted` expects a Retry control that the product
does not show for a model-rejected failure. It fails until a product
decision is made.
- `recovery.retry-exhausted` no longer covers infrastructure exhaustion;
the two other recovery cases cover the new automatic path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 17:26:50 -06:00
..

Paperclip feature map

Start here when reproducing a user-facing bug, verifying a change, or deciding which surfaces a fix must cover. Each recipe describes what a person can do, where they can do it, the expected result, and the evidence needed to verify it. Product behavior is still governed by the code and its tests.

The sibling agent access map records APIs, agent entry points and permission boundaries for the same capability IDs. Follow the shared map contract when maintaining either map.

The chat scenario catalog describes how people might request these capabilities in conversation, including setup and permission branches. These are planned Product E2E journeys, not live coverage results.

IDs and references

Use the existing IDs in issues, PRs, Storybook annotations, tests, and verification evidence:

Reference Example Defined by
Feature ID onboarding The recipe filename without .md.
Capability ID onboarding/company The feature ID plus a key from its Sub-features list, separated by /.
Entry-point reference onboarding.md#web-wizard The recipe filename plus its entry-point heading. The local entry-point ID is web-wizard.

For example, report verifying onboarding/company through web-wizard. Capabilities describe behavior; entry points identify the surface used to exercise it. Entry-point IDs are separate from generated capability IDs. Both maps and Storybook use the same full capability IDs. Each child page states its feature ID; recipe bullets show the full capability ID beside the local key, and access companions show it beneath the local heading. Local keys remain unchanged for parsing and existing anchors.

Keep recipe filenames, sub-feature keys, and entry-point keys stable when changing display titles or navigation. Renaming a filename or key changes its references. The section headings below are navigation labels, not additional IDs.

Browse features

Use this index to open a recipe or copy its feature and full capability IDs.

Getting started and access

Feature ID Recipe Capability IDs
onboarding Onboarding and first work onboarding/instance, onboarding/company, onboarding/adapter, onboarding/first-task, onboarding/resume
access Login, invitations, and access access/login, access/invitations, access/membership, access/resources, access/cli-auth
companies Companies and portability companies/selection, companies/identity, companies/decision-models, companies/prompt-injection-prevention, companies/fast-responses, companies/agent-observability, companies/archive, companies/portability

Tasks and conversations

Feature ID Recipe Capability IDs
tasks Task creation and lifecycle tasks/create, tasks/organize, tasks/properties, tasks/discussion, tasks/completion, tasks/privacy
task-coordination Delegation, dependencies, and signoff task-coordination/delegation, task-coordination/dependencies, task-coordination/signoff, task-coordination/tree-control
inbox-search Inbox, decisions, and search inbox-search/views, inbox-search/triage, inbox-search/decisions, inbox-search/search, inbox-search/toasts
agent-chat Agent conversations and project handoff agent-chat/discovery, agent-chat/history, agent-chat/handoff, agent-chat/conference-room
questions-and-approvals Questions and approvals questions-and-approvals/questions, questions-and-approvals/dismiss-and-reopen, questions-and-approvals/confirmations, questions-and-approvals/plan-review, questions-and-approvals/audience, questions-and-approvals/governance, questions-and-approvals/tool-review, questions-and-approvals/continuation
steering Steering and queued messages steering/queue, steering/edit-order-discard, steering/steer, steering/interrupt, steering/stop-and-pause, steering/drafts, steering/approval-queue
documents-artifacts Documents, attachments, and work products documents-artifacts/documents, documents-artifacts/annotations, documents-artifacts/attachments, documents-artifacts/work-products, documents-artifacts/library

Agents and reusable capabilities

Feature ID Recipe Capability IDs
agents Hiring, configuration, and organization agents/hire, agents/grok-bot, agents/instructions, agents/organization, agents/primary-agent, agents/lifecycle, agents/built-in, agents/api-keys, agents/managed-agent-setup
runs-adapters Runs, harnesses, and model accounts runs-adapters/availability, runs-adapters/validation, runs-adapters/runs, runs-adapters/session
skills Skills and Skill Studio skills/discovery, skills/authoring, skills/versions, skills/policy, skills/testing
teams Team packages and installation teams/discovery, teams/preview, teams/install, teams/configuration

Projects and execution

Feature ID Recipe Capability IDs
projects Projects and repositories projects/lifecycle, projects/task-context, projects/repositories, projects/defaults, projects/privacy
goals Goals and work alignment goals/hierarchy, goals/alignment, goals/navigation
workspaces Workspaces, services, and files workspaces/binding, workspaces/provisioning, workspaces/services, workspaces/files-git, workspaces/closure
execution-environments Execution environments execution-environments/configuration, execution-environments/capabilities, execution-environments/probe, execution-environments/assignment, execution-environments/images
recovery Recovery recovery/diagnosis, recovery/bounded-continuation, recovery/source-action, recovery/exhaustion, recovery/workspace, recovery/unknown-outcome, recovery/user-continuation, recovery/reconnect

Apps, channels, and extensions

Feature ID Recipe Capability IDs
connection-setup Connection setup connection-setup/catalog, connection-setup/identity-and-access, connection-setup/credentials, connection-setup/handoff, connection-setup/draft-recovery, connection-setup/task-request, connection-setup/channel-setup, connection-setup/failure
assistant-connections Assistant connections (MCP) assistant-connections/setup, assistant-connections/consent, assistant-connections/device-login, assistant-connections/client-use, assistant-connections/revocation
app-permissions App access and action permissions app-permissions/agent-access, app-permissions/action-policy, app-permissions/catalog-change, app-permissions/test-action, app-permissions/account-maintenance
chat-channels External chat and email chat-channels/providers, chat-channels/identity, chat-channels/conversation, chat-channels/attachments, chat-channels/lifecycle, chat-channels/outbound-email
gateways-profiles Tool gateways and access profiles gateways-profiles/gateway, gateways-profiles/profiles, gateways-profiles/client-connect, gateways-profiles/tokens, gateways-profiles/activity
secrets Secrets and proposals secrets/company-secrets, secrets/user-secrets, secrets/proposals, secrets/vault-import, secrets/runtime-delivery
plugins Plugins plugins/installation, plugins/configuration, plugins/contributions, plugins/lifecycle

Automation and structured work

Feature ID Recipe Capability IDs
routines Routines, schedules, and triggers routines/definition, routines/triggers, routines/run-now, routines/history, routines/management
cases Cases cases/browse, cases/structure, cases/history, cases/task-links, cases/handoffs
status-cards Status cards status-cards/authoring, status-cards/generation, status-cards/inspection, status-cards/archive

Oversight and operation

Feature ID Recipe Capability IDs
budgets-costs Costs and budgets budgets-costs/cost-reporting, budgets-costs/budgets, budgets-costs/incidents, budgets-costs/resolution, budgets-costs/subscriptions, budgets-costs/accounting-recovery
activity Dashboards and audit trails activity/overview, activity/activity, activity/runs, activity/routine-audit, activity/timeline, activity/agent-commentary
navigation-preferences Navigation, profile, and announcements navigation-preferences/navigation, navigation-preferences/preferences, navigation-preferences/profile, navigation-preferences/announcements, navigation-preferences/fallbacks, navigation-preferences/command-palette
instance-operations Instance operations instance-operations/install-update, instance-operations/server-service, instance-operations/configuration, instance-operations/diagnostics, instance-operations/backup, instance-operations/runtime-setup
cli-operations CLI/API and local worktrees cli-operations/context, cli-operations/domain-commands, cli-operations/outputs, cli-operations/run-controls, cli-operations/worktree-instances, cli-operations/feedback-export

Contributor surfaces

Feature ID Recipe Capability IDs
developer-labs Developer previews and diagnostic labs developer-labs/design, developer-labs/interaction-labs, developer-labs/performance, developer-labs/route-gates

Open the published Storybook guide for the assembled feature and capability index. The Markdown recipes in this directory are the authored source. Each has a same-named agent-access companion.

Run node scripts/storybook-catalog.mjs to generate local navigation in maps/generated/feature-map.md, maps/generated/agent-access-map.md, and the Storybook guide. Recipes are discovered automatically from their navigation metadata; generation does not depend on this high-level index.

Before driving a journey

  1. Use this checkout's isolated test drive for manual product checks. It creates a temporary instance and prints its URL and data directory. Do not guess that a server on port 3100 belongs to you.
  2. Record the commit, URL, company, login role, relevant experimental settings, adapter, runtime mode, and live versus simulated dependencies. Use disposable tasks and provider resources. A fresh test drive creates a company and CEO, but no task or first run.
  3. Use the current navigation and company prefix. Paths in recipes omit that prefix. Record whether the streamlined or production shell is selected; Agent Chat, chat connectors, and the combined Inbox/Tasks view have separate gates. A hidden surface is an unmet prerequisite, not a successful test.
  4. Run the smallest relevant test first. Vitest commands below run from the repository root after installing dependencies. Playwright recipes use their named configuration and its isolated environment, not an unrelated running instance. Reserve expensive runner/provider tests for the behavior at issue.
  5. Exercise the actual user action, inspect its result, reload, and verify the persisted state or continuation. Use read-only API checks to corroborate UI evidence; creating state through the API does not prove the creation UI.

For agent-driven acceptance work, start with the Paperclip verification skill for an isolated launch, ownership/readiness checks, browser driving, evidence, and cleanup. It uses this directory as its feature map. The existing dev-workspace run/verify skill and evaluation skill describe runtime ownership and evidence handling. Reuse them; this map introduces no environment launcher, credentials, scheduled job, or second test framework.

Evidence contract

Report each entry point as passed, failed, blocked (with its missing prerequisite), or not run. Include the feature ID, full capability IDs and entry-point ID (for example, onboarding/company through web-wizard), plus the recipe link, commit/environment, user action, expected and observed result, command/exit code, and evidence links. Screenshots or traces should show both the action and the discriminating result. Include task/run/request IDs when relevant, without secrets.

A fix is verified across its affected surfaces only when each has evidence or an explicit reason it does not apply. Shared code alone does not prove parity. Keep product regressions visible; do not change the recipe to bless a failure. For Paperclip-assigned work, attach evidence through the artifact workflow.

Inventory and remaining depth

The generated maps/generated/coverage.json lists the current non-test TSX page modules and the recipes that directly reference each one. An unreferenced module is a documentation candidate, not proof that its feature is absent or untested. Remaining workflow limits live in each recipe. Inventory and example associations describe documentation scope, not runtime health.

Maintaining the reference

The map documents user journeys. It does not execute them automatically; generated inventories record documentation scope rather than live test health. Product E2E reports reuse these capability IDs through the journey associations. Their coverage view shows which scenarios in a campaign are linked to each capability and the associated test results. An association or passing scenario does not certify the whole capability; unselected and unmapped journeys remain distinct.

When maintaining a recipe, compare its entry points and expected results with the current source. Check that linked tests still exercise the stated behavior and that local references still exist. Keep the same-named agent access companion and stable sub-feature IDs aligned when behavior or access changes. Source-only access inventory remains distinct from live verification. Run node scripts/check-maps.mjs after map edits for structural/reference checks; follow the shared contract for the source and behavioral checks it cannot perform.

Each recipe starts with a small frontmatter navigation record, then an H1:

---
navigation: {"section":"Tasks and conversations","sectionOrder":20,"order":10}
---

The H1 supplies the feature title, and the first paragraph supplies its summary. Orders are local to the section; leave gaps for additions. Equal orders use the stable filename as a tie-breaker. Keep a section's name/order consistent across its recipes. Discovery uses the recipe and companion; keep the readable ID index above aligned when features or capability IDs change.

Optional ### Storybook verification guidance lives inside the recipe's Gotchas section. Each level-four heading contains a backtick-quoted capability key and has Scope (ui, cli, or browser), Storybook, Production, and Sources fields. These are verification instructions and intentional scope limits, not run results or a required story for every API. The generated gap report combines this guidance with current story associations.

After frontmatter, each recipe starts with an H1 and a user-visible description, then exactly:

  1. Sub-features — stable backticked local keys, full capability IDs, and observable behavior/states, such as - `company`: `onboarding/company` — create a company.
  2. How to get to it (user POV) — one H3 backticked entry-point ID per surface.
  3. Driving it with — starts with Preconditions:; repeat each entry-point H3, with Automated: and Manual: paragraphs. Name test scope and gaps honestly.
  4. Gotchas — misleading look-alikes, feature gates, and invalid evidence.

New or updated entries follow Poteto's feature-map examples. Existing entries with a Driving it heading can adopt the named harness when their recipes are updated. The original format was inspired by Omnigent's feature map. Paperclip's recipes and checks follow its own task model and test infrastructure.