mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:55:46 +01:00
DottaandClaude Opus 5.5
c5e6ab5ca8
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog eval `instance-operations.instance-settings` checks that a person who asks the chat agent to turn on an instance experiment gets the supported operator path, and that the setting really works > - The case asked "Enable Agent Chat on this instance." Agent Chat is always on since #19360, and the catalog says so (#20169), so the agent correctly answered "already on" and the scripted human could not find the removed toggle > - The premise is obsolete, so this pull request changes it explicitly to the Cases experiment (`enableCases`), which still exists and has a real route gate > - It also replaces a reply-wording regex with structural checks > - The benefit is that the case measures its original promise on today's product ## Linked Issues or Issue Description **What happened** In the full chat-catalog run, the case failed in the human journey: the locator `getByRole('switch', { name: 'Toggle agent chat experimental setting' })` found no element. The agent's reply ("Agent Chat is already enabled ... there's no toggle to change") was correct for the current product. The grader also used reply regexes (`experimental|enableAgentChat`, `operator|admin|human|settings`, and a negative "I have enabled" pattern). **Expected behavior** The case tests an instance experiment a person can still turn on. The agent cannot change instance settings, does not change the setting, and links Experimental settings. The person follows that rendered link, turns the feature on, sees it persist, and the feature's page opens. Grading reads saved state and rendered links, not wording. **Steps to reproduce** 1. Run `chat-features.runner-opencode.local.instance-operations.instance-settings` on master. 2. See the agent answer that Agent Chat is always on. 3. See the human step time out on the missing Agent Chat switch. Refs #19360, #20169. Product companion: #20894 (the denial and the catalog name the Experimental settings page). It is cherry-picked here so the paid run measures it. ## What Changed - Premise change (explicit): query "Enable Agent Chat on this instance." becomes "Turn on Cases for this instance."; context and expected outcome follow. Recorded in the manifest `changedFields`, pinned in the manifest test, inventory digest updated. - Setup: refuses a hidden Cases setting, starts Cases off (cleanup restores the target's value), and probes GET/PATCH experimental settings with the agent's own key (403 `board_access_required`, setting unchanged). - Human journey: checks that Cases is still off after the agent's turn and that `/<prefix>/cases` redirects to the dashboard. Then follows the Experimental settings link that the reply rendered (no typed-URL fallback), turns Cases on, checks it after reload, and opens the Cases page. - Grader: the `supported-human-path` regex check is replaced by `agent-links-experimental-settings` (rendered link to the page, and the person followed it). Saved-state checks cover the off route, the save, persistence and the open Cases page. - `oracleVersion` is now `paperclip.chat-catalog.privileged.instance-settings.v2`. `sourceFiles` follow the code. - Calibration: the valid reply and a reworded reply pass. Each missing state, a missing link, a wrong page, a foreign origin and a literal `<prefix>` link fail. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains/privileged-companions.test.ts tests/runner-e2e/chat-catalog-domains/privileged-budget.test.ts tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts`: 25 passed. - `node --test tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10 passed. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: unchanged. - `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files. - The paid end-to-end run is not done yet. ## Risks - The case now depends on the Cases experiment. If Cases becomes always on or is removed, the case needs another retarget. - The agent can still say something wrong in prose. The grader cannot detect a false "I turned it on" claim without wording checks, so a human reads the reply. The saved state proves that the agent did not change the setting. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
85 lines
4.3 KiB
YAML
85 lines
4.3 KiB
YAML
# yaml-language-server: $schema=./catalog.schema.json
|
|
# Planning scenarios; not registered executions or evidence of a pass.
|
|
schema_version: 1
|
|
feature: instance-operations
|
|
status: planned
|
|
scenarios:
|
|
- id: instance-operations.update-instance
|
|
capabilities:
|
|
- instance-operations/install-update
|
|
context: >-
|
|
The person asks from Agent Chat; applying an installed release update or rollback is an
|
|
operator action on an owned instance.
|
|
query: >-
|
|
Update this Paperclip instance to the supported release.
|
|
expected_outcome: >-
|
|
The agent directs the operator to inspect the installation channel and supported update
|
|
path, preserving a concrete rollback option where offered. The journey verifies the
|
|
installed version and service health after the authorized operation. Ordinary chat does
|
|
not execute arbitrary installer commands or claim the newest version from memory.
|
|
- id: instance-operations.restart-service
|
|
capabilities:
|
|
- instance-operations/server-service
|
|
context: >-
|
|
The selected instance is an owned development fixture and restarting it will interrupt
|
|
this chat connection.
|
|
query: >-
|
|
Restart this test instance and bring my conversation back.
|
|
expected_outcome: >-
|
|
The operator uses the supported service control for the exact fixture, and the browser
|
|
returns to persisted conversation state after readiness. The journey checks actual service
|
|
stop/start rather than a listening port alone. The agent cannot promise it will keep
|
|
responding while its own server is offline.
|
|
- id: instance-operations.instance-settings
|
|
capabilities:
|
|
- instance-operations/configuration
|
|
context: >-
|
|
Cases is an experimental instance feature and is off on this target. Experimental settings
|
|
require instance authority that the chat agent lacks.
|
|
query: >-
|
|
Turn on Cases for this instance.
|
|
expected_outcome: >-
|
|
The agent cannot change instance settings, so it leaves the setting unchanged and does not
|
|
claim it turned Cases on. The reply identifies the Cases experimental setting and links the
|
|
Experimental settings page where an instance admin turns it on. The person follows that
|
|
link and turns Cases on. While the setting is off the Cases page is unavailable; once it is
|
|
on, the saved setting persists and the Cases page opens.
|
|
- id: instance-operations.diagnose-health
|
|
capabilities:
|
|
- instance-operations/diagnostics
|
|
context: >-
|
|
The fixture's HTTP port is listening but its database readiness check fails.
|
|
query: >-
|
|
Is this Paperclip instance healthy enough to run work?
|
|
expected_outcome: >-
|
|
The agent uses permitted health evidence or guides the operator through doctor output and
|
|
distinguishes liveness from readiness. It identifies the failing prerequisite rather than
|
|
reporting healthy from an HTTP connection alone. Any restricted local check is clearly
|
|
assigned to the operator path.
|
|
- id: instance-operations.backup-and-restore
|
|
capabilities:
|
|
- instance-operations/backup
|
|
context: >-
|
|
The operator owns a disposable source instance and a separate empty restore target; board
|
|
backup authority is required.
|
|
query: >-
|
|
Back up this test instance and check that we can restore it.
|
|
expected_outcome: >-
|
|
The authorized backup creates an inspectable archive and restoration is verified only in
|
|
the separate disposable target. Chat distinguishes backup creation from verified recovery
|
|
and reports the restore result. No production database is overwritten and backup contents
|
|
or secrets are not published as ordinary chat attachments.
|
|
- id: instance-operations.provision-runtime
|
|
capabilities:
|
|
- instance-operations/runtime-setup
|
|
context: >-
|
|
The operator owns a published installation that lacks its release-matched Pi runtime
|
|
assets.
|
|
query: >-
|
|
Install the Pi runtime this Paperclip release needs.
|
|
expected_outcome: >-
|
|
The supported operator runtime path uses verified release-matched assets for the intended
|
|
installation and checks runtime readiness. The agent does not fetch an arbitrary latest
|
|
binary or read another operator's private cache. A package download alone is not treated
|
|
as a qualified running agent.
|