mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:38:22 +01:00
DottaandClaude Opus 5.5
b7446969d7
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog evals check that agents answer real user questions through the product. > - The case `execution-environments.choose-capable-target` asks "Which computer can run Morgan with Claude Code?" while environment reads are board-only. > - Its setup saved a built-in fake sandbox as the "unsupported" computer. The product refuses to save fake sandboxes (since #4449), so setup failed with 422 before any agent run, and the test API hid the reason. > - No computer that the product can save is advertised as unable to run Claude Code, so the original contrast cannot be built through supported APIs. > - This pull request changes the contrast to advertised support versus runtime health, grades the reply by structure instead of keywords, and makes setup failures show the product's validation reason. > - The benefit is that the case can measure its user promise, and a future setup 422 says why. ## Linked Issues or Issue Description **What happened** In the full chat catalog run, setup failed with `POST /api/companies/<id>/environments returned 422; body withheld`. A focused server test shows the reason: "Built-in fake sandbox environments are reserved for internal probes and cannot be saved." The fixture payload was `driver: "sandbox", config: { provider: "fake" }`. **Expected behavior** Setup builds its computers only through supported APIs, the agent run starts, and the grader checks the user promise without reply keywords. A setup validation failure states the product's reason. **Steps to reproduce** 1. Run `chat-features.runner-opencode.local.execution-environments.choose-capable-target` on master. 2. Setup posts a fake sandbox environment and gets 422. 3. `cd server && pnpm exec vitest run src/__tests__/environment-routes.test.ts -t "rejects persisted fake sandbox environments"` shows the same rule. ## What Changed - `runtime-capability-choice.ts`: setup saves an SSH computer on a reserved `.invalid` host (no key, never probed) beside the local fixture computer. Production capabilities advertise Claude Code on both. The `enableEnvironments` experimental setting is still set only through `PATCH /api/instance/settings/experimental` and restored in cleanup. - Grader: removed the reply regex and keyword checks. New checks: agent gets 403 on capabilities and probe; the reply renders a link to `/<prefix>/company/settings/instance/environments` (or a target's edit page); the reply has no board-only SSH settings (host, user, path); targets (including status and metadata) and the worker are unchanged; exactly one chat turn and no tasks. `verifyResult` clicks the rendered link and checks that the page shows both computers and loads the same capabilities. - `execution-environments.yaml`: context changed from "one target supports the harness and another does not" to support versus health, with the reason. Recorded in `chat-full-catalog-testops-manifest.json` `changedFields`, pinned in the manifest test, and the inventory digest is updated. - `oracleVersion` is now `paperclip.chat-catalog.runtime-capability-choice.v2` in code and in `chat-catalog-descriptors.json`; `sourceFiles` adds `internal-support.ts`, `server/src/routes/authz.ts` and `server/src/services/environment-config.ts`. - `api.ts`: cookie-session 400/409/422 failures now include the product's `error` message, at most 240 characters, with request string values (8 or more characters), the session cookie and long token-like strings redacted. `details` and all other statuses stay withheld. - Includes a cherry-pick of #20880 (agent environment denials link the environment settings page). Without it, the agent has no route to give. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts` on `runtime-capability-choice.test.ts`, `runtime-computers.test.ts`, `chat-catalog-contract.test.ts`, `chat-catalog-consistency.test.ts`, `api-private-transport.test.ts`, `api.test.ts`, `api-response-reading.test.ts`: 73 passed. - Grader calibration: valid and reworded replies with the settings link pass. Missing, unprefixed, wrong-prefix, cross-origin and wrong-page links fail. Replies with the SSH host, user or path fail. Unsupported capability, reachable host, 200 boundary, changed target or worker, an extra run and a created task fail. - `node --test tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10 passed. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: unchanged. - `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files. - Paid run still needed: `chat-features.runner-opencode.local.execution-environments.choose-capable-target`. ## Risks - Premise change: the case no longer has a saved computer that is advertised as unable to run Claude Code, because the product cannot save one. The "avoid an incompatible target" clause stays in the expected outcome, but no saved target exercises it. - The reply's claims about health are not graded by words. A reviewer must read the paid run's reply. - The 400/409/422 reason can show product text that the test API hid before. It is bounded and redacted. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
76 lines
3.8 KiB
YAML
76 lines
3.8 KiB
YAML
# yaml-language-server: $schema=./catalog.schema.json
|
|
# Planning scenarios; not registered executions or evidence of a pass.
|
|
schema_version: 1
|
|
feature: execution-environments
|
|
status: planned
|
|
scenarios:
|
|
- id: execution-environments.add-computer
|
|
capabilities:
|
|
- execution-environments/configuration
|
|
context: >-
|
|
No SSH computer is configured; the person is an authorized instance operator, but the chat
|
|
agent has no operator identity.
|
|
query: >-
|
|
Add my development server as a computer agents can use.
|
|
expected_outcome: >-
|
|
The agent points to the supported environment setup and explains the small set of
|
|
connection details the operator must provide securely. It does not execute host
|
|
provisioning under ordinary company-agent authority. Completion requires a saved target
|
|
and successful qualification, not just a friendly name in chat.
|
|
- id: execution-environments.choose-capable-target
|
|
capabilities:
|
|
- execution-environments/capabilities
|
|
- runs-adapters/availability
|
|
context: >-
|
|
Two saved computers are advertised as able to run the requested coding harness, and
|
|
neither has been tested for it; one is an SSH computer that cannot be reached. Paperclip
|
|
cannot save a computer that is advertised as unable to run that harness. Environment
|
|
inventory, capabilities and tests are board-only.
|
|
query: >-
|
|
Which computer can run Morgan with Claude Code?
|
|
expected_outcome: >-
|
|
The answer uses available admitted runtime context or directs the person to the
|
|
environment capability view when agent reads are restricted. It distinguishes advertised
|
|
capability from runtime health and avoids selecting an incompatible target or claiming
|
|
hidden inventory was inspected.
|
|
- id: execution-environments.test-computer
|
|
capabilities:
|
|
- execution-environments/probe
|
|
- runs-adapters/validation
|
|
context: >-
|
|
An SSH environment exists but may lack the selected runtime; environment probes require an
|
|
operator.
|
|
query: >-
|
|
Check whether that computer is ready to run Morgan.
|
|
expected_outcome: >-
|
|
The agent routes the person to the actual-target probe and explains its result after an
|
|
authorized run. A failed login or missing runtime is reported as the concrete
|
|
prerequisite. A successful local probe or reachable host alone is not represented as
|
|
qualification of the remote execution environment.
|
|
- id: execution-environments.assign-computer
|
|
capabilities:
|
|
- execution-environments/assignment
|
|
- agents/hire
|
|
context: >-
|
|
Morgan and a qualified cloud computer belong to the same company; the CEO may save this
|
|
configuration and no protected change remains.
|
|
query: >-
|
|
Have Morgan use our cloud computer for future work.
|
|
expected_outcome: >-
|
|
The permitted agent configuration binds Morgan to the intended qualified target and
|
|
persists after reload. Chat explains that future admission uses the new target while an
|
|
active run remains subject to lifecycle rules. Unsupported or protected changes receive
|
|
their own required human step rather than a silent fallback.
|
|
- id: execution-environments.prepare-image
|
|
capabilities:
|
|
- execution-environments/images
|
|
context: >-
|
|
A sandbox provider supports custom images; preparing one requires operator administration.
|
|
query: >-
|
|
Make a reusable cloud image with the tools our website agents need.
|
|
expected_outcome: >-
|
|
The agent helps specify the tool requirements and points the authorized operator to the
|
|
supported image preparation path. The resulting journey must inspect preparation status
|
|
and a usable pinned image before claiming readiness. Ordinary chat cannot manufacture
|
|
provider provisioning rights or treat an image name as a qualified runtime.
|