Files
PaperClipAI/tests
DottaandClaude Opus 5.5 fa50b8d662 fix(evals): give choose-capable-target a saveable computer, grade it structurally and surface setup validation reasons
The fixture saved a built-in fake sandbox as its unsupported control. The
product has refused to save fake sandboxes since #4449 ("reserved for
internal probes"), so setup returned 422 before any agent run, and the
test API withheld the reason. No saveable computer is advertised as unable
to run Claude Code, so the case now contrasts advertised support with
runtime health: the fixture saves an SSH computer on a reserved .invalid
host that is advertised for Claude Code but never tested. The yaml
context records this premise change (manifest changedFields pinned).

The grader no longer reads reply keywords. It requires the agent
boundary (403 on capabilities and probe), a rendered reply link to the
environment settings page that the person then clicks to see both
computers and the checked capabilities, no board-only SSH settings in the
reply, unchanged targets and worker, and exactly one chat turn.

Cookie-session API failures with 400/409/422 now report the product's
own validation message, bounded and with request values and the session
redacted. Details and other statuses stay withheld.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 11:57:55 -06:00
..