mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:55:46 +01:00
DottaandClaude Opus 5.5
5a332517da
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat eval catalog has a case that checks how chat handles a request to set up a throwaway Paperclip instance > - In this case, a scripted operator creates an owned development instance with the supported CLI after the agent answers > - The operator step always failed, so the eval never measured the agent follow-up > - The fixture started `paperclipai run` with an unsupported `--yes` flag, and it discarded the child output, so the cause was not visible > - This pull request starts the owned instance with a supported command, keeps the child output for diagnosis, and replaces reply keyword checks with structural checks > - The benefit is a valid eval that measures the case promise, and a readable cause if the operator step fails again ## Linked Issues or Issue Description **What happened** The `cli-operations.owned-dev-instance` chat eval failed every time with "Stopped waiting for Actual owned development target readiness: Owned dev server exited before readiness". The agent reply was correct. The scripted operator phase failed. The fixture ignored the child stdio, so the evidence did not contain the cause. **Expected behavior** The fixture starts the owned development instance with a supported CLI command. The instance becomes healthy. The eval then grades cleanup, isolation and the agent follow-up. If the child exits, the error names a bounded and redacted reason. **Steps to reproduce** 1. Start a disposable source instance with an embedded database. 2. Copy the checkout like the fixture does, then run `paperclipai worktree init --instance X --home H --from-config <source config> --server-port P --db-port D --seed-mode minimal`. 3. Run `paperclipai run --config <checkout>/.paperclip/config.json --yes`. It exits 1 with `error: unknown option '--yes'`. 4. Run the same command without `--yes`. It exits with `TS2688: Cannot find type definition file for 'node'` because the copied plugin SDK has no linked `node_modules`. ## What Changed - Remove `--yes` from the owned `paperclipai run` command. `run` has no such flag. It onboards only when the config is missing, and `worktree init` writes the config. - Link the installed `node_modules` of every workspace package into the copied checkout. Before, the fixture linked only six packages, so `run` could not compile the copied plugin SDK. - Write the owned CLI output to a private `0600` log under the attempt temp root, outside the owned root that cleanup deletes. - Add a bounded reason (last 12 lines, 1200 characters at most, terminal codes removed, known secrets and secret shapes redacted) to readiness and CLI failure errors. - Replace the `operator-dev-guidance` reply keyword check with `initial-chat-left-host-unchanged`. The check compares the digests of every Paperclip installation config under the attempt root (agent HOME, workspace and runtime directories) before and after the initial turn. - Change `native-reports-real-target` to require only the exact `host:port` that the human receipt supplied (bare, linked or with the scheme). It no longer matches outcome words. - Bump the case `oracleVersion` to `paperclip.chat-catalog.privileged.owned-dev-instance.v2` in code and in the descriptor. ## Verification - Local reproduction with a disposable source instance: the old command exits with `unknown option '--yes'`. Without `--yes` and with the old links, `run` exits with TS2688. With this change, the owned target reports healthy in 16 to 19 seconds. The seeded canary task reads back from the target. `worktree:cleanup` removes the instance. - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains/privileged-operators.test.ts tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts`: 17 passed. - Calibration: the retained CI reply and reworded receipt reports pass. A changed host installation, a missing target, a longer port, a different port, an unattributed reply and an empty initial reply fail. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD sha> --check`: no change. - `npx tsc --noEmit -p tests/runner-e2e`: no errors in the changed files. Remaining errors come from a local dependency mismatch in unrelated packages. - The paid eval `chat-features.runner-opencode.local.cli-operations.owned-dev-instance` is not run yet. ## Risks - Low risk. The change is in one eval fixture and its unit test. - The installation inventory walks the attempt root. It skips `node_modules` and `.git`, and it stops with an error after 1,000,000 entries. - The initial-turn check does not grade the reply wording. A reply that falsely claims to have created an instance, without host changes, needs human review of the retained reply. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>