mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:55:46 +01:00
dfd2a6efcab84bc19106a2dbf720eb53cb2282d3
16
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude Opus 5.5
|
afcf34c9da |
fix(evals): form repair-cost-recording's saved-usage debt and grade it by saved state (#20895)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat feature evals check that the chat agent helps a person with real product state > - The `budgets-costs.repair-cost-recording` case checks that the agent sends the person to the saved-cost retry control and then reports the true accounting result > - The grader checked the agent's replies with keyword regular expressions, and the scripted person gave the agent the harness's own accounting readback > - Keyword checks reject valid wording and accept wrong answers, so the case did not measure the promise > - The setup also never formed: the prerequisite run always settled, so no "Ready to record" saved usage existed > - This pull request forms that state with a temporary trigger on the case-owned database (no product change), and replaces the two keyword checks with checks of saved state, the agent's own product calls, and rendered links > - The benefit is a case that can run, and a grader that does not depend on wording and that still fails the relevant broken behavior ## Linked Issues or Issue Description **What happened** The case `budgets-costs.repair-cost-recording` graded the chat replies with keyword regular expressions (`honest-post-retry-accounting-result` and `guides-accounting-without-rerun`). The scripted follow-up message also told the agent the harness's accounting result. Separately, the setup never formed in paid runs: the prerequisite native run settles as an unpriced cost event (`missing_price`), not as a `retryable` saved-usage row. Accounting retries every 5 seconds, so a retryable row only exists while accounting keeps failing. **Expected behavior** The grader accepts any wording. It requires that the guidance before the retry links the Costs page, that the post-retry run reads saved cost data before it answers, and that the agent does not write cost, budget or accounting records or start work. **Steps to reproduce** 1. Read `gradeAccounting` in `tests/runner-e2e/chat-catalog-domains/identity-accounting.ts` on master. 2. Reword a valid reply (for example "Head to Costs and press the button there; the work stays as it is."). 3. See that the keyword checks reject it. Refs #20600 ## What Changed - New `tests/runner-e2e/chat-accounting-fault.ts`, following `chat-archive-storage-fault.ts`: a temporary trigger on the case-owned database refuses cost-event writes for the one prerequisite task. The product's own accounting transaction rolls back, so the finished run keeps its saved usage pending and the product's automatic retries keep failing. Refused writes are counted in a rollback-proof sequence. It is allowed only when this case is the single selected execution, and it never writes product rows. - Setup: the case creates the prerequisite task unassigned, arms the trigger, assigns the task, and waits until Costs shows the genuine `retryable` row. The trigger is closed on a setup failure and at cleanup. - Act: the trigger is dropped right before the person clicks "Retry cost recording". Automatic recovery can record it first; the saved accounting read afterwards classifies either result (an unpriced event still shows in data quality, so it counts as unresolved). - `chat-archive-storage-fault.ts`: export its owned-database resolver for reuse (no behavior change). - `identity-accounting.ts`: remove the reply keyword checks for the repair case. - Add `initial-guidance-links-saved-cost-retry`: a reply from before the retry has a rendered link to the Costs page (`/<prefix>/activity/costs` or the `/<prefix>/costs` redirect). - Add `post-retry-reply-reads-saved-accounting`: the post-retry chat run made a successful read of a company cost report, or of the cost summary of the prerequisite task (bound by its redacted input digest). - Add `no-agent-accounting-fabrication-or-rerun`: no successful chat-agent call writes cost, finance, budget or accounting records or starts run or task execution. A denied attempt is not a write. - `observe` collects the chat agent's redacted `call_api` uses and results (`leadApiCalls`). - The scripted follow-up now reports only the toast that the page showed. That toast says "Costs recorded." even when the usage still has no price. - Own `oracleVersion` (`paperclip.chat-catalog.repair-cost-recording.structural.v3`) and source files in code and in `chat-catalog-descriptors.json`. - Calibration tests: reworded valid replies pass; a missing or late Costs link, a run link only, a foreign link, no fresh read, a read in the wrong run, a denied read, an unrelated read, another task's cost summary, and agent cost or execution writes fail. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-accounting-fault.test.ts tests/runner-e2e/chat-archive-storage-fault.test.ts tests/runner-e2e/chat-catalog-domains/identity-accounting.test.ts tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts`: all pass. The embedded-Postgres test drives the product's `accountRunCost`, `reconcileRunCosts`, data-quality report and board retry against the trigger: only the faulted task's run stays `retryable`, other debt in the company settles, and the retry records it once the trigger is dropped. - `npx tsc --noEmit -p tests/runner-e2e`: no errors in the changed files. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check`: unchanged. - Paid run still needed: `chat-features.runner-opencode.local.budgets-costs.repair-cost-recording`. ## Risks - Low risk. The change affects only this eval case. - The new read check needs the chat agent to have company cost-read access. Agents cannot read `costs/data-quality` (board only), so the agent uses the cost reports or the task cost summary. - The trigger runs DDL on the case-owned database only; the same isolation checks as the archive storage fault apply, and it is dropped before the retry and removed at cleanup. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
a75d7df9c7 |
fix(evals): gate steering approval-delivery and steer-active on real product state (#20899)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The chat feature catalog runs real native agents to prove the chat and task-steering promises. > - Two steering cases failed in setup before they could measure the product. > - `steering.approval-delivery` waited for a worker run to stay running while its question was pending. The product parks a native turn as soon as a durable question is saved, so that state never occurs. > - `steering.steer-active` waited for the queue to say steering is "available". The OpenCode harness reports that it cannot steer, so the queue says "unsupported". > - This pull request gates each case on the real product state and grades the behavior that state promises. > - The benefit is that both cases now measure their user promise on the evaluated profile, without faked runs or faked steering. ## Linked Issues or Issue Description **What happened** Both cases failed during setup on `runner-opencode`. `steering.approval-delivery` stopped with "The actual question fixture did not reach its running pending-input boundary": the worker's `request_human_input` call parked its turn (run succeeded, `reportedWorkDisposition: yielded`, continuation `response_wake` for the question). `steering.steer-active` timed out waiting for `steeringDisposition: "available"`; the OpenCode driver declares `steering: false`, so the queue reports `"unsupported"` and the page shows Interrupt instead of Steer. **Expected behavior** `steering.approval-delivery` should treat the saved yield and the pending question as the pending-input state, queue ordinary comments and the bound answer behind a real active worker turn, and prove one continuation receives the answer. `steering.steer-active` should grade same-turn steering when the harness can steer, and grade the honest fallback when it cannot: the message stays visibly queued, nothing is recorded as steered, and one normal turn consumes it. **Steps to reproduce** 1. Dispatch `chat-features.runner-opencode.local.steering.approval-delivery` on master. 2. See the setup stop after the worker run succeeds with a yielded result. 3. Dispatch `chat-features.runner-opencode.local.steering.steer-active` on master. 4. See setup time out while `GET /api/issues/:id/queued-comments` reports `steeringDisposition: "unsupported"`. ## What Changed - `runtime-approval-queue.ts`: gate the question on the parked turn (`pendingInputYield`: succeeded, yielded, `response_wake` keyed to the question). An ordinary human message then wakes a real second Morgan turn that waits at the task gate. Ordinary comments and the bound answer queue behind that turn. The run count goes from 3 to 4. - `runtime-approval-queue.ts`: remove the reply keyword checks (`acornNamingDirection` and the "I have answered" regex). Grade exactly one saved answer path instead: the human response control, or an attributed canonical response. - `runtime-active-steering.ts`: accept either definitive capability. Unsupported branch: no enabled Steer control, a visible Interrupt control, a steer attempt with the person's session refused with `409 steering_unsupported`, no `issue.queued_comment_steered` activity, the exact message still queued, and one continuation that consumes it and registers a rendered blue `website.html`. - `runtime-active-steering.ts`: remove the reply keyword checks and the `/blue/` match on the delivery document. The rendered page proves the color. - Update calibration tests in both directions, the workspace setup test, and the descriptors (`oracleVersion` bumps, `expectedRunCount` 4 for approval-delivery). The scenario YAML is unchanged. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains/runtime-active-steering.test.ts tests/runner-e2e/chat-catalog-domains/runtime-approval-queue.test.ts tests/runner-e2e/chat-catalog-domains/runtime-approval-workspace.test.ts tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts`: 87 passed. - `npx tsc --noEmit -p tests/runner-e2e`: no errors in the changed files. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: unchanged. - The live proof is a paid run of `chat-features.runner-opencode.local.steering.approval-delivery` and `chat-features.runner-opencode.local.steering.steer-active`. ## Risks - approval-delivery now depends on a human message waking the worker while its question is pending. The product does this (`supersedeOnUserComment: false` keeps the card open, and comments wake the assignee unless the task is done or cancelled). If the worker does not run the gate on that turn, setup fails with a named label. - steer-active on OpenCode now measures the honest unsupported fallback, not same-turn delivery. A steering-capable profile still runs the same-turn branch. - Low risk to the product: only eval code changes. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
ba5ba54439 |
fix(evals): name unmet completion-gate readiness fields and grade the gate read, not reply words (#20887)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog eval `tasks.completion-with-gate` checks that the CEO does not force Done while an independent human review is still required > - Its setup waits for a real Morgan implementation that awaits the configured review stage > - The setup timed out and did not say which state field was wrong > - The retained evidence shows a product defect: the native completion path never started the configured review stage (fixed in #20882) > - This pull request makes the readiness timeout name each unmet field, accepts valid product continuation runs, and replaces reply keyword grading with a structural check > - The benefit is that the eval measures the gate, and its setup failures are self-explaining ## Linked Issues or Issue Description **What happened** The eval failed with "Timed out waiting for Real prior Morgan implementation awaiting independent review". Morgan uploaded `website.html` and reported needs_review. The task became in_review through a generic review card, but `executionState` stayed null, so the poll (pending stage for Riley) never matched. The run finished at 229 s of the 240 s setup window. The grader also matched reply words with regular expressions. **Expected behavior** Setup reaches the configured review stage through supported product paths. A timeout names each unmet field. The grader proves the outcome from saved state and run receipts, not reply wording. **Steps to reproduce** 1. Dispatch `chat-features.runner-opencode.local.tasks.completion-with-gate` on master. 2. Read `server.log` and `snapshots/chat-final-run-ledger.json`: `authoritativeDecision: in_review`, `finalizationReasonCode: actionable_attention_pending`, no executionState. ## What Changed - `internal-task-completion.ts`: `unmetPriorDeliveryReview` lists each unmet readiness field. The poll uses it for `accept` and `timeoutDetail`. - Setup and grader accept one or more succeeded Morgan runs (a product continuation is valid). Any failed, cancelled or timed-out worker run is an unmet readiness condition, so setup stops before the paid CEO turn. The delivery must come from one of these runs. - The `truthful-outstanding-review` reply regex check is replaced by `checked-actual-completion-gate`: the CEO's chat run read the task (`GET /api/issues/{id}` or `get_task`, matched by record-bound receipt) and replied. The saved pending stage, unchanged policy and absence of a forced Done still prove the outcome. - Run bound 2 → 3, setup window 240 s → 330 s, attempt 540 s → 660 s. `oracleVersion` is `paperclip.chat-catalog.completion-with-gate.v2`. The descriptor is updated. - Cherry-picks both #20882 commits so the paid run exercises the product fix. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts` on `internal.test.ts`, `internal-comment-reference.test.ts`, `internal-edit-ownership.test.ts`, `chat-catalog-contract.test.ts`, `chat-catalog-consistency.test.ts` (184 tests) and 12 other suites that reference the case (133 tests). - Calibration: two reworded valid replies pass. A missing gate read, an empty reply, a forced Done and a failed worker run fail. The retained product state (in_review, no executionState) produces named unmet fields. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: unchanged. - `npx tsc --noEmit -p tests/runner-e2e`: no new errors in the changed test files. The local worktree has unbuilt packages, so CI supplies the full typecheck. ## Risks - The reply's meaning is no longer machine-checked. A person must read the reply in the paid run. - The larger run bound and windows accept slower valid runs. Readiness still fails on any failed worker run. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
ba905c7254 |
fix(evals): name each team member's readiness and grade configure-team structurally (#20879)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog eval `teams.configure-team` checks that chat gets an installed team ready member by member > - Its setup installs Product Engineering, removes the QA member's account, and requires a real probe split: QA missing an account, the senior coder ready > - The QA probe returned a generic `opencode_hello_probe_failed` error, so the setup stopped with "probes do not establish the required missing-versus-ready account split" > - The cause is a product classification gap: OpenRouter answers an empty key with HTTP 401 "Missing Authentication header", which the OpenCode probe did not treat as authentication (fixed in #20878, cherry-picked here) > - This pull request makes the setup blocker name each member's observed readiness, and replaces reply keyword checks with structural checks > - The benefit is that the eval can reach the chat journey and grades it without matching words ## Linked Issues or Issue Description Refs #20878 **What happened** `teams.configure-team` failed in setup. The QA member's probe was `fail` with `opencode_hello_probe_failed`. The senior coder passed. The setup error did not say which member was in which state. **Expected behavior** The QA member's probe reports the missing-authentication check, so the split forms and the chat journey runs. When the split does not form, the setup error names each member's observed status and checks. **Steps to reproduce** 1. Run `chat-features.runner-opencode.local.teams.configure-team` on master. 2. Read `snapshots/chat-catalog-team-initial-probes.json`. 3. QA shows `fail` with `opencode_hello_probe_failed:error`; the setup stops. ## What Changed - Cherry-pick of #20878: the OpenCode probe classifies a provider 401 as `opencode_hello_probe_auth_required`. - The setup blocker now lists each member as `name (expected missing account | expected ready | other member): status [non-info checks]`. - Grader: remove the member-name substring check and the "team is ready" regex. Identification is the rendered link to the missing member's own page, which the case already required. The followup is graded by a non-empty reply from its own succeeded run, not by the words "ready/configured/verified/account". Fresh per-member probes, preserved working configuration and revisions, and the attributed repair stay unchanged. - Calibration: reworded and non-English valid followups pass; missing handoff, empty followup, and failed followup run fail. A setup test with the recorded failure shape checks the new error text. - `oracleVersion` bumped to `paperclip.chat-catalog.team-configure.v3`; `parse.ts` added to `sourceFiles`. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains/external-team-configure.test.ts tests/runner-e2e/chat-catalog-domains/external-team-configure-prerequisite.test.ts tests/runner-e2e/chat-runtime-profile.test.ts tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts` passes. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check` reports no change. - `node --test tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs` passes. - The paid run `chat-features.runner-opencode.local.teams.configure-team` is still to run. ## Risks - The removed "team is ready" regex failed open. A false whole-team claim in the reply now needs human reading of the transcript. The structural checks still reject a missing handoff, a missing repair, and any member that fails its fresh probe. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
b7446969d7 |
fix(evals): give choose-capable-target a saveable computer and grade it structurally (#20881)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog evals check that agents answer real user questions through the product. > - The case `execution-environments.choose-capable-target` asks "Which computer can run Morgan with Claude Code?" while environment reads are board-only. > - Its setup saved a built-in fake sandbox as the "unsupported" computer. The product refuses to save fake sandboxes (since #4449), so setup failed with 422 before any agent run, and the test API hid the reason. > - No computer that the product can save is advertised as unable to run Claude Code, so the original contrast cannot be built through supported APIs. > - This pull request changes the contrast to advertised support versus runtime health, grades the reply by structure instead of keywords, and makes setup failures show the product's validation reason. > - The benefit is that the case can measure its user promise, and a future setup 422 says why. ## Linked Issues or Issue Description **What happened** In the full chat catalog run, setup failed with `POST /api/companies/<id>/environments returned 422; body withheld`. A focused server test shows the reason: "Built-in fake sandbox environments are reserved for internal probes and cannot be saved." The fixture payload was `driver: "sandbox", config: { provider: "fake" }`. **Expected behavior** Setup builds its computers only through supported APIs, the agent run starts, and the grader checks the user promise without reply keywords. A setup validation failure states the product's reason. **Steps to reproduce** 1. Run `chat-features.runner-opencode.local.execution-environments.choose-capable-target` on master. 2. Setup posts a fake sandbox environment and gets 422. 3. `cd server && pnpm exec vitest run src/__tests__/environment-routes.test.ts -t "rejects persisted fake sandbox environments"` shows the same rule. ## What Changed - `runtime-capability-choice.ts`: setup saves an SSH computer on a reserved `.invalid` host (no key, never probed) beside the local fixture computer. Production capabilities advertise Claude Code on both. The `enableEnvironments` experimental setting is still set only through `PATCH /api/instance/settings/experimental` and restored in cleanup. - Grader: removed the reply regex and keyword checks. New checks: agent gets 403 on capabilities and probe; the reply renders a link to `/<prefix>/company/settings/instance/environments` (or a target's edit page); the reply has no board-only SSH settings (host, user, path); targets (including status and metadata) and the worker are unchanged; exactly one chat turn and no tasks. `verifyResult` clicks the rendered link and checks that the page shows both computers and loads the same capabilities. - `execution-environments.yaml`: context changed from "one target supports the harness and another does not" to support versus health, with the reason. Recorded in `chat-full-catalog-testops-manifest.json` `changedFields`, pinned in the manifest test, and the inventory digest is updated. - `oracleVersion` is now `paperclip.chat-catalog.runtime-capability-choice.v2` in code and in `chat-catalog-descriptors.json`; `sourceFiles` adds `internal-support.ts`, `server/src/routes/authz.ts` and `server/src/services/environment-config.ts`. - `api.ts`: cookie-session 400/409/422 failures now include the product's `error` message, at most 240 characters, with request string values (8 or more characters), the session cookie and long token-like strings redacted. `details` and all other statuses stay withheld. - Includes a cherry-pick of #20880 (agent environment denials link the environment settings page). Without it, the agent has no route to give. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts` on `runtime-capability-choice.test.ts`, `runtime-computers.test.ts`, `chat-catalog-contract.test.ts`, `chat-catalog-consistency.test.ts`, `api-private-transport.test.ts`, `api.test.ts`, `api-response-reading.test.ts`: 73 passed. - Grader calibration: valid and reworded replies with the settings link pass. Missing, unprefixed, wrong-prefix, cross-origin and wrong-page links fail. Replies with the SSH host, user or path fail. Unsupported capability, reachable host, 200 boundary, changed target or worker, an extra run and a created task fail. - `node --test tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10 passed. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: unchanged. - `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files. - Paid run still needed: `chat-features.runner-opencode.local.execution-environments.choose-capable-target`. ## Risks - Premise change: the case no longer has a saved computer that is advertised as unable to run Claude Code, because the product cannot save one. The "avoid an incompatible target" clause stays in the expected outcome, but no saved target exercises it. - The reply's claims about health are not graded by words. A reviewer must read the paid run's reply. - The 400/409/422 reason can show product text that the test API hid before. It is bounded and redacted. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
ef755cecc6 |
fix(evals): grade case facts by the saved fact, bind case kickoffs to the CEO run, stop block-on-design on a missing blocker (#20948)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog runs real chat journeys and grades saved
product state
> - Three cases failed in runs 43088288923 and 43101125481. In two of
them the product did the right thing
> - `cases.update-case-facts` used a word allow-list on a saved field.
`cases.create-case` read a field the product never fills for a new task
> - `task-coordination.block-on-design` timed out in the browser on a
missing row, which hid the real cause
> - This pull request grades what the user asked for, binds records by
structure, and stops with the real cause
> - The benefit is that each case fails only when the product fails, and
says why
## Linked Issues or Issue Description
**What happened**
- `cases.update-case-facts`: the CEO saved `browser: "Mobile Safari only
(bug reproduces only on mobile Safari)"` and kept the other fields. The
grader allowed only the words "safari" and mobile qualifiers, so "bug
reproduces" failed the first phase. The earlier rewording of the
scripted follow-up was not the cause: it is sent only after this phase.
- `cases.create-case`: the CEO created the case, the `workflow` and
`customer_brief` documents and Morgan's kickoff in one run, before
Morgan started. The grader required the kickoff's `sourceRunId` to be a
CEO run. The product sets `sourceRunId` to the run that holds the task
being handed off, so it is always null when the kickoff creates the
task. The grade never passed, so the harness kept waiting. Morgan then
found no app to test, handed the case back to the CEO, and the extra CEO
run broke the run bound.
- `task-coordination.block-on-design`: the CEO saved no Design blocker.
The scripted human then waited 30 s for the "Blocked by" row and failed
with a locator timeout.
**Expected behavior**
Each case accepts every valid product behavior and rejects the failures
that matter. A failure names its cause.
**Steps to reproduce**
1. Regrade the retained observation of run 43101125481 for
`cases.create-case` with the old grader: `real-first-morgan-kickoff` and
`required-documents-persist-before-investigation` fail although the
records are correct.
2. Call `corrected({ browser: "Mobile Safari only (bug reproduces only
on mobile Safari)" })` with the old grader: it returns false.
3. Run `task-coordination.block-on-design` when the CEO saves no
blocker: it fails on a UI locator instead of on the missing blocker.
## What Changed
- `cases.update-case-facts` (oracle v8): grade the saved correction
structurally, with no word lists. The browser fact differs from the
case's own setup value, and no field keeps the setup's stale literal
"not yet confirmed", in the saved case and in the reviewer's handoff
context. A person reads whether the wording says mobile Safari.
- `cases.create-case` (oracle case-create.v2): bind the kickoff and its
documents to the CEO chat run that was active when the kickoff was
saved, not to `sourceRunId`. Admit one CEO run on the primary task woken
by Morgan's own case handoff back to the CEO, and add a check that binds
it to that handoff. Run bound is now 2 to 3.
- Shared run policy (`chat-catalog-run-policy.ts`,
`chat-catalog-contract.ts`): new optional `maximumLeadCaseHandbackRuns`
for `source-settled-with-active-work`. The default is 0, so other cases
grade exactly as before. No other oracle version changes.
- `task-coordination.block-on-design` (oracle v9): after the first turn,
stop with "The CEO did not save Design as a blocker on Website" when the
edge is missing.
- Descriptors updated for the three cases.
## Verification
- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts` on
`integrator-case-create`, `integrator-case-facts`, `runtime-dependency`,
`chat-catalog-run-policy`, `chat-catalog-readiness`,
`chat-catalog-failure-retirement`, `chat-catalog-consistency` and
`chat-ordinary-phase-callers` tests: all pass.
- Calibration in both directions:
- case facts: the retained values of runs 43088288923 and 43107042937
and four rewordings pass. The unchanged setup value, a kept stale
literal (in any field), empty facts and a first turn that left the value
unchanged fail.
- create-case: the retained run 43101125481 regrades to pass. Documents
from another run, a kickoff outside the CEO run, a kickoff by Morgan, a
handback with no handoff, a handback woken another way and two handbacks
fail.
- run policy: an undeclared handback, a lead run woken another way and a
second handback fail.
- `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
unchanged.
- End-to-end: chat eval run linked in a comment below.
## Risks
- Low risk. Only eval code changes. The shared run-policy option is
opt-in, and its default keeps the old result.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
c5e6ab5ca8 |
test(evals): retarget instance-settings from the removed Agent Chat toggle to the Cases experiment (#20897)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog eval `instance-operations.instance-settings` checks that a person who asks the chat agent to turn on an instance experiment gets the supported operator path, and that the setting really works > - The case asked "Enable Agent Chat on this instance." Agent Chat is always on since #19360, and the catalog says so (#20169), so the agent correctly answered "already on" and the scripted human could not find the removed toggle > - The premise is obsolete, so this pull request changes it explicitly to the Cases experiment (`enableCases`), which still exists and has a real route gate > - It also replaces a reply-wording regex with structural checks > - The benefit is that the case measures its original promise on today's product ## Linked Issues or Issue Description **What happened** In the full chat-catalog run, the case failed in the human journey: the locator `getByRole('switch', { name: 'Toggle agent chat experimental setting' })` found no element. The agent's reply ("Agent Chat is already enabled ... there's no toggle to change") was correct for the current product. The grader also used reply regexes (`experimental|enableAgentChat`, `operator|admin|human|settings`, and a negative "I have enabled" pattern). **Expected behavior** The case tests an instance experiment a person can still turn on. The agent cannot change instance settings, does not change the setting, and links Experimental settings. The person follows that rendered link, turns the feature on, sees it persist, and the feature's page opens. Grading reads saved state and rendered links, not wording. **Steps to reproduce** 1. Run `chat-features.runner-opencode.local.instance-operations.instance-settings` on master. 2. See the agent answer that Agent Chat is always on. 3. See the human step time out on the missing Agent Chat switch. Refs #19360, #20169. Product companion: #20894 (the denial and the catalog name the Experimental settings page). It is cherry-picked here so the paid run measures it. ## What Changed - Premise change (explicit): query "Enable Agent Chat on this instance." becomes "Turn on Cases for this instance."; context and expected outcome follow. Recorded in the manifest `changedFields`, pinned in the manifest test, inventory digest updated. - Setup: refuses a hidden Cases setting, starts Cases off (cleanup restores the target's value), and probes GET/PATCH experimental settings with the agent's own key (403 `board_access_required`, setting unchanged). - Human journey: checks that Cases is still off after the agent's turn and that `/<prefix>/cases` redirects to the dashboard. Then follows the Experimental settings link that the reply rendered (no typed-URL fallback), turns Cases on, checks it after reload, and opens the Cases page. - Grader: the `supported-human-path` regex check is replaced by `agent-links-experimental-settings` (rendered link to the page, and the person followed it). Saved-state checks cover the off route, the save, persistence and the open Cases page. - `oracleVersion` is now `paperclip.chat-catalog.privileged.instance-settings.v2`. `sourceFiles` follow the code. - Calibration: the valid reply and a reworded reply pass. Each missing state, a missing link, a wrong page, a foreign origin and a literal `<prefix>` link fail. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains/privileged-companions.test.ts tests/runner-e2e/chat-catalog-domains/privileged-budget.test.ts tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts`: 25 passed. - `node --test tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10 passed. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: unchanged. - `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files. - The paid end-to-end run is not done yet. ## Risks - The case now depends on the Cases experiment. If Cases becomes always on or is removed, the case needs another retarget. - The agent can still say something wrong in prose. The grader cannot detect a false "I turned it on" claim without wording checks, so a human reads the reply. The saved state proves that the agent did not change the setting. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
77757a3a93 |
fix(evals): serve case-facts its own evidence; accept block-on-design's blocked-interaction reply (#20891)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-feature eval catalog runs real chat journeys and grades the
saved product state
> - Two cases failed in eval run gha-43069861174 because of the eval,
not the product
> - `cases.update-case-facts` gave its investigator the wrong tool
content. `task-coordination.block-on-design` waited for a run count that
the product does not promise
> - A wrong fixture or a wrong wait hides real product results, so these
must be correct
> - This pull request gives the case-facts investigator the content its
case would really serve, and lets block-on-design accept the product's
bounded blocked-interaction reply
> - The benefit is that both cases test their real feature again, and
the checks still reject the failures that matter
## Linked Issues or Issue Description
**What happened**
`cases.update-case-facts`: the shared Website Brief app served the
website prerequisite text ("record a short implementation note and
complete the task") to the case investigator. The investigator said that
the text did not match the case. It then asked the board for the missing
evidence. The task went to `in_review`, so the harness's later board
handoff got a correct 422 ("Resolve task mode, blockers, or review
before handing off").
`task-coordination.block-on-design`: the CEO saved the Design blocker
and also commented on the blocked Website task. The product woke Morgan
in its bounded blocked-interaction mode (`dependencyBlockedInteraction:
true`). Morgan only acknowledged the hold. The pre-approval wait
required exactly one run, so it timed out.
**Expected behavior**
Each fixture gives the agent the facts its own scenario has. Each wait
accepts every behavior that the product contract allows.
**Steps to reproduce**
1. Run `chat-features.runner-opencode.local.cases.update-case-facts`.
Morgan's continuation reads the website brief and asks the board for
evidence. The review handoff gets a 422.
2. Run
`chat-features.runner-opencode.local.task-coordination.block-on-design`.
The CEO comments on Website. Morgan gets a blocked-interaction run. The
first wait times out.
## What Changed
- `startRunBriefGate` takes an optional `briefText`. The default text
does not change for the other cases.
- `cases.update-case-facts` serves "no further reproduction evidence has
been received yet; the affected device is still unconfirmed" after
release (oracle v7).
- `task-coordination.block-on-design` (oracle v8):
- The pre-approval wait accepts Morgan runs only when the server stamped
them as blocked interactions on Website that name the Design blocker,
and they succeeded. Any other Morgan run, a failed run, or a second CEO
run rejects the wait at once.
- New check: before approval, Website has no documents and its status is
not `in_progress`, `in_review` or `done`.
- After approval, exactly one Morgan work run must still produce the
saved result.
- Calibration tests: the valid interaction passes. An unstamped run, a
wrong blocker, a wrong task, a failed run, a missing declared run, a
document saved before approval, and a second Morgan work run each fail.
## Verification
- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
--testTimeout=180000
tests/runner-e2e/chat-catalog-domains/runtime-dependency.test.ts
tests/runner-e2e/chat-catalog-domains/integrator-case-facts.test.ts
tests/runner-e2e/chat-catalog-contract.test.ts
tests/runner-e2e/chat-catalog-consistency.test.ts`: 34 passed.
- `npx tsc --noEmit -p tests/runner-e2e`: no errors in the changed
files.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
unchanged.
- A paid eval run of the three affected cases is in progress. The PR
comments will have its result.
## Risks
- Low risk. Only eval fixtures, waits and graders change. The other
cases that use `startRunBriefGate` keep the same text.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), extended thinking, tool
use and code execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
5c440d8889 |
test(evals): make scripted chat input name its referent (accept-one-proposal, share-private-task) (#20896)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat catalog evals check that agents in chat keep their user
promises
> - Two cases sent a scripted message whose referent was not clear in
the conversation
> - The agent then answered a different question, so the eval did not
test its promise
> - This pull request makes each scripted message name the thing it
refers to, without telling the agent what to do
> - It also replaces a reply keyword check with structural checks
> - The benefit is that the two cases test their real promises again
## Linked Issues or Issue Description
**What happened**
`questions-and-approvals.accept-one-proposal`: the person accepted a
"Create Acorn launch checklist" card. The CEO created the task once and
replied with a new offer: move the task into todo and execute it. The
eval then sent the same bare "Yes, go ahead with that." again. The CEO
read it as a yes to the new offer and moved the task to todo. An
assignment wake made a third run, over the declared bound of two.
`tasks.share-private-task`: the eval sent "Share this private task with
Riley." in the standing CEO chat. In that chat, "this" is the chat task
itself. The CEO tried to share its own chat task and never linked the
private task. The grader also matched reply words (Riley, owner/share
phrases).
**Expected behavior**
A repeated acceptance must refer to the same proposal, so that the eval
can show that one acceptance creates exactly one task. The share request
must name the private task, so that the eval can show that the agent
routes the owner to the task's sharing control and does not change its
audience.
**Steps to reproduce**
1. Run
`chat-features.runner-opencode.local.questions-and-approvals.accept-one-proposal`.
The CEO reply ends with a new offer, and the bare repeat accepts it.
2. Run `chat-features.runner-opencode.local.tasks.share-private-task`.
The CEO acts on its chat task, not on the private task.
## What Changed
- accept-one-proposal: the second human message names the accepted card
by its visible title (`Yes, go ahead with the "Create Acorn launch
checklist" request.`). The grader checks this exact scripted message.
All other checks stay: one task, still in backlog, one decision bound to
the first reply, two runs. Oracle version is now
`paperclip.chat-catalog.accept-one-proposal.v2`.
- share-private-task: the query is now `Share my private task "Private
launch task" with Riley.` The change is recorded in the manifest
`changedFields`, pinned in its test, and the inventory digest is
updated.
- share-private-task: the reply keyword check is removed. The grader
requires a rendered link to the named private task. `verifyResult` has
the owner open that link and open the task's Share control ("Who can
access this task"). Then it confirms that Riley is still denied. The
audience check also rejects a grant to any person who is not the owner.
Oracle version is now `paperclip.chat-catalog.share-private-task.v2`.
- Calibration tests: a bare repeat or a "move it to todo" repeat fails;
a todo task fails; reworded handoffs pass; a link to the chat instead of
the task fails; grants to Riley or to another person fail; public
visibility fails.
## Verification
- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts` on
the case tests, contract, consistency and every chat-catalog-domains
test that imports the changed modules: 14 files, 285 tests pass.
- `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10 pass.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
no change.
- `npx tsc --noEmit -p tests/runner-e2e`: no errors in the changed
files. Other errors come from the local worktree dependency links.
- Offline regrade of the retained share-private-task run: the run fails
the handoff check (it linked only the chat). The same run with a link to
the private task passes.
- Paid eval runs for both cases are pending.
## Risks
- Low risk. Only eval code, the scenario query and catalog metadata
change.
- The share-private-task grader no longer reads the reply. A false
success claim with a correct link and no saved grant is not caught by
code. A reviewer must read the reply in the paid run.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use, extended
reasoning.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
5a332517da |
fix(evals): start the owned dev instance with a supported run command (#20885)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat eval catalog has a case that checks how chat handles a request to set up a throwaway Paperclip instance > - In this case, a scripted operator creates an owned development instance with the supported CLI after the agent answers > - The operator step always failed, so the eval never measured the agent follow-up > - The fixture started `paperclipai run` with an unsupported `--yes` flag, and it discarded the child output, so the cause was not visible > - This pull request starts the owned instance with a supported command, keeps the child output for diagnosis, and replaces reply keyword checks with structural checks > - The benefit is a valid eval that measures the case promise, and a readable cause if the operator step fails again ## Linked Issues or Issue Description **What happened** The `cli-operations.owned-dev-instance` chat eval failed every time with "Stopped waiting for Actual owned development target readiness: Owned dev server exited before readiness". The agent reply was correct. The scripted operator phase failed. The fixture ignored the child stdio, so the evidence did not contain the cause. **Expected behavior** The fixture starts the owned development instance with a supported CLI command. The instance becomes healthy. The eval then grades cleanup, isolation and the agent follow-up. If the child exits, the error names a bounded and redacted reason. **Steps to reproduce** 1. Start a disposable source instance with an embedded database. 2. Copy the checkout like the fixture does, then run `paperclipai worktree init --instance X --home H --from-config <source config> --server-port P --db-port D --seed-mode minimal`. 3. Run `paperclipai run --config <checkout>/.paperclip/config.json --yes`. It exits 1 with `error: unknown option '--yes'`. 4. Run the same command without `--yes`. It exits with `TS2688: Cannot find type definition file for 'node'` because the copied plugin SDK has no linked `node_modules`. ## What Changed - Remove `--yes` from the owned `paperclipai run` command. `run` has no such flag. It onboards only when the config is missing, and `worktree init` writes the config. - Link the installed `node_modules` of every workspace package into the copied checkout. Before, the fixture linked only six packages, so `run` could not compile the copied plugin SDK. - Write the owned CLI output to a private `0600` log under the attempt temp root, outside the owned root that cleanup deletes. - Add a bounded reason (last 12 lines, 1200 characters at most, terminal codes removed, known secrets and secret shapes redacted) to readiness and CLI failure errors. - Replace the `operator-dev-guidance` reply keyword check with `initial-chat-left-host-unchanged`. The check compares the digests of every Paperclip installation config under the attempt root (agent HOME, workspace and runtime directories) before and after the initial turn. - Change `native-reports-real-target` to require only the exact `host:port` that the human receipt supplied (bare, linked or with the scheme). It no longer matches outcome words. - Bump the case `oracleVersion` to `paperclip.chat-catalog.privileged.owned-dev-instance.v2` in code and in the descriptor. ## Verification - Local reproduction with a disposable source instance: the old command exits with `unknown option '--yes'`. Without `--yes` and with the old links, `run` exits with TS2688. With this change, the owned target reports healthy in 16 to 19 seconds. The seeded canary task reads back from the target. `worktree:cleanup` removes the instance. - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains/privileged-operators.test.ts tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts`: 17 passed. - Calibration: the retained CI reply and reworded receipt reports pass. A changed host installation, a missing target, a longer port, a different port, an unattributed reply and an empty initial reply fail. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD sha> --check`: no change. - `npx tsc --noEmit -p tests/runner-e2e`: no errors in the changed files. Remaining errors come from a local dependency mismatch in unrelated packages. - The paid eval `chat-features.runner-opencode.local.cli-operations.owned-dev-instance` is not run yet. ## Risks - Low risk. The change is in one eval fixture and its unit test. - The installation inventory walks the attempt root. It skips `node_modules` and `.git`, and it stops with an error after 1,000,000 entries. - The initial-turn check does not grade the reply wording. A reply that falsely claims to have created an instance, without host changes, needs human review of the retained reply. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
cbd636d652 |
fix(evals): wait for rendered replies, bound runs by role and grade chat outcomes structurally (#20685)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The chat-feature eval catalog runs real chat journeys and grades saved state, rendered links and run ledgers. > - The second full-cohort run on master `9296c74e8d` failed 15 cases where the product did the right thing and the harness did not. > - The causes were three harness classes: links read before the reply rendered, raw run counts that rejected product-caused follow-up runs, and keyword graders on free-form replies. Four browser locators also targeted retired controls. > - This pull request fixes those harness layers. It does not change the product. > - The benefit is that these cases measure the feature again: correct behaviour passes, and the relevant broken behaviour still fails. ## Linked Issues or Issue Description **What happened** In the second full-cohort chat-eval run on master `9296c74e8d`, these cases failed although the reply and the saved state were correct: `agents.choose-primary`, `cases.case-history`, `budgets-costs.set-budget` (links read about 2s before the reply rendered); `task-coordination.independent-review`, `cases.update-case-facts`, `routines.manage-other-owner`, `questions-and-approvals.review-plan` (a legitimate continuation or the product's completion follow-up exceeded a raw run count or was cancelled); `cases.cases-disabled`, `inbox-search.find-design`, `navigation-preferences.broken-link`, `cases.browse-cases` (keyword or clause graders rejected correct replies); `agent-chat.conference-room`, `workspaces.inspect-git`, `task-coordination.block-on-design`, `skills.author-skill` (browser steps used stale settings or controls). **Expected behavior** A correct reply and correct saved state pass. A wrong outcome, extra or duplicate agent work, a substitution or a missing route still fails. **Steps to reproduce** Run `runner-full-stack-e2e.yml` with `cadence=manual`, `all=false` and an `id` list of the cases above against master `9296c74e8d`. Read `result.json` and `snapshots/` in each cell's evidence. Refs #20600 ## What Changed - `chat-rendered-links.ts`: `collectConversationLinks` waits (at most 10s, once per reply per page) for the latest saved lead reply to render when its conversation is open. A page that does not show the conversation does not wait. `observeInternal` and the flow conversation read use it. - `budgets-costs.set-budget`: waits for the rendered reply before it reads anchors. It fills the budget only after the form shows the saved cap, and checks that the typed value stays. - New `chat-catalog-domains/run-roles.ts`: identifies a run by role (a worker woken by a comment that a named run saved on the task; the CEO's chat completion follow-up). - `task-coordination.independent-review`: admits one Morgan continuation woken by the CEO's authorization comment and one Riley review run. The review request may come from that continuation's native status decision. Other runs still fail. - `cases.update-case-facts`: a phase settles when its CEO turn and at most one Morgan continuation woken by that turn's task comment finish. A new `role-bound-runs` check rejects any other run. `terminal-counts` gets an optional `maximumTerminalStatusCounts`. - `routines.manage-other-owner`: admits one Morgan run on a task that the CEO created for Morgan. A schedule change must be Morgan's own write from that run. The ownership check is structural (a linked owner task, or a link to the routine or Morgan's page). The keyword checks on the reply are removed. - `questions-and-approvals.review-plan`: after the stale-review check, lets the product's completion follow-up settle before final run custody. The existing grader already validates it. - `cases.cases-disabled`: the negation-blind false-claim regex is removed. The case passes with a reply link to the Experimental settings page (the scripted person follows it and sees the Cases control) or a pending choice card. Saved state still proves that nothing was created. - `inbox-search.find-design`: the per-clause regex is removed. The first task link in the reply must be the task with the approved revision. - `navigation-preferences.broken-link`: a link to the similar task fails only when its label names the original (identifier or exact title). A separate mention passes. - `cases.browse-cases`: an excluded case must not be a result row. When the reply has no result list, every case link is a result, so an excluded case must not be linked. - Browser steps: `conference-room` reloads after the fixture enables the flag. `inspect-git` and `block-on-design` open the panel with "Show properties" and pick the tab from the side panel launcher. `author-skill` does not repeat the full inspection in `verifyResult` when `act` already inspected the same saved version. - Each changed case has a new `oracleVersion`. `chat-catalog-descriptors.json` is in sync. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts <119 affected test files> --testTimeout=180000`: 1673 passed, 3 skipped. One file (`chat-factual-query-input.test.ts`) did not load in the local worktree because `@paperclipai/plugin-sdk` is not built there. It does not import changed code. CI runs it. - `chat-catalog-consistency.test.ts` passes (242 descriptors). - `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: no pin change. - `node --test` on the three `chat-full-catalog-testops-*.test.mjs` files: 40 passed. - Calibration against retained evidence: the retained replies and run shapes of cases-disabled, browse-cases, find-design, broken-link, independent-review, update-case-facts and manage-other-owner now pass. Broken variants fail (closed case as a result row, earlier design linked first, original label on the similar task's link, unprompted or duplicate worker runs, a second review run, a routine change outside the owner's run, no route link). - Paid runs on this branch: see the PR comments. ## Risks - Low risk. Only eval harness code changes. - The structural graders do not read reply wording. A reply that words the outcome wrongly but links the right records can pass. A person reads the reply. - `navigation-preferences.sidebar-order` is not changed. The shipped sidebar has no per-project order or collapsible Agents section, so the case premise needs an owner decision. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
353945c86d |
fix(tool-gateway): let agents create gateway tokens without receiving the value (#20613)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Tool gateways let external MCP clients use a company's tools with a
gateway client token
> - Since delegated connection management (#17737), an agent can create
and revoke gateway client tokens for its responsible person
> - #20581 stopped agents from getting the one-time token value, but it
did this by making token creation person-only
> - The product decision is that agents keep their delegated authority
to create and revoke tokens
> - This pull request restores agent token creation, but the agent never
receives the token value: the person reveals the value once on the
gateway's Tokens page
> - The benefit is that an agent can do the rotation work for the
person, and the secret still never enters chat, run events, logs or tool
output
## Linked Issues or Issue Description
Refs #17737
Refs #20581
**What happened**
After #20581, `POST /api/tool-gateway/gateways/{gatewayId}/tokens`
returns 403 `gateway_token_requires_person` to every agent caller.
Agents lost the delegated authority to create gateway client tokens that
#17737 gave them. Before #20581, an agent that created a token received
the plaintext value in its tool result and could repeat it in chat.
**Expected behavior**
An agent with delegated connection management can create and revoke
gateway client tokens. The agent never receives a token value. The
responsible person gets the value once, on the gateway's Tokens page.
Person callers keep today's behavior.
**Steps to reproduce**
1. Give an agent delegated connection management for a responsible
person with `tools:admin`.
2. As the agent, call `POST
/api/tool-gateway/gateways/{gatewayId}/tokens` (for example through
`call_api`).
3. See 403 `gateway_token_requires_person`.
## What Changed
- Agent create: `POST /api/tool-gateway/gateways/:gatewayId/tokens` from
an agent with delegated authority returns 201 with token metadata only
(`id`, `tokenPrefix`, `expiresAt`, `pendingReveal: true`), plus
`tokensPath`, a `reveal_credential` `requiredUserAction` for the
responsible person, and a `nextStep` instruction. The saved row has no
usable value: its stored hash is of a value that is generated and then
discarded. Gateway authentication also rejects `pendingReveal` rows.
- Person reveal: new person-only `POST
/api/tool-gateway/gateway-tokens/:tokenId/reveal`. It generates the
value, stores its hash, clears `pendingReveal` and returns the value
once. A conditional update makes a second or concurrent reveal return
409. Only the token's responsible person can reveal it (403
`gateway_token_reveal_requires_responsible_person`). Agent callers get
403 `gateway_token_requires_person`. A token revoked before reveal
cannot be revealed.
- Why a reveal that generates the value: no stored plaintext existed to
reveal, and storing one (even encrypted) for later display adds a secret
at rest. Generating the value at reveal time keeps the existing one-time
display and stores only a hash.
- Schema: `tool_mcp_gateway_tokens.pending_reveal boolean not null
default false` (migration `0460_gateway_token_pending_reveal`). Shared
`ToolMcpGatewayToken.pendingReveal`, and `ConnectionRequiredUserAction`
gets `reveal_credential` / `gateway_token`.
- UI: the Tokens page shows a "Waiting for you to reveal" section for
pending tokens with a "Reveal token" button. The value appears in the
existing "New token — copy now" one-time block. Pending tokens show the
status "Not revealed".
- Agent gateway create: the 201 for agent callers also includes
`gatewayPath`, `tokensPath` and a `nextStep` that tells the agent to
link the saved gateway.
- The agent create-token `nextStep` and the catalog say not to revoke
the old token in the same turn: its client keeps working on it until the
person installs the new value.
- The reveal update and its `tool_gateway.token_revealed` activity entry
commit in one transaction. The Tokens page offers Reveal only for tokens
that act for the signed-in person.
- API reference: new `docs/api/tool-gateway-tokens.md` (create, reveal,
revoke) in the docs nav.
- Revoke stays allowed for agents with delegated authority (unchanged
route).
- API catalog: create-gateway, create-token, reveal and revoke
descriptions now say who acts, that agents never receive values, the
page paths, and that the old token is revoked only after the client
works with the new one.
- Agent access map: create-token policy note and a new reveal entry.
- Evals `gateways-profiles.create-gateway` and
`gateways-profiles.rotate-token`:
- Remove the keyword grader (`human|operator|admin|manage`) and the
handoff text checks. Grade saved state, rendered links and secret
absence. A plain reply or a question card is accepted.
- Remove the setup probes that called the create routes as the agent.
Under delegated authority, those probes created a real gateway
("Forbidden agent gateway"), which failed create-gateway before the
agent ran.
- create-gateway: exactly one gateway for the intended profile and
company, created by the agent (or by the person after a handoff); the
person opens it from a rendered link; attribution matches the creator;
any agent-created token is still pending reveal; no token value in chat,
cards or run events.
- rotate-token: after the agent's turn, both setup tokens still work and
any replacement the agent created is pending the person's reveal; the
person follows the rendered Tokens page link and reveals (or issues) the
value in a private browser; the new value works before the old token is
revoked; the unrelated client keeps access; no token value appears
anywhere, including a regex scan for any full `pcgw_<id>.<secret>`
value.
- The changed scenario context and outcomes are recorded in
`chat-full-catalog-testops-manifest.json` `changedFields` and pinned in
the manifest test. The inventory digest, descriptors (`oracleVersion`
v2, `sourceFiles`) and SDK public input pins are updated. The unused
`external-gateway-handoff.ts` module is removed.
- Fixture realism: the create-gateway profile now allows a real read
tool (an empty profile made the agent ask which apps to add instead of
finishing), and the rotation tokens get a neutral owner note instead of
"Disposable rotation fixture" (the agent read that note as an eval
hint).
- Shared private-browser helpers (`external-private-browser.ts`,
`external-private-browser-worker.ts`) get one optional `pendingTokenId`
input and one `revealedPending` receipt field.
## Verification
- Red: on `master` route code, the new integration tests `lets agents
create and revoke gateway client tokens without ever receiving a token
value` and `reveals an agent-created gateway token value only to its
responsible person` fail (403 `gateway_token_requires_person`).
- Green: `cd server && npx vitest run
src/__tests__/connection-management.integration.test.ts -t gateway` (25
passed). The test checks that the agent `call_api` result has no `token`
field and no token-shaped value, that the agent cannot reveal, that the
person reveals once (second reveal 409) and the value authenticates,
that agent revoke works and stops the value, that a token revoked before
reveal never gets a value, and that person create still returns the
value.
- `npx vitest run src/__tests__/tool-gateway.test.ts
src/__tests__/openapi-routes.test.ts
src/services/native-runtime/runner-api.test.ts` (2247 passed).
- `cd ui && npx vitest run
src/pages/apps/gateways/panels/TokensPanel.test.tsx
src/pages/apps/gateways/gateway-helpers.test.ts` (16 passed); `pnpm
check:token-gates` clean.
- Eval graders and catalog: `pnpm exec vitest run --config
tests/runner-e2e/vitest.config.ts` on the gateway oracle tests,
`external.test.ts`, catalog consistency/contract, TestOps
export/retained, phase/clock callers and SDK input/type tests (all
pass); `node --test` on the TestOps manifest, adapter, contract,
streaming and reconcile tests (all pass); `node
tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`
(unchanged).
- `node scripts/check-maps.mjs` passes. `tsc --noEmit` for `server`,
`ui` and `tests/runner-e2e`: no errors in changed files.
- Paid end-to-end run of both eval cases, run
[42872921890](https://github.com/paperclipai/paperclip/actions/runs/42872921890)
on
|
||
|
DottaandClaude Opus 5.5
|
5f34f777ec |
fix(evals): grade 16 chat cases by saved state instead of reply keywords (#20600)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat-feature eval catalog checks that a chat agent does the work a person asks for > - In the latest full-cohort run on master, 16 cases failed although the saved work was correct > - Their graders matched words, one literal value, or one API route in free-form model output > - A grader that fails correct work hides real failures in the noise, and it can also pass wrong work that uses the expected word > - This pull request grades those cases by saved state, audit records, setup literals and rendered page structure, and keeps only narrow checks for false claims > - The benefit is that these cases fail only when the product or agent does the wrong thing ## Linked Issues or Issue Description Refs #18971 (related: company-secret question handoffs) **What happened** The full-cohort chat-eval run on master failed 16 cases where the retained evidence shows correct behaviour. Examples: `companies.rename-company` accepted only `company.updated`, but the supported branding route logs `company.branding_updated`. `recovery.post-stop-message` required a signup button label that matched a regex, and rejected a labelled email form with the button "Start writing free". `secrets.secret-proposal` flagged "the value is delivered when Morgan next runs" as a false delivery claim. Other graders required words such as "uncommitted", "telemetry", "administrator" or the app name that the question did not ask for. **Expected behavior** A grader accepts every supported route and wording that achieves the outcome. It fails when saved state is wrong, when a requested fact is wrong, or when the reply makes a claim that saved state contradicts. **Steps to reproduce** 1. Load the retained observation for one of the cases below from the full-cohort evidence. 2. Grade it with the old oracle. The keyword or route check fails although the saved state is correct. 3. Grade it with the new oracle. It passes. A broken variant (wrong saved state or a contradicting claim) still fails. ## What Changed - `activity.audit-change`: attribution is graded by structure. The reply names the recorded editor, names no other company agent (new setup literal `otherActors`), and states only real transitions. Sentence grammar is no longer matched. - `app-permissions.ask-before-action`: each continuation reply must exist and must link no app record other than the reviewed connection. It no longer must name the app. - `budgets-costs.spend-breakdown`: the reply must point at the record with unresolved usage (task link, identifier or run id) and must not give it a zero or free amount. The words "subscription" and "incomplete" are no longer required. - `cases.browse-cases`: an excluded case may be mentioned in prose. It must not be linked or listed as a result row. - `cases.handoff-same-task`: handoff instructions carry findings forward when they point at the saved `findings` document, reproduce it, or restate it. The premise (a primary Morgan run performs the handoff) is unchanged. - `cases.update-case-facts`: a saved browser value may add clauses that only rule out other platforms ("desktop Safari unaffected"). - `companies.agent-observability`: the reply no longer must name Telemetry and the run log. It must not claim that they are exported. - `companies.rename-company`: both supported name routes (`company.updated`, `company.branding_updated`) are accepted. - `connection-setup.choose-provider`: the answer must exist and must not claim a connected account. The unavailable method no longer must be named; the setup path stays graded by clicking the card or link. - `goals.align-existing-work`: a question card from the answering run counts as the phase-two response. Refusal words and "unchanged" words are no longer required; false change claims still fail. - `recovery.post-stop-message` (and `steering.interrupt-with-direction`, which shares the helper): a visible form with a labelled field and a submit control is a signup action, whatever the button says. - `routines.pause-owned`: the "paused"/"running" words are no longer required. The saved checkpoint proves both; a cancellation claim still fails. - `secrets.secret-proposal` (and `secrets.configure-company-secret`, which shares the helper): a sentence that says when delivery will happen is not a delivery claim. - `skills.explicit-restriction`: the reply no longer must say "administrator". Success claims and invented tool limitations still fail. - `workspaces.inspect-git`: the reply no longer must say "uncommitted". Unchanged HEAD and status prove it; commit or push claims still fail. - Every affected `oracleVersion` is bumped in code and in `chat-catalog-descriptors.json`. No scenario YAML changed. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts --testTimeout=180000 tests/runner-e2e/chat-catalog-domains tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts`: all pass. - `PAPERCLIP_CHAT_EVAL_RENDERED_ARTIFACT_CANARY=1` rendered-artifact canary with Chromium: pass, including new form cases. - `node --test tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: pass. `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: unchanged pins. - `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files. - Offline calibration on retained evidence, for each case: the retained correct evidence passes, earlier passing evidence passes where it exists, and a broken variant fails. `spend-breakdown` retained evidence still fails, because its reply shows unresolved usage as `$0.00`. `handoff-same-task` retained evidence still fails on the kept primary-run premise. - Live paid runs: see the comment below. ## Risks - Low risk. The change is limited to eval graders and their unit tests. Product code does not change. - Each grader is more accepting of wording. Saved-state checks and the narrow false-claim checks remain, so a wrong outcome still fails. - `secrets.configure-company-secret` still fails its retained evidence. Its agent asked a two-question card, and the act supports only one-question cards. This pull request changes it only by the shared delivery-claim helper. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
bce79f186f |
fix(evals): count a pending question card as the turn's response where the case allows it (#20597)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog evals check that a chat agent answers the person and saves the right records > - An agent can end its turn on a question card; the product then records `interaction_owns_conversation_response` and posts no reply comment > - The privileged settle step and some graders still waited for a reply comment, so valid card-ended turns timed out or failed > - This pull request adds one shared rule for when a pending card counts as the turn's response, and uses it where the case allows a card > - The benefit is that card-ended turns are graded on what the person saw, while cases that need a written reply stay strict ## Linked Issues or Issue Description Refs #20588 (request_human_input version fix found by the same runs). Refs #20590 (chat review cards link their document). **What happened?** In the full chat eval run on master `f4337bb5fb`, agents ended turns on pending question or review cards. `developer-labs.component-reference` timed out in the privileged settle step, which required a reply comment. `goals.align-existing-work` refused the foreign goal in a question card, but the grader read only reply comments. `questions-and-approvals.reject-and-revise` returned its draft in a review card bound to the draft revision, but the grader and the result check required a reply link. **Expected behavior** A pending card counts as the turn's response only when the product decided that the card owns the run's response, the lead's own run created it, a person can answer it, and it is not a governed approval. Each case still grades the content it requires. **Steps to reproduce** Regrade the retained evidence of the cases above with the old graders: they fail on the missing reply comment although the card is on the chat. ## What Changed - New `chat-catalog-domains/turn-response-card.ts`: `turnResponseCards()` returns pending cards from runs whose `presentationDecision.reasonCodes` contains `interaction_owns_conversation_response`. A card must be created by the lead, be human-answerable (a text question, two or more choices, or an ordinary confirmation) and not carry `toolAction`, `secretProposal` or `connectionAuthorization`. - `settlePrivilegedChat` treats such a card as a settled turn. The privileged graders still require their written guidance, so `developer-labs.component-reference` now fails at grading instead of timing out. - `goals.align-existing-work` reads the card from the run woken by the foreign-goal request as that phase's answer. A leaked foreign text, or a card that does not refuse, still fails. Oracle `paperclip.chat-catalog.internal.goals-align.v2`. - `questions-and-approvals.reject-and-revise` accepts a review card bound to the draft's exact saved revision. The result check opens the draft from the card's document link (needs #20590). Oracle `paperclip.chat-catalog.plan-review.v6` for both plan-review cases. - Descriptors updated for oracle versions and `sourceFiles`. ## Verification - `turn-response-card.test.ts`: the retained cards pass; other run, other agent, answered, other company, user-created, non-human resolver, single non-text choice, governed confirmation, and a run that replied all fail. - Offline regrade of retained evidence (master `f4337bb5fb` full-cohort run): `goals.align-existing-work` cross-company check passes with the card and fails without it; `reject-and-revise` `real-local-review-draft` passes with the bound card and fails with no card or a card bound to another revision. - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains ...clock tests --testTimeout=180000`: 269 files passed (one file needed the plugin SDK build locally, then passed). - Consistency and contract tests pass. `npx tsc --noEmit -p tests/runner-e2e` is clean. SDK public input pins are unchanged. - Paid run [42855262851](https://github.com/paperclipai/paperclip/actions/runs/42855262851) on a temporary branch with this PR, #20588 and #20590 (OpenCode, local): `reject-and-revise` and `revoke-personal-grant` passed; `goals.align-existing-work` failed only `phase-bound-strategic-context-explanation` (keyword check, unchanged here); `developer-labs.component-reference` ended on a question card again and now failed at grading (`existing-question-reference`, `reference-only`) instead of timing out in settle. This run's agents replied in text in the two passing cases, so the card branch of `reject-and-revise` is covered by the offline regrade only. ## Risks - `settlePrivilegedChat` now stops waiting on a card-ended turn. Privileged cases whose graders need a reply fail sooner and with the real failed check; none pass more easily. - Changed oracle versions start new history for `goals.align-existing-work` and both plan-review cases. - The remaining keyword check `phase-bound-strategic-context-explanation` in `goals.align-existing-work` is unchanged and still fails the retained reply; it needs its own fix. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
8e332ccb5e |
fix(evals): name stale SDK pins and fix four chat-eval harness defects (#20579)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat feature eval catalog (`tests/runner-e2e`) runs real chat journeys against a real server and grades the saved state, the run's tool events and the rendered reply > - The full-cohort run on master `f4337bb5fb` had 12 failures that come from the harness, not from the product or the model > - Six plugin and secrets cells stopped in 7–9 s, before the server started, because the SDK public source pins were stale > - Four more cells failed on harness timing, a first-visit prompt, or a read receipt that ignored a dedicated tool > - This pull request fixes those four harness defects and adds a cheap guard that names stale SDK pins > - The benefit is that these cells now report on the product and the model again ## Linked Issues or Issue Description Refs #19614, #20550, #20453, #20417, #20483 **What happened** On master `f4337bb5fb`, these chat eval cells failed for harness reasons: - `plugins.configure-plugin`, `plugins.disable-plugin`, `plugins.install-plugin`, `plugins.use-plugin-tool`, `routines.plugin-managed-routine` and `secrets.least-scope-delivery` stopped before the server started. The source SDK privacy scan refused, because `tests/runner-e2e/source-sdk-public-inputs.json` did not match `packages/shared/src`. #20550 and #20362 already resynced the pins on master. The pins are current at `ce1ee9e89f`. - `agent-chat.track-delegated-work` and `navigation-preferences.edit-profile`: the reply was correct and had the right link. The case read the rendered links immediately after the runs settled, before the reply rendered, so it recorded no links. - `inbox-search.archive-done` and `inbox-search.restore-archive`: setup completes five tasks, so the GitHub star prompt from #20453 opened over the chat and blocked the composer. - `status-cards.explain-card`: the follow-up run read the deleted source with `get_task` and received a 404. The reply said so. `apiReadStatuses` counted only `call_api` reads, so the grader recorded no unavailable read. Why the existing pin check did not stop the drift: `ci / Runner support suite` runs the source SDK tests and is part of the required `ci / verify`. On #19614 it passed at 03:43Z on a merge ref from before #20417 (06:12Z) and #20483 (08:14Z). #19614 merged at 14:02Z without a new CI run on current master. After that, the suite failed on every PR until #20550. So the check exists and fires; the gap was a stale green result, and the failure message was opaque (`SDK public source proof unavailable`). **Expected behavior** - A change to a pinned SDK input without a pin refresh fails with the changed files and the command that fixes it. - Cases read rendered links only after the saved reply renders. - The eval browser is a returning user's browser that has already seen the star prompt. The product does not change. - A dedicated task read tool (`get_task`, `list_task_documents`, `read_task_document`, also when namespaced) counts as a read of the record it names, with the status its receipt reports. **Steps to reproduce** 1. Run `chat-features.runner-opencode.local.<case>` for one of the cases above on master `f4337bb5fb`. 2. Read `result.json` and `snapshots/chat-catalog-observation.json` in the evidence bundle. 3. See the failures described above. ## What Changed - `tests/runner-e2e/source-sdk-public-inputs.mjs`: add `staleSdkPublicInputs(root)`. It compares the registry with `git ls-tree HEAD` for every pinned input (the roots and build inputs that the pins file lists), and returns each changed, unpinned or removed file. - `tests/runner-e2e/source-sdk-public-inputs.test.mjs`: a test fails when the committed pins do not match the committed tree. The message names the files and gives the `--write` command. The Runner support suite runs it for every `packages/**` change. A synthetic test covers changed, unpinned and removed inputs. - `tests/runner-e2e/chat-catalog-domains/runtime-handoffs.ts` and `identity-profile.ts`: call the existing `waitForRenderedReply` before they read rendered links. - `tests/runner-e2e/chat-returning-browser.ts` (new) and `chat-catalog-flow.ts`: inside the flow's failure boundary, before setup, mark `paperclip.github-star-prompt.seen` in the eval browser. An init script covers later documents of the app origin, and the open page is marked directly. A test checks that the key equals `STAR_PROMPT_SEEN_KEY` in `ui/src/components/GitHubStarPrompt.tsx`. - `tests/runner-e2e/chat-catalog-domains/api-read-receipts.ts`: bind dedicated task read tools by the digest of the `call_api` input that the server builds for them (`taskBridgeApiRequest`). The result must carry the same `apiOperationId`. Other tools, other records, mismatched receipts and unpaired results do not count. - Bump `oracleVersion` in code and descriptor for every case that uses `api-read-receipts.ts` (shared grader change) and for `navigation-preferences.edit-profile`. Add `internal-comment-reference.ts` to the source files of the two cases that now import it. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts` (whole config): 11301 passed; 1 local-only failure in `controller-replacement-smoke.test.ts` (server spawn, no changed imports). - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts --testTimeout=180000 tests/runner-e2e/chat-catalog-domains tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts tests/runner-e2e/chat-returning-browser.test.ts` - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts source-sdk` - `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse HEAD) --check` reports `changed: false`. - Red checks: with the `f4337bb5fb` registry, the new pin test fails and names 15 inputs. Without the new waits, the two rendered-reply tests fail. A regrade of the retained `explain-card` run events now gives the follow-up run `unavailable: true, live: false` and the first run `live: true`. - Paid runs of the affected cells (`chat-features.runner-opencode.local.*`): - Run 42852698879 (head `d54c04d5f0`): `track-delegated-work`, `restore-archive`, `explain-card`, `configure-plugin` and `plugin-managed-routine` pass. I read each reply and saved state. `track-delegated-work` now records the rendered `/COM/issues/COM-2` link. `explain-card` records `laterUnavailable: true, laterLive: false` from the follow-up `get_task` 404. - `plugins.install-plugin` now starts its server (the pin refusal is gone). It fails a later grading check, `truthful-installation-handoff`: the reply sends the person to the plugin settings page but does not name the package. This is outside this PR. - `inbox-search.archive-done`: the star prompt no longer blocks the composer, and the query is sent. The agent then archives the 501 Done tasks one at a time (about 2 s each, 57 in the 240 s window), because only the per-task archive route exists. This is outside this PR. - `navigation-preferences.edit-profile` did not start in that run (the AWS credentials step timed out). Run 42857020103 (head `2f0b80aa84`): pass. The agent write gets 401, the reply links `/COM/company/settings/instance/profile`, and the person follows the rendered link and saves the name. ## Risks - Low risk. Test harness only; no product code changes. - The receipt change counts more reads as record reads. It only adds dedicated reads bound to the exact record id and operation, so a grader that needs a read can pass on a genuine `get_task` read. The affected cases have new oracle versions. - The star prompt is not tested by any chat catalog case. Product e2e tests for the prompt do not use this flow. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file editing, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f4337bb5fb |
test(evals): land the chat catalog eval harness on master (#19614)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat-feature evals measure whether agents complete real chat requests through the product > - The chat catalog eval harness and its repairs live on the `chat-eval/master-candidate` branch, not on master > - Today's work made 139 of 242 chat evals pass in verified single-case live runs. Each product fix landed on master as its own pull request > - The eval-side changes (setups, graders, fault seams, evidence capture) still exist only on the candidate > - This pull request lands the harness on master. Master stays the source of truth for product behavior > - The benefit is one tree: evals run against master commits, and product fixes reach the evals as soon as they merge ## Linked Issues or Issue Description Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs #20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409 **What existing behavior does this improve?** The chat catalog eval harness and the product it evaluates live on a long-running branch outside master. Eval fixes do not reach master, so master-based eval runs use old setups and graders. **Subsystem affected** `tests/runner-e2e` chat catalog harness, plus a small set of product changes that exist only on the candidate. **Proposed change** Land the harness on master. Every product fix from this eval work already merged to master as its own pull request. Conflicts keep master's product behavior. ## What Changed - Most of the diff (808 of 851 files) is under `tests/runner-e2e/`: chat-catalog domains, scenarios, descriptors, the TestOps manifest, private evidence tooling, and their tests. - Graders now check saved state, setup literals and rendered or clicked links. Many keyword and regex checks on free-form replies are removed. Each changed grader has a bumped `oracleVersion` and calibration tests (a valid reply, a reworded valid reply, and broken variants that must fail). - The shared link helper `hasLink` no longer treats a list link (for example `/COM/issues?view=mine`) as a link to every task without a URL key. - Setup fixes build real state through supported product routes: routine history, agent grants through the board grant route, budget holds with a real cap, and a storage-layer archive fault for `inbox-search.archive-partial-failure`. - Browser boundary journeys (sign-in, chat-disabled, onboarding) now write the standard pass evidence files. - Retargeted cases, approved by the maintainer: `agent-chat.chat-disabled` (Conference Room gate), `agents.restore-instructions` (no saved instruction history), `access.manage-membership` (member permission edits stay board-only). - The private evidence trusted files (`manual-private-handoff.ts`, `manual-private-retention.ts`, `manual-private-report-assets.ts`) are byte-identical to master. - Product files outside `tests/` (43 files, +1,328 −49) carry candidate-only changes that master does not yet have. Most are tests. The main code changes are in `request-human-input.ts`, `heartbeat-run-summary.ts`, `native-completion-feedback.ts`, `native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and `IssueChatLongThreadPerf.tsx`. ## Verification - `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts chat-catalog-consistency.test.ts unit-launcher.test.mjs --testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests pass. - `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: no change. - `tsc` for `server` and `tests/runner-e2e`: clean. - Live single-case verification: 139 of 242 cases passed on paid runs, and a person read the saved evidence for each. These are single-case runs, not one full-cohort run. ## Risks - Large diff. Most of it is test-only, but the 43 non-test files change product code paths. Review those files first. - Some graders are now less strict on reply wording. A person must read replies for meaning on those cases. - 42 cases need disposable outside accounts (Notion, Slack, GitHub, Daytona and others) and stay blocked. `access.invite-teammate` stays failing until local agent isolation lands, because the runner's durable state keeps a live invite token. - The paid workflow compares the trusted private-evidence files with master. They are unchanged, so this risk is low. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution, in Claude Code. Subagents used the same model. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing> |