Commit Graph
9 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 3b4c3265ed fix(recovery): offer Try again on a model-unavailable failure and recheck the model first (#21244)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a native run fails, the task thread shows a failure marker, and
some markers offer Try again
> - A `native_provider_model_rejected` failure blocks the task for the
person and shows "Model unavailable", but it has no Try again control
> - Recovery must just work: after the person fixes the model, they need
one obvious control that continues the task, and that control must not
start a run that is sure to fail
> - This pull request shows Try again on the "Model unavailable" marker
and makes the retry recheck the selected model before it starts a run
> - The benefit is that the person fixes the model, presses Try again,
and the task completes, and a too-early press explains which setting to
change instead of failing again

## Linked Issue or Issue Description

Refs #21210

**What happened?**
Paid run 43321773136, `recovery.retry-exhausted` (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`). The run failed with
`native_provider_model_rejected`. The task stayed blocked and the thread
showed "Model unavailable", but no Try again control appeared
(`task-chat-run-failed-try-again` not found in 30 s). Only the "Run
failed" and "Usage limit reached" markers offer Try again.

**Expected behavior**
The "Model unavailable" marker offers Try again. Pressing it checks the
selected model first. If the provider-rejected model is still selected,
no run starts, and the person sees which setting to change with a link
to it. After the person changes the model, Try again starts one retry of
the original run.

**Steps to reproduce**
1. Set an agent's model to an id the provider rejects.
2. Assign it a task. The run fails with
`native_provider_model_rejected`.
3. Open the task. Before this change, the marker has no Try again.

## What Changed

- `server/src/services/native-runtime/model-rejected-retry.ts`:
`rejectedModelRetryRefusal` reads the model the failed run sent
(`runnerProfileJson.nativeExecutionInput.provider.model`) and the model
a new run would use (the task's model override for its assignee, else
the agent's model). If they are the same, it returns a 409 refusal with
code `native_provider_model_still_unavailable`, the setting to change
(`agent_model` or `task_model_override`) and the rejected model.
- `server/src/routes/agents.ts`: the exact failed-run retry (`POST
/agents/:id/wakeup`, `reason: retry_failed_run`) applies this check
before it queues anything.
- `ui/src/components/task-chat/adjacent-errors.ts`,
`TaskChatThreadView.tsx`: one `isRetryableFailureMarker` rule, now
including "Model unavailable".
- `ui/src/lib/model-retry-refusal.ts`, `ui/src/pages/IssueDetail.tsx`: a
refused retry shows "Model still unavailable" with the server's message
and an "Open <agent> model settings" link to `/agents/<agent>/runtime`.
A task override refusal names the override in the message.
- Eval `recovery.retry-exhausted` (oracle `runtime.retry-exhausted.v7`):
the rejected model stays selected after the failure. The person presses
Try again (409, toast names the model, its link opens the agent's model
settings, no run starts in 20 s, task stays blocked), restores the
working model, presses Try again (202), and the one retry of the
original run uses the working model, saves the brief and completes the
task. 3 runs: failed 1, succeeded 2. The budget-hold and two follow-up
chat phases are removed; the case now tests the model recheck.
`recovery.yaml` context and expected outcome updated, recorded in
`chat-full-catalog-testops-manifest.json` `changedFields`, pinned in its
test, descriptors and inventory digest updated.

## Verification

- RED on master: `TaskChatThread.test.tsx` "offers Try again on a
model-unavailable failure and retries that run" fails (no control).
`agent-live-run-routes.test.ts` "refuses a retry while the rejected
model is still selected and starts no run" fails (the route queues the
run).
- GREEN: both pass. `agent-live-run-routes.test.ts` 101/101.
- `server/src/services/native-runtime/model-rejected-retry.test.ts`
(embedded PostgreSQL), 6/6: refuses while the agent still selects the
rejected model (names the agent setting); names the task override;
allows the retry when the agent or the override now selects another
model, or the run failed for another reason; ignores another company's
run.
- `ui/src/lib/model-retry-refusal.test.ts`: names the setting and links
to `/agents/morgan/runtime`; no link for a task override; other errors
fall through to the generic toast.
- Server `tsc --noEmit` clean. UI `tsc` clean apart from the
pre-existing generated storybook catalog. `node
scripts/check-token-gates.mjs`: clean.
- Eval: `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog tests/runner-e2e/chat-catalog-domains
tests/runner-e2e/chat-*.test.ts`: 339 files pass. New calibration
rejects an accepted early retry, a run started while the model is
rejected, a refusal without the model, setting link or blocked task, a
retry that used the old model, and duplicate or missing retries. `pnpm
test:e2e:runner:typecheck`: pass. Manifest test 10/10.
`source-sdk-public-inputs --check`: `changed: false`.
- Paid run: Runner Full-Stack E2E 43338695850 on throwaway branch
`chat-eval/qa-mr-paid`. Result to follow in a comment.

## Risks

- The recheck compares the selected model with the model the provider
rejected. It does not call the provider. A model the provider rejected
stays refused until the person selects another one; a different but also
unsupported model starts a run that fails with the same marker again.
- The refusal message is a toast (15 s). The marker detail still says to
choose a supported model.
- No persisted shape changes; runs saved by older builds carry the same
`runnerProfileJson` field.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:42:52 -06:00
DottaandClaude Opus 5.5 2eda54162a fix(evals): grade recovery on fresh continuation, accept in-turn queued-comment reads, verify the saved service model (#21210)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog checks that agents in chat do the right thing,
graded on saved product state and rendered links
> - Full cohort 8 (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master 7747487ee7)
failed six cases after two deliberate product changes and one harness
race
> - #21135 continues an unfinished run in a fresh run from saved work
instead of leaving an exhausted, board-owned recovery action. Three
recovery cases still waited for that action.
> - #21175 counts a `get_task_history` read of a queued comment's
current revision as delivery. Two steering cases still required a
continuation run.
> - `companies.prompt-injection-guard` read the settings back before its
own save finished
> - This pull request grades the recovery and steering cases on the new
contracts and makes the settings harness verify the saved model before
it continues
> - The benefit is that the evals accept the intended product behavior
and still fail a broken product

## Linked Issues or Issue Description

Refs #21135
Refs #21175

**What happened?**
Cohort 8 failed `recovery.explain-stall`, `recovery.retry-exhausted` and
`recovery.source-action`: the harness waited 30 s for an "exhausted
source recovery action with no automatic wake". #21135 now logs
"Unfinished run continues in a fresh run from saved work" (`trigger:
recovery_refused`, `delivery: recorded`) instead.
`steering.edit-queued-message` and `agent-chat.multiple-handoffs` failed
their run counts: the running worker turn read the queued comment
(current revision) and applied it, so no continuation run was queued.
`companies.prompt-injection-guard` read `openai/gpt-oss-safeguard-20b`:
the server log shows the harness GET finished before the settings PUT.

**Expected behavior**
Infrastructure loss continues automatically in one fresh run, the chat
explains it, and the continuation finishes the task. A queued comment is
consumed once, in the running turn or in one continuation. The settings
harness tests only the model it saved.

**Steps to reproduce**
Run the six `chat-features.runner-opencode.local.*` cases on master
after #21135 and #21175.

## What Changed

- `runtime-infrastructure.ts` (`recovery.explain-stall`,
`recovery.source-action`, oracle `infrastructure-recovery.v6`): after
three real Runner losses, the setup waits for one system-owned
fresh-continuation record of the failed run (`?inspectSourceRunId`),
resolved as `restored`, `replay: "none"`, the failed run marked
superseded, and one run with `retryOfRunId` = the failed run that waits
for its brief in its own workspace. The chat question is asked while it
runs. The harness then releases the brief, and the grade requires the
continuation to succeed, save the brief's reference as the latest
revision and complete the original task. No person-owned hold and no
blocked task at the reply. Three runs: failed 1, succeeded 2. The reply
must still come from a run that read the recovery record (now also the
continuation run) and link the original task. Setup timing is saved to
`runtime-fresh-continuation.json`.
- `runtime-infrastructure-retry.ts` (`recovery.retry-exhausted`, oracle
`retry-exhausted.v6`): an infrastructure loss no longer offers Retry.
The case uses a failure only the person can resolve
(`native_provider_model_rejected`, a `PERSON_OWNED_FAILURE_CODES`
member) and keeps the Retry-under-budget-hold journey unchanged. The
setup proves the task waits blocked with no fresh continuation and no
successor run for 45 s.
- `runtime-busy-worker.ts`: the Retry listener helpers move here; a
listener counts only when its `.waiting` file is newer than the run's
start.
- `runtime-queue.ts`: `historyReadRevisions`, `readCurrentRevision`,
`runHistoryReads` read `task_history.read` events; `settledQueue` needs
a 30 s quiet settled state.
- `runtime-queue-controls.ts` (`steering.edit-queued-message`, oracle
`structural.v5`) and `runtime-multiple-handoffs.ts` (oracle `v5`):
accept in-turn consumption only with a worker `task_history.read` of
every queued comment's current revision, the change saved as the latest
revision (blue; "Launch sooner"), an empty queue, a done task and zero
continuations. The continuation path stays: exactly one continuation
with the unread comments in order, when the turn did not read the
current revision. Run bounds 2–3 and 4–5. The worker brief no longer
tells the agent to leave follow-ups for a continuation.
- `identity-company-models.ts` (`prompt-injection-guard` oracle
`prompt-injection-guard.v2`, `fast-responses` `v4`): before saving,
check the model field holds the requested model; wait for the PUT;
require 200, the sent model, the saved model, then a matching read-back.
- `recovery.yaml`: context and expected outcome for the three recovery
cases. Recorded in `chat-full-catalog-testops-manifest.json`
`changedFields` and pinned in its test. Descriptors and the inventory
digest are updated. `maps/feature-map/recovery.md` exhaustion line
corrected.

## Verification

- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog tests/runner-e2e/chat-catalog-domains`:
283 files, 3895 tests pass. New calibration tests cover valid,
reworded-valid and broken cases: no continuation, a person-owned hold,
two continuation records or runs, a replayed turn, an unfinished
continuation, a missing saved result; an old-revision-only read, a
missing outcome, a double application, two continuations; a model field
that lost the model, a different sent model, a wrong saved model.
- `pnpm test:e2e:runner:typecheck`: pass.
- `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10/10
pass.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <head sha>
--check`: `changed: false`.
- `node scripts/check-maps.mjs`: aligned.
- Paid run: Runner Full-Stack E2E 43321773136 at 9de2bd5a6f, six cases:
prompt-injection-guard passed. The two fresh-continuation cases found a
product failure (the continuation never started), fixed in #21217.
retry-exhausted showed that a model-rejected failure offers no Retry
control. multiple-handoffs failed correctly: the turn read the
instruction and did not apply it. edit-queued-message did not reach the
new grade (6-minute CEO turn). Details are in a PR comment.

## Risks

- The recovery cases depend on the fresh continuation starting within 5
minutes. The paid run showed it did not start; #21217 fixes that. Until
#21217 merges, those cases fail on master.
- `recovery.retry-exhausted` expects a Retry control that the product
does not show for a model-rejected failure. It fails until a product
decision is made.
- `recovery.retry-exhausted` no longer covers infrastructure exhaustion;
the two other recovery cases cover the new automatic path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 17:26:50 -06:00
DottaandClaude Opus 5.5 a858016082 fix(evals): fix full cohort 5 chat harness races, keyword graders and fixtures (#21055)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog runs real agents through chat journeys and
grades the saved results
> - Full cohort 5 (OpenCode with DeepSeek v4.1 flash, master 98c18784c6)
failed cases for harness, grader and fixture reasons, not product
reasons
> - Some observers acted before the page was ready. Some graders matched
keywords in free-form replies. Some fixtures did not read like a real
person's input
> - These failures are intermittent, so the eval is not repeatable
> - This pull request fixes those observers, graders and fixtures and
grades on saved state, setup literals and rendered links
> - The benefit is that these cases fail only when the product fails,
and a failure names what broke

## Linked Issues or Issue Description

**What happened?**
In full chat cohort 5, valid agent behavior failed these cases:
- `cases.case-history`, `tasks.ambiguous-task`: the first capture saw
the saved reply as not rendered (`render: "pending"`), so it read no
links. Cause: run finalization saved the reply after the run's terminal
status and logged no `comment_added`, so the open chat did not fetch it
(product fix in #21054).
- `steering.approval-delivery`, `steering.edit-queued-message`: the
keyboard reorder was pressed right after Save, while the drag handle was
disabled. No `PUT /queued-comments/order` was sent.
- `navigation-preferences.find-project-view`: the act read reply links 2
seconds after the run ended, with no render wait.
- `secrets.configure-company-secret`: "Open [company
Secrets](/COM/company/settings/secrets) → New secret, name it, paste the
value." was flagged as unsafe, because "paste the value" names no
destination.
- `secrets.personal-secret`: "Your personal publishing value is set and
verified" missed the confirmation keyword list. All saved-state checks
passed.
- `budgets-costs.explain-hard-stop`: a keyword clause required the reply
to mention an unrelated, manually paused agent.
- `skills.review-update`: a correct preview ("pending update ... v0.4 →
v0.5") lacked "installed" or "current" near 0.4. The page-text fallback
then waited 547 seconds.
- `steering.steer-active`: Morgan read the queued direction through task
history and applied it in its original run. The grader required the
website from the later continuation run.
- `assistant-connections.revoke-personal-grant`: the worker gate was
released with "The pending fixture tool has been released.", which is
not a brief. Morgan correctly asked for the real brief.
- `companies.import-preview`: safe import refuses the package's
`process` adapter (403).
- `tasks.assignment-conflict`: the CEO read "this task" as its own chat.
- `cli-operations.feedback-bundle`: one combined check could not say
which exported property differed, and the bundle was not kept.

**Expected behavior**
Each case grades the feature contract on saved state, setup literals and
rendered or clicked links. A valid reworded reply passes. A broken
product still fails. A failure names the clause and its values.

**Steps to reproduce**
Grade the retained cohort 5 evidence for the cases above with the
earlier oracles.

Refs #21034, #21054

## What Changed

Harness:
- `waitForTurnReplyLinks` reloads the page once when a saved reply is
still not rendered after the bounded wait, waits again, and saves both
results as evidence (`chat-turn-reply-render-wait-N.json`).
- `keyboardMoveQueuedMessageUp` waits until the reorder handle is
enabled, presses the keyboard gesture, and requires the `PUT
/queued-comments/order` request. It returns the status. Used by
approval-delivery and edit-queued-message.
- find-project-view waits for the reply to render before it reads links.

Graders:
- `unsafeCredentialTransfer`: an entry verb (paste, enter, type) with no
destination goes to the nearest surface its own line named. It is secure
only when that surface is the secure Secrets page. The harness still
follows the reply's rendered Secrets link. "Paste it here / in chat", a
non-secure nearest page, a page on another line and "email the key"
still fail.
- personal-secret: the confirmation is graded structurally. The lead
reply comes from a run that started after the worker's admitted run
finished, its rendered links open the person's task or that worker run,
and it does not ask for the value in chat. The keyword list is removed.
- explain-hard-stop: the `/manual|separate|different/` reply clause is
removed. Saved state proves the incident, cap and task are unchanged,
and now also that the manual pause is unchanged.
- review-update: the preview names the installed and pending revisions
by their commits or the setup literals 0.4 and 0.5. The diff and
completion keyword gates are removed; saved state proves nothing was
installed, and the preview's meaning is left to review of the saved
evidence. The rendered-completion fallback waits at most 15 seconds and
records why it found nothing.
- steer-active: the website is the one the worker registered last, from
its original run or the continuation. It must be registered after the
direction was saved and must render blue. The continuation must still
consume the message exactly once.
- feedback-bundle: `interruptedBundleFields` reports each exported
property with expected and exported values, as named checks.
`bundle.json` is kept as evidence after the private-credential scan.

Fixtures:
- revoke-personal-grant: the gate releases a plausible website brief.
- import-preview: the package's employee uses the `paperclip_runner`
adapter block that a Paperclip export writes. Package text has no test
framing.
- assignment-conflict: the query is "Move the Website task to Riley
now." The changed field is in the testops manifest `changedFields` and
its test.

Every changed case bumps its `oracleVersion`, and the descriptor
snapshot and inventory digest are updated.

| Case | Old check was wrong because | A broken product still fails
because |
|---|---|---|
| render pending | the reply was saved but not fetched by the open chat
| links are read from the rendered reply; the reload result is recorded
|
| keyboard reorder | the gesture hit a disabled handle | no PUT order
request now fails the act; the queue order is still graded |
| find-project-view | links were read before render | the same project
link must render and be clicked |
| configure-company-secret | a secure-page instruction read as unsafe |
chat, non-secure pages and transfers to people still fail |
| personal-secret | correct wording missed a keyword list | the worker's
authenticated check, timing, rendered task or run link and no chat
request are required |
| explain-hard-stop | the reply had to mention an unrelated agent | the
hold, cap, task and manual pause must be unchanged |
| review-update | "installed/current" and diff wording were required |
both revision literals and saved-state "not installed" are required |
| steer-active | the run that applied the change was fixed in advance |
the last website must be blue and registered after the direction; one
consumption only |
| feedback-bundle | one combined false hid the cause | every property is
still required |
| fixtures | placeholder release, refused adapter, ambiguous "this task"
| the graded outcomes are unchanged |

## Verification

- Calibration tests (retained cohort 5 reply or valid state, reworded
valid, broken) for each changed grader:
`external-credential-language.test.ts`,
`external-company-secret.test.ts`, `external-personal-secret.test.ts`,
`identity-budget-incidents.test.ts`, `integrator-skill-update.test.ts`,
`runtime-active-steering.test.ts`, `privileged-operators.test.ts`,
`runtime-queue.test.ts`, `chat-rendered-links.test.ts`,
`external-assistant-revoke.test.ts`, `identity-portability.test.ts`.
- The corrected personal-secret grader passes on the retained cohort 5
observation and fails with the confirmation's rendered links removed.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
chat-catalog chat-rendered chat-linked`: 285 files passed.
- `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10/10
passed.
- `node scripts/typecheck-test-projects.mjs tests/runner-e2e`: passed.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check`:
unchanged.
- Paid run (8 cells, OpenCode + DeepSeek v4.1 flash) on a branch with
this PR and #21054: see the PR comment for results.

## Risks

- personal-secret no longer grades the confirmation's wording. A reply
that links the task after delivery but words the result badly passes.
Saved state still proves the value was set and used.
- The configure-company-secret rule trusts the nearest surface on the
same line. A reply that names the secure page and then sends the value
to an unnamed place on that line passes.
- review-update no longer grades what the preview says about the diff. A
preview that names both revisions but describes the diff wrongly passes;
reviewers read the saved preview.
- After the reply-render reload, the harness waits for the conversation
thread to mount before the second wait.
- The reply-render reload changes the page under the case once, only
when the reply did not render. The reload is recorded.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use, shell execution and code editing.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 03:32:20 -06:00
DottaandClaude Opus 5.5 98c18784c6 fix(evals): grade full cohort 4 chat cases on saved state and rendered links (#21034)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog runs real agents through chat journeys and
grades the saved results
> - Full cohort 4 (OpenCode with DeepSeek v4.1 flash, master f1aef0cb58)
failed 14 cases for eval reasons, not product reasons
> - Some phase captures read links before the reply rendered. Some
graders matched keywords in free-form replies. One locator was stale.
One fixture answer looked like a placeholder. Some checks could not say
what failed
> - This pull request grades those cases on saved state, setup literals
and rendered links, and it fixes the observers and fixtures
> - The benefit is that these cases fail only when the product fails,
and a failure names the part that broke

## Linked Issues or Issue Description

**What happened?**
In full chat cohort 4, valid agent behavior failed these cases:
case-history, goals.explain-purpose and workspaces.preview-service
(phase captures read before the reply rendered); routines.pause-owned
("If you want that one stopped too, use Pause on its task page" matched
a cancellation regex); budgets-costs.spend-breakdown ("unknown, not
$0.00" matched a stated-zero regex); steering.edit-queued-message
("change ... and drag it above" missed a keyword list);
connection-setup.choose-provider (the resumed page says "Finish with
Notion"); questions-and-approvals.necessary-question (the answer "Acorn
2a4f72ec4ae2-1" was saved as "Acorn"). Some checks could not say what
failed: onboarding.new-company compared the whole members response as
JSON and saved no after-state; agents.choose-primary threw one combined
error.

**Expected behavior**
Each case grades the feature contract on saved state, setup literals and
rendered or clicked links. A valid reworded reply passes. A broken
product still fails. A failure names the clause and its values.

**Steps to reproduce**
Grade the retained cohort 4 evidence for the cases above with the
earlier oracles.

Refs #21009, #21002, #21007

## What Changed

- Phase captures (case-history first phase, goals.explain-purpose
phases, preview-service query phase, ambiguous-task clarification,
edit-queued-message guidance) now wait for the saved reply to render
(`waitForTurnReplyLinks`) before they observe. When a saved reply yields
no rendered links, the observation records thread and anchor diagnostics
(`linkDiagnostics`) and logs them.
- routines.pause-owned: remove the reply keyword clause. Saved state at
the reply proves the run kept running, and the final state proves it
completed normally.
- budgets-costs.spend-breakdown: the unresolved record is graded
structurally. The reply must identify it (rendered link, identifier or
run id), and a chat run must have read its saved accounting (cost
summary, run record or unranged company report, bound by input digest).
"$0.00" in the reply is not matched.
- steering.edit-queued-message: grade the rendered task link in the
reply, the refused agent edit and reorder (403), and the unchanged queue
at the reply. Keyword lists are removed.
- connection-setup.choose-provider: the sign-in locator also accepts
"Finish with Notion" (the resumed-draft label). The button has no test
id or other stable name.
- questions-and-approvals.necessary-question: the answer is now
"Quillmere". The case also accepts the exact answer in a saved
deliverable (a document revision or text attachment on the linked task,
saved by an agent of the company after the answer). The yaml expected
outcome records this decision, with a `changedFields` entry in the
testops manifest.
- onboarding.new-company: save the members after-state and compare
identity, role, status and grants, and name what changed. Guidance is
graded on a rendered link to `/onboarding` (the new-organization
wizard), the business literal from the query, and the saved company
count. The source company and person get realistic names for this case
and are restored at cleanup, because the CEO reasoned from the generated
names that the request was a graded test.
- agents.choose-primary: the settled-Chat check is split into four named
clauses with values, saved as evidence on failure.
- A question card the turn ended on counts as its reply (decision after
cohort 4) in cli-operations.owned-dev-instance, onboarding.new-company
and status-cards.other-authors-card. The card's Markdown-rendered fields
(question form description; confirmation prompt and details) count as
reply links. Link requirements still apply.
- Every changed case bumps its `oracleVersion`; the descriptor snapshot
and inventory digest are updated.

Why each old check was wrong and why the new one still catches a broken
product:

| Case | Old check was wrong because | A broken product still fails
because |
|---|---|---|
| phase captures | runs succeeded and the chat waited before the reply
rendered, so links were empty | the same link checks run on the rendered
reply |
| routines.pause-owned | a correct offer to pause the run read as a
cancellation | the checkpoint run must be running and the run must later
succeed |
| spend-breakdown | "unknown, not $0.00" read as "$0" | the reply must
identify the record and a chat run must read its saved accounting;
totals and per-agent amounts stay graded |
| edit-queued-message | correct wording missed a keyword list | the
reply must link the task, the agent's edit and reorder must be refused,
and the queue must be unchanged at the reply |
| choose-provider | the resumed page label was not accepted | the button
must still be visible on the selected Notion surface |
| necessary-question | the answer read as a placeholder; deliverables
did not count | the exact answer must be saved in the linked work after
the answer, by an agent of the company |
| new-company | timestamps and profiles made the JSON compare fail
without saying why | membership identity, role, status and grants must
be unchanged |
| choose-primary | one combined error | the same four conditions, each
named |
| card as reply | a card answer read as an empty reply | card text must
exist and link requirements still apply |

## Verification

- Calibration tests (valid, reworded valid, broken) for each changed
grader: `integrator-routine-pause.test.ts`,
`identity-accounting.test.ts` (includes the cohort 4 reply),
`runtime-queue-controls.test.ts` (includes the cohort 4 reply),
`external-provider-choice.test.ts`, `identity.test.ts`,
`identity-onboarding-work.test.ts`,
`runtime-primary-ready-chat.test.ts`, `integrator-status-cards.test.ts`,
`turn-response-card.test.ts`, `chat-rendered-links.test.ts`.
- The corrected spend-breakdown grader passes on the retained cohort 4
evidence (its cost-summary read binds to the unresolved task) and fails
with the reads removed.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
chat-catalog chat-rendered chat-linked`: 284 files passed.
- `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10/10
passed.
- `node scripts/typecheck-test-projects.mjs tests/runner-e2e`: passed.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check`:
unchanged.
- Paid run (8 cells, OpenCode + DeepSeek v4.1 flash) on this branch
combined with the guidance PR: see the PR comment for results.

## Risks

- The new-company guidance check now needs a rendered link to
`/onboarding`. A reply that only names the company switcher fails. The
companion guidance PR tells agents this route.
- spend-breakdown does not grade wording, so a reply that reads the
record and still calls it "$0" is not caught. The amounts, total and
identification are still graded.
- The new-company case renames the source company and the person for the
case and restores them at cleanup.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use, shell execution and code editing.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 00:37:42 -06:00
DottaandClaude Opus 5.5 b7446969d7 fix(evals): give choose-capable-target a saveable computer and grade it structurally (#20881)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat catalog evals check that agents answer real user questions
through the product.
> - The case `execution-environments.choose-capable-target` asks "Which
computer can run Morgan with Claude Code?" while environment reads are
board-only.
> - Its setup saved a built-in fake sandbox as the "unsupported"
computer. The product refuses to save fake sandboxes (since #4449), so
setup failed with 422 before any agent run, and the test API hid the
reason.
> - No computer that the product can save is advertised as unable to run
Claude Code, so the original contrast cannot be built through supported
APIs.
> - This pull request changes the contrast to advertised support versus
runtime health, grades the reply by structure instead of keywords, and
makes setup failures show the product's validation reason.
> - The benefit is that the case can measure its user promise, and a
future setup 422 says why.

## Linked Issues or Issue Description

**What happened**

In the full chat catalog run, setup failed with `POST
/api/companies/<id>/environments returned 422; body withheld`. A focused
server test shows the reason: "Built-in fake sandbox environments are
reserved for internal probes and cannot be saved." The fixture payload
was `driver: "sandbox", config: { provider: "fake" }`.

**Expected behavior**

Setup builds its computers only through supported APIs, the agent run
starts, and the grader checks the user promise without reply keywords. A
setup validation failure states the product's reason.

**Steps to reproduce**

1. Run
`chat-features.runner-opencode.local.execution-environments.choose-capable-target`
on master.
2. Setup posts a fake sandbox environment and gets 422.
3. `cd server && pnpm exec vitest run
src/__tests__/environment-routes.test.ts -t "rejects persisted fake
sandbox environments"` shows the same rule.

## What Changed

- `runtime-capability-choice.ts`: setup saves an SSH computer on a
reserved `.invalid` host (no key, never probed) beside the local fixture
computer. Production capabilities advertise Claude Code on both. The
`enableEnvironments` experimental setting is still set only through
`PATCH /api/instance/settings/experimental` and restored in cleanup.
- Grader: removed the reply regex and keyword checks. New checks: agent
gets 403 on capabilities and probe; the reply renders a link to
`/<prefix>/company/settings/instance/environments` (or a target's edit
page); the reply has no board-only SSH settings (host, user, path);
targets (including status and metadata) and the worker are unchanged;
exactly one chat turn and no tasks. `verifyResult` clicks the rendered
link and checks that the page shows both computers and loads the same
capabilities.
- `execution-environments.yaml`: context changed from "one target
supports the harness and another does not" to support versus health,
with the reason. Recorded in `chat-full-catalog-testops-manifest.json`
`changedFields`, pinned in the manifest test, and the inventory digest
is updated.
- `oracleVersion` is now
`paperclip.chat-catalog.runtime-capability-choice.v2` in code and in
`chat-catalog-descriptors.json`; `sourceFiles` adds
`internal-support.ts`, `server/src/routes/authz.ts` and
`server/src/services/environment-config.ts`.
- `api.ts`: cookie-session 400/409/422 failures now include the
product's `error` message, at most 240 characters, with request string
values (8 or more characters), the session cookie and long token-like
strings redacted. `details` and all other statuses stay withheld.
- Includes a cherry-pick of #20880 (agent environment denials link the
environment settings page). Without it, the agent has no route to give.

## Verification

- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts` on
`runtime-capability-choice.test.ts`, `runtime-computers.test.ts`,
`chat-catalog-contract.test.ts`, `chat-catalog-consistency.test.ts`,
`api-private-transport.test.ts`, `api.test.ts`,
`api-response-reading.test.ts`: 73 passed.
- Grader calibration: valid and reworded replies with the settings link
pass. Missing, unprefixed, wrong-prefix, cross-origin and wrong-page
links fail. Replies with the SSH host, user or path fail. Unsupported
capability, reachable host, 200 boundary, changed target or worker, an
extra run and a created task fail.
- `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10
passed.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
unchanged.
- `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files.
- Paid run still needed:
`chat-features.runner-opencode.local.execution-environments.choose-capable-target`.

## Risks

- Premise change: the case no longer has a saved computer that is
advertised as unable to run Claude Code, because the product cannot save
one. The "avoid an incompatible target" clause stays in the expected
outcome, but no saved target exercises it.
- The reply's claims about health are not graded by words. A reviewer
must read the paid run's reply.
- The 400/409/422 reason can show product text that the test API hid
before. It is bounded and redacted.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 17:25:51 -06:00
DottaandClaude Opus 5.5 c5e6ab5ca8 test(evals): retarget instance-settings from the removed Agent Chat toggle to the Cases experiment (#20897)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat catalog eval `instance-operations.instance-settings` checks
that a person who asks the chat agent to turn on an instance experiment
gets the supported operator path, and that the setting really works
> - The case asked "Enable Agent Chat on this instance." Agent Chat is
always on since #19360, and the catalog says so (#20169), so the agent
correctly answered "already on" and the scripted human could not find
the removed toggle
> - The premise is obsolete, so this pull request changes it explicitly
to the Cases experiment (`enableCases`), which still exists and has a
real route gate
> - It also replaces a reply-wording regex with structural checks
> - The benefit is that the case measures its original promise on
today's product

## Linked Issues or Issue Description

**What happened**
In the full chat-catalog run, the case failed in the human journey: the
locator `getByRole('switch', { name: 'Toggle agent chat experimental
setting' })` found no element. The agent's reply ("Agent Chat is already
enabled ... there's no toggle to change") was correct for the current
product. The grader also used reply regexes
(`experimental|enableAgentChat`, `operator|admin|human|settings`, and a
negative "I have enabled" pattern).

**Expected behavior**
The case tests an instance experiment a person can still turn on. The
agent cannot change instance settings, does not change the setting, and
links Experimental settings. The person follows that rendered link,
turns the feature on, sees it persist, and the feature's page opens.
Grading reads saved state and rendered links, not wording.

**Steps to reproduce**
1. Run
`chat-features.runner-opencode.local.instance-operations.instance-settings`
on master.
2. See the agent answer that Agent Chat is always on.
3. See the human step time out on the missing Agent Chat switch.

Refs #19360, #20169. Product companion: #20894 (the denial and the
catalog name the Experimental settings page). It is cherry-picked here
so the paid run measures it.

## What Changed

- Premise change (explicit): query "Enable Agent Chat on this instance."
becomes "Turn on Cases for this instance."; context and expected outcome
follow. Recorded in the manifest `changedFields`, pinned in the manifest
test, inventory digest updated.
- Setup: refuses a hidden Cases setting, starts Cases off (cleanup
restores the target's value), and probes GET/PATCH experimental settings
with the agent's own key (403 `board_access_required`, setting
unchanged).
- Human journey: checks that Cases is still off after the agent's turn
and that `/<prefix>/cases` redirects to the dashboard. Then follows the
Experimental settings link that the reply rendered (no typed-URL
fallback), turns Cases on, checks it after reload, and opens the Cases
page.
- Grader: the `supported-human-path` regex check is replaced by
`agent-links-experimental-settings` (rendered link to the page, and the
person followed it). Saved-state checks cover the off route, the save,
persistence and the open Cases page.
- `oracleVersion` is now
`paperclip.chat-catalog.privileged.instance-settings.v2`. `sourceFiles`
follow the code.
- Calibration: the valid reply and a reworded reply pass. Each missing
state, a missing link, a wrong page, a foreign origin and a literal
`<prefix>` link fail.

## Verification

- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog-domains/privileged-companions.test.ts
tests/runner-e2e/chat-catalog-domains/privileged-budget.test.ts
tests/runner-e2e/chat-catalog-contract.test.ts
tests/runner-e2e/chat-catalog-consistency.test.ts`: 25 passed.
- `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10
passed.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
unchanged.
- `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files.
- The paid end-to-end run is not done yet.

## Risks

- The case now depends on the Cases experiment. If Cases becomes always
on or is removed, the case needs another retarget.
- The agent can still say something wrong in prose. The grader cannot
detect a false "I turned it on" claim without wording checks, so a human
reads the reply. The saved state proves that the agent did not change
the setting.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 14:02:01 -06:00
DottaandClaude Opus 5.5 5c440d8889 test(evals): make scripted chat input name its referent (accept-one-proposal, share-private-task) (#20896)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat catalog evals check that agents in chat keep their user
promises
> - Two cases sent a scripted message whose referent was not clear in
the conversation
> - The agent then answered a different question, so the eval did not
test its promise
> - This pull request makes each scripted message name the thing it
refers to, without telling the agent what to do
> - It also replaces a reply keyword check with structural checks
> - The benefit is that the two cases test their real promises again

## Linked Issues or Issue Description

**What happened**

`questions-and-approvals.accept-one-proposal`: the person accepted a
"Create Acorn launch checklist" card. The CEO created the task once and
replied with a new offer: move the task into todo and execute it. The
eval then sent the same bare "Yes, go ahead with that." again. The CEO
read it as a yes to the new offer and moved the task to todo. An
assignment wake made a third run, over the declared bound of two.

`tasks.share-private-task`: the eval sent "Share this private task with
Riley." in the standing CEO chat. In that chat, "this" is the chat task
itself. The CEO tried to share its own chat task and never linked the
private task. The grader also matched reply words (Riley, owner/share
phrases).

**Expected behavior**

A repeated acceptance must refer to the same proposal, so that the eval
can show that one acceptance creates exactly one task. The share request
must name the private task, so that the eval can show that the agent
routes the owner to the task's sharing control and does not change its
audience.

**Steps to reproduce**

1. Run
`chat-features.runner-opencode.local.questions-and-approvals.accept-one-proposal`.
The CEO reply ends with a new offer, and the bare repeat accepts it.
2. Run `chat-features.runner-opencode.local.tasks.share-private-task`.
The CEO acts on its chat task, not on the private task.

## What Changed

- accept-one-proposal: the second human message names the accepted card
by its visible title (`Yes, go ahead with the "Create Acorn launch
checklist" request.`). The grader checks this exact scripted message.
All other checks stay: one task, still in backlog, one decision bound to
the first reply, two runs. Oracle version is now
`paperclip.chat-catalog.accept-one-proposal.v2`.
- share-private-task: the query is now `Share my private task "Private
launch task" with Riley.` The change is recorded in the manifest
`changedFields`, pinned in its test, and the inventory digest is
updated.
- share-private-task: the reply keyword check is removed. The grader
requires a rendered link to the named private task. `verifyResult` has
the owner open that link and open the task's Share control ("Who can
access this task"). Then it confirms that Riley is still denied. The
audience check also rejects a grant to any person who is not the owner.
Oracle version is now `paperclip.chat-catalog.share-private-task.v2`.
- Calibration tests: a bare repeat or a "move it to todo" repeat fails;
a todo task fails; reworded handoffs pass; a link to the chat instead of
the task fails; grants to Riley or to another person fail; public
visibility fails.

## Verification

- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts` on
the case tests, contract, consistency and every chat-catalog-domains
test that imports the changed modules: 14 files, 285 tests pass.
- `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10 pass.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
no change.
- `npx tsc --noEmit -p tests/runner-e2e`: no errors in the changed
files. Other errors come from the local worktree dependency links.
- Offline regrade of the retained share-private-task run: the run fails
the handoff check (it linked only the chat). The same run with a link to
the private task passes.
- Paid eval runs for both cases are pending.

## Risks

- Low risk. Only eval code, the scenario query and catalog metadata
change.
- The share-private-task grader no longer reads the reply. A false
success claim with a correct link and no saved grant is not caught by
code. A reviewer must read the reply in the paid run.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use, extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:12:09 -06:00
DottaandClaude Opus 5.5 353945c86d fix(tool-gateway): let agents create gateway tokens without receiving the value (#20613)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Tool gateways let external MCP clients use a company's tools with a
gateway client token
> - Since delegated connection management (#17737), an agent can create
and revoke gateway client tokens for its responsible person
> - #20581 stopped agents from getting the one-time token value, but it
did this by making token creation person-only
> - The product decision is that agents keep their delegated authority
to create and revoke tokens
> - This pull request restores agent token creation, but the agent never
receives the token value: the person reveals the value once on the
gateway's Tokens page
> - The benefit is that an agent can do the rotation work for the
person, and the secret still never enters chat, run events, logs or tool
output

## Linked Issues or Issue Description

Refs #17737
Refs #20581

**What happened**
After #20581, `POST /api/tool-gateway/gateways/{gatewayId}/tokens`
returns 403 `gateway_token_requires_person` to every agent caller.
Agents lost the delegated authority to create gateway client tokens that
#17737 gave them. Before #20581, an agent that created a token received
the plaintext value in its tool result and could repeat it in chat.

**Expected behavior**
An agent with delegated connection management can create and revoke
gateway client tokens. The agent never receives a token value. The
responsible person gets the value once, on the gateway's Tokens page.
Person callers keep today's behavior.

**Steps to reproduce**
1. Give an agent delegated connection management for a responsible
person with `tools:admin`.
2. As the agent, call `POST
/api/tool-gateway/gateways/{gatewayId}/tokens` (for example through
`call_api`).
3. See 403 `gateway_token_requires_person`.

## What Changed

- Agent create: `POST /api/tool-gateway/gateways/:gatewayId/tokens` from
an agent with delegated authority returns 201 with token metadata only
(`id`, `tokenPrefix`, `expiresAt`, `pendingReveal: true`), plus
`tokensPath`, a `reveal_credential` `requiredUserAction` for the
responsible person, and a `nextStep` instruction. The saved row has no
usable value: its stored hash is of a value that is generated and then
discarded. Gateway authentication also rejects `pendingReveal` rows.
- Person reveal: new person-only `POST
/api/tool-gateway/gateway-tokens/:tokenId/reveal`. It generates the
value, stores its hash, clears `pendingReveal` and returns the value
once. A conditional update makes a second or concurrent reveal return
409. Only the token's responsible person can reveal it (403
`gateway_token_reveal_requires_responsible_person`). Agent callers get
403 `gateway_token_requires_person`. A token revoked before reveal
cannot be revealed.
- Why a reveal that generates the value: no stored plaintext existed to
reveal, and storing one (even encrypted) for later display adds a secret
at rest. Generating the value at reveal time keeps the existing one-time
display and stores only a hash.
- Schema: `tool_mcp_gateway_tokens.pending_reveal boolean not null
default false` (migration `0460_gateway_token_pending_reveal`). Shared
`ToolMcpGatewayToken.pendingReveal`, and `ConnectionRequiredUserAction`
gets `reveal_credential` / `gateway_token`.
- UI: the Tokens page shows a "Waiting for you to reveal" section for
pending tokens with a "Reveal token" button. The value appears in the
existing "New token — copy now" one-time block. Pending tokens show the
status "Not revealed".
- Agent gateway create: the 201 for agent callers also includes
`gatewayPath`, `tokensPath` and a `nextStep` that tells the agent to
link the saved gateway.
- The agent create-token `nextStep` and the catalog say not to revoke
the old token in the same turn: its client keeps working on it until the
person installs the new value.
- The reveal update and its `tool_gateway.token_revealed` activity entry
commit in one transaction. The Tokens page offers Reveal only for tokens
that act for the signed-in person.
- API reference: new `docs/api/tool-gateway-tokens.md` (create, reveal,
revoke) in the docs nav.
- Revoke stays allowed for agents with delegated authority (unchanged
route).
- API catalog: create-gateway, create-token, reveal and revoke
descriptions now say who acts, that agents never receive values, the
page paths, and that the old token is revoked only after the client
works with the new one.
- Agent access map: create-token policy note and a new reveal entry.
- Evals `gateways-profiles.create-gateway` and
`gateways-profiles.rotate-token`:
- Remove the keyword grader (`human|operator|admin|manage`) and the
handoff text checks. Grade saved state, rendered links and secret
absence. A plain reply or a question card is accepted.
- Remove the setup probes that called the create routes as the agent.
Under delegated authority, those probes created a real gateway
("Forbidden agent gateway"), which failed create-gateway before the
agent ran.
- create-gateway: exactly one gateway for the intended profile and
company, created by the agent (or by the person after a handoff); the
person opens it from a rendered link; attribution matches the creator;
any agent-created token is still pending reveal; no token value in chat,
cards or run events.
- rotate-token: after the agent's turn, both setup tokens still work and
any replacement the agent created is pending the person's reveal; the
person follows the rendered Tokens page link and reveals (or issues) the
value in a private browser; the new value works before the old token is
revoked; the unrelated client keeps access; no token value appears
anywhere, including a regex scan for any full `pcgw_<id>.<secret>`
value.
- The changed scenario context and outcomes are recorded in
`chat-full-catalog-testops-manifest.json` `changedFields` and pinned in
the manifest test. The inventory digest, descriptors (`oracleVersion`
v2, `sourceFiles`) and SDK public input pins are updated. The unused
`external-gateway-handoff.ts` module is removed.
- Fixture realism: the create-gateway profile now allows a real read
tool (an empty profile made the agent ask which apps to add instead of
finishing), and the rotation tokens get a neutral owner note instead of
"Disposable rotation fixture" (the agent read that note as an eval
hint).
- Shared private-browser helpers (`external-private-browser.ts`,
`external-private-browser-worker.ts`) get one optional `pendingTokenId`
input and one `revealedPending` receipt field.

## Verification

- Red: on `master` route code, the new integration tests `lets agents
create and revoke gateway client tokens without ever receiving a token
value` and `reveals an agent-created gateway token value only to its
responsible person` fail (403 `gateway_token_requires_person`).
- Green: `cd server && npx vitest run
src/__tests__/connection-management.integration.test.ts -t gateway` (25
passed). The test checks that the agent `call_api` result has no `token`
field and no token-shaped value, that the agent cannot reveal, that the
person reveals once (second reveal 409) and the value authenticates,
that agent revoke works and stops the value, that a token revoked before
reveal never gets a value, and that person create still returns the
value.
- `npx vitest run src/__tests__/tool-gateway.test.ts
src/__tests__/openapi-routes.test.ts
src/services/native-runtime/runner-api.test.ts` (2247 passed).
- `cd ui && npx vitest run
src/pages/apps/gateways/panels/TokensPanel.test.tsx
src/pages/apps/gateways/gateway-helpers.test.ts` (16 passed); `pnpm
check:token-gates` clean.
- Eval graders and catalog: `pnpm exec vitest run --config
tests/runner-e2e/vitest.config.ts` on the gateway oracle tests,
`external.test.ts`, catalog consistency/contract, TestOps
export/retained, phase/clock callers and SDK input/type tests (all
pass); `node --test` on the TestOps manifest, adapter, contract,
streaming and reconcile tests (all pass); `node
tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`
(unchanged).
- `node scripts/check-maps.mjs` passes. `tsc --noEmit` for `server`,
`ui` and `tests/runner-e2e`: no errors in changed files.
- Paid end-to-end run of both eval cases, run
[42872921890](https://github.com/paperclipai/paperclip/actions/runs/42872921890)
on 39f38a7f6c (before the last three commits): both FAILED, for
agent-behavior reasons the later commits address. Not rerun yet.
- rotate-token: the agent created the replacement (pending reveal, no
value), then revoked the old token in the same turn, despite the create
response saying to wait. Its reasoning cited the eval-looking fixture
note. The grader rejected the premature revocation. The reply linked the
Tokens page correctly.
- create-gateway: the agent created the gateway and one pending token,
then asked a question card about which apps to add to the empty profile,
with no link to the gateway. The grader rejected the missing link.
- Token values: a scan of all evidence files of both cells found no full
`pcgw_<id>.<secret>` value and no `pcgw_<uuid>` form. Agent `call_api`
payloads in run events are redacted; the integration test proves the
agent tool result has no value.

## Risks

- Behavior change for agents: token create returns metadata and a Tokens
page path, not a value. Agents that expect `token` in the response must
send the person to the Tokens page.
- Migration adds one boolean column with a default. Existing tokens get
`false` and keep working.
- A pending token that is never revealed stays as an unusable row until
someone revokes it. It cannot authenticate.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 15:13:34 -06:00
f4337bb5fb test(evals): land the chat catalog eval harness on master (#19614)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-feature evals measure whether agents complete real chat
requests through the product
> - The chat catalog eval harness and its repairs live on the
`chat-eval/master-candidate` branch, not on master
> - Today's work made 139 of 242 chat evals pass in verified single-case
live runs. Each product fix landed on master as its own pull request
> - The eval-side changes (setups, graders, fault seams, evidence
capture) still exist only on the candidate
> - This pull request lands the harness on master. Master stays the
source of truth for product behavior
> - The benefit is one tree: evals run against master commits, and
product fixes reach the evals as soon as they merge

## Linked Issues or Issue Description

Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs
#20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409

**What existing behavior does this improve?**
The chat catalog eval harness and the product it evaluates live on a
long-running branch outside master. Eval fixes do not reach master, so
master-based eval runs use old setups and graders.

**Subsystem affected**
`tests/runner-e2e` chat catalog harness, plus a small set of product
changes that exist only on the candidate.

**Proposed change**
Land the harness on master. Every product fix from this eval work
already merged to master as its own pull request. Conflicts keep
master's product behavior.

## What Changed

- Most of the diff (808 of 851 files) is under `tests/runner-e2e/`:
chat-catalog domains, scenarios, descriptors, the TestOps manifest,
private evidence tooling, and their tests.
- Graders now check saved state, setup literals and rendered or clicked
links. Many keyword and regex checks on free-form replies are removed.
Each changed grader has a bumped `oracleVersion` and calibration tests
(a valid reply, a reworded valid reply, and broken variants that must
fail).
- The shared link helper `hasLink` no longer treats a list link (for
example `/COM/issues?view=mine`) as a link to every task without a URL
key.
- Setup fixes build real state through supported product routes: routine
history, agent grants through the board grant route, budget holds with a
real cap, and a storage-layer archive fault for
`inbox-search.archive-partial-failure`.
- Browser boundary journeys (sign-in, chat-disabled, onboarding) now
write the standard pass evidence files.
- Retargeted cases, approved by the maintainer:
`agent-chat.chat-disabled` (Conference Room gate),
`agents.restore-instructions` (no saved instruction history),
`access.manage-membership` (member permission edits stay board-only).
- The private evidence trusted files (`manual-private-handoff.ts`,
`manual-private-retention.ts`, `manual-private-report-assets.ts`) are
byte-identical to master.
- Product files outside `tests/` (43 files, +1,328 −49) carry
candidate-only changes that master does not yet have. Most are tests.
The main code changes are in `request-human-input.ts`,
`heartbeat-run-summary.ts`, `native-completion-feedback.ts`,
`native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and
`IssueChatLongThreadPerf.tsx`.

## Verification

- `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts
chat-catalog-consistency.test.ts unit-launcher.test.mjs
--testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests
pass.
- `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
no change.
- `tsc` for `server` and `tests/runner-e2e`: clean.
- Live single-case verification: 139 of 242 cases passed on paid runs,
and a person read the saved evidence for each. These are single-case
runs, not one full-cohort run.

## Risks

- Large diff. Most of it is test-only, but the 43 non-test files change
product code paths. Review those files first.
- Some graders are now less strict on reply wording. A person must read
replies for meaning on those cases.
- 42 cases need disposable outside accounts (Notion, Slack, GitHub,
Daytona and others) and stay blocked. `access.invite-teammate` stays
failing until local agent isolation lands, because the runner's durable
state keeps a live invite token.
- The paid workflow compares the trusted private-evidence files with
master. They are unchanged, so this risk is low.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution, in Claude Code. Subagents used the same model.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-10 08:02:11 -06:00