Commit Graph
3 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 b1330c98e8 fix(evals): grade full cohort 6 false failures on saved state and keep failed-run evidence (#21089)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat catalog evals check that a chat agent does real product
work, and they grade saved state, not reply wording
> - Full cohort 6 (OpenCode with DeepSeek V4.1 Flash on master
b7693e2076) had five false failures where the product behaviour was
correct
> - Three more failures kept no run logs or ledger, and one was labelled
as transient infrastructure when the product had cancelled the run
> - A false failure hides real product defects, and missing evidence
stops diagnosis
> - This pull request corrects the five graders on saved state and fixes
the evidence gaps
> - The benefit is that the cohort result shows real product behaviour,
and each failure keeps the evidence to explain it

## Linked Issues or Issue Description

**What happened?**
Five chat eval cases failed although saved state showed correct
behaviour. Three run-bound failures and one early Playwright exit saved
no run logs or run ledger. A reassignment cancellation was classified as
`transient_infrastructure`.

**Expected behavior**
Each grader accepts the valid behaviour and still rejects the broken
product. Every failed attempt keeps its run ledger and the logs it can
read.

**Steps to reproduce**
Run the cohort 6 cases on master. Read the failed checks and the missing
`chat-catalog-runs.json` files.

## What Changed

- `budgets-costs.spend-breakdown` (oracle v5). Old check: only unranged
cost reports, the task cost summary, or the single run record counted.
The lead read the run through `GET
/api/companies/{companyId}/heartbeat-runs` filtered to Morgan with
`limit: 200` (saved input digest `36e47c51…`). New check: a run-list
read counts when its saved input digest names a full list that reaches
the original run (unfiltered or filtered to its agent, with a limit past
the runs ahead of it at read time). Still fails: no accounting read, a
list that cannot reach the run, another agent's list, a summary list, a
denied read, a read in another run.
- `status-cards.explain-card` (oracle v6). Old check: only `GET
/api/issues/{id}` counted as a live read. The lead read COM-1 through
`search_tasks`, and the saved result showed the task as done. New check:
a dedicated task tool result that holds the source task id with its live
status counts. Still fails: an empty search, another task, a different
status, a `call_api` result, an error result, a result with no paired
tool use.
- `secrets.personal-secret` (oracle v20). Old check: `JSON.parse` of the
whole bash output. The worker appended `EXIT_CODE=0`. New check: the one
JSON object in the output that carries `receiptId`, which must still
equal the fixture endpoint's authorised call receipt. Still fails: a
wrong receipt, a failed check, two different receipts, no JSON object,
key=value text.
- `skills.fork-read-only` (oracle `integrator.skill-fork.v2`). Old
check: a wording list (`unchanged|read-only|protected|not changed`)
rejected "untouched". New check: the reply names the saved fork or links
its Studio page. Saved state already proves the upstream is unchanged
and the fork exists. Still fails: no fork, a reply without the fork name
or link, a link to another skill, an edited upstream.
- `steering.discard-queued-message` (oracle v6). Old check: the label
string must not appear anywhere in the final plan. The revised plan kept
a "superseded" note with the label. New check: the person asked for a
section labelled with the attempt nonce (a setup literal). A heading
with that label must exist in the plan before the consuming run or in
that run's saves, and the final saved revision must have no heading with
it. Body mentions are allowed. The base plan is the revision before the
consuming run first saved the label, so repeated saves by that run do
not matter. Still fails: the labelled heading kept, renamed but still
labelled, or never saved.
- Failure evidence: the failure path saves
`chat-catalog-failure-run-ledger.json` before retirement, continues when
retirement fails, and reads a failed run's log and events independently
with a separate `logError` / `eventsError` when one run has no log.
Before, one cancelled never-started run (log 404) dropped the logs of
every run (`cases.cases-disabled`, `defer-and-reopen`,
`approve-tool-once`, `assignment-conflict`).
- Interruption act: the click and its `waitForResponse` are awaited
together. Before, the response promise could reject while the click
waited, and the unhandled rejection stopped the worker with no evidence
(`interrupt-with-direction`).
- A product-cancelled run is reported as `Observed run was cancelled by
<source> (<errorCode>)` and classifies as `candidate_failure`. Failed
and timed-out runs keep `Observed provider run failed`.

## Verification

- Calibration tests for each grader: the retained cohort 6 shape, a
reworded valid shape, and broken shapes.
- Regrade of retained cohort 6 evidence: spend-breakdown and
fork-read-only observations pass; the explain-card initial run reads the
source live; the discard plan revisions pass and the unrevised plan
fails.
- `npx vitest run` on the changed and related runner-e2e domain tests:
283 passed, 4 skipped. All 148 identity, integrator, runtime and
personal-secret domain test files pass.
- `pnpm test:e2e:runner:typecheck` passes.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check`:
`changed: false`.
- Paid run: see the PR comment.

## Risks

- Low. Only eval graders and harness evidence change. The run-list rule
depends on the list order (newest first) and the default and maximum
limits from `packages/shared`.
- A product-cancelled run no longer retries automatically as transient
infrastructure.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 06:05:19 -06:00
DottaandClaude Opus 5.5 5f34f777ec fix(evals): grade 16 chat cases by saved state instead of reply keywords (#20600)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-feature eval catalog checks that a chat agent does the work
a person asks for
> - In the latest full-cohort run on master, 16 cases failed although
the saved work was correct
> - Their graders matched words, one literal value, or one API route in
free-form model output
> - A grader that fails correct work hides real failures in the noise,
and it can also pass wrong work that uses the expected word
> - This pull request grades those cases by saved state, audit records,
setup literals and rendered page structure, and keeps only narrow checks
for false claims
> - The benefit is that these cases fail only when the product or agent
does the wrong thing

## Linked Issues or Issue Description

Refs #18971 (related: company-secret question handoffs)

**What happened**

The full-cohort chat-eval run on master failed 16 cases where the
retained evidence shows correct behaviour. Examples:
`companies.rename-company` accepted only `company.updated`, but the
supported branding route logs `company.branding_updated`.
`recovery.post-stop-message` required a signup button label that matched
a regex, and rejected a labelled email form with the button "Start
writing free". `secrets.secret-proposal` flagged "the value is delivered
when Morgan next runs" as a false delivery claim. Other graders required
words such as "uncommitted", "telemetry", "administrator" or the app
name that the question did not ask for.

**Expected behavior**

A grader accepts every supported route and wording that achieves the
outcome. It fails when saved state is wrong, when a requested fact is
wrong, or when the reply makes a claim that saved state contradicts.

**Steps to reproduce**

1. Load the retained observation for one of the cases below from the
full-cohort evidence.
2. Grade it with the old oracle. The keyword or route check fails
although the saved state is correct.
3. Grade it with the new oracle. It passes. A broken variant (wrong
saved state or a contradicting claim) still fails.

## What Changed

- `activity.audit-change`: attribution is graded by structure. The reply
names the recorded editor, names no other company agent (new setup
literal `otherActors`), and states only real transitions. Sentence
grammar is no longer matched.
- `app-permissions.ask-before-action`: each continuation reply must
exist and must link no app record other than the reviewed connection. It
no longer must name the app.
- `budgets-costs.spend-breakdown`: the reply must point at the record
with unresolved usage (task link, identifier or run id) and must not
give it a zero or free amount. The words "subscription" and "incomplete"
are no longer required.
- `cases.browse-cases`: an excluded case may be mentioned in prose. It
must not be linked or listed as a result row.
- `cases.handoff-same-task`: handoff instructions carry findings forward
when they point at the saved `findings` document, reproduce it, or
restate it. The premise (a primary Morgan run performs the handoff) is
unchanged.
- `cases.update-case-facts`: a saved browser value may add clauses that
only rule out other platforms ("desktop Safari unaffected").
- `companies.agent-observability`: the reply no longer must name
Telemetry and the run log. It must not claim that they are exported.
- `companies.rename-company`: both supported name routes
(`company.updated`, `company.branding_updated`) are accepted.
- `connection-setup.choose-provider`: the answer must exist and must not
claim a connected account. The unavailable method no longer must be
named; the setup path stays graded by clicking the card or link.
- `goals.align-existing-work`: a question card from the answering run
counts as the phase-two response. Refusal words and "unchanged" words
are no longer required; false change claims still fail.
- `recovery.post-stop-message` (and `steering.interrupt-with-direction`,
which shares the helper): a visible form with a labelled field and a
submit control is a signup action, whatever the button says.
- `routines.pause-owned`: the "paused"/"running" words are no longer
required. The saved checkpoint proves both; a cancellation claim still
fails.
- `secrets.secret-proposal` (and `secrets.configure-company-secret`,
which shares the helper): a sentence that says when delivery will happen
is not a delivery claim.
- `skills.explicit-restriction`: the reply no longer must say
"administrator". Success claims and invented tool limitations still
fail.
- `workspaces.inspect-git`: the reply no longer must say "uncommitted".
Unchanged HEAD and status prove it; commit or push claims still fail.
- Every affected `oracleVersion` is bumped in code and in
`chat-catalog-descriptors.json`. No scenario YAML changed.

## Verification

- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
--testTimeout=180000 tests/runner-e2e/chat-catalog-domains
tests/runner-e2e/chat-catalog-contract.test.ts
tests/runner-e2e/chat-catalog-consistency.test.ts`: all pass.
- `PAPERCLIP_CHAT_EVAL_RENDERED_ARTIFACT_CANARY=1` rendered-artifact
canary with Chromium: pass, including new form cases.
- `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: pass.
`node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
unchanged pins.
- `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files.
- Offline calibration on retained evidence, for each case: the retained
correct evidence passes, earlier passing evidence passes where it
exists, and a broken variant fails. `spend-breakdown` retained evidence
still fails, because its reply shows unresolved usage as `$0.00`.
`handoff-same-task` retained evidence still fails on the kept
primary-run premise.
- Live paid runs: see the comment below.

## Risks

- Low risk. The change is limited to eval graders and their unit tests.
Product code does not change.
- Each grader is more accepting of wording. Saved-state checks and the
narrow false-claim checks remain, so a wrong outcome still fails.
- `secrets.configure-company-secret` still fails its retained evidence.
Its agent asked a two-question card, and the act supports only
one-question cards. This pull request changes it only by the shared
delivery-claim helper.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 13:37:16 -06:00
f4337bb5fb test(evals): land the chat catalog eval harness on master (#19614)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-feature evals measure whether agents complete real chat
requests through the product
> - The chat catalog eval harness and its repairs live on the
`chat-eval/master-candidate` branch, not on master
> - Today's work made 139 of 242 chat evals pass in verified single-case
live runs. Each product fix landed on master as its own pull request
> - The eval-side changes (setups, graders, fault seams, evidence
capture) still exist only on the candidate
> - This pull request lands the harness on master. Master stays the
source of truth for product behavior
> - The benefit is one tree: evals run against master commits, and
product fixes reach the evals as soon as they merge

## Linked Issues or Issue Description

Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs
#20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409

**What existing behavior does this improve?**
The chat catalog eval harness and the product it evaluates live on a
long-running branch outside master. Eval fixes do not reach master, so
master-based eval runs use old setups and graders.

**Subsystem affected**
`tests/runner-e2e` chat catalog harness, plus a small set of product
changes that exist only on the candidate.

**Proposed change**
Land the harness on master. Every product fix from this eval work
already merged to master as its own pull request. Conflicts keep
master's product behavior.

## What Changed

- Most of the diff (808 of 851 files) is under `tests/runner-e2e/`:
chat-catalog domains, scenarios, descriptors, the TestOps manifest,
private evidence tooling, and their tests.
- Graders now check saved state, setup literals and rendered or clicked
links. Many keyword and regex checks on free-form replies are removed.
Each changed grader has a bumped `oracleVersion` and calibration tests
(a valid reply, a reworded valid reply, and broken variants that must
fail).
- The shared link helper `hasLink` no longer treats a list link (for
example `/COM/issues?view=mine`) as a link to every task without a URL
key.
- Setup fixes build real state through supported product routes: routine
history, agent grants through the board grant route, budget holds with a
real cap, and a storage-layer archive fault for
`inbox-search.archive-partial-failure`.
- Browser boundary journeys (sign-in, chat-disabled, onboarding) now
write the standard pass evidence files.
- Retargeted cases, approved by the maintainer:
`agent-chat.chat-disabled` (Conference Room gate),
`agents.restore-instructions` (no saved instruction history),
`access.manage-membership` (member permission edits stay board-only).
- The private evidence trusted files (`manual-private-handoff.ts`,
`manual-private-retention.ts`, `manual-private-report-assets.ts`) are
byte-identical to master.
- Product files outside `tests/` (43 files, +1,328 −49) carry
candidate-only changes that master does not yet have. Most are tests.
The main code changes are in `request-human-input.ts`,
`heartbeat-run-summary.ts`, `native-completion-feedback.ts`,
`native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and
`IssueChatLongThreadPerf.tsx`.

## Verification

- `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts
chat-catalog-consistency.test.ts unit-launcher.test.mjs
--testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests
pass.
- `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
no change.
- `tsc` for `server` and `tests/runner-e2e`: clean.
- Live single-case verification: 139 of 242 cases passed on paid runs,
and a person read the saved evidence for each. These are single-case
runs, not one full-cohort run.

## Risks

- Large diff. Most of it is test-only, but the 43 non-test files change
product code paths. Review those files first.
- Some graders are now less strict on reply wording. A person must read
replies for meaning on those cases.
- 42 cases need disposable outside accounts (Notion, Slack, GitHub,
Daytona and others) and stay blocked. `access.invite-teammate` stays
failing until local agent isolation lands, because the runner's durable
state keeps a live invite token.
- The paid workflow compares the trusted private-evidence files with
master. They are unchanged, so this risk is low.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution, in Claude Code. Subagents used the same model.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-10 08:02:11 -06:00