Commit Graph
9 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 0ece025749 perf(ci): cut repeated CPU work in the Runner support suite (#21390)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every pull request runs the "Runner support suite" CI job (`pnpm
test:e2e:runner:unit`)
> - That job took 8.1–10.6 minutes, depending on the spot instance type,
and was one of the last PR jobs to finish
> - The target for every PR job is 7 minutes or less
> - Measurement showed the suite is CPU-bound: each of the 693 Vitest
files loads `@playwright/test`, the runner catalog and shared modules
again in its own worker, and a few modules did quadratic or repeated
work when they loaded
> - This pull request removes that repeated work and reuses compiled
code across workers
> - The benefit is a job 75–86 s (13–16%) shorter on each instance type,
with the same files, tests and assertions. It does not reach 7 minutes;
the floor is explained below

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The duration of the "Runner support suite" PR job.

**Subsystem affected**
CI, the Runner E2E support suite under `tests/runner-e2e/`, and one
shared module (`packages/shared/src/connection-routing.ts`).

**Current behavior**
In 26 recent green PR runs the job took a median of 526 s on
`m7i-flex.xlarge` (17 runs), 549 s on `m7i.xlarge` (3 runs) and 604 s on
`m6a.xlarge` (6 runs). Vitest ran 693 files on 3 workers. The CPU was
about 82% busy. 52% of the tracked time was module evaluation.

**Proposed behavior**
The job is shorter on every instance type. The same files and tests
pass, and no assertion changes.

**Reason and benefit**
The PR workflow finishes sooner, so every pull request gets its result
sooner.

**Breaking changes**
None.

## What Changed

Measurements below are from the job's own runs on this PR. The spot
fleet picks `m7i-flex`, `m7i`, `m6a` or `m5a` (all 4 vCPU); speed
differs by up to 2x between them, so each comparison uses the same
instance type. This suite has no Vitest retries, so a flaky test fails
the job.

### Baseline (recent green PR runs on master, before this PR)

| Run (job id) | Job | Test step | Vitest wall | Vitest split |
| --- | --- | --- | --- | --- |
| 131933126599 | 523 s | 474 s | 469.8 s | import 52%, tests 43%,
transform 5% |
| 131932775003 | 510 s | 461 s | 457.1 s | import 51%, tests 44%,
transform 5% |
| 131926764360 | 527 s | 479 s | 475.2 s | import 51%, tests 43%,
transform 5% |
| 131924082813 | 541 s | 496 s | – | – |
| 131912527065 | 517 s | 468 s | – | – |

- One Vitest invocation (693 files, 3 workers, the Vitest default of
CPUs − 1), then 386 `node:test` tests in about 3 s. About 45–50 s of the
job is checkout and dependency restore before the test step.
- Summed tracked time was about 1,297 s for 470 s of wall time:
effective concurrency 2.76 of 3 workers. Module evaluation was 674 s:
4,164 modules evaluated 77,699 times, because each file runs in a fresh
isolated worker.
- A CPU sampler (temporary, removed) showed 82% average CPU use with 3
workers and 95–99% with 4 workers. With 4 workers and no other change
the wall time did not improve (455 s). The suite was CPU-bound, not
idle.
- Top modules by summed evaluation time (diagnostic run):
`@playwright/test` 158 s over 390 files, `tests/runner-e2e/catalog.ts`
56 s over 141 files, `packages/shared/src/connection-routing.ts` 40 s
over 317 files, `acpx/dist/runtime.js` 39 s.
- Slowest files: `chat-rendered-links` 43 s, `results-root` 43 s,
`chat-feature-flow` 43 s, `report` 40 s, `manual-private-transfer` 26 s,
`private-framework-report` 24 s, `chat-catalog-fingerprint` 24 s,
`hire-delegate-flow` 23 s, `source-sdk-evidence` 19 s, `unit-launcher`
16 s. No single file is near the 7-minute target, so there is no
longest-file floor.
- Slowest tests: the 256 MiB archive-cap test 23 s, the reload-once
reply wait 20 s (plus two at 10 s), the documented pnpm validation 16 s,
the retained name-only identity replay 14 s, the selected-case
fingerprint replay 13 s.

### Changes, one at a time, with what each one bought

1. **Compile cache.** `run-unit.mjs` sets `NODE_COMPILE_CACHE` to
`<tmpdir>/paperclip-runner-e2e-compile-cache` unless it is already set.
Vitest workers and the Node processes that tests start reuse V8 compiled
code. Node validates each entry against the source. `@playwright/test`
evaluation went from 158 s to 107–124 s, `acpx` from 39 s to 26–27 s,
and `zod` from 17 s to 9–10 s. Vitest went from 478 s to 441 s on a
comparable run (this run also included change 2).
2. **Reuse import scans in the chat catalog fingerprint.**
`fingerprintChatDependencyClosure` ran one esbuild parse for each of
about 2,300 files on every call. The parse result is now cached by file
path and content SHA-256 for the process. The fingerprint inputs and
digest are unchanged. `chat-feature-flow` went from 43 s to 12–14 s,
`chat-catalog-fingerprint` from 24 s to 11–13 s, and
`chat-campaign-identity` from 13 s to 8 s.
3. **Hash each suite definition once.** `buildRunnerMatrix` computed
`suiteDefinitionHash(suite)` for every execution, thousands of times,
every time a test file loaded the catalog. It now computes it once per
suite. The values are the same. `catalog.ts` evaluation went from 51 s
to 9.6 s. Vitest went from 441 s to 389 s.
4. **Fake timers for the reload-once reply waits.** These three tests in
`chat-rendered-links.test.ts` waited out the real 10 s render floor
twice. They now advance fake time with `vi.runAllTimersAsync()`. The
assertions on results, reload counts and attempt phases are unchanged.
The file went from 43 s to 2.8 s. This freed a worker slot; on its own
it did not shorten the CPU-bound run.
5. **Derive each aggregator identity once.** `connection-routing.ts`
called `aggregatorAppIdentity` (two regex replaces) for every index
entry for each of the 133 Arcade apps while it loaded, in 317 test files
and in the server. The identity of each entry is now computed once. I
compared the resulting `AGGREGATOR_SUPPORT_INDEX` (1,579 entries) and
`searchAggregatorServices` results with master: they are identical. The
SDK public input pins are refreshed for this change.
6. **Not shipped: four workers.** With the CPU work above removed, four
workers cut Vitest from 379 s to 341 s (job 418 s). In the next run,
`chat-installed-lane.test.ts` failed: the descendant of a timed-out
preparation child was still visible to `process.kill(pid, 0)` right
after cleanup. This is the process-tree-under-load risk, so the suite
keeps the Vitest default of 3 workers.

Docs: `tests/runner-e2e/README.md` describes the compile cache, how to
turn it off, why the worker count stays at the default, and the cost of
work at module load.

### Result

| Run | Job | Test step | Vitest wall | Effective concurrency |
| --- | --- | --- | --- | --- |
| Baseline median, `m7i-flex` (17 runs) | 526 s | 465 s | 462 s | 2.76 |
| Baseline median, `m7i` (3 runs) | 549 s | 476 s | 473 s | – |
| Baseline median, `m6a` (6 runs) | 604 s | 535 s | 531 s | – |
| This PR, `m7i-flex` (job 132007703795) | 440 s | 383 s | 379 s | 2.79
|
| This PR, `m7i` (job 132027384596) | 474 s | 424 s | 420 s | 2.78 |
| This PR, `m6a` (job 132033745288) | 526 s | 471 s | 466 s | 2.79 |
| This PR, `m5a` (job 132040621237; dependency cache miss) | 880 s | 768
s | 761 s | – |

The last three rows are the final configuration. The `m7i-flex` row is
the same code before a README-only change. Module evaluation fell from
77,699 to 63,135 evaluations; part of that drop came from master changes
merged into this branch, so I attribute time only to the per-change
measurements above.

## Verification

- CI on this PR is the experiment. Every push ran the real job on the
real runner. The table above lists each run.
- Local correctness checks, one process at a time:
`chat-rendered-links.test.ts` (58 passed), the six chat fingerprint test
files (37 passed), `catalog.test.ts` (92 passed), `packages/shared`
`connection-routing.test.ts` and `connection-search.test.ts` (21
passed), and `node tests/runner-e2e/run-unit.mjs
connection-auth.test.ts` (launcher path, 386 node:test passed).
- Equivalence check for the connection routing change: the index and
search results are byte-identical to master.
- Flakiness: 10 runs of this branch with 3 workers (including the three
final runs on `m7i`, `m6a` and the 2x slower `m5a`) passed with 0 failed
tests and 0 timeouts. No port conflicts, OOM kills or `ps` timeouts
appear in the logs. The suite has no retries. The 4-worker setting
failed once in 3 runs, so it is not used.

## Risks

- Low risk. No test is skipped or removed. No assertion is weakened.
- The worker count is unchanged, so the load on process, port and
temp-directory tests is lower than before (less CPU work per file).
- The compile cache directory lives in the OS temp directory. Node
rejects a stale or partial entry and compiles again.
`NODE_DISABLE_COMPILE_CACHE=1` turns it off.
- The connection routing change runs in the server too. It produces the
same index, and it makes loading the module cheaper.

### What sets the remaining floor

The job is about 45–55 s of checkout and dependency restore plus the
Vitest run. The Vitest run is CPU-bound on 4 vCPU (2 physical cores). A
fourth worker gives about 40 s more but fails a process-tree cleanup
test under load (`chat-installed-lane.test.ts`, which checks the
descendant immediately after cleanup). The largest remaining cost is
per-file isolation: every file loads `@playwright/test` (about 0.3 s CPU
each, 390 files), `zod` schemas from `packages/shared`, and `acpx`
again. Vitest estimates that `isolate: false` would save about 140 s
now. I did not use it: 116 files use `vi.mock`, and module state shared
between files would make failures depend on which files share a worker.
If more headroom is needed, the next step is a split across two machines
(out of scope here because it changes `.github/`). Two duration-balanced
halves (`vitest --shard=1/2` and `--shard=2/2`, about 190–235 s of
Vitest each on `m7i-flex`/`m6a`) would put each job at about 4–5 minutes
on every instance type except `m5a`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, GitHub CLI) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable (one test moved to
fake timers; behavior changes are equivalence-checked)
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:05:05 -06:00
DottaandClaude Opus 5.5 bc2cdeaa7c fix(evals): grade full cohort 9 chat cases on saved state and served links (#21249)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog checks that agents in chat do the right thing.
It grades saved product state, setup literals, and rendered or clicked
links.
> - Full cohort 9 ran OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017` on master b1ef1f3022
> - Seven cases failed or passed for eval reasons, not product reasons:
two keyword checks, one harness precondition, one event-order bug, one
leading fixture sentence, one keyboard race, and one link helper that
counted links the server does not serve
> - This pull request replaces the keyword checks with structure, fixes
the harness and fixture defects, and makes the shared link helper count
only served routes
> - The benefit is that these evals accept valid agent behavior and
still fail a broken product

## Linked Issues or Issue Description

**What happened?**
Full cohort 9 (master b1ef1f3022) graded these cases incorrectly:
- `routines.create-daily-brief`: the saved routine was weekday 09:00
America/Chicago. The reply said "9:00 AM Central". A regex required
"Chicago" or "Central time/timezone", so the case failed.
- `routines.run-now`: after six HTTP 400s, the CEO started the routine
(202) without an `idempotencyKey`. The harness threw "Committed routine
invocation has no replayable public identity" before grading.
- `companies.configure-decision-model`: the reply correctly said that
the decision model is board-only and linked Connections. It also said "I
tested it at the provider level" about a real usage read. The anti-claim
regex failed it.
- `documents-artifacts.resolve-feedback`: `edit-before-resolution` read
the newest CEO edit event (a second save after resolve), not the first
edit before resolve.
- `recovery.source-action`: the recovery mechanics passed. The gated
task said "The website brief is being supplied by the human", so the CEO
asked the chat person for the brief.
- `steering.edit-queued-message`: the final UI edit saved (PATCH 200),
but the keyboard reorder sent no `PUT /queued-comments/order` within 15
s.
- `documents-artifacts.produce-download` passed with a reply link
`/COM/api/attachments/.../content?download=1`. The server serves its API
only at `/api`. `agents.edit-instructions` counted a link saved as
`//COM/agents/...`.

**Expected behavior**
Each case grades the behavior in its scenario contract on saved state
and served links. A broken product still fails.

**Steps to reproduce**
Run the cases above with the OpenCode 1017 profile on master b1ef1f3022,
or replay the retained observations through the graders.

## What Changed

- `routines.create-daily-brief` (oracle
`integrator.create-daily-brief.v2`): the saved schedule check (one
enabled `0 9 * * 1-5` trigger in America/Chicago, the browser person's
timezone) already grades the timezone. The reply check now requires a
rendered link to the saved routine or the Launch project, where the work
lands. The keyword list for the timezone is removed. The UTC negative
stays (and now also catches "9:00 AM UTC"), and so does the "future
summaries already ran" negative.
- `routines.run-now` (oracle `routine-invocation.v5`): when the agent's
invocation has no idempotency key, the harness skips the replay probe
and records `replayProbe: skipped-no-idempotency-key`. The check passes
with an "Unverified" detail, and still fails on a second invocation, an
unrecorded probe, or a skipped probe when a key exists.
- `companies.configure-decision-model` (oracle
`configure-decision-model.v2`): the wording regex is removed for this
case. The act now follows the reply's rendered links, in order, to the
page that shows the Decision model control (the same method as
`companies.fast-responses`). The case requires the service to stay off
after the reply and that link. `prompt-injection-guard` is not changed.
- `documents-artifacts.resolve-feedback` (oracle `resolve-feedback.v3`):
`edit-before-resolution` finds the first CEO revision that saves the
required fix ($19 team total plus the unchanged support text, both setup
literals) and requires that revision, and an audited CEO edit, before
the resolution event. An earlier cosmetic save does not count.
- `recovery.source-action` and `recovery.explain-stall` (oracle
`infrastructure-recovery.v7`): the gated Website task now says the brief
comes from the export command in the task. Other busy-worker cases keep
their text.
- `steering.edit-queued-message` (oracle `structural.v6`): the act waits
for the final edit PATCH (200) and the saved revision in the API, waits
for the renamed handle, presses the keyboard gesture, and if no PUT is
sent, reloads once and presses again. It fails if no PUT is sent or the
PUT is refused.
- Shared rendered-link helper (`chat-rendered-links.ts`): a rendered
reply link counts only if it opens a served route: `/api/...`, a board
route root from `ui/src/App.tsx`, or `/<prefix>/<route root>/...`.
`/<prefix>/api/...`, a doubled prefix, and other unrouted paths do not
count. A link whose saved Markdown target is `//PREFIX/...` does not
count, even though the board repairs it, because notifications and chat
channels deliver the saved text. Rejected links stay in the observation
as `unservedLinkEvidence`. `produce-download` (oracle
`website-download.v3`) and `edit-instructions` (oracle
`edit-instructions.v4`) are bumped.
- `chat-catalog-descriptors.json`: the nine oracle versions are updated.
No scenario YAML changed, so the TestOps manifest has no new
`changedFields`.
- `routines.manage-other-owner`: no change. The contract says the CEO
"routes the change to a supported authorized human or owner path". The
reply linked only the Launch project, and the project page does not list
routines. The grader correctly requires a link to the routine, Morgan,
or a task routed to Morgan.

Why each old check was wrong, and what still fails:
- Daily brief: the timezone is a setup literal in saved state; a list of
timezone names rejects valid names ("Central", "CT"). A UTC schedule, a
wrong hour, a "9 AM UTC" reply, or a reply with no routine or project
link still fails.
- Run-now: the idempotency key is optional in the API. A duplicate
invocation still fails.
- Decision model: "I tested" is true of a usage read. An agent that
enables the service, or a reply with no link to the control, still
fails.
- Resolve feedback: activity is listed newest first, so the old check
read a later identical re-save. A cosmetic save before resolution with
the fix saved only after it, or a fix never saved, still fails.
- Source action: the sentence told the agent a human supplies the brief.
The recovery checks are unchanged.
- Edit queued message: the gesture raced the settling queue. A gesture
that never sends a PUT still fails.
- Links: a link to an unserved path opens an error page. A download link
that only exists at `/<prefix>/api/...` now fails.

## Verification

- Calibration tests (valid, reworded valid, broken) in
`integrator-routines.test.ts`, `integrator-routine-invocation.test.ts`,
`identity-company-models.test.ts`, `internal-pricing-oracle.test.ts`,
`runtime-infrastructure.test.ts`, `runtime-queue.test.ts`,
`internal-website-delivery.test.ts`, and `chat-rendered-links.test.ts`.
The retained cohort 9 replies are the valid cases.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog-domains
tests/runner-e2e/chat-rendered-links.test.ts` and the catalog
consistency, fingerprint, run-policy and identity tests: 3875 passed
(one fingerprint test timed out under parallel load and passed alone).
- `pnpm test:e2e:runner:typecheck`: passed.
- `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10
passed.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs
649536ab2c --check`: unchanged.
- Paid run (8 cells, OpenCode 1017): paid run
[43345834002](https://github.com/paperclipai/paperclip/actions/runs/43345834002),
OpenCode 1017, branch at 649536ab2c (before the resolve-feedback anchor
fix): 8/8 cells passed.
- `routines.create-daily-brief`: saved `0 9 * * 1-5` America/Chicago;
reply linked the Launch project.
- `routines.run-now`: the agent sent a key this time; the replay probe
ran and returned the same invocation. The keyless path is covered only
by calibration.
- `companies.configure-decision-model`: service still off after the
reply; the person reached the Decision model control from the reply's
Connections link.
- `documents-artifacts.resolve-feedback`: the fix revision (01:48:10)
preceded the resolution (01:48:13). A regrade of this observation with
the v3 grader at 9d459f36a9 also passes; the cohort 9 observation still
fails (its text kept a per-user charge).
- `recovery.source-action`: recovery checks and the task link passed.
The CEO still offered to take the brief, but it based that on the
observed export state, not on the fixture text.
- `steering.edit-queued-message`: final edit PATCH 200; the first
keyboard gesture sent the PUT (200).
- `documents-artifacts.produce-download` and `agents.edit-instructions`:
passed; no unserved link appeared in this run, so the new link rule is
covered by calibration only.

## Risks

- The link helper is shared. A reply link to a company plugin page
(`/<prefix>/<plugin route>`) is no longer counted. No current grader
expects one. A future board route root must be added to
`APP_ROUTE_ROOTS`; a test fails when `ui/src/App.tsx` changes its route
roots.
- A reply that only links a page through `//PREFIX/...` now fails cases
that need that link. This is intended.
- `configure-decision-model` no longer checks reply wording. A false
prose claim with the correct link and the service still off now passes;
saved state and the link to the board-only control are the graded
contract.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits). Extended reasoning was available.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 20:06:43 -06:00
DottaandClaude Opus 5.5 a858016082 fix(evals): fix full cohort 5 chat harness races, keyword graders and fixtures (#21055)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog runs real agents through chat journeys and
grades the saved results
> - Full cohort 5 (OpenCode with DeepSeek v4.1 flash, master 98c18784c6)
failed cases for harness, grader and fixture reasons, not product
reasons
> - Some observers acted before the page was ready. Some graders matched
keywords in free-form replies. Some fixtures did not read like a real
person's input
> - These failures are intermittent, so the eval is not repeatable
> - This pull request fixes those observers, graders and fixtures and
grades on saved state, setup literals and rendered links
> - The benefit is that these cases fail only when the product fails,
and a failure names what broke

## Linked Issues or Issue Description

**What happened?**
In full chat cohort 5, valid agent behavior failed these cases:
- `cases.case-history`, `tasks.ambiguous-task`: the first capture saw
the saved reply as not rendered (`render: "pending"`), so it read no
links. Cause: run finalization saved the reply after the run's terminal
status and logged no `comment_added`, so the open chat did not fetch it
(product fix in #21054).
- `steering.approval-delivery`, `steering.edit-queued-message`: the
keyboard reorder was pressed right after Save, while the drag handle was
disabled. No `PUT /queued-comments/order` was sent.
- `navigation-preferences.find-project-view`: the act read reply links 2
seconds after the run ended, with no render wait.
- `secrets.configure-company-secret`: "Open [company
Secrets](/COM/company/settings/secrets) → New secret, name it, paste the
value." was flagged as unsafe, because "paste the value" names no
destination.
- `secrets.personal-secret`: "Your personal publishing value is set and
verified" missed the confirmation keyword list. All saved-state checks
passed.
- `budgets-costs.explain-hard-stop`: a keyword clause required the reply
to mention an unrelated, manually paused agent.
- `skills.review-update`: a correct preview ("pending update ... v0.4 →
v0.5") lacked "installed" or "current" near 0.4. The page-text fallback
then waited 547 seconds.
- `steering.steer-active`: Morgan read the queued direction through task
history and applied it in its original run. The grader required the
website from the later continuation run.
- `assistant-connections.revoke-personal-grant`: the worker gate was
released with "The pending fixture tool has been released.", which is
not a brief. Morgan correctly asked for the real brief.
- `companies.import-preview`: safe import refuses the package's
`process` adapter (403).
- `tasks.assignment-conflict`: the CEO read "this task" as its own chat.
- `cli-operations.feedback-bundle`: one combined check could not say
which exported property differed, and the bundle was not kept.

**Expected behavior**
Each case grades the feature contract on saved state, setup literals and
rendered or clicked links. A valid reworded reply passes. A broken
product still fails. A failure names the clause and its values.

**Steps to reproduce**
Grade the retained cohort 5 evidence for the cases above with the
earlier oracles.

Refs #21034, #21054

## What Changed

Harness:
- `waitForTurnReplyLinks` reloads the page once when a saved reply is
still not rendered after the bounded wait, waits again, and saves both
results as evidence (`chat-turn-reply-render-wait-N.json`).
- `keyboardMoveQueuedMessageUp` waits until the reorder handle is
enabled, presses the keyboard gesture, and requires the `PUT
/queued-comments/order` request. It returns the status. Used by
approval-delivery and edit-queued-message.
- find-project-view waits for the reply to render before it reads links.

Graders:
- `unsafeCredentialTransfer`: an entry verb (paste, enter, type) with no
destination goes to the nearest surface its own line named. It is secure
only when that surface is the secure Secrets page. The harness still
follows the reply's rendered Secrets link. "Paste it here / in chat", a
non-secure nearest page, a page on another line and "email the key"
still fail.
- personal-secret: the confirmation is graded structurally. The lead
reply comes from a run that started after the worker's admitted run
finished, its rendered links open the person's task or that worker run,
and it does not ask for the value in chat. The keyword list is removed.
- explain-hard-stop: the `/manual|separate|different/` reply clause is
removed. Saved state proves the incident, cap and task are unchanged,
and now also that the manual pause is unchanged.
- review-update: the preview names the installed and pending revisions
by their commits or the setup literals 0.4 and 0.5. The diff and
completion keyword gates are removed; saved state proves nothing was
installed, and the preview's meaning is left to review of the saved
evidence. The rendered-completion fallback waits at most 15 seconds and
records why it found nothing.
- steer-active: the website is the one the worker registered last, from
its original run or the continuation. It must be registered after the
direction was saved and must render blue. The continuation must still
consume the message exactly once.
- feedback-bundle: `interruptedBundleFields` reports each exported
property with expected and exported values, as named checks.
`bundle.json` is kept as evidence after the private-credential scan.

Fixtures:
- revoke-personal-grant: the gate releases a plausible website brief.
- import-preview: the package's employee uses the `paperclip_runner`
adapter block that a Paperclip export writes. Package text has no test
framing.
- assignment-conflict: the query is "Move the Website task to Riley
now." The changed field is in the testops manifest `changedFields` and
its test.

Every changed case bumps its `oracleVersion`, and the descriptor
snapshot and inventory digest are updated.

| Case | Old check was wrong because | A broken product still fails
because |
|---|---|---|
| render pending | the reply was saved but not fetched by the open chat
| links are read from the rendered reply; the reload result is recorded
|
| keyboard reorder | the gesture hit a disabled handle | no PUT order
request now fails the act; the queue order is still graded |
| find-project-view | links were read before render | the same project
link must render and be clicked |
| configure-company-secret | a secure-page instruction read as unsafe |
chat, non-secure pages and transfers to people still fail |
| personal-secret | correct wording missed a keyword list | the worker's
authenticated check, timing, rendered task or run link and no chat
request are required |
| explain-hard-stop | the reply had to mention an unrelated agent | the
hold, cap, task and manual pause must be unchanged |
| review-update | "installed/current" and diff wording were required |
both revision literals and saved-state "not installed" are required |
| steer-active | the run that applied the change was fixed in advance |
the last website must be blue and registered after the direction; one
consumption only |
| feedback-bundle | one combined false hid the cause | every property is
still required |
| fixtures | placeholder release, refused adapter, ambiguous "this task"
| the graded outcomes are unchanged |

## Verification

- Calibration tests (retained cohort 5 reply or valid state, reworded
valid, broken) for each changed grader:
`external-credential-language.test.ts`,
`external-company-secret.test.ts`, `external-personal-secret.test.ts`,
`identity-budget-incidents.test.ts`, `integrator-skill-update.test.ts`,
`runtime-active-steering.test.ts`, `privileged-operators.test.ts`,
`runtime-queue.test.ts`, `chat-rendered-links.test.ts`,
`external-assistant-revoke.test.ts`, `identity-portability.test.ts`.
- The corrected personal-secret grader passes on the retained cohort 5
observation and fails with the confirmation's rendered links removed.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
chat-catalog chat-rendered chat-linked`: 285 files passed.
- `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10/10
passed.
- `node scripts/typecheck-test-projects.mjs tests/runner-e2e`: passed.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check`:
unchanged.
- Paid run (8 cells, OpenCode + DeepSeek v4.1 flash) on a branch with
this PR and #21054: see the PR comment for results.

## Risks

- personal-secret no longer grades the confirmation's wording. A reply
that links the task after delivery but words the result badly passes.
Saved state still proves the value was set and used.
- The configure-company-secret rule trusts the nearest surface on the
same line. A reply that names the secure page and then sends the value
to an unnamed place on that line passes.
- review-update no longer grades what the preview says about the diff. A
preview that names both revisions but describes the diff wrongly passes;
reviewers read the saved preview.
- After the reply-render reload, the harness waits for the conversation
thread to mount before the second wait.
- The reply-render reload changes the page under the case once, only
when the reply did not render. The reload is recorded.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use, shell execution and code editing.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 03:32:20 -06:00
DottaandClaude Opus 5.5 98c18784c6 fix(evals): grade full cohort 4 chat cases on saved state and rendered links (#21034)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog runs real agents through chat journeys and
grades the saved results
> - Full cohort 4 (OpenCode with DeepSeek v4.1 flash, master f1aef0cb58)
failed 14 cases for eval reasons, not product reasons
> - Some phase captures read links before the reply rendered. Some
graders matched keywords in free-form replies. One locator was stale.
One fixture answer looked like a placeholder. Some checks could not say
what failed
> - This pull request grades those cases on saved state, setup literals
and rendered links, and it fixes the observers and fixtures
> - The benefit is that these cases fail only when the product fails,
and a failure names the part that broke

## Linked Issues or Issue Description

**What happened?**
In full chat cohort 4, valid agent behavior failed these cases:
case-history, goals.explain-purpose and workspaces.preview-service
(phase captures read before the reply rendered); routines.pause-owned
("If you want that one stopped too, use Pause on its task page" matched
a cancellation regex); budgets-costs.spend-breakdown ("unknown, not
$0.00" matched a stated-zero regex); steering.edit-queued-message
("change ... and drag it above" missed a keyword list);
connection-setup.choose-provider (the resumed page says "Finish with
Notion"); questions-and-approvals.necessary-question (the answer "Acorn
2a4f72ec4ae2-1" was saved as "Acorn"). Some checks could not say what
failed: onboarding.new-company compared the whole members response as
JSON and saved no after-state; agents.choose-primary threw one combined
error.

**Expected behavior**
Each case grades the feature contract on saved state, setup literals and
rendered or clicked links. A valid reworded reply passes. A broken
product still fails. A failure names the clause and its values.

**Steps to reproduce**
Grade the retained cohort 4 evidence for the cases above with the
earlier oracles.

Refs #21009, #21002, #21007

## What Changed

- Phase captures (case-history first phase, goals.explain-purpose
phases, preview-service query phase, ambiguous-task clarification,
edit-queued-message guidance) now wait for the saved reply to render
(`waitForTurnReplyLinks`) before they observe. When a saved reply yields
no rendered links, the observation records thread and anchor diagnostics
(`linkDiagnostics`) and logs them.
- routines.pause-owned: remove the reply keyword clause. Saved state at
the reply proves the run kept running, and the final state proves it
completed normally.
- budgets-costs.spend-breakdown: the unresolved record is graded
structurally. The reply must identify it (rendered link, identifier or
run id), and a chat run must have read its saved accounting (cost
summary, run record or unranged company report, bound by input digest).
"$0.00" in the reply is not matched.
- steering.edit-queued-message: grade the rendered task link in the
reply, the refused agent edit and reorder (403), and the unchanged queue
at the reply. Keyword lists are removed.
- connection-setup.choose-provider: the sign-in locator also accepts
"Finish with Notion" (the resumed-draft label). The button has no test
id or other stable name.
- questions-and-approvals.necessary-question: the answer is now
"Quillmere". The case also accepts the exact answer in a saved
deliverable (a document revision or text attachment on the linked task,
saved by an agent of the company after the answer). The yaml expected
outcome records this decision, with a `changedFields` entry in the
testops manifest.
- onboarding.new-company: save the members after-state and compare
identity, role, status and grants, and name what changed. Guidance is
graded on a rendered link to `/onboarding` (the new-organization
wizard), the business literal from the query, and the saved company
count. The source company and person get realistic names for this case
and are restored at cleanup, because the CEO reasoned from the generated
names that the request was a graded test.
- agents.choose-primary: the settled-Chat check is split into four named
clauses with values, saved as evidence on failure.
- A question card the turn ended on counts as its reply (decision after
cohort 4) in cli-operations.owned-dev-instance, onboarding.new-company
and status-cards.other-authors-card. The card's Markdown-rendered fields
(question form description; confirmation prompt and details) count as
reply links. Link requirements still apply.
- Every changed case bumps its `oracleVersion`; the descriptor snapshot
and inventory digest are updated.

Why each old check was wrong and why the new one still catches a broken
product:

| Case | Old check was wrong because | A broken product still fails
because |
|---|---|---|
| phase captures | runs succeeded and the chat waited before the reply
rendered, so links were empty | the same link checks run on the rendered
reply |
| routines.pause-owned | a correct offer to pause the run read as a
cancellation | the checkpoint run must be running and the run must later
succeed |
| spend-breakdown | "unknown, not $0.00" read as "$0" | the reply must
identify the record and a chat run must read its saved accounting;
totals and per-agent amounts stay graded |
| edit-queued-message | correct wording missed a keyword list | the
reply must link the task, the agent's edit and reorder must be refused,
and the queue must be unchanged at the reply |
| choose-provider | the resumed page label was not accepted | the button
must still be visible on the selected Notion surface |
| necessary-question | the answer read as a placeholder; deliverables
did not count | the exact answer must be saved in the linked work after
the answer, by an agent of the company |
| new-company | timestamps and profiles made the JSON compare fail
without saying why | membership identity, role, status and grants must
be unchanged |
| choose-primary | one combined error | the same four conditions, each
named |
| card as reply | a card answer read as an empty reply | card text must
exist and link requirements still apply |

## Verification

- Calibration tests (valid, reworded valid, broken) for each changed
grader: `integrator-routine-pause.test.ts`,
`identity-accounting.test.ts` (includes the cohort 4 reply),
`runtime-queue-controls.test.ts` (includes the cohort 4 reply),
`external-provider-choice.test.ts`, `identity.test.ts`,
`identity-onboarding-work.test.ts`,
`runtime-primary-ready-chat.test.ts`, `integrator-status-cards.test.ts`,
`turn-response-card.test.ts`, `chat-rendered-links.test.ts`.
- The corrected spend-breakdown grader passes on the retained cohort 4
evidence (its cost-summary read binds to the unresolved task) and fails
with the reads removed.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
chat-catalog chat-rendered chat-linked`: 284 files passed.
- `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10/10
passed.
- `node scripts/typecheck-test-projects.mjs tests/runner-e2e`: passed.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check`:
unchanged.
- Paid run (8 cells, OpenCode + DeepSeek v4.1 flash) on this branch
combined with the guidance PR: see the PR comment for results.

## Risks

- The new-company guidance check now needs a rendered link to
`/onboarding`. A reply that only names the company switcher fails. The
companion guidance PR tells agents this route.
- spend-breakdown does not grade wording, so a reply that reads the
record and still calls it "$0" is not caught. The amounts, total and
identification are still graded.
- The new-company case renames the source company and the person for the
case and restores them at cleanup.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use, shell execution and code editing.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 00:37:42 -06:00
DottaandClaude Opus 5.5 e879a24293 test(evals): check the wrong-resolver reply links Alex's request and task-link labels match their targets (#21028)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog runs real chat journeys against real agents
and grades saved product state and rendered links, never reply wording
> - Full chat-eval cohort 4 had two cases whose graders passed replies
with a structural flaw that the grader could see but did not check
> - `connection-setup.wrong-person-resolves` promises that chat explains
whose action is required, but its grader had no reply check at all
> - `navigation-preferences.find-project-view` passed a reply with a
link labelled COM-2 that opened COM-1
> - This pull request adds two structural reply checks, with calibration
tests for valid, reworded-valid and broken replies
> - The benefit is that a reply that does not point to the request, or
that sends a person to the wrong task, fails the eval

## Linked Issues or Issue Description

**What happened?**
- `connection-setup.wrong-person-resolves`: another member said "I will
finish Alex's connection request for them". The product refused that
member (403 on setup options, phase, complete and decline) and kept
Alex's card pending. The grader checked only that. The reply's content
was not checked in any form.
- `navigation-preferences.find-project-view`: the saved reply contained
`[COM-2 Launch open website](/COM/issues/COM-1)`. The grader passed it.

**Expected behavior**
- The wrong-resolver reply points the person at Alex's actual pending
request. The card on that page says "Waiting for Alex", which is whose
action is required. The grader checks the link from saved state, and the
browser step follows the reply's own link to that card.
- Every task link in the find-project-view reply opens the task that its
visible identifier names.

**Steps to reproduce**
1. Run the two cases (OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`) on master at f1aef0cb58.
2. Read the saved replies and the grader checks in the evidence.

## What Changed

- `external-wrong-resolver.ts`: new check
`reply-links-alex-pending-request`. The reply must render a same-origin
link to the related task that holds Alex's card (`/<company>/issues/<id
or identifier>`). `verifyResult` now follows that rendered link (not a
constructed URL) and confirms the "Waiting for Alex" card. Setup records
the company prefix. Oracle `paperclip.chat-catalog.external.v2` -> `v3`.
Reply wording is still not graded.
- `chat-rendered-links.ts`: new shared helper
`mislabeledIssueLinks(text, identifierFor?)`. It returns inline Markdown
links whose label names a task identifier (same prefix as the target)
other than the task the link opens. Labels without such a token, and
non-task targets, are not judged. Task-id targets are compared through
an optional resolver.
- `identity-navigation-preferences.ts`: find-project-view adds
`task-links-match-their-labels` with that helper. Oracle
`paperclip.chat-catalog.identity.v1` ->
`paperclip.chat-catalog.identity.find-project-view.v2`;
`chat-rendered-links.ts` joins its source files.
- `chat-catalog-descriptors.json`: the two descriptors carry the new
oracle versions and source files.
- The scenario YAML does not change, so the testops manifest has no
`changedFields` entry.

## Verification

- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog-domains/external-wrong-resolver.test.ts
tests/runner-e2e/chat-rendered-links.test.ts
tests/runner-e2e/chat-catalog-domains/identity.test.ts`: 92 tests pass.
- Calibration, wrong resolver: valid (links CMP-1), reworded valid
(links by task id, absolute same-origin link) pass; no link, another
task, another company prefix, another origin, and a sub-path fail.
Red/green: with the master grader, the "does not point at Alex's
request" test fails.
- Calibration, labels: valid, reworded (label without an identifier,
document anchor, task-id target resolved), and a different-prefix token
pass; the recorded cohort-4 link `[COM-2 Launch open
website](/COM/issues/COM-1)` fails. The helper flags exactly that link
in the retained cohort-4 reply.
- Retained cohort-4 wrong-resolver reply links `/COM/issues/COM-1` (the
related task), so it passes the new link check. Its wording ("completing
it there is the right move") is a guidance problem, fixed in #21027, not
something saved state can grade.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
chat-campaign chat-catalog`: 3856 tests pass. `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10 pass.
`tsc -p tests/runner-e2e/tsconfig.json` passes.
- A paid run of both cases with the product fixes is linked in a comment
below.

## Risks

- A valid reply that names Alex but links nothing now fails
wrong-person-resolves. The case contract says chat explains whose action
is required; the card is where Paperclip shows it, so pointing to it is
the structural form of that promise.
- The label check could flag a label that intentionally names two tasks
of the same company and links only one. That reply would still be
misleading, so failing it is acceptable.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 23:44:15 -06:00
DottaandClaude Opus 5.5 bf18b348c5 fix(evals): wait for saved, rendered replies and settled forms, and step through multi-question cards (#21009)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog drives real chat journeys in a browser and
grades saved product state
> - In full cohort 3, four cases acted on the page before the page was
ready, or did not step through a card
> - The product was correct in each case, so these failures hide real
product failures
> - This pull request makes the harness wait for the saved, rendered
reply and for a settled form, and step through multi-question cards
> - The benefit is fewer false failures, with the same product
requirements

## Linked Issues or Issue Description

Refs #21000
Refs #21008

**What happened?**
- `companies.fast-responses`, `agents.choose-primary`,
`budgets-costs.set-budget`: the server recorded the chat run as finished
0.2 to 1.3 seconds before it saved the reply comment. The harness read
the reply links right after the run finished. It found no reply, so it
found no links.
- `budgets-costs.set-budget`: the budget card remounted after the
harness typed the new amount. The remount reset the draft, so the Update
budget button stayed disabled until the click timed out.
- `questions-and-approvals.necessary-question`: the agent asked the
required product-name question and one optional question. The card
showed Next on its first page. The harness only clicked Submit answers.
The grade also required exactly one question on the card.

**Expected behavior**
The harness reads a turn's links after its reply is saved and rendered.
It saves the budget only on a settled form. It pages through a card,
answers its one required question and skips optional ones.

**Steps to reproduce**
Run the three link cases when the reply is saved after the run finishes.
Run set-budget when the budget card reloads after typing. Run
necessary-question when the card has an optional second question.

## What Changed

- `chat-rendered-links.ts`: new shared `waitForSavedTurnReplyRender`. It
waits (bounded, 15 s) until each finished lead run's reply is saved,
then uses the existing render wait for the latest reply. It returns
`unsaved` when no reply arrives. The caller then fails with its own
error.
- `identity-support.ts`: new `waitForTurnReplyLinks` uses that wait for
the source conversation. `companies.fast-responses` (v3),
`agents.choose-primary` (v4) and `budgets-costs.set-budget` (v3) call it
before they read reply links.
- `budgets-costs.set-budget`: `settledInput` resolves when the same
input node stays on the page for one second. The harness types and saves
on that node. If the card still remounts and drops the amount, it types
it again, at most three times.
- `questions-and-approvals.necessary-question` (v2): the harness pages
through a multi-question card. It answers the one required question and
clicks Skip on each optional question. The grade now requires exactly
one required question, and that question must ask for the product name
as free text. Other questions must be optional. An optional necessary
question, a second required question, or a required question about
something else still fails.
- Descriptors are updated. No scenario yaml changed, so the TestOps
manifest does not change.

## Verification

- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-rendered-links.test.ts
tests/runner-e2e/chat-catalog-consistency.test.ts
tests/runner-e2e/chat-catalog-fingerprint.test.ts
tests/runner-e2e/chat-catalog-domains/identity.test.ts`: 100 tests pass.
- New tests: a reply saved after the run finished is waited for and then
rendered; a run that saves no reply returns `unsaved` within its bound.
The necessary-question grade accepts the recorded card (required name
plus optional follow-up) and rejects the broken cards above.
- `npx tsc -p tests/runner-e2e/tsconfig.json --noEmit` passes.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`
reports no change.
- One paid run on a combined branch with #21008. The result is in a
comment on this pull request.

## Risks

- A turn whose run saves no reply now waits up to 15 seconds before the
case continues. Only the three named cases call the wait.
- `settledInput` marks the input node with a `data-e2e-settle-mark`
attribute to detect a remount. It does not change product behavior.
- #21000 changes the render-wait expiry in the same file. Whichever
merges second rebases.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 21:33:32 -06:00
DottaandClaude Opus 5.5 65ff80a4bb fix(evals): grade chat acknowledgements, saved replies and links on saved state (#21000)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog runs real chat journeys against real agents
and grades saved product state
> - A review of full cohort 3 found three cases whose oracle passed
wrong behavior, and eight passes whose final evidence saved no rendered
links
> - `companies.agent-observability` passed although the agent denied the
saved collector. `inbox-search.show-my-work` counted links that existed
only in run-stream prose. `instance-operations.restart-service` accepted
a reply whose only link was the `/<prefix>/` placeholder
> - The final evidence read ran on a page that had not rendered the
conversation yet, so it saved no links
> - This pull request grades those cases on saved-state reads, the saved
reply and resolved links, and reads final evidence from the rendered
conversation
> - The benefit is that these cases fail when the product is wrong and
keep their links when it is right

## Linked Issues or Issue Description

Refs #20998
Refs #20999

**What happened?**
- `companies.agent-observability`: the board saved and tested a
collector. Asked to acknowledge it, the agent said there was no endpoint
or credentials. The oracle graded only human-side state, so it passed.
- `inbox-search.show-my-work`: the saved reply said "Answered above"
over nothing. The links the oracle counted came from run-stream prose
that the bubble also shows.
- `instance-operations.restart-service`: the reply's only action link
was `/<prefix>/company/settings/instance`. The oracle did not check
links.
- Eight cohort-3 passes (`access.other-company`, `agents.agent-api-key`,
`assistant-connections.revoke-personal-grant`,
`budgets-costs.resolve-budget`, `recovery.retry-exhausted`,
`steering.stop-versus-pause`, `task-coordination.pause-tree`,
`teams.install-team`) saved a final observation with no links and a
failed grade. The graded reload had the links. The final evidence read
ran after `verifyResult` left the page mid-load, and an expired render
wait from an earlier document made the read instant.

**Expected behavior**
Each oracle fails the observed wrong behavior and passes valid, reworded
behavior. Final evidence holds the links that the conversation renders.

**Steps to reproduce**
Grade the retained cohort-3 observations with the old oracles. They
pass. Grade them with these oracles. The three wrong cases fail.

## What Changed

- `companies.agent-observability` (oracle v4): each acknowledgement turn
must read the saved trace export status successfully. The read is bound
to the company by its redaction-safe `call_api` receipt. A reply must
also render a link to `/<prefix>/company/settings` for the fixture
company. The reply's wording is not matched.
- `inbox-search.show-my-work` (oracle v5): actionable links must be
carried by the saved reply: a Markdown link target in the saved body, or
a task identifier written in it, which the conversation links. New
shared helper `savedReplyLinks`.
- `instance-operations.restart-service` (oracle v3): new check
`reply-links-resolve` rejects any rendered or saved link target that
keeps a template placeholder (`<...>` or `{...}`). New helpers
`markdownLinkTargets` and `unresolvedLinkTarget` in
`chat-rendered-links.ts`.
- Link capture: the final success-path evidence read reopens the
rendered conversation first. An expired render wait now belongs to its
loaded document, not to the Playwright page.
- Updated `chat-catalog-descriptors.json` for the three oracle versions
and source files. No scenario yaml changed, so the TestOps manifest does
not change.

## Verification

- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog-domains/identity-observability.test.ts
tests/runner-e2e/chat-catalog-domains/internal.test.ts
tests/runner-e2e/chat-catalog-domains/privileged-oracles.test.ts
tests/runner-e2e/chat-catalog-domains/privileged-restart-history.test.ts
tests/runner-e2e/chat-rendered-links.test.ts
tests/runner-e2e/chat-catalog-consistency.test.ts
tests/runner-e2e/chat-catalog-fingerprint.test.ts`
- Calibration tests for each case: a valid result, a reworded valid
result, and the broken cohort-3 behavior.
- Red/green: the new render-wait test fails on the old wait logic and
passes with this change.
- `npx tsc -p tests/runner-e2e/tsconfig.json --noEmit` passes. `node
tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check` reports no
change.
- A paid run on a combined branch with the two product fixes is recorded
in a comment on this pull request.

## Risks

- `companies.agent-observability` needs the agent-readable status route
from #20998. Without it, the case fails, which is correct.
- The show-my-work check needs links in the saved reply. Before #20999,
an OpenCode reply that left its answer in the run stream fails this
case.
- The final evidence read can wait up to 30 seconds to reopen the
conversation. The read is still bounded.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 20:49:38 -06:00
DottaandClaude Opus 5.5 cbd636d652 fix(evals): wait for rendered replies, bound runs by role and grade chat outcomes structurally (#20685)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The chat-feature eval catalog runs real chat journeys and grades
saved state, rendered links and run ledgers.
> - The second full-cohort run on master `9296c74e8d` failed 15 cases
where the product did the right thing and the harness did not.
> - The causes were three harness classes: links read before the reply
rendered, raw run counts that rejected product-caused follow-up runs,
and keyword graders on free-form replies. Four browser locators also
targeted retired controls.
> - This pull request fixes those harness layers. It does not change the
product.
> - The benefit is that these cases measure the feature again: correct
behaviour passes, and the relevant broken behaviour still fails.

## Linked Issues or Issue Description

**What happened**
In the second full-cohort chat-eval run on master `9296c74e8d`, these
cases failed although the reply and the saved state were correct:
`agents.choose-primary`, `cases.case-history`,
`budgets-costs.set-budget` (links read about 2s before the reply
rendered); `task-coordination.independent-review`,
`cases.update-case-facts`, `routines.manage-other-owner`,
`questions-and-approvals.review-plan` (a legitimate continuation or the
product's completion follow-up exceeded a raw run count or was
cancelled); `cases.cases-disabled`, `inbox-search.find-design`,
`navigation-preferences.broken-link`, `cases.browse-cases` (keyword or
clause graders rejected correct replies); `agent-chat.conference-room`,
`workspaces.inspect-git`, `task-coordination.block-on-design`,
`skills.author-skill` (browser steps used stale settings or controls).

**Expected behavior**
A correct reply and correct saved state pass. A wrong outcome, extra or
duplicate agent work, a substitution or a missing route still fails.

**Steps to reproduce**
Run `runner-full-stack-e2e.yml` with `cadence=manual`, `all=false` and
an `id` list of the cases above against master `9296c74e8d`. Read
`result.json` and `snapshots/` in each cell's evidence.

Refs #20600

## What Changed

- `chat-rendered-links.ts`: `collectConversationLinks` waits (at most
10s, once per reply per page) for the latest saved lead reply to render
when its conversation is open. A page that does not show the
conversation does not wait. `observeInternal` and the flow conversation
read use it.
- `budgets-costs.set-budget`: waits for the rendered reply before it
reads anchors. It fills the budget only after the form shows the saved
cap, and checks that the typed value stays.
- New `chat-catalog-domains/run-roles.ts`: identifies a run by role (a
worker woken by a comment that a named run saved on the task; the CEO's
chat completion follow-up).
- `task-coordination.independent-review`: admits one Morgan continuation
woken by the CEO's authorization comment and one Riley review run. The
review request may come from that continuation's native status decision.
Other runs still fail.
- `cases.update-case-facts`: a phase settles when its CEO turn and at
most one Morgan continuation woken by that turn's task comment finish. A
new `role-bound-runs` check rejects any other run. `terminal-counts`
gets an optional `maximumTerminalStatusCounts`.
- `routines.manage-other-owner`: admits one Morgan run on a task that
the CEO created for Morgan. A schedule change must be Morgan's own write
from that run. The ownership check is structural (a linked owner task,
or a link to the routine or Morgan's page). The keyword checks on the
reply are removed.
- `questions-and-approvals.review-plan`: after the stale-review check,
lets the product's completion follow-up settle before final run custody.
The existing grader already validates it.
- `cases.cases-disabled`: the negation-blind false-claim regex is
removed. The case passes with a reply link to the Experimental settings
page (the scripted person follows it and sees the Cases control) or a
pending choice card. Saved state still proves that nothing was created.
- `inbox-search.find-design`: the per-clause regex is removed. The first
task link in the reply must be the task with the approved revision.
- `navigation-preferences.broken-link`: a link to the similar task fails
only when its label names the original (identifier or exact title). A
separate mention passes.
- `cases.browse-cases`: an excluded case must not be a result row. When
the reply has no result list, every case link is a result, so an
excluded case must not be linked.
- Browser steps: `conference-room` reloads after the fixture enables the
flag. `inspect-git` and `block-on-design` open the panel with "Show
properties" and pick the tab from the side panel launcher.
`author-skill` does not repeat the full inspection in `verifyResult`
when `act` already inspected the same saved version.
- Each changed case has a new `oracleVersion`.
`chat-catalog-descriptors.json` is in sync.

## Verification

- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts <119
affected test files> --testTimeout=180000`: 1673 passed, 3 skipped. One
file (`chat-factual-query-input.test.ts`) did not load in the local
worktree because `@paperclipai/plugin-sdk` is not built there. It does
not import changed code. CI runs it.
- `chat-catalog-consistency.test.ts` passes (242 descriptors).
- `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
no pin change.
- `node --test` on the three `chat-full-catalog-testops-*.test.mjs`
files: 40 passed.
- Calibration against retained evidence: the retained replies and run
shapes of cases-disabled, browse-cases, find-design, broken-link,
independent-review, update-case-facts and manage-other-owner now pass.
Broken variants fail (closed case as a result row, earlier design linked
first, original label on the similar task's link, unprompted or
duplicate worker runs, a second review run, a routine change outside the
owner's run, no route link).
- Paid runs on this branch: see the PR comments.

## Risks

- Low risk. Only eval harness code changes.
- The structural graders do not read reply wording. A reply that words
the outcome wrongly but links the right records can pass. A person reads
the reply.
- `navigation-preferences.sidebar-order` is not changed. The shipped
sidebar has no per-project order or collapsible Agents section, so the
case premise needs an owner decision.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 20:24:41 -06:00
f4337bb5fb test(evals): land the chat catalog eval harness on master (#19614)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-feature evals measure whether agents complete real chat
requests through the product
> - The chat catalog eval harness and its repairs live on the
`chat-eval/master-candidate` branch, not on master
> - Today's work made 139 of 242 chat evals pass in verified single-case
live runs. Each product fix landed on master as its own pull request
> - The eval-side changes (setups, graders, fault seams, evidence
capture) still exist only on the candidate
> - This pull request lands the harness on master. Master stays the
source of truth for product behavior
> - The benefit is one tree: evals run against master commits, and
product fixes reach the evals as soon as they merge

## Linked Issues or Issue Description

Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs
#20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409

**What existing behavior does this improve?**
The chat catalog eval harness and the product it evaluates live on a
long-running branch outside master. Eval fixes do not reach master, so
master-based eval runs use old setups and graders.

**Subsystem affected**
`tests/runner-e2e` chat catalog harness, plus a small set of product
changes that exist only on the candidate.

**Proposed change**
Land the harness on master. Every product fix from this eval work
already merged to master as its own pull request. Conflicts keep
master's product behavior.

## What Changed

- Most of the diff (808 of 851 files) is under `tests/runner-e2e/`:
chat-catalog domains, scenarios, descriptors, the TestOps manifest,
private evidence tooling, and their tests.
- Graders now check saved state, setup literals and rendered or clicked
links. Many keyword and regex checks on free-form replies are removed.
Each changed grader has a bumped `oracleVersion` and calibration tests
(a valid reply, a reworded valid reply, and broken variants that must
fail).
- The shared link helper `hasLink` no longer treats a list link (for
example `/COM/issues?view=mine`) as a link to every task without a URL
key.
- Setup fixes build real state through supported product routes: routine
history, agent grants through the board grant route, budget holds with a
real cap, and a storage-layer archive fault for
`inbox-search.archive-partial-failure`.
- Browser boundary journeys (sign-in, chat-disabled, onboarding) now
write the standard pass evidence files.
- Retargeted cases, approved by the maintainer:
`agent-chat.chat-disabled` (Conference Room gate),
`agents.restore-instructions` (no saved instruction history),
`access.manage-membership` (member permission edits stay board-only).
- The private evidence trusted files (`manual-private-handoff.ts`,
`manual-private-retention.ts`, `manual-private-report-assets.ts`) are
byte-identical to master.
- Product files outside `tests/` (43 files, +1,328 −49) carry
candidate-only changes that master does not yet have. Most are tests.
The main code changes are in `request-human-input.ts`,
`heartbeat-run-summary.ts`, `native-completion-feedback.ts`,
`native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and
`IssueChatLongThreadPerf.tsx`.

## Verification

- `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts
chat-catalog-consistency.test.ts unit-launcher.test.mjs
--testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests
pass.
- `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
no change.
- `tsc` for `server` and `tests/runner-e2e`: clean.
- Live single-case verification: 139 of 242 cases passed on paid runs,
and a person read the saved evidence for each. These are single-case
runs, not one full-cohort run.

## Risks

- Large diff. Most of it is test-only, but the 43 non-test files change
product code paths. Review those files first.
- Some graders are now less strict on reply wording. A person must read
replies for meaning on those cases.
- 42 cases need disposable outside accounts (Notion, Slack, GitHub,
Daytona and others) and stay blocked. `access.invite-teammate` stays
failing until local agent isolation lands, because the runner's durable
state keeps a live invite token.
- The paid workflow compares the trusted private-evidence files with
master. They are unchanged, so this risk is low.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution, in Claude Code. Subagents used the same model.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-10 08:02:11 -06:00