Files
PaperClipAI/tests/runner-e2e/chat-returning-browser.ts
T
DottaandClaude Opus 5.5 8e332ccb5e fix(evals): name stale SDK pins and fix four chat-eval harness defects (#20579)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat feature eval catalog (`tests/runner-e2e`) runs real chat
journeys against a real server and grades the saved state, the run's
tool events and the rendered reply
> - The full-cohort run on master `f4337bb5fb` had 12 failures that come
from the harness, not from the product or the model
> - Six plugin and secrets cells stopped in 7–9 s, before the server
started, because the SDK public source pins were stale
> - Four more cells failed on harness timing, a first-visit prompt, or a
read receipt that ignored a dedicated tool
> - This pull request fixes those four harness defects and adds a cheap
guard that names stale SDK pins
> - The benefit is that these cells now report on the product and the
model again

## Linked Issues or Issue Description

Refs #19614, #20550, #20453, #20417, #20483

**What happened**

On master `f4337bb5fb`, these chat eval cells failed for harness
reasons:

- `plugins.configure-plugin`, `plugins.disable-plugin`,
`plugins.install-plugin`, `plugins.use-plugin-tool`,
`routines.plugin-managed-routine` and `secrets.least-scope-delivery`
stopped before the server started. The source SDK privacy scan refused,
because `tests/runner-e2e/source-sdk-public-inputs.json` did not match
`packages/shared/src`. #20550 and #20362 already resynced the pins on
master. The pins are current at `ce1ee9e89f`.
- `agent-chat.track-delegated-work` and
`navigation-preferences.edit-profile`: the reply was correct and had the
right link. The case read the rendered links immediately after the runs
settled, before the reply rendered, so it recorded no links.
- `inbox-search.archive-done` and `inbox-search.restore-archive`: setup
completes five tasks, so the GitHub star prompt from #20453 opened over
the chat and blocked the composer.
- `status-cards.explain-card`: the follow-up run read the deleted source
with `get_task` and received a 404. The reply said so. `apiReadStatuses`
counted only `call_api` reads, so the grader recorded no unavailable
read.

Why the existing pin check did not stop the drift: `ci / Runner support
suite` runs the source SDK tests and is part of the required `ci /
verify`. On #19614 it passed at 03:43Z on a merge ref from before #20417
(06:12Z) and #20483 (08:14Z). #19614 merged at 14:02Z without a new CI
run on current master. After that, the suite failed on every PR until
#20550. So the check exists and fires; the gap was a stale green result,
and the failure message was opaque (`SDK public source proof
unavailable`).

**Expected behavior**

- A change to a pinned SDK input without a pin refresh fails with the
changed files and the command that fixes it.
- Cases read rendered links only after the saved reply renders.
- The eval browser is a returning user's browser that has already seen
the star prompt. The product does not change.
- A dedicated task read tool (`get_task`, `list_task_documents`,
`read_task_document`, also when namespaced) counts as a read of the
record it names, with the status its receipt reports.

**Steps to reproduce**

1. Run `chat-features.runner-opencode.local.<case>` for one of the cases
above on master `f4337bb5fb`.
2. Read `result.json` and `snapshots/chat-catalog-observation.json` in
the evidence bundle.
3. See the failures described above.

## What Changed

- `tests/runner-e2e/source-sdk-public-inputs.mjs`: add
`staleSdkPublicInputs(root)`. It compares the registry with `git ls-tree
HEAD` for every pinned input (the roots and build inputs that the pins
file lists), and returns each changed, unpinned or removed file.
- `tests/runner-e2e/source-sdk-public-inputs.test.mjs`: a test fails
when the committed pins do not match the committed tree. The message
names the files and gives the `--write` command. The Runner support
suite runs it for every `packages/**` change. A synthetic test covers
changed, unpinned and removed inputs.
- `tests/runner-e2e/chat-catalog-domains/runtime-handoffs.ts` and
`identity-profile.ts`: call the existing `waitForRenderedReply` before
they read rendered links.
- `tests/runner-e2e/chat-returning-browser.ts` (new) and
`chat-catalog-flow.ts`: inside the flow's failure boundary, before
setup, mark `paperclip.github-star-prompt.seen` in the eval browser. An
init script covers later documents of the app origin, and the open page
is marked directly. A test checks that the key equals
`STAR_PROMPT_SEEN_KEY` in `ui/src/components/GitHubStarPrompt.tsx`.
- `tests/runner-e2e/chat-catalog-domains/api-read-receipts.ts`: bind
dedicated task read tools by the digest of the `call_api` input that the
server builds for them (`taskBridgeApiRequest`). The result must carry
the same `apiOperationId`. Other tools, other records, mismatched
receipts and unpaired results do not count.
- Bump `oracleVersion` in code and descriptor for every case that uses
`api-read-receipts.ts` (shared grader change) and for
`navigation-preferences.edit-profile`. Add
`internal-comment-reference.ts` to the source files of the two cases
that now import it.

## Verification

- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts`
(whole config): 11301 passed; 1 local-only failure in
`controller-replacement-smoke.test.ts` (server spawn, no changed
imports).
- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
--testTimeout=180000 tests/runner-e2e/chat-catalog-domains
tests/runner-e2e/chat-catalog-contract.test.ts
tests/runner-e2e/chat-catalog-consistency.test.ts
tests/runner-e2e/chat-returning-browser.test.ts`
- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
source-sdk`
- `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse
HEAD) --check` reports `changed: false`.
- Red checks: with the `f4337bb5fb` registry, the new pin test fails and
names 15 inputs. Without the new waits, the two rendered-reply tests
fail. A regrade of the retained `explain-card` run events now gives the
follow-up run `unavailable: true, live: false` and the first run `live:
true`.
- Paid runs of the affected cells
(`chat-features.runner-opencode.local.*`):
- Run 42852698879 (head `d54c04d5f0`): `track-delegated-work`,
`restore-archive`, `explain-card`, `configure-plugin` and
`plugin-managed-routine` pass. I read each reply and saved state.
`track-delegated-work` now records the rendered `/COM/issues/COM-2`
link. `explain-card` records `laterUnavailable: true, laterLive: false`
from the follow-up `get_task` 404.
- `plugins.install-plugin` now starts its server (the pin refusal is
gone). It fails a later grading check, `truthful-installation-handoff`:
the reply sends the person to the plugin settings page but does not name
the package. This is outside this PR.
- `inbox-search.archive-done`: the star prompt no longer blocks the
composer, and the query is sent. The agent then archives the 501 Done
tasks one at a time (about 2 s each, 57 in the 240 s window), because
only the per-task archive route exists. This is outside this PR.
- `navigation-preferences.edit-profile` did not start in that run (the
AWS credentials step timed out). Run 42857020103 (head `2f0b80aa84`):
pass. The agent write gets 401, the reply links
`/COM/company/settings/instance/profile`, and the person follows the
rendered link and saves the name.

## Risks

- Low risk. Test harness only; no product code changes.
- The receipt change counts more reads as record reads. It only adds
dedicated reads bound to the exact record id and operation, so a grader
that needs a read can pass on a genuine `get_task` read. The affected
cases have new oracle versions.
- The star prompt is not tested by any chat catalog case. Product e2e
tests for the prompt do not use this flow.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with
tool use (shell, file editing, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 11:01:44 -06:00

33 lines
1.8 KiB
TypeScript

import type { Page } from "@playwright/test";
/**
* Once-per-browser prompts a returning person has already dismissed. Chat
* journeys test chat, not first-visit nudges: the GitHub star prompt (#20453)
* opens after five done tasks and covers the composer. The product is
* unchanged; the eval browser carries the same saved choice a returning user's
* browser would. Keys mirror ui/src/components/GitHubStarPrompt.tsx (checked
* by chat-returning-browser.test.ts).
*/
export const RETURNING_BROWSER_SEEN_KEYS = ["paperclip.github-star-prompt.seen"] as const;
const SEEN_AT = "2026-01-01T00:00:00.000Z";
function markSeen({ origin, keys, seenAt }: { origin: string; keys: readonly string[]; seenAt: string }) {
try {
if (window.location.origin !== origin) return;
for (const key of keys) if (window.localStorage.getItem(key) === null) window.localStorage.setItem(key, seenAt);
} catch { /* opaque or storage-less document */ }
}
/** Marks the prompts seen for every later document of this app origin, and for the one already open. */
export async function markReturningBrowser(page: Page, appOrigin: string) {
// Flow unit tests drive the flow with page stand-ins that have no browser context; there is nothing to mark.
const context = typeof page?.context === "function" ? page.context() : undefined;
if (!context || typeof appOrigin !== "string") return;
const args = { origin: new URL(appOrigin).origin, keys: [...RETURNING_BROWSER_SEEN_KEYS], seenAt: SEEN_AT };
await context.addInitScript(markSeen, args);
let current: string | undefined;
try { current = new URL(page.url()).origin; } catch { current = undefined; }
// The prompt rereads storage before it opens, so an already-open page needs no reload.
if (current === args.origin) await page.evaluate(markSeen, args);
}