PaperClipAI

Jannik/PaperClipAI

0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-13 15:55:46 +01:00

Commit Graph

Author	SHA1	Message	Date
DottaandClaude Opus 5.5	e2cf9a670f	fix(runner-e2e): resync SDK public lock pins and make the lockfile bot keep them current (#20623 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner support suite proves the plugin SDK public types from pinned sources in `tests/runner-e2e/source-sdk-public-inputs.json` > - That registry pins the committed `pnpm-lock.yaml` blob and a `lockSha256` that must equal the `outputSha256` in `tests/runner-e2e/report-lock.json` > - The lockfile bot refresh #20615 changed the lock and `report-lock.json`, but the bot does not write the SDK registry > - The bot merges without CI, so the drift guard from #20579 could not block it, and every PR now fails `ci / Runner support suite` > - The guard also checked only the source blobs, so it did not name the `lockSha256` drift, and the sync script could not update `lockSha256` > - This pull request regenerates the registry, derives `lockSha256` from the reviewed policy, and makes the bot resync and check the registry in its own commit > - The benefit is that master CI is green again, and the next lockfile refresh cannot leave the SDK proof stale ## Linked Issues or Issue Description Refs #20615. Refs #20579. Refs #20550. Refs #20613. Overlaps #20618, which refreshes only the registry data. This PR has the same data change and also closes the gap that let the drift land. What happened? After #20615 merged, `ci / Runner support suite` failed on master and on every PR (for example runs 42877750315 and 42877750957, and PR #20613). `source-sdk-public-type.test.ts` and `source-sdk-evidence.test.ts` failed with "Reviewed SDK reference policy does not match public proof". `source-sdk-public-inputs.test.mjs` failed with "pnpm-lock.yaml (changed)". Expected behavior A lockfile refresh keeps the SDK public proof registry in step with the lock and the reviewed report-lock policy. If the registry drifts, the failure names the stale pin and the fix command. Steps to reproduce 1. Check out master at `3bf7af6c77`. 2. Run `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts source-sdk`. 3. See the "Reviewed SDK reference policy does not match public proof" failure. ## What Changed - Root cause: #20615 changed `report-lock.json` from output `2360d5e9…` to `b3208306…` and changed the `pnpm-lock.yaml` blob from `851e65af…` to `6d825f97…`. The registry kept the old values. `refresh-lockfile.yml` commits only the lockfiles and the report-lock proof, and it merges without CI. - `source-sdk-public-inputs.json`: regenerated with `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --write`. Only `inputs["pnpm-lock.yaml"]` and `lockSha256` change. The compiler, parser and Zod pins do not change. - `source-sdk-public-inputs.mjs`: the sync now copies `lockSha256` from the committed `report-lock.json` `outputSha256`. It refuses when the committed lock is not that policy's reviewed input or output. This is the same rule that `prepareReviewedSdkReference` applies. Only the lockfile bot can write that policy, so a contributor cannot choose `lockSha256`. - `staleSdkPublicInputs()`: also reports a `lockSha256` that differs from the committed report-lock output. The guard test then names both stale pins and the `--write` command. - `source-sdk-reviewed-fixture.ts`: the "Reviewed SDK reference policy does not match public proof" error now names the two values and the fix command. The check itself is unchanged. - `refresh-lockfile.yml`: after the lock commit, the bot runs the SDK sync with `--write`, amends the commit, and runs `--check` before it pushes. `head_sha` is still read after the amend, so the bot merges the exact commit it checked. - Tests: new sync tests for a lockfile refresh and for an unreviewed lock, and a workflow-order test in `lockfile-refresh-merge.test.mjs`. - Docs: `doc/RELEASE-AUTOMATION-SETUP.md` §8.1 and `tests/runner-e2e/OPERATIONS.md` describe the resync. ## Verification - Reproduced on master: `source-sdk-public-type.test.ts` and `source-sdk-evidence.test.ts` failed with the policy mismatch. - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts source-sdk prepare-report-lock sdk-canary workflow-security`: 9 files, 135 tests pass. - Before the registry change, the extended guard reported `pnpm-lock.yaml (changed), tests/runner-e2e/report-lock.json outputSha256 (lockSha256 changed)` with the fix command. - `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse HEAD) --check` reports `changed: false`. - `node --test .github/scripts/tests/lockfile-refresh-merge.test.mjs .github/scripts/tests/lockfile-refresh-workflows.test.mjs .github/scripts/tests/lockfile-trust.test.mjs`: 39 tests pass. - `pnpm test:e2e:runner:unit` on a host with load near 28: 6 files timed out, and none of them are the SDK proof files. All 15 of those files pass when run alone. CI runs the full suite. ## Risks - The bot step now runs one more repository script. If that script refuses (for example, an unreviewed committed lock), the refresh PR is not pushed. Before this change, the bot merged a refresh that made master red. - `lockSha256` now comes from the reviewed policy, not from the old registry value. The proof check is unchanged. It still needs the lock bytes to hash to `lockSha256`. - If #20618 merges first, this branch rebases cleanly, because the registry change is identical. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit) and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-11 06:57:51 -08:00
DottaandClaude Opus 5.5	8e332ccb5e	fix(evals): name stale SDK pins and fix four chat-eval harness defects (#20579 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat feature eval catalog (`tests/runner-e2e`) runs real chat journeys against a real server and grades the saved state, the run's tool events and the rendered reply > - The full-cohort run on master `f4337bb5fb` had 12 failures that come from the harness, not from the product or the model > - Six plugin and secrets cells stopped in 7–9 s, before the server started, because the SDK public source pins were stale > - Four more cells failed on harness timing, a first-visit prompt, or a read receipt that ignored a dedicated tool > - This pull request fixes those four harness defects and adds a cheap guard that names stale SDK pins > - The benefit is that these cells now report on the product and the model again ## Linked Issues or Issue Description Refs #19614, #20550, #20453, #20417, #20483 What happened On master `f4337bb5fb`, these chat eval cells failed for harness reasons: - `plugins.configure-plugin`, `plugins.disable-plugin`, `plugins.install-plugin`, `plugins.use-plugin-tool`, `routines.plugin-managed-routine` and `secrets.least-scope-delivery` stopped before the server started. The source SDK privacy scan refused, because `tests/runner-e2e/source-sdk-public-inputs.json` did not match `packages/shared/src`. #20550 and #20362 already resynced the pins on master. The pins are current at `ce1ee9e89f`. - `agent-chat.track-delegated-work` and `navigation-preferences.edit-profile`: the reply was correct and had the right link. The case read the rendered links immediately after the runs settled, before the reply rendered, so it recorded no links. - `inbox-search.archive-done` and `inbox-search.restore-archive`: setup completes five tasks, so the GitHub star prompt from #20453 opened over the chat and blocked the composer. - `status-cards.explain-card`: the follow-up run read the deleted source with `get_task` and received a 404. The reply said so. `apiReadStatuses` counted only `call_api` reads, so the grader recorded no unavailable read. Why the existing pin check did not stop the drift: `ci / Runner support suite` runs the source SDK tests and is part of the required `ci / verify`. On #19614 it passed at 03:43Z on a merge ref from before #20417 (06:12Z) and #20483 (08:14Z). #19614 merged at 14:02Z without a new CI run on current master. After that, the suite failed on every PR until #20550. So the check exists and fires; the gap was a stale green result, and the failure message was opaque (`SDK public source proof unavailable`). Expected behavior - A change to a pinned SDK input without a pin refresh fails with the changed files and the command that fixes it. - Cases read rendered links only after the saved reply renders. - The eval browser is a returning user's browser that has already seen the star prompt. The product does not change. - A dedicated task read tool (`get_task`, `list_task_documents`, `read_task_document`, also when namespaced) counts as a read of the record it names, with the status its receipt reports. Steps to reproduce 1. Run `chat-features.runner-opencode.local.<case>` for one of the cases above on master `f4337bb5fb`. 2. Read `result.json` and `snapshots/chat-catalog-observation.json` in the evidence bundle. 3. See the failures described above. ## What Changed - `tests/runner-e2e/source-sdk-public-inputs.mjs`: add `staleSdkPublicInputs(root)`. It compares the registry with `git ls-tree HEAD` for every pinned input (the roots and build inputs that the pins file lists), and returns each changed, unpinned or removed file. - `tests/runner-e2e/source-sdk-public-inputs.test.mjs`: a test fails when the committed pins do not match the committed tree. The message names the files and gives the `--write` command. The Runner support suite runs it for every `packages/*` change. A synthetic test covers changed, unpinned and removed inputs. - `tests/runner-e2e/chat-catalog-domains/runtime-handoffs.ts` and `identity-profile.ts`: call the existing `waitForRenderedReply` before they read rendered links. - `tests/runner-e2e/chat-returning-browser.ts` (new) and `chat-catalog-flow.ts`: inside the flow's failure boundary, before setup, mark `paperclip.github-star-prompt.seen` in the eval browser. An init script covers later documents of the app origin, and the open page is marked directly. A test checks that the key equals `STAR_PROMPT_SEEN_KEY` in `ui/src/components/GitHubStarPrompt.tsx`. - `tests/runner-e2e/chat-catalog-domains/api-read-receipts.ts`: bind dedicated task read tools by the digest of the `call_api` input that the server builds for them (`taskBridgeApiRequest`). The result must carry the same `apiOperationId`. Other tools, other records, mismatched receipts and unpaired results do not count. - Bump `oracleVersion` in code and descriptor for every case that uses `api-read-receipts.ts` (shared grader change) and for `navigation-preferences.edit-profile`. Add `internal-comment-reference.ts` to the source files of the two cases that now import it. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts` (whole config): 11301 passed; 1 local-only failure in `controller-replacement-smoke.test.ts` (server spawn, no changed imports). - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts --testTimeout=180000 tests/runner-e2e/chat-catalog-domains tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts tests/runner-e2e/chat-returning-browser.test.ts` - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts source-sdk` - `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse HEAD) --check` reports `changed: false`. - Red checks: with the `f4337bb5fb` registry, the new pin test fails and names 15 inputs. Without the new waits, the two rendered-reply tests fail. A regrade of the retained `explain-card` run events now gives the follow-up run `unavailable: true, live: false` and the first run `live: true`. - Paid runs of the affected cells (`chat-features.runner-opencode.local.`): - Run 42852698879 (head `d54c04d5f0`): `track-delegated-work`, `restore-archive`, `explain-card`, `configure-plugin` and `plugin-managed-routine` pass. I read each reply and saved state. `track-delegated-work` now records the rendered `/COM/issues/COM-2` link. `explain-card` records `laterUnavailable: true, laterLive: false` from the follow-up `get_task` 404. - `plugins.install-plugin` now starts its server (the pin refusal is gone). It fails a later grading check, `truthful-installation-handoff`: the reply sends the person to the plugin settings page but does not name the package. This is outside this PR. - `inbox-search.archive-done`: the star prompt no longer blocks the composer, and the query is sent. The agent then archives the 501 Done tasks one at a time (about 2 s each, 57 in the 240 s window), because only the per-task archive route exists. This is outside this PR. - `navigation-preferences.edit-profile` did not start in that run (the AWS credentials step timed out). Run 42857020103 (head `2f0b80aa84`): pass. The agent write gets 401, the reply links `/COM/company/settings/instance/profile`, and the person follows the rendered link and saves the name. ## Risks - Low risk. Test harness only; no product code changes. - The receipt change counts more reads as record reads. It only adds dedicated reads bound to the exact record id and operation, so a grader that needs a read can pass on a genuine `get_task` read. The affected cases have new oracle versions. - The star prompt is not tested by any chat catalog case. Product e2e tests for the prompt do not use this flow. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file editing, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 11:01:44 -06:00
Dotta Claude Opus 5.5 Paperclip	f4337bb5fb	test(evals): land the chat catalog eval harness on master (#19614 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat-feature evals measure whether agents complete real chat requests through the product > - The chat catalog eval harness and its repairs live on the `chat-eval/master-candidate` branch, not on master > - Today's work made 139 of 242 chat evals pass in verified single-case live runs. Each product fix landed on master as its own pull request > - The eval-side changes (setups, graders, fault seams, evidence capture) still exist only on the candidate > - This pull request lands the harness on master. Master stays the source of truth for product behavior > - The benefit is one tree: evals run against master commits, and product fixes reach the evals as soon as they merge ## Linked Issues or Issue Description Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs #20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409 What existing behavior does this improve? The chat catalog eval harness and the product it evaluates live on a long-running branch outside master. Eval fixes do not reach master, so master-based eval runs use old setups and graders. Subsystem affected `tests/runner-e2e` chat catalog harness, plus a small set of product changes that exist only on the candidate. Proposed change Land the harness on master. Every product fix from this eval work already merged to master as its own pull request. Conflicts keep master's product behavior. ## What Changed - Most of the diff (808 of 851 files) is under `tests/runner-e2e/`: chat-catalog domains, scenarios, descriptors, the TestOps manifest, private evidence tooling, and their tests. - Graders now check saved state, setup literals and rendered or clicked links. Many keyword and regex checks on free-form replies are removed. Each changed grader has a bumped `oracleVersion` and calibration tests (a valid reply, a reworded valid reply, and broken variants that must fail). - The shared link helper `hasLink` no longer treats a list link (for example `/COM/issues?view=mine`) as a link to every task without a URL key. - Setup fixes build real state through supported product routes: routine history, agent grants through the board grant route, budget holds with a real cap, and a storage-layer archive fault for `inbox-search.archive-partial-failure`. - Browser boundary journeys (sign-in, chat-disabled, onboarding) now write the standard pass evidence files. - Retargeted cases, approved by the maintainer: `agent-chat.chat-disabled` (Conference Room gate), `agents.restore-instructions` (no saved instruction history), `access.manage-membership` (member permission edits stay board-only). - The private evidence trusted files (`manual-private-handoff.ts`, `manual-private-retention.ts`, `manual-private-report-assets.ts`) are byte-identical to master. - Product files outside `tests/` (43 files, +1,328 −49) carry candidate-only changes that master does not yet have. Most are tests. The main code changes are in `request-human-input.ts`, `heartbeat-run-summary.ts`, `native-completion-feedback.ts`, `native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and `IssueChatLongThreadPerf.tsx`. ## Verification - `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts chat-catalog-consistency.test.ts unit-launcher.test.mjs --testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests pass. - `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: no change. - `tsc` for `server` and `tests/runner-e2e`: clean. - Live single-case verification: 139 of 242 cases passed on paid runs, and a person read the saved evidence for each. These are single-case runs, not one full-cohort run. ## Risks - Large diff. Most of it is test-only, but the 43 non-test files change product code paths. Review those files first. - Some graders are now less strict on reply wording. A person must read replies for meaning on those cases. - 42 cases need disposable outside accounts (Notion, Slack, GitHub, Daytona and others) and stay blocked. `access.invite-teammate` stays failing until local agent isolation lands, because the runner's durable state keeps a live invite token. - The paid workflow compares the trusted private-evidence files with master. They are unchanged, so this risk is low. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution, in Claude Code. Subagents used the same model. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-10 08:02:11 -06:00

Author

SHA1

Message

Date

DottaandClaude Opus 5.5

e2cf9a670f

fix(runner-e2e): resync SDK public lock pins and make the lockfile bot keep them current (#20623 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner support suite proves the plugin SDK public types from
pinned sources in `tests/runner-e2e/source-sdk-public-inputs.json`
> - That registry pins the committed `pnpm-lock.yaml` blob and a
`lockSha256` that must equal the `outputSha256` in
`tests/runner-e2e/report-lock.json`
> - The lockfile bot refresh #20615 changed the lock and
`report-lock.json`, but the bot does not write the SDK registry
> - The bot merges without CI, so the drift guard from #20579 could not
block it, and every PR now fails `ci / Runner support suite`
> - The guard also checked only the source blobs, so it did not name the
`lockSha256` drift, and the sync script could not update `lockSha256`
> - This pull request regenerates the registry, derives `lockSha256`
from the reviewed policy, and makes the bot resync and check the
registry in its own commit
> - The benefit is that master CI is green again, and the next lockfile
refresh cannot leave the SDK proof stale

## Linked Issues or Issue Description

Refs #20615. Refs #20579. Refs #20550. Refs #20613. Overlaps #20618,
which refreshes only the registry data. This PR has the same data change
and also closes the gap that let the drift land.

**What happened?**

After #20615 merged, `ci / Runner support suite` failed on master and on
every PR (for example runs 42877750315 and 42877750957, and PR #20613).
`source-sdk-public-type.test.ts` and `source-sdk-evidence.test.ts`
failed with "Reviewed SDK reference policy does not match public proof".
`source-sdk-public-inputs.test.mjs` failed with "pnpm-lock.yaml
(changed)".

**Expected behavior**

A lockfile refresh keeps the SDK public proof registry in step with the
lock and the reviewed report-lock policy. If the registry drifts, the
failure names the stale pin and the fix command.

**Steps to reproduce**

1. Check out master at `3bf7af6c77`.
2. Run `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
source-sdk`.
3. See the "Reviewed SDK reference policy does not match public proof"
failure.

## What Changed

- Root cause: #20615 changed `report-lock.json` from output `2360d5e9…`
to `b3208306…` and changed the `pnpm-lock.yaml` blob from `851e65af…` to
`6d825f97…`. The registry kept the old values. `refresh-lockfile.yml`
commits only the lockfiles and the report-lock proof, and it merges
without CI.
- `source-sdk-public-inputs.json`: regenerated with `node
tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --write`. Only
`inputs["pnpm-lock.yaml"]` and `lockSha256` change. The compiler, parser
and Zod pins do not change.
- `source-sdk-public-inputs.mjs`: the sync now copies `lockSha256` from
the committed `report-lock.json` `outputSha256`. It refuses when the
committed lock is not that policy's reviewed input or output. This is
the same rule that `prepareReviewedSdkReference` applies. Only the
lockfile bot can write that policy, so a contributor cannot choose
`lockSha256`.
- `staleSdkPublicInputs()`: also reports a `lockSha256` that differs
from the committed report-lock output. The guard test then names both
stale pins and the `--write` command.
- `source-sdk-reviewed-fixture.ts`: the "Reviewed SDK reference policy
does not match public proof" error now names the two values and the fix
command. The check itself is unchanged.
- `refresh-lockfile.yml`: after the lock commit, the bot runs the SDK
sync with `--write`, amends the commit, and runs `--check` before it
pushes. `head_sha` is still read after the amend, so the bot merges the
exact commit it checked.
- Tests: new sync tests for a lockfile refresh and for an unreviewed
lock, and a workflow-order test in `lockfile-refresh-merge.test.mjs`.
- Docs: `doc/RELEASE-AUTOMATION-SETUP.md` §8.1 and
`tests/runner-e2e/OPERATIONS.md` describe the resync.

## Verification

- Reproduced on master: `source-sdk-public-type.test.ts` and
`source-sdk-evidence.test.ts` failed with the policy mismatch.
- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
source-sdk prepare-report-lock sdk-canary workflow-security`: 9 files,
135 tests pass.
- Before the registry change, the extended guard reported
`pnpm-lock.yaml (changed), tests/runner-e2e/report-lock.json
outputSha256 (lockSha256 changed)` with the fix command.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse
HEAD) --check` reports `changed: false`.
- `node --test .github/scripts/tests/lockfile-refresh-merge.test.mjs
.github/scripts/tests/lockfile-refresh-workflows.test.mjs
.github/scripts/tests/lockfile-trust.test.mjs`: 39 tests pass.
- `pnpm test:e2e:runner:unit` on a host with load near 28: 6 files timed
out, and none of them are the SDK proof files. All 15 of those files
pass when run alone. CI runs the full suite.

## Risks

- The bot step now runs one more repository script. If that script
refuses (for example, an unreviewed committed lock), the refresh PR is
not pushed. Before this change, the bot merged a refresh that made
master red.
- `lockSha256` now comes from the reviewed policy, not from the old
registry value. The proof check is unchanged. It still needs the lock
bytes to hash to `lockSha256`.
- If #20618 merges first, this branch rebases cleanly, because the
registry change is identical.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit) and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

2026-11-11 06:57:51 -08:00

DottaandClaude Opus 5.5

8e332ccb5e

fix(evals): name stale SDK pins and fix four chat-eval harness defects (#20579 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat feature eval catalog (`tests/runner-e2e`) runs real chat
journeys against a real server and grades the saved state, the run's
tool events and the rendered reply
> - The full-cohort run on master `f4337bb5fb` had 12 failures that come
from the harness, not from the product or the model
> - Six plugin and secrets cells stopped in 7–9 s, before the server
started, because the SDK public source pins were stale
> - Four more cells failed on harness timing, a first-visit prompt, or a
read receipt that ignored a dedicated tool
> - This pull request fixes those four harness defects and adds a cheap
guard that names stale SDK pins
> - The benefit is that these cells now report on the product and the
model again

## Linked Issues or Issue Description

Refs #19614, #20550, #20453, #20417, #20483

**What happened**

On master `f4337bb5fb`, these chat eval cells failed for harness
reasons:

- `plugins.configure-plugin`, `plugins.disable-plugin`,
`plugins.install-plugin`, `plugins.use-plugin-tool`,
`routines.plugin-managed-routine` and `secrets.least-scope-delivery`
stopped before the server started. The source SDK privacy scan refused,
because `tests/runner-e2e/source-sdk-public-inputs.json` did not match
`packages/shared/src`. #20550 and #20362 already resynced the pins on
master. The pins are current at `ce1ee9e89f`.
- `agent-chat.track-delegated-work` and
`navigation-preferences.edit-profile`: the reply was correct and had the
right link. The case read the rendered links immediately after the runs
settled, before the reply rendered, so it recorded no links.
- `inbox-search.archive-done` and `inbox-search.restore-archive`: setup
completes five tasks, so the GitHub star prompt from #20453 opened over
the chat and blocked the composer.
- `status-cards.explain-card`: the follow-up run read the deleted source
with `get_task` and received a 404. The reply said so. `apiReadStatuses`
counted only `call_api` reads, so the grader recorded no unavailable
read.

Why the existing pin check did not stop the drift: `ci / Runner support
suite` runs the source SDK tests and is part of the required `ci /
verify`. On #19614 it passed at 03:43Z on a merge ref from before #20417
(06:12Z) and #20483 (08:14Z). #19614 merged at 14:02Z without a new CI
run on current master. After that, the suite failed on every PR until
#20550. So the check exists and fires; the gap was a stale green result,
and the failure message was opaque (`SDK public source proof
unavailable`).

**Expected behavior**

- A change to a pinned SDK input without a pin refresh fails with the
changed files and the command that fixes it.
- Cases read rendered links only after the saved reply renders.
- The eval browser is a returning user's browser that has already seen
the star prompt. The product does not change.
- A dedicated task read tool (`get_task`, `list_task_documents`,
`read_task_document`, also when namespaced) counts as a read of the
record it names, with the status its receipt reports.

**Steps to reproduce**

1. Run `chat-features.runner-opencode.local.<case>` for one of the cases
above on master `f4337bb5fb`.
2. Read `result.json` and `snapshots/chat-catalog-observation.json` in
the evidence bundle.
3. See the failures described above.

## What Changed

- `tests/runner-e2e/source-sdk-public-inputs.mjs`: add
`staleSdkPublicInputs(root)`. It compares the registry with `git ls-tree
HEAD` for every pinned input (the roots and build inputs that the pins
file lists), and returns each changed, unpinned or removed file.
- `tests/runner-e2e/source-sdk-public-inputs.test.mjs`: a test fails
when the committed pins do not match the committed tree. The message
names the files and gives the `--write` command. The Runner support
suite runs it for every `packages/**` change. A synthetic test covers
changed, unpinned and removed inputs.
- `tests/runner-e2e/chat-catalog-domains/runtime-handoffs.ts` and
`identity-profile.ts`: call the existing `waitForRenderedReply` before
they read rendered links.
- `tests/runner-e2e/chat-returning-browser.ts` (new) and
`chat-catalog-flow.ts`: inside the flow's failure boundary, before
setup, mark `paperclip.github-star-prompt.seen` in the eval browser. An
init script covers later documents of the app origin, and the open page
is marked directly. A test checks that the key equals
`STAR_PROMPT_SEEN_KEY` in `ui/src/components/GitHubStarPrompt.tsx`.
- `tests/runner-e2e/chat-catalog-domains/api-read-receipts.ts`: bind
dedicated task read tools by the digest of the `call_api` input that the
server builds for them (`taskBridgeApiRequest`). The result must carry
the same `apiOperationId`. Other tools, other records, mismatched
receipts and unpaired results do not count.
- Bump `oracleVersion` in code and descriptor for every case that uses
`api-read-receipts.ts` (shared grader change) and for
`navigation-preferences.edit-profile`. Add
`internal-comment-reference.ts` to the source files of the two cases
that now import it.

## Verification

- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts`
(whole config): 11301 passed; 1 local-only failure in
`controller-replacement-smoke.test.ts` (server spawn, no changed
imports).
- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
--testTimeout=180000 tests/runner-e2e/chat-catalog-domains
tests/runner-e2e/chat-catalog-contract.test.ts
tests/runner-e2e/chat-catalog-consistency.test.ts
tests/runner-e2e/chat-returning-browser.test.ts`
- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
source-sdk`
- `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse
HEAD) --check` reports `changed: false`.
- Red checks: with the `f4337bb5fb` registry, the new pin test fails and
names 15 inputs. Without the new waits, the two rendered-reply tests
fail. A regrade of the retained `explain-card` run events now gives the
follow-up run `unavailable: true, live: false` and the first run `live:
true`.
- Paid runs of the affected cells
(`chat-features.runner-opencode.local.*`):
- Run 42852698879 (head `d54c04d5f0`): `track-delegated-work`,
`restore-archive`, `explain-card`, `configure-plugin` and
`plugin-managed-routine` pass. I read each reply and saved state.
`track-delegated-work` now records the rendered `/COM/issues/COM-2`
link. `explain-card` records `laterUnavailable: true, laterLive: false`
from the follow-up `get_task` 404.
- `plugins.install-plugin` now starts its server (the pin refusal is
gone). It fails a later grading check, `truthful-installation-handoff`:
the reply sends the person to the plugin settings page but does not name
the package. This is outside this PR.
- `inbox-search.archive-done`: the star prompt no longer blocks the
composer, and the query is sent. The agent then archives the 501 Done
tasks one at a time (about 2 s each, 57 in the 240 s window), because
only the per-task archive route exists. This is outside this PR.
- `navigation-preferences.edit-profile` did not start in that run (the
AWS credentials step timed out). Run 42857020103 (head `2f0b80aa84`):
pass. The agent write gets 401, the reply links
`/COM/company/settings/instance/profile`, and the person follows the
rendered link and saves the name.

## Risks

- Low risk. Test harness only; no product code changes.
- The receipt change counts more reads as record reads. It only adds
dedicated reads bound to the exact record id and operation, so a grader
that needs a read can pass on a genuine `get_task` read. The affected
cases have new oracle versions.
- The star prompt is not tested by any chat catalog case. Product e2e
tests for the prompt do not use this flow.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with
tool use (shell, file editing, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

2026-11-10 11:01:44 -06:00

f4337bb5fb

test(evals): land the chat catalog eval harness on master (#19614 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-feature evals measure whether agents complete real chat
requests through the product
> - The chat catalog eval harness and its repairs live on the
`chat-eval/master-candidate` branch, not on master
> - Today's work made 139 of 242 chat evals pass in verified single-case
live runs. Each product fix landed on master as its own pull request
> - The eval-side changes (setups, graders, fault seams, evidence
capture) still exist only on the candidate
> - This pull request lands the harness on master. Master stays the
source of truth for product behavior
> - The benefit is one tree: evals run against master commits, and
product fixes reach the evals as soon as they merge

## Linked Issues or Issue Description

Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs
#20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409

**What existing behavior does this improve?**
The chat catalog eval harness and the product it evaluates live on a
long-running branch outside master. Eval fixes do not reach master, so
master-based eval runs use old setups and graders.

**Subsystem affected**
`tests/runner-e2e` chat catalog harness, plus a small set of product
changes that exist only on the candidate.

**Proposed change**
Land the harness on master. Every product fix from this eval work
already merged to master as its own pull request. Conflicts keep
master's product behavior.

## What Changed

- Most of the diff (808 of 851 files) is under `tests/runner-e2e/`:
chat-catalog domains, scenarios, descriptors, the TestOps manifest,
private evidence tooling, and their tests.
- Graders now check saved state, setup literals and rendered or clicked
links. Many keyword and regex checks on free-form replies are removed.
Each changed grader has a bumped `oracleVersion` and calibration tests
(a valid reply, a reworded valid reply, and broken variants that must
fail).
- The shared link helper `hasLink` no longer treats a list link (for
example `/COM/issues?view=mine`) as a link to every task without a URL
key.
- Setup fixes build real state through supported product routes: routine
history, agent grants through the board grant route, budget holds with a
real cap, and a storage-layer archive fault for
`inbox-search.archive-partial-failure`.
- Browser boundary journeys (sign-in, chat-disabled, onboarding) now
write the standard pass evidence files.
- Retargeted cases, approved by the maintainer:
`agent-chat.chat-disabled` (Conference Room gate),
`agents.restore-instructions` (no saved instruction history),
`access.manage-membership` (member permission edits stay board-only).
- The private evidence trusted files (`manual-private-handoff.ts`,
`manual-private-retention.ts`, `manual-private-report-assets.ts`) are
byte-identical to master.
- Product files outside `tests/` (43 files, +1,328 −49) carry
candidate-only changes that master does not yet have. Most are tests.
The main code changes are in `request-human-input.ts`,
`heartbeat-run-summary.ts`, `native-completion-feedback.ts`,
`native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and
`IssueChatLongThreadPerf.tsx`.

## Verification

- `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts
chat-catalog-consistency.test.ts unit-launcher.test.mjs
--testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests
pass.
- `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
no change.
- `tsc` for `server` and `tests/runner-e2e`: clean.
- Live single-case verification: 139 of 242 cases passed on paid runs,
and a person read the saved evidence for each. These are single-case
runs, not one full-cohort run.

## Risks

- Large diff. Most of it is test-only, but the 43 non-test files change
product code paths. Review those files first.
- Some graders are now less strict on reply wording. A person must read
replies for meaning on those cases.
- 42 cases need disposable outside accounts (Notion, Slack, GitHub,
Daytona and others) and stay blocked. `access.invite-teammate` stays
failing until local agent isolation lands, because the runner's durable
state keeps a live invite token.
- The paid workflow compares the trusted private-evidence files with
master. They are unchanged, so this risk is low.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution, in Claude Code. Subagents used the same model.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-11-10 08:02:11 -06:00