mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-14 00:26:28 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat feature eval catalog (`tests/runner-e2e`) runs real chat journeys against a real server and grades the saved state, the run's tool events and the rendered reply > - The full-cohort run on master `f4337bb5fb` had 12 failures that come from the harness, not from the product or the model > - Six plugin and secrets cells stopped in 7–9 s, before the server started, because the SDK public source pins were stale > - Four more cells failed on harness timing, a first-visit prompt, or a read receipt that ignored a dedicated tool > - This pull request fixes those four harness defects and adds a cheap guard that names stale SDK pins > - The benefit is that these cells now report on the product and the model again ## Linked Issues or Issue Description Refs #19614, #20550, #20453, #20417, #20483 **What happened** On master `f4337bb5fb`, these chat eval cells failed for harness reasons: - `plugins.configure-plugin`, `plugins.disable-plugin`, `plugins.install-plugin`, `plugins.use-plugin-tool`, `routines.plugin-managed-routine` and `secrets.least-scope-delivery` stopped before the server started. The source SDK privacy scan refused, because `tests/runner-e2e/source-sdk-public-inputs.json` did not match `packages/shared/src`. #20550 and #20362 already resynced the pins on master. The pins are current at `ce1ee9e89f`. - `agent-chat.track-delegated-work` and `navigation-preferences.edit-profile`: the reply was correct and had the right link. The case read the rendered links immediately after the runs settled, before the reply rendered, so it recorded no links. - `inbox-search.archive-done` and `inbox-search.restore-archive`: setup completes five tasks, so the GitHub star prompt from #20453 opened over the chat and blocked the composer. - `status-cards.explain-card`: the follow-up run read the deleted source with `get_task` and received a 404. The reply said so. `apiReadStatuses` counted only `call_api` reads, so the grader recorded no unavailable read. Why the existing pin check did not stop the drift: `ci / Runner support suite` runs the source SDK tests and is part of the required `ci / verify`. On #19614 it passed at 03:43Z on a merge ref from before #20417 (06:12Z) and #20483 (08:14Z). #19614 merged at 14:02Z without a new CI run on current master. After that, the suite failed on every PR until #20550. So the check exists and fires; the gap was a stale green result, and the failure message was opaque (`SDK public source proof unavailable`). **Expected behavior** - A change to a pinned SDK input without a pin refresh fails with the changed files and the command that fixes it. - Cases read rendered links only after the saved reply renders. - The eval browser is a returning user's browser that has already seen the star prompt. The product does not change. - A dedicated task read tool (`get_task`, `list_task_documents`, `read_task_document`, also when namespaced) counts as a read of the record it names, with the status its receipt reports. **Steps to reproduce** 1. Run `chat-features.runner-opencode.local.<case>` for one of the cases above on master `f4337bb5fb`. 2. Read `result.json` and `snapshots/chat-catalog-observation.json` in the evidence bundle. 3. See the failures described above. ## What Changed - `tests/runner-e2e/source-sdk-public-inputs.mjs`: add `staleSdkPublicInputs(root)`. It compares the registry with `git ls-tree HEAD` for every pinned input (the roots and build inputs that the pins file lists), and returns each changed, unpinned or removed file. - `tests/runner-e2e/source-sdk-public-inputs.test.mjs`: a test fails when the committed pins do not match the committed tree. The message names the files and gives the `--write` command. The Runner support suite runs it for every `packages/**` change. A synthetic test covers changed, unpinned and removed inputs. - `tests/runner-e2e/chat-catalog-domains/runtime-handoffs.ts` and `identity-profile.ts`: call the existing `waitForRenderedReply` before they read rendered links. - `tests/runner-e2e/chat-returning-browser.ts` (new) and `chat-catalog-flow.ts`: inside the flow's failure boundary, before setup, mark `paperclip.github-star-prompt.seen` in the eval browser. An init script covers later documents of the app origin, and the open page is marked directly. A test checks that the key equals `STAR_PROMPT_SEEN_KEY` in `ui/src/components/GitHubStarPrompt.tsx`. - `tests/runner-e2e/chat-catalog-domains/api-read-receipts.ts`: bind dedicated task read tools by the digest of the `call_api` input that the server builds for them (`taskBridgeApiRequest`). The result must carry the same `apiOperationId`. Other tools, other records, mismatched receipts and unpaired results do not count. - Bump `oracleVersion` in code and descriptor for every case that uses `api-read-receipts.ts` (shared grader change) and for `navigation-preferences.edit-profile`. Add `internal-comment-reference.ts` to the source files of the two cases that now import it. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts` (whole config): 11301 passed; 1 local-only failure in `controller-replacement-smoke.test.ts` (server spawn, no changed imports). - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts --testTimeout=180000 tests/runner-e2e/chat-catalog-domains tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts tests/runner-e2e/chat-returning-browser.test.ts` - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts source-sdk` - `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse HEAD) --check` reports `changed: false`. - Red checks: with the `f4337bb5fb` registry, the new pin test fails and names 15 inputs. Without the new waits, the two rendered-reply tests fail. A regrade of the retained `explain-card` run events now gives the follow-up run `unavailable: true, live: false` and the first run `live: true`. - Paid runs of the affected cells (`chat-features.runner-opencode.local.*`): - Run 42852698879 (head `d54c04d5f0`): `track-delegated-work`, `restore-archive`, `explain-card`, `configure-plugin` and `plugin-managed-routine` pass. I read each reply and saved state. `track-delegated-work` now records the rendered `/COM/issues/COM-2` link. `explain-card` records `laterUnavailable: true, laterLive: false` from the follow-up `get_task` 404. - `plugins.install-plugin` now starts its server (the pin refusal is gone). It fails a later grading check, `truthful-installation-handoff`: the reply sends the person to the plugin settings page but does not name the package. This is outside this PR. - `inbox-search.archive-done`: the star prompt no longer blocks the composer, and the query is sent. The agent then archives the 501 Done tasks one at a time (about 2 s each, 57 in the 240 s window), because only the per-task archive route exists. This is outside this PR. - `navigation-preferences.edit-profile` did not start in that run (the AWS credentials step timed out). Run 42857020103 (head `2f0b80aa84`): pass. The agent write gets 401, the reply links `/COM/company/settings/instance/profile`, and the person follows the rendered link and saves the name. ## Risks - Low risk. Test harness only; no product code changes. - The receipt change counts more reads as record reads. It only adds dedicated reads bound to the exact record id and operation, so a grader that needs a read can pass on a genuine `get_task` read. The affected cases have new oracle versions. - The star prompt is not tested by any chat catalog case. Product e2e tests for the prompt do not use this flow. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file editing, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
94 lines
7.4 KiB
JavaScript
94 lines
7.4 KiB
JavaScript
import { strict as assert } from "node:assert";
|
|
import { execFileSync } from "node:child_process";
|
|
import { createHash } from "node:crypto";
|
|
import { mkdir, mkdtemp, readFile, realpath, rm, symlink, writeFile } from "node:fs/promises";
|
|
import os from "node:os";
|
|
import path from "node:path";
|
|
import { afterEach, test } from "vitest";
|
|
import { fileURLToPath } from "node:url";
|
|
import { staleSdkPublicInputs, synchronizeSdkPublicInputs } from "./source-sdk-public-inputs.mjs";
|
|
|
|
const registry = "tests/runner-e2e/source-sdk-public-inputs.json";
|
|
const roots = [];
|
|
const digest = bytes => createHash("sha256").update(bytes).digest("hex");
|
|
const blob = bytes => createHash("sha1").update(`blob ${Buffer.byteLength(bytes)}\0`).update(bytes).digest("hex");
|
|
const git = (root, ...args) => execFileSync("git", ["-c", "core.hooksPath=/dev/null", "-c", "commit.gpgsign=false", ...args], { cwd: root, encoding: "utf8", stdio: ["ignore", "pipe", "pipe"] });
|
|
async function put(root, file, bytes) { await mkdir(path.dirname(path.join(root, file)), { recursive: true }); await writeFile(path.join(root, file), bytes); }
|
|
async function fixture() {
|
|
const root = await realpath(await mkdtemp(path.join(os.tmpdir(), "sdk-input-sync-"))); roots.push(root);
|
|
const files = ["package.json", "packages/plugins/sdk/package.json", "packages/plugins/sdk/tsconfig.json",
|
|
"packages/shared/package.json", "packages/shared/tsconfig.json", "pnpm-lock.yaml", "scripts/ensure-plugin-build-deps.mjs",
|
|
"tsconfig.base.json", "tsconfig.json", "packages/shared/src/schema.ts", "packages/plugins/sdk/src/types.ts"];
|
|
for (const file of files) await put(root, file, `synthetic original ${file}\n`);
|
|
const baseline = { inputs: Object.fromEntries(files.map(file => [file, blob(`synthetic stale ${file}\n`)])),
|
|
lockSha256: digest("synthetic authenticated restored lock"), compilerPackageSha256: "a".repeat(64),
|
|
parserPackageSha256: "b".repeat(64), parserEntrySha256: "c".repeat(64),
|
|
zodPackageSha256: "d".repeat(64), zodEntrySha256: "e".repeat(64), zodTypeFiles: { "index.d.cts": "f".repeat(64) } };
|
|
await put(root, registry, `${JSON.stringify(baseline, null, 2)}\n`);
|
|
git(root, "init", "-q"); git(root, "add", ".");
|
|
git(root, "-c", "user.name=Synthetic unit", "-c", "user.email=synthetic@example.invalid", "commit", "-qm", "synthetic source");
|
|
return { root, baseline, revision: git(root, "rev-parse", "HEAD").trim() };
|
|
}
|
|
afterEach(async () => { await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true }))); });
|
|
|
|
test("synchronizes the committed source graph, preserves every runtime pin and is idempotent", async () => {
|
|
const { root, baseline } = await fixture();
|
|
await put(root, "packages/shared/src/added.ts", "synthetic added source\n");
|
|
git(root, "add", "."); git(root, "-c", "user.name=Synthetic unit", "-c", "user.email=synthetic@example.invalid", "commit", "-qm", "additional source");
|
|
const revision = git(root, "rev-parse", "HEAD").trim();
|
|
assert.equal((await synchronizeSdkPublicInputs(root, revision)).changed, true);
|
|
assert.deepEqual(JSON.parse(await readFile(path.join(root, registry), "utf8")), baseline, "check does not write");
|
|
const changed = await synchronizeSdkPublicInputs(root, revision, { write: true });
|
|
const updated = JSON.parse(await readFile(path.join(root, registry), "utf8"));
|
|
assert.equal(updated.inputs["packages/shared/src/added.ts"], blob("synthetic added source\n"));
|
|
for (const [file, pin] of Object.entries(updated.inputs)) assert.equal(pin, git(root, "rev-parse", `${revision}:${file}`).trim());
|
|
const { inputs: oldInputs, ...runtime } = baseline, { inputs: newInputs, ...afterRuntime } = updated;
|
|
assert.deepEqual(afterRuntime, runtime); assert.equal(changed.runtimePinsChanged, false);
|
|
assert.equal(changed.registrySha256, digest(await readFile(path.join(root, registry))));
|
|
assert.equal((await synchronizeSdkPublicInputs(root, revision)).changed, false);
|
|
});
|
|
for (const mutation of ["dirty", "missing", "extra", "symlink", "registry", "revision"]) {
|
|
test(`refuses ${mutation} source custody without writing the registry`, async () => {
|
|
const { root, revision } = await fixture();
|
|
const target = path.join(root, "packages/shared/src/schema.ts");
|
|
if (mutation === "dirty") await writeFile(target, "uncommitted synthetic source");
|
|
if (mutation === "missing") await rm(target);
|
|
if (mutation === "extra") await put(root, "packages/shared/src/uncommitted.ts", "uncommitted");
|
|
if (mutation === "symlink") { await rm(target); await symlink("../package.json", target); }
|
|
if (mutation === "registry") { const current = JSON.parse(await readFile(path.join(root, registry), "utf8")); current.parserEntrySha256 = "0".repeat(64); await writeFile(path.join(root, registry), JSON.stringify(current)); }
|
|
const before = await readFile(path.join(root, registry));
|
|
await assert.rejects(synchronizeSdkPublicInputs(root, mutation === "revision" ? "0".repeat(40) : revision, { write: true }));
|
|
assert.deepEqual(await readFile(path.join(root, registry)), before);
|
|
});
|
|
}
|
|
test("admits only the separately pinned restored lock overlay without changing its identities", async () => {
|
|
const { root, baseline, revision } = await fixture();
|
|
await writeFile(path.join(root, "pnpm-lock.yaml"), "synthetic authenticated restored lock");
|
|
await synchronizeSdkPublicInputs(root, revision, { write: true });
|
|
const updated = JSON.parse(await readFile(path.join(root, registry), "utf8"));
|
|
assert.equal(updated.lockSha256, baseline.lockSha256);
|
|
assert.equal(updated.inputs["pnpm-lock.yaml"], git(root, "rev-parse", `${revision}:pnpm-lock.yaml`).trim());
|
|
await writeFile(path.join(root, "pnpm-lock.yaml"), "unknown lock overlay");
|
|
await assert.rejects(synchronizeSdkPublicInputs(root, revision));
|
|
});
|
|
|
|
// The pins only prove the SDK public types while they match the committed
|
|
// source. A PR that edits packages/shared/src (or another pinned input) without
|
|
// refreshing them breaks every SDK-dependent Runner cell before its server
|
|
// starts, so fail here, cheaply, with the inputs and the fix command.
|
|
test("the committed SDK public source pins match this checkout's committed tree", async () => {
|
|
const root = fileURLToPath(new URL("../..", import.meta.url));
|
|
const stale = await staleSdkPublicInputs(root);
|
|
assert.deepEqual(stale, [], `${registry} is stale for ${stale.length} pinned input(s): ${stale.slice(0, 10).map(row => `${row.file} (${row.change})`).join(", ")}${stale.length > 10 ? ", ..." : ""}. `
|
|
+ "Commit your change, run `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse HEAD) --write`, and commit the updated registry.");
|
|
});
|
|
test("the drift guard names changed, unpinned and removed inputs and clears after a resync", async () => {
|
|
const { root, revision } = await fixture();
|
|
assert.deepEqual((await staleSdkPublicInputs(root)).map(row => row.change), Array(11).fill("changed"));
|
|
await synchronizeSdkPublicInputs(root, revision, { write: true });
|
|
assert.deepEqual(await staleSdkPublicInputs(root), []);
|
|
await put(root, "packages/shared/src/added.ts", "synthetic added source\n"); await rm(path.join(root, "packages/plugins/sdk/src/types.ts"));
|
|
git(root, "add", "-A"); git(root, "-c", "user.name=Synthetic unit", "-c", "user.email=synthetic@example.invalid", "commit", "-qm", "drift");
|
|
assert.deepEqual(await staleSdkPublicInputs(root), [{ file: "packages/plugins/sdk/src/types.ts", change: "removed" }, { file: "packages/shared/src/added.ts", change: "unpinned" }]);
|
|
});
|