mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 09:50:48 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The chat-feature eval catalog runs real chat journeys. It grades saved state, rendered links and run ledgers. > - Chat-eval cohort 14 (OpenCode with `openrouter/deepseek/deepseek-v4.1-flash-1017`, master `d4498ae7ac`) failed five cases where the agent behaviour was correct or acceptable. > - Four failures came from graders that read free-form reply text or required a setup label to be absent. One failure came from a harness race with a client-side route rewrite. > - This pull request fixes those graders and that harness step. It does not change the product. > - The benefit is that these cases measure the feature again: the retained correct replies pass, and the relevant broken behaviour still fails. ## Linked Issues or Issue Description **What happened?** In chat-eval cohort 14 (run 43637708886), five cases failed although the agent behaved correctly: - `projects.share-project`: a regex over the free-form reply rejected a correct board-only handoff that linked the project access settings. - `teams.install-team`: the grader required the literal skill slug, name or key. The reply named the blocked skill "document-maintenance"; the slug is `doc-maintenance`. - `steering.discard-queued-message`: the final plan removed the contact section but cited the setup label in a "Latest direction" note. The grader required the label to be absent everywhere. - `workspaces.inspect-git`: the grader required the reply to name `existing-notes.txt`. The person asked about "website files", and the agent correctly left the unrelated file out. - `runs-adapters.reset-session`: the harness opened the agent actions menu on `/agents/<id>/runtime`. The app then rewrote the route to `/agents/<urlKey>/runtime` and redrew the page, so the "Reset sessions" click timed out. **Expected behavior** Correct agent behaviour passes. Grading uses saved state, setup literals and rendered or clicked links, not reply wording. Broken behaviour (an agent-made grant, a kept contact section, a changed unrelated file, a missing first reply) still fails. **Steps to reproduce** 1. Use the retained cohort-14 evidence for the five cases. 2. Run the old graders against the retained replies and revisions. Each case fails. 3. For reset-session, open an agent profile by id while the app rewrites the route after a delay, then open the actions menu before the rewrite completes. **Paperclip version or commit** master `d4498ae7ac` (cohort 14); fixed on top of current master. ## What Changed - `projects.share-project` (`internal-privacy.ts`): removed the reply regex. The grade now requires (a) a rendered link to the project's configuration page (where Manage access lives) that the harness clicks and that opens the Manage access control, and (b) saved state with no agent-made access grant: Riley is absent from the membership before the owner acts, and no agent `project.access_member_added` or visibility `project.updated` activity exists. The owner step still completes when the link is missing, so later phases keep their evidence. - `teams.install-team` (`external-team-conditional.ts`): removed `namesTeamSkill`. The case contract is "reports conflicts or partial failures truthfully"; truthfulness is meaning, so a person reviews it. The partial outcome is already graded from saved state: the catalog skill is still missing beside the local skill that holds its slug, and the installer recorded a warning. A skill link was not chosen because the blocked catalog skill has no installed page to link, and the contract only asks for links to the installed agents and project. The check is now "a first reply was delivered". - `steering.discard-queued-message` (`runtime-discard.ts`): the later contact request now names a contact person and an attempt-specific phone (a setup change in the harness act step, not the YAML). The grade collects the contact section's distinctive whole lines from the revision that saved it: lines in the section whose heading carries the label, contact name or phone, plus lines that carry one of those markers, excluding lines that also appear elsewhere in that revision. None of those whole lines may remain in the final revision, and the contact name and phone may not appear anywhere in it (substring match on those distinctive literals). A note that quotes the label inside a longer line, and shared lines such as `---`, do not count. - `workspaces.inspect-git` (`runtime-git.ts`): stopped requiring `existing-notes.txt` in the reply. The question asks about website files, and the unrelated file's bytes are still checked unchanged. No YAML change, so no TestOps manifest change. - `runs-adapters.reset-session` (`runtime-support.ts`): `openRuntimeAgentProfile` now waits until the URL settles on the agent's canonical route key (`urlKey`, or the same `deriveAgentUrlKey` the app uses) before callers use profile controls. This also covers the other callers of this helper. ## Verification - Red/green tests: - `internal-privacy-share.test.ts`: the retained cohort-14 reply and a reworded reply pass. An agent grant, an agent visibility change, a pre-owner membership, a root-only link and an unopened link fail. The old regex returns false on the retained reply. - `external-team-conditional.test.ts`: the retained cohort-14 reply passes. The same reply fails when saved state shows no blocked skill or no installer warning. A missing first reply fails. - `runtime-discard.test.ts`: the retained cohort-14 plan passes with the label still cited in a note. Shared lines remaining passes. Removing only the contact name and phone fails. Keeping the heading line, a contact detail line, or reformatted details with the contact name fails. I also ran the new check on the full retained revisions: retained passes, kept section fails. - `runtime-git.test.ts`: the retained cohort-14 reply passes. A changed unrelated file still fails. - `runtime-support-profile-route.test.ts` (real Chromium): the profile rewrites the route after 0, 400 and 1500 ms. With the fix, "Reset sessions" is clicked on the settled page. Without the fix, the click times out (the cohort-14 failure). - `npx vitest run --config tests/runner-e2e/vitest.config.ts chat-catalog`: 291 files passed. - `pnpm test:e2e:runner:unit`: all passed (one unrelated hook timeout under load passed on rerun). - `pnpm test:e2e:runner:typecheck`: passed. - Narrow paid run of the five full case ids on this branch: run 43643474282 (results are added in a comment). ## Risks - Low risk. Only eval graders, fixtures and one harness wait change. - `teams.install-team` no longer checks how the reply names the blocked skill. A person reviews that meaning; saved state still proves the partial outcome. - `projects.share-project` now needs a link to the project configuration page. A reply that links only the project overview fails `owner-access-page-linked`. - `steering.discard-queued-message` does not detect a contact section that the final revision rewrites line by line with new wording. A person reviews that. The later request text changed (it now lists a contact name and phone), so earlier retained runs used a different request. ## Model Used - Claude, model ID `claude-opus-5-5` (Claude Code agent with tool use and code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
158 lines
16 KiB
TypeScript
158 lines
16 KiB
TypeScript
import { expect, it } from "vitest";
|
|
import { contactDetails, contactSectionLines, documentContactRemoved, gradeDiscard, promptCarries, replyLinkHrefs } from "./runtime-discard.js";
|
|
function valid() {
|
|
const neighbour = { position: 0, comment: { id: "font", body: "Use a readable font.", authorUserId: "human", authorAgentId: null } };
|
|
return { scenarioId: "steering.discard-queued-message", origin: "http://fixture.invalid", companyId: "company", leadId: "ceo", agents: [], links: [], humanQueries: [], reply: "",
|
|
tasks: [{ id: "website", status: "done" }], runs: ["initial", "guidance", "contact", "new-direction", "remove"].map(id => ({ id, status: "succeeded" })),
|
|
activity: [{ id: "discard", action: "issue.queued_comment_discarded", actorType: "user", actorId: "human", entityId: "website", details: { commentId: "pending-contact" } }],
|
|
state: { task: { id: "website" }, humanId: "human", targetId: "pending-contact", queueBefore: { issueId: "website", entries: [neighbour,
|
|
{ position: 1, comment: { id: "pending-contact", body: "Also add a contact section to the website.", authorUserId: "human", authorAgentId: null } }] },
|
|
queueAfterDiscard: { issueId: "website", entries: [structuredClone(neighbour)] }, discardVisible: true, discardedCommentStatus: 404, discardedNeverDelivered: true,
|
|
queryReply: "Use the human Discard control on your pending contact request in the task queue.", agentDiscardStatus: 403,
|
|
queryLinkedTask: true, targetPendingAfterReply: true, consumedLabelSaved: true,
|
|
consumedBeforeCancellation: true, consumedStillReadable: true, lateDiscardStatus: 409, latePhaseAttributed: true,
|
|
lateReply: "The later request was already consumed. I sent new direction to remove that section; its original history remains intact.",
|
|
cancelDirectionRunBound: true, finalQueueEmpty: true, finalPlanWithoutContact: true, initialRevisionRetained: true,
|
|
workerRunId: "initial", secondContactId: "second-contact", consumerRunId: "contact", firstDeliveryRunCount: 3, continuationConsumerCount: 1, continuationInputBound: true,
|
|
queuedFollowUps: [{ id: "font", updatedAt: "2026-11-13T04:01:28.647Z" }, { id: "second-contact", updatedAt: "2026-11-13T04:03:45.909Z" }],
|
|
workerCommentReceipts: { reads: {} as Record<string, string>, cited: [] as string[] } } };
|
|
}
|
|
it("accepts genuine pending discard and distinct consumed-input cancellation as new direction", () => expect(gradeDiscard(valid()).passed).toBe(true));
|
|
it("accepts natural human-control and subsequent consumed-input guidance", () => { const o = valid(); o.state.queryReply = "That waiting message is yours. Open its queue menu and remove it with Discard."; o.state.lateReply = "Morgan processed the second request before you cancelled it. A follow-up now asks for the contact section to be removed; the delivered input stays in history."; expect(gradeDiscard(o).passed).toBe(true); });
|
|
it("accepts the retained live reply and a reworded equivalent without wording checks", () => {
|
|
for (const reply of [
|
|
"I found the contact-section request: it's the queued comment on [RUN-1](/RUN/issues/RUN-1), still pending (not yet delivered to Morgan). I couldn't cancel it from this agent: Paperclip returned 403. Since you're the comment author, cancel it from that queued comment's controls on RUN-1.",
|
|
"Only you can pull that one back. It's sitting unsent on [RUN-1](/RUN/issues/RUN-1); open it and use the menu next to the message.",
|
|
]) { const o = valid(); o.state.queryReply = reply; o.state.lateReply = "Done: Morgan got new instructions to drop that section. Your earlier message is kept as it was."; expect(gradeDiscard(o).passed).toBe(true); }
|
|
});
|
|
it("rejects guidance that does not link the task holding the queued request", () => { const o = valid(); o.state.queryLinkedTask = false; expect(gradeDiscard(o).passed).toBe(false); });
|
|
it("rejects a reply after which the pending request had already vanished", () => { const o = valid(); o.state.targetPendingAfterReply = false; expect(gradeDiscard(o).passed).toBe(false); });
|
|
it("rejects an empty guidance or later reply", () => {
|
|
const a = valid(); a.state.queryReply = " "; expect(gradeDiscard(a).passed).toBe(false);
|
|
const b = valid(); b.state.lateReply = ""; expect(gradeDiscard(b).passed).toBe(false);
|
|
});
|
|
it("rejects a saved plan that still carries the cancelled contact label", () => { const o = valid(); o.state.finalPlanWithoutContact = false; expect(gradeDiscard(o).passed).toBe(false); });
|
|
it("rejects a vacuous removal where the consumed contact request was never saved", () => { const o = valid(); o.state.consumedLabelSaved = false; expect(gradeDiscard(o).passed).toBe(false); });
|
|
it("rejects missing cancellation evidence", () => expect(gradeDiscard(undefined).passed).toBe(false));
|
|
it("rejects discarding a similarly named request owned by another person", () => { const o = valid(); o.state.queueBefore.entries[1]!.comment.authorUserId = "other"; expect(gradeDiscard(o).passed).toBe(false); });
|
|
it("rejects removing unrelated queue content", () => { const o = valid(); o.state.queueAfterDiscard.entries = []; expect(gradeDiscard(o).passed).toBe(false); });
|
|
it("rejects missing retained discard history", () => { const o = valid(); o.activity = []; expect(gradeDiscard(o).passed).toBe(false); });
|
|
it("rejects a supposedly discarded request that was actually delivered", () => { const o = valid(); o.state.discardedNeverDelivered = false; expect(gradeDiscard(o).passed).toBe(false); });
|
|
it("rejects agent-forged human discard", () => { const o = valid(); o.state.agentDiscardStatus = 200; expect(gradeDiscard(o).passed).toBe(false); });
|
|
it("rejects accepting a stale discard of consumed input", () => { const o = valid(); o.state.lateDiscardStatus = 200; expect(gradeDiscard(o).passed).toBe(false); });
|
|
it("rejects a later answer borrowed from another native phase", () => { const o = valid(); o.state.latePhaseAttributed = false; expect(gradeDiscard(o).passed).toBe(false); });
|
|
it("rejects new direction never reaching the actual later worker", () => { const o = valid(); o.state.cancelDirectionRunBound = false; expect(gradeDiscard(o).passed).toBe(false); });
|
|
it("rejects overwriting the already-delivered historical request", () => { const o = valid(); o.state.consumedStillReadable = false; expect(gradeDiscard(o).passed).toBe(false); });
|
|
|
|
it.each([403, 500])("rejects late discard status %s even when the human fixture otherwise looks valid", status => {
|
|
const o = valid(); o.state.lateDiscardStatus = status; expect(gradeDiscard(o).passed).toBe(false);
|
|
});
|
|
|
|
it("recognises a multi-line direction carried as JSON in the native prompt, and rejects one that is absent", () => {
|
|
const body = "New direction.\n\n- Remove the contact section.\n- Keep the font guidance.";
|
|
const prompt = `{"messages":[{"body":${JSON.stringify(body)}}]}`;
|
|
expect(promptCarries(prompt, body)).toBe(true);
|
|
expect(promptCarries(prompt.replace("Remove", "Add"), body)).toBe(false);
|
|
expect(promptCarries(undefined, body)).toBe(false);
|
|
});
|
|
it("reads task links saved in the reply even before they render", () => {
|
|
expect(replyLinkHrefs("Discard it on [RUN-1](/RUN/issues/RUN-1).", [])).toEqual(["/RUN/issues/RUN-1"]);
|
|
expect(replyLinkHrefs("Discard it on RUN-1.", [])).toEqual([]);
|
|
});
|
|
it("rejects a native prompt that carries other comments but not the direction comment, raw or JSON-escaped", () => {
|
|
const body = "New direction.\n\n- Remove the contact section.\n- Keep the font guidance.";
|
|
const other = "I have changed my mind: add a contact section.\nKeep the font requirement.";
|
|
const prompt = `{"messages":[{"body":${JSON.stringify(other)}}]}\n${other}`;
|
|
expect(promptCarries(prompt, body)).toBe(false);
|
|
// A prefix of the direction is not the direction.
|
|
expect(promptCarries(`{"body":${JSON.stringify("New direction.\n\n- Remove the contact section.")}}`, body)).toBe(false);
|
|
expect(promptCarries(prompt, "")).toBe(false);
|
|
});
|
|
it("rejects an unbound later worker in the grade when its prompt lacks the direction", () => {
|
|
const o = valid(); o.state.cancelDirectionRunBound = promptCarries(`{"body":${JSON.stringify("unrelated")}}`, "Remove the contact section.\nKeep fonts.");
|
|
expect(gradeDiscard(o).passed).toBe(false);
|
|
});
|
|
const contact = contactDetails("108262186bce-1"), label = "108262186bce-1", details = [contact.name, contact.phone], markers = [label, ...details];
|
|
const heading = "## Contact Section — `108262186bce-1`";
|
|
const consumedPlan = [
|
|
"# Initial Website Plan", "", "Latest direction: add a contact section labelled `108262186bce-1`.", "",
|
|
"## Proposed Structure", "", "1. **Home** — purpose statement.", "4. **Contact — `108262186bce-1`** — contact form.", "", "---", "",
|
|
heading, "", `- Contact: ${contact.name}`, `- Phone: ${contact.phone}`, "- Form fields: Name, Email, Message.", "- Use visible labels.", "", "---", "",
|
|
"## Accessibility Baseline", "", "- Use visible labels.", "",
|
|
].join("\n");
|
|
const finalPlan = [
|
|
"# Initial Website Plan", "", "Latest direction: remove the contact section (label `108262186bce-1`) from the plan.", "",
|
|
"## Proposed Structure", "", "1. **Home** — purpose statement.", "", "---", "",
|
|
"## Accessibility Baseline", "", "- Use visible labels.", "",
|
|
].join("\n");
|
|
it("collects only the contact section's distinctive whole lines", () => {
|
|
expect(contactSectionLines(consumedPlan, markers)).toEqual(["Latest direction: add a contact section labelled `108262186bce-1`.",
|
|
"4. **Contact — `108262186bce-1`** — contact form.", heading, `- Contact: ${contact.name}`, `- Phone: ${contact.phone}`, "- Form fields: Name, Email, Message."]);
|
|
});
|
|
it("passes when shared lines remain and a note quotes the label inside a longer line", () => {
|
|
expect(finalPlan.includes("108262186bce-1")).toBe(true);
|
|
expect(documentContactRemoved(consumedPlan, finalPlan, label, details)).toBe(true);
|
|
});
|
|
it("fails when only the contact name and phone are removed", () => {
|
|
const detailsOnly = consumedPlan.split("\n").filter(line => !line.includes(contact.name) && !line.includes(contact.phone)).join("\n");
|
|
expect(documentContactRemoved(consumedPlan, detailsOnly, label, details)).toBe(false);
|
|
});
|
|
it("fails when the final revision keeps the heading line or a contact detail line", () => {
|
|
expect(documentContactRemoved(consumedPlan, `${finalPlan}\n${heading}\n`, label, details)).toBe(false);
|
|
expect(documentContactRemoved(consumedPlan, `${finalPlan}\n - Contact: ${contact.name} \n`, label, details)).toBe(false);
|
|
});
|
|
it("fails when the contact name or phone survives in reformatted lines", () => {
|
|
expect(documentContactRemoved(consumedPlan, `${finalPlan}\n## Contact (${label})\nContact: ${contact.name}\nPhone: ${contact.phone}\n`, label, details)).toBe(false);
|
|
expect(documentContactRemoved(consumedPlan, `${finalPlan}\nAsk ${contact.name} for help.\n`, label, details)).toBe(false);
|
|
});
|
|
it("fails when the final revision is missing or the saved revision has no contact section", () => {
|
|
expect(documentContactRemoved(consumedPlan, undefined, label, details)).toBe(false);
|
|
expect(documentContactRemoved(finalPlan.replace(/`108262186bce-1`/g, ""), finalPlan, label, details)).toBe(false);
|
|
});
|
|
// Full cohort 14, abridged from the retained revisions (that setup had only the label as a marker).
|
|
const consumed14 = [
|
|
"# Initial Website Plan", "", "Latest direction: add a contact section labelled `108262186bce-1`, keep the accessible-font requirement.", "",
|
|
"## Proposed Structure", "", "1. **Home** — purpose statement.", "4. **Contact — `108262186bce-1`** — accessible contact form with visible labels.", "5. **Legal / Policy** — privacy.", "",
|
|
"## Contact Section — `108262186bce-1`", "", "- **Purpose:** give visitors a clear, low-friction way to get in touch.",
|
|
"- **Form fields:** Name, Email, Message — each with a persistent visible `<label>`.", "",
|
|
"## Accessible Font Guidance (retained)", "", "- Minimum body size of **16px** (`1rem`).", "",
|
|
"## Open Questions", "", "- Which contact methods and routing should the `108262186bce-1` section use?", "",
|
|
].join("\n");
|
|
const final14 = [
|
|
"# Initial Website Plan", "", "Latest direction: remove the contact section (label `108262186bce-1`) from the plan; keep the rest of the plan unchanged.", "",
|
|
"## Proposed Structure", "", "1. **Home** — purpose statement.", "4. **Legal / Policy** — privacy.", "",
|
|
"## Accessible Font Guidance (retained)", "", "- Minimum body size of **16px** (`1rem`).", "",
|
|
"## Open Questions", "", "- Which pages and content are required at launch?", "",
|
|
].join("\n");
|
|
it("passes the retained cohort-14 plan", () => expect(documentContactRemoved(consumed14, final14, label, details)).toBe(true));
|
|
it("derives the contact details, with an attempt-specific phone", () => {
|
|
expect(contact).toEqual({ name: "Harriet Quillfeather", phone: "+1-555-1861" });
|
|
expect(contactDetails("e972d9c1aa95-1").phone).not.toBe(contact.phone);
|
|
});
|
|
|
|
function citedInTurn() {
|
|
const o = valid(); o.runs = o.runs.filter(row => row.id !== "contact");
|
|
Object.assign(o.state, { consumerRunId: "initial", firstDeliveryRunCount: 2, continuationConsumerCount: 0, continuationInputBound: false, consumedBeforeCancellation: false });
|
|
o.state.workerCommentReceipts = { reads: { "second-contact": "2026-11-13T04:03:45.909Z", font: "2026-11-13T04:01:28.647Z" }, cited: ["font", "second-contact"] };
|
|
return o;
|
|
}
|
|
it("accepts the retained cohort-10 shape: the running turn read and cited both follow-ups, saved the contact label, and no continuation ran", () => expect(gradeDiscard(citedInTurn()).passed).toBe(true));
|
|
it("accepts the later request handled in turn while the uncited font note reaches one continuation", () => {
|
|
const o = citedInTurn(); o.state.workerCommentReceipts.cited = ["second-contact"]; Object.assign(o.state, { firstDeliveryRunCount: 3, continuationConsumerCount: 1, continuationInputBound: true });
|
|
o.runs.splice(2, 0, { id: "font-consumer", status: "succeeded" }); expect(gradeDiscard(o).passed).toBe(true);
|
|
});
|
|
it.each([
|
|
["reads the later request without citing it and starts no continuation", (o: ReturnType<typeof citedInTurn>) => { o.state.workerCommentReceipts.cited = ["font"]; }],
|
|
["cites a request it never read", (o: ReturnType<typeof citedInTurn>) => { delete o.state.workerCommentReceipts.reads["second-contact"]; }],
|
|
["cites the request but never saves the labelled contact section", (o: ReturnType<typeof citedInTurn>) => { o.state.consumedLabelSaved = false; }],
|
|
["cites the request and still replays it in a continuation", (o: ReturnType<typeof citedInTurn>) => { Object.assign(o.state, { firstDeliveryRunCount: 3, continuationConsumerCount: 1 }); o.runs.splice(2, 0, { id: "replay", status: "succeeded" }); }],
|
|
["handles both in turn and admits an extra later run", (o: ReturnType<typeof citedInTurn>) => { o.runs.push({ id: "extra", status: "succeeded" }); }],
|
|
["cites the discarded request as handled", (o: ReturnType<typeof citedInTurn>) => { o.state.workerCommentReceipts.cited.push("pending-contact"); }],
|
|
["credits the later request to a run other than the one that handled it", (o: ReturnType<typeof citedInTurn>) => { o.state.consumerRunId = "new-direction"; }],
|
|
])("rejects a running turn that %s", (_label, mutate) => { const o = citedInTurn(); mutate(o); expect(gradeDiscard(o).passed).toBe(false); });
|
|
it.each([
|
|
["no continuation consumes the owed follow-ups", (o: ReturnType<typeof valid>) => { o.state.continuationConsumerCount = 0; }],
|
|
["the continuation input lacks an owed follow-up", (o: ReturnType<typeof valid>) => { o.state.continuationInputBound = false; }],
|
|
["the continuation never received the later request", (o: ReturnType<typeof valid>) => { o.state.consumedBeforeCancellation = false; }],
|
|
])("rejects the continuation path when %s", (_label, mutate) => { const o = valid(); mutate(o); expect(gradeDiscard(o).passed).toBe(false); });
|