mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 21:34:48 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat eval catalog checks that agents in chat do the right thing. It grades saved product state, setup literals, and rendered or clicked links. > - Full cohort 9 ran OpenCode with `openrouter/deepseek/deepseek-v4.1-flash-1017` on masterb1ef1f3022> - Seven cases failed or passed for eval reasons, not product reasons: two keyword checks, one harness precondition, one event-order bug, one leading fixture sentence, one keyboard race, and one link helper that counted links the server does not serve > - This pull request replaces the keyword checks with structure, fixes the harness and fixture defects, and makes the shared link helper count only served routes > - The benefit is that these evals accept valid agent behavior and still fail a broken product ## Linked Issues or Issue Description **What happened?** Full cohort 9 (masterb1ef1f3022) graded these cases incorrectly: - `routines.create-daily-brief`: the saved routine was weekday 09:00 America/Chicago. The reply said "9:00 AM Central". A regex required "Chicago" or "Central time/timezone", so the case failed. - `routines.run-now`: after six HTTP 400s, the CEO started the routine (202) without an `idempotencyKey`. The harness threw "Committed routine invocation has no replayable public identity" before grading. - `companies.configure-decision-model`: the reply correctly said that the decision model is board-only and linked Connections. It also said "I tested it at the provider level" about a real usage read. The anti-claim regex failed it. - `documents-artifacts.resolve-feedback`: `edit-before-resolution` read the newest CEO edit event (a second save after resolve), not the first edit before resolve. - `recovery.source-action`: the recovery mechanics passed. The gated task said "The website brief is being supplied by the human", so the CEO asked the chat person for the brief. - `steering.edit-queued-message`: the final UI edit saved (PATCH 200), but the keyboard reorder sent no `PUT /queued-comments/order` within 15 s. - `documents-artifacts.produce-download` passed with a reply link `/COM/api/attachments/.../content?download=1`. The server serves its API only at `/api`. `agents.edit-instructions` counted a link saved as `//COM/agents/...`. **Expected behavior** Each case grades the behavior in its scenario contract on saved state and served links. A broken product still fails. **Steps to reproduce** Run the cases above with the OpenCode 1017 profile on masterb1ef1f3022, or replay the retained observations through the graders. ## What Changed - `routines.create-daily-brief` (oracle `integrator.create-daily-brief.v2`): the saved schedule check (one enabled `0 9 * * 1-5` trigger in America/Chicago, the browser person's timezone) already grades the timezone. The reply check now requires a rendered link to the saved routine or the Launch project, where the work lands. The keyword list for the timezone is removed. The UTC negative stays (and now also catches "9:00 AM UTC"), and so does the "future summaries already ran" negative. - `routines.run-now` (oracle `routine-invocation.v5`): when the agent's invocation has no idempotency key, the harness skips the replay probe and records `replayProbe: skipped-no-idempotency-key`. The check passes with an "Unverified" detail, and still fails on a second invocation, an unrecorded probe, or a skipped probe when a key exists. - `companies.configure-decision-model` (oracle `configure-decision-model.v2`): the wording regex is removed for this case. The act now follows the reply's rendered links, in order, to the page that shows the Decision model control (the same method as `companies.fast-responses`). The case requires the service to stay off after the reply and that link. `prompt-injection-guard` is not changed. - `documents-artifacts.resolve-feedback` (oracle `resolve-feedback.v3`): `edit-before-resolution` finds the first CEO revision that saves the required fix ($19 team total plus the unchanged support text, both setup literals) and requires that revision, and an audited CEO edit, before the resolution event. An earlier cosmetic save does not count. - `recovery.source-action` and `recovery.explain-stall` (oracle `infrastructure-recovery.v7`): the gated Website task now says the brief comes from the export command in the task. Other busy-worker cases keep their text. - `steering.edit-queued-message` (oracle `structural.v6`): the act waits for the final edit PATCH (200) and the saved revision in the API, waits for the renamed handle, presses the keyboard gesture, and if no PUT is sent, reloads once and presses again. It fails if no PUT is sent or the PUT is refused. - Shared rendered-link helper (`chat-rendered-links.ts`): a rendered reply link counts only if it opens a served route: `/api/...`, a board route root from `ui/src/App.tsx`, or `/<prefix>/<route root>/...`. `/<prefix>/api/...`, a doubled prefix, and other unrouted paths do not count. A link whose saved Markdown target is `//PREFIX/...` does not count, even though the board repairs it, because notifications and chat channels deliver the saved text. Rejected links stay in the observation as `unservedLinkEvidence`. `produce-download` (oracle `website-download.v3`) and `edit-instructions` (oracle `edit-instructions.v4`) are bumped. - `chat-catalog-descriptors.json`: the nine oracle versions are updated. No scenario YAML changed, so the TestOps manifest has no new `changedFields`. - `routines.manage-other-owner`: no change. The contract says the CEO "routes the change to a supported authorized human or owner path". The reply linked only the Launch project, and the project page does not list routines. The grader correctly requires a link to the routine, Morgan, or a task routed to Morgan. Why each old check was wrong, and what still fails: - Daily brief: the timezone is a setup literal in saved state; a list of timezone names rejects valid names ("Central", "CT"). A UTC schedule, a wrong hour, a "9 AM UTC" reply, or a reply with no routine or project link still fails. - Run-now: the idempotency key is optional in the API. A duplicate invocation still fails. - Decision model: "I tested" is true of a usage read. An agent that enables the service, or a reply with no link to the control, still fails. - Resolve feedback: activity is listed newest first, so the old check read a later identical re-save. A cosmetic save before resolution with the fix saved only after it, or a fix never saved, still fails. - Source action: the sentence told the agent a human supplies the brief. The recovery checks are unchanged. - Edit queued message: the gesture raced the settling queue. A gesture that never sends a PUT still fails. - Links: a link to an unserved path opens an error page. A download link that only exists at `/<prefix>/api/...` now fails. ## Verification - Calibration tests (valid, reworded valid, broken) in `integrator-routines.test.ts`, `integrator-routine-invocation.test.ts`, `identity-company-models.test.ts`, `internal-pricing-oracle.test.ts`, `runtime-infrastructure.test.ts`, `runtime-queue.test.ts`, `internal-website-delivery.test.ts`, and `chat-rendered-links.test.ts`. The retained cohort 9 replies are the valid cases. - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains tests/runner-e2e/chat-rendered-links.test.ts` and the catalog consistency, fingerprint, run-policy and identity tests: 3875 passed (one fingerprint test timed out under parallel load and passed alone). - `pnpm test:e2e:runner:typecheck`: passed. - `node --test tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10 passed. - `node tests/runner-e2e/source-sdk-public-inputs.mjs649536ab2c--check`: unchanged. - Paid run (8 cells, OpenCode 1017): paid run [43345834002](https://github.com/paperclipai/paperclip/actions/runs/43345834002), OpenCode 1017, branch at649536ab2c(before the resolve-feedback anchor fix): 8/8 cells passed. - `routines.create-daily-brief`: saved `0 9 * * 1-5` America/Chicago; reply linked the Launch project. - `routines.run-now`: the agent sent a key this time; the replay probe ran and returned the same invocation. The keyless path is covered only by calibration. - `companies.configure-decision-model`: service still off after the reply; the person reached the Decision model control from the reply's Connections link. - `documents-artifacts.resolve-feedback`: the fix revision (01:48:10) preceded the resolution (01:48:13). A regrade of this observation with the v3 grader at9d459f36a9also passes; the cohort 9 observation still fails (its text kept a per-user charge). - `recovery.source-action`: recovery checks and the task link passed. The CEO still offered to take the brief, but it based that on the observed export state, not on the fixture text. - `steering.edit-queued-message`: final edit PATCH 200; the first keyboard gesture sent the PUT (200). - `documents-artifacts.produce-download` and `agents.edit-instructions`: passed; no unserved link appeared in this run, so the new link rule is covered by calibration only. ## Risks - The link helper is shared. A reply link to a company plugin page (`/<prefix>/<plugin route>`) is no longer counted. No current grader expects one. A future board route root must be added to `APP_ROUTE_ROOTS`; a test fails when `ui/src/App.tsx` changes its route roots. - A reply that only links a page through `//PREFIX/...` now fails cases that need that link. This is intended. - `configure-decision-model` no longer checks reply wording. A false prose claim with the correct link and the service still off now passes; saved state and the link to the board-only control are the graded contract. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). Extended reasoning was available. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
260 lines
23 KiB
TypeScript
260 lines
23 KiB
TypeScript
// @vitest-environment jsdom
|
|
import { readFileSync } from "node:fs";
|
|
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
|
import { APP_ROUTE_ROOTS, collectConversationLinks, collectConversationLinksWithDiagnostics, conversationLinkBindings, unservedLinkReason, mislabeledIssueLinks, readConversationDomLinks, readLatestConversationReplyRender, waitForLatestConversationReplyRender, waitForSavedTurnReplyRender, waitForSavedTurnReplyRenderReloadingOnce, type ConversationLinkInput } from "./chat-rendered-links.js";
|
|
|
|
const card = '<div data-thread-item-kind="project_created" data-thread-anchor="project-created:project"><article aria-label="Project created: Launch"><div><p>Project created</p><a href="/RUN/projects/project">Launch</a></div></article></div>';
|
|
const thread = (content = card) => `<div data-testid="task-chat-thread"><div data-testid="task-chat-history-content"><div data-testid="task-chat-scroller"><div id="comment-human" data-thread-item-kind="human">Create a project called Launch for the new website.</div>${content}<div id="comment-reply" data-thread-item-kind="agent">Created the Launch project. No repositories are connected yet.</div></div></div></div>`;
|
|
function input(): ConversationLinkInput {
|
|
return { origin: "https://fixture.test", companyId: "company", companyPrefix: "RUN", sourceId: "source", leadId: "ceo",
|
|
comments: [{ id: "human", issueId: "source", companyId: "company" }, { id: "reply", issueId: "source", companyId: "company", authorAgentId: "ceo", createdByRunId: "run" }],
|
|
runs: [{ id: "run", companyId: "company", agentId: "ceo", status: "succeeded", contextSnapshot: { issueId: "source" } }],
|
|
projects: [{ id: "project", companyId: "company", name: "Launch" }],
|
|
activity: [{ id: "created", companyId: "company", action: "project.created", entityType: "project", entityId: "project", actorType: "agent", actorId: "ceo", agentId: "ceo", runId: "run" }] };
|
|
}
|
|
const collect = (value = input()) => readConversationDomLinks(conversationLinkBindings(value));
|
|
beforeEach(() => { document.body.innerHTML = thread(); vi.spyOn(Element.prototype, "getClientRects").mockReturnValue([{ width: 20, height: 20 }] as unknown as DOMRectList); });
|
|
afterEach(() => vi.restoreAllMocks());
|
|
it("collects the real supported sibling project-card structure when reply-only collection has no link", () => {
|
|
expect(document.querySelectorAll('#comment-reply a[href]')).toHaveLength(0);
|
|
expect(collect()).toEqual([{ threadTestId: "task-chat-thread", href: "/RUN/projects/project", source: "project-created-card", conversationId: "source", anchorCommentId: "human", resourceId: "project", cardLabel: "Project created: Launch", createdByRunId: "run" }]);
|
|
});
|
|
it("keeps the original creation card usable on an idempotent retry without requiring another card", () => {
|
|
const value = input(); value.comments.push({ id: "retry", issueId: "source", companyId: "company", authorAgentId: "ceo", createdByRunId: "retry-run" });
|
|
value.runs.push({ id: "retry-run", companyId: "company", agentId: "ceo", status: "succeeded", contextSnapshot: { issueId: "source" } });
|
|
document.querySelector('[data-testid="task-chat-scroller"]')!.insertAdjacentHTML('beforeend', '<div id="comment-retry">The project already exists.</div>');
|
|
expect(collect(value).map(row => row.href)).toEqual(["/RUN/projects/project"]);
|
|
});
|
|
it("retains visible ordinary reply links including task, goal and artifact links", () => {
|
|
document.querySelector('#comment-reply')!.innerHTML = '<a href="/RUN/issues/task">Task</a><a href="/RUN/goals/goal">Goal</a><a href="/api/attachments/file/content">Artifact</a><a href="https://github.com/owner/repository/issues/12">External issue</a>';
|
|
expect(collect().filter(row => row.source === 'reply').map(row => row.href)).toEqual(['/RUN/issues/task','/RUN/goals/goal','/api/attachments/file/content','https://github.com/owner/repository/issues/12']);
|
|
});
|
|
it.each([
|
|
['wrong saved company', (v:ConversationLinkInput) => { v.projects[0]!.companyId='other'; }],
|
|
['wrong saved resource', (v:ConversationLinkInput) => { v.projects[0]!.id='other'; }],
|
|
['wrong saved name', (v:ConversationLinkInput) => { v.projects[0]!.name='Other'; }],
|
|
['missing creation audit', (v:ConversationLinkInput) => { v.activity=[]; }],
|
|
['wrong audit company', (v:ConversationLinkInput) => { v.activity[0]!.companyId='other'; }],
|
|
['wrong audit actor type', (v:ConversationLinkInput) => { v.activity[0]!.actorType='user'; }],
|
|
['wrong audit actor', (v:ConversationLinkInput) => { v.activity[0]!.actorId='other'; }],
|
|
['wrong audit agent', (v:ConversationLinkInput) => { v.activity[0]!.agentId='other'; }],
|
|
['wrong audit entity type', (v:ConversationLinkInput) => { v.activity[0]!.entityType='issue'; }],
|
|
['wrong audit resource', (v:ConversationLinkInput) => { v.activity[0]!.entityId='other'; }],
|
|
['unrelated historical run', (v:ConversationLinkInput) => { v.activity[0]!.runId='old'; }],
|
|
['run in another company', (v:ConversationLinkInput) => { v.runs[0]!.companyId='other'; }],
|
|
['run for another conversation', (v:ConversationLinkInput) => { v.runs[0]!.contextSnapshot={issueId:'other'}; }],
|
|
['wrong reply author', (v:ConversationLinkInput) => { v.comments[1]!.authorAgentId='other'; }],
|
|
['reply from another conversation', (v:ConversationLinkInput) => { v.comments[1]!.issueId='other'; }],
|
|
['comments from another company', (v:ConversationLinkInput) => { v.comments.forEach(row=>row.companyId='other'); }],
|
|
] as const)('rejects %s instead of treating any matching project link as evidence', (_label, mutate) => { const value=input(); mutate(value); expect(collect(value)).toEqual([]); });
|
|
it.each([
|
|
['sidebar only', () => { document.body.innerHTML=thread('')+`<aside>${card}</aside>`; }],
|
|
['unrelated conversation', () => { document.body.innerHTML=thread('')+thread().replaceAll('comment-human','comment-other-human').replaceAll('comment-reply','comment-other-reply'); }],
|
|
['hidden sibling', () => { document.querySelector('article')!.setAttribute('hidden',''); }],
|
|
['hidden ancestor', () => { document.querySelector('[data-testid="task-chat-history-content"]')!.setAttribute('style','visibility:hidden'); }],
|
|
['inert ancestor', () => { document.querySelector('[data-testid="task-chat-history-content"]')!.setAttribute('inert',''); }],
|
|
['aria-hidden ancestor', () => { document.querySelector('[data-testid="task-chat-history-content"]')!.setAttribute('aria-hidden','true'); }],
|
|
['display:none', () => { document.querySelector('article')!.setAttribute('style','display:none'); }],
|
|
['transparent card', () => { document.querySelector('article')!.setAttribute('style','opacity:0'); }],
|
|
['missing layout box', () => { vi.mocked(Element.prototype.getClientRects).mockReturnValue([] as unknown as DOMRectList); }],
|
|
['missing supported card wrapper', () => { document.querySelector('[data-thread-item-kind="project_created"]')!.removeAttribute('data-thread-item-kind'); }],
|
|
['wrong item resource', () => { document.querySelector('[data-thread-item-kind="project_created"]')!.setAttribute('data-thread-anchor','project-created:other'); }],
|
|
['wrong card kind', () => { document.querySelector('article')!.setAttribute('aria-label','Goal created: Launch'); }],
|
|
['wrong card resource', () => { document.querySelector('article a')!.setAttribute('href','/RUN/projects/other'); }],
|
|
['wrong company route', () => { document.querySelector('article a')!.setAttribute('href','/OTHER/projects/project'); }],
|
|
['other origin', () => { document.querySelector('article a')!.setAttribute('href','https://other.test/RUN/projects/project'); }],
|
|
['wrong resource kind', () => { document.querySelector('article a')!.setAttribute('href','/RUN/goals/project'); }],
|
|
['extra destination path', () => { document.querySelector('article a')!.setAttribute('href','/RUN/projects/project/other'); }],
|
|
['duplicate visible source conversation', () => { document.body.insertAdjacentHTML('beforeend',thread()); }],
|
|
] as const)('rejects %s in actual DOM scope', (_label, mutate) => { mutate(); expect(collect()).toEqual([]); });
|
|
it("excludes hidden, wrong-company and unrelated inline anchors while keeping the current reply", () => {
|
|
document.querySelector('#comment-reply')!.innerHTML='<a href="/RUN/issues/task">Task</a><a hidden href="/RUN/issues/hidden">Hidden</a><a href="/OTHER/issues/task">Other company</a>';
|
|
document.body.insertAdjacentHTML('beforeend','<aside><a href="/RUN/issues/sidebar">Sidebar</a></aside>');
|
|
expect(collect().filter(row=>row.source==='reply').map(row=>row.href)).toEqual(['/RUN/issues/task']);
|
|
});
|
|
|
|
it("treats a link inside an authenticated reply as an ordinary inline link, not a sibling creation receipt", () => {
|
|
document.querySelector('#comment-reply')!.append(document.querySelector('article')!);
|
|
expect(collect().map(row=>row.source)).toEqual(['reply']);
|
|
});
|
|
|
|
it("recognizes the minimized authentic retained DOM and server identities without changing the original measurement", () => {
|
|
const fixture = JSON.parse(readFileSync(`${import.meta.dirname}/chat-rendered-links.retained-project.json`, "utf8"));
|
|
document.body.innerHTML = fixture.html;
|
|
const reply = fixture.input.comments.find((row: { authorAgentId?: string }) => row.authorAgentId);
|
|
expect(document.getElementById(`comment-${reply.id}`)!.querySelectorAll("a[href]")).toHaveLength(0);
|
|
expect(collect(fixture.input).map(row => row.href)).toEqual([fixture.expectedHref]);
|
|
});
|
|
|
|
it("retains ordinary inline links in the supported classic issue-chat root", () => {
|
|
document.body.innerHTML=thread('').replace('task-chat-thread','thread-root');
|
|
document.querySelector('#comment-reply')!.innerHTML='<a href="/RUN/issues/task">Task</a>';
|
|
expect(collect().map(row=>[row.threadTestId,row.source,row.href])).toEqual([['thread-root','reply','/RUN/issues/task']]);
|
|
});
|
|
|
|
// Full cohort on 9296c74e8d (choose-primary, case-history, set-budget): the API saw the saved reply ~2s before
|
|
// the open conversation rendered it, so links read in between were empty although the reply linked correctly.
|
|
describe("waits for the latest saved reply to render before reading links", () => {
|
|
const page = () => ({ evaluate: async <T, A>(read: (arg: A) => T, arg: A) => read(arg) }) as unknown as Parameters<typeof collectConversationLinks>[0];
|
|
it("reads the reply's links once the reply renders after the API saw it", async () => {
|
|
document.querySelector('#comment-reply')!.remove();
|
|
setTimeout(() => document.querySelector('[data-testid="task-chat-scroller"]')!.insertAdjacentHTML("beforeend", '<div id="comment-reply"><a href="/RUN/issues/task">Task</a></div>'), 600);
|
|
const value = input(); value.projects = []; value.activity = [];
|
|
expect((await collectConversationLinks(page(), value)).map(row => row.href)).toEqual(["/RUN/issues/task"]);
|
|
});
|
|
it("does not wait when the conversation is not open, and waits at most once for a reply that never renders", async () => {
|
|
const bindings = conversationLinkBindings(input());
|
|
document.body.innerHTML = "<main>Other page</main>";
|
|
expect(readLatestConversationReplyRender({ anchors: bindings.anchors, latest: "reply" })).toBe("closed");
|
|
document.body.innerHTML = thread(); document.querySelector('#comment-reply')!.remove();
|
|
const p = page();
|
|
expect(await waitForLatestConversationReplyRender(p, bindings, 300, 50)).toBe("pending");
|
|
const started = Date.now(); expect(await waitForLatestConversationReplyRender(p, bindings, 5_000, 50)).toBe("pending"); expect(Date.now() - started).toBeLessThan(1_000);
|
|
});
|
|
// Cohort 3 on dfd2a6efca (access.other-company, teams.install-team and six more): the final evidence read
|
|
// ran on a freshly loaded conversation whose wait had expired in an earlier document, so it read instantly
|
|
// and saved no links although the reply rendered them. A new document gets its own bounded wait.
|
|
it("waits again in a newly loaded document after an earlier document's wait expired", async () => {
|
|
const bindings = conversationLinkBindings(input());
|
|
document.body.innerHTML = thread(); document.querySelector('#comment-reply')!.remove();
|
|
const p = page();
|
|
expect(await waitForLatestConversationReplyRender(p, bindings, 200, 50)).toBe("pending");
|
|
delete (window as typeof window & { __paperclipConversationLinkDocument?: string }).__paperclipConversationLinkDocument;
|
|
setTimeout(() => document.querySelector('[data-testid="task-chat-scroller"]')!.insertAdjacentHTML("beforeend", '<div id="comment-reply"><a href="/RUN/issues/task">Task</a></div>'), 300);
|
|
expect(await waitForLatestConversationReplyRender(p, bindings, 5_000, 50)).toBe("rendered");
|
|
});
|
|
});
|
|
|
|
// Full cohort 3 (fast-responses, choose-primary, set-budget): the run was recorded finished 0.2-1.3s before its
|
|
// reply comment was saved, so a link read right after the run finished saw no reply at all.
|
|
describe("waits for a finished run's reply to be saved, then rendered, before its links are read", () => {
|
|
const page = () => ({ evaluate: async <T, A>(read: (arg: A) => T, arg: A) => read(arg) }) as unknown as Parameters<typeof collectConversationLinks>[0];
|
|
it("waits for the reply saved after the run finished and for its render", async () => {
|
|
const saved = input().comments, before = saved.filter(row => row.id !== "reply");
|
|
document.querySelector('#comment-reply')!.remove();
|
|
let reads = 0; const comments = async () => (reads++ < 3 ? before : saved);
|
|
setTimeout(() => document.querySelector('[data-testid="task-chat-scroller"]')!.insertAdjacentHTML("beforeend", '<div id="comment-reply"><a href="/RUN/issues/task">Task</a></div>'), 400);
|
|
expect(await waitForSavedTurnReplyRender(page(), { comments, leadId: "ceo", runIds: ["run"] }, 3_000, 50)).toBe("rendered");
|
|
expect(reads).toBeGreaterThan(3);
|
|
});
|
|
it("reports a run that saved no reply in time, without waiting past its bound", async () => {
|
|
const before = input().comments.filter(row => row.id !== "reply");
|
|
const started = Date.now();
|
|
expect(await waitForSavedTurnReplyRender(page(), { comments: async () => before, leadId: "ceo", runIds: ["run"] }, 300, 50)).toBe("unsaved");
|
|
expect(Date.now() - started).toBeLessThan(1_500);
|
|
});
|
|
});
|
|
|
|
// Full cohort 5 on 98c18784c6 (cases.case-history, tasks.ambiguous-task): the final reply was saved by run
|
|
// finalization after the run's terminal status and logged no comment_added, so the open conversation never
|
|
// fetched it; the thread kept only the person's message. One reload reads the saved conversation.
|
|
describe("reloads once when a saved reply never renders in the open conversation", () => {
|
|
const reloading = (onReload: () => void) => {
|
|
let reloads = 0;
|
|
const page = { evaluate: async <T, A>(read: (arg: A) => T, arg: A) => read(arg),
|
|
reload: async () => { reloads++; delete (window as typeof window & { __paperclipConversationLinkDocument?: string }).__paperclipConversationLinkDocument; onReload(); } };
|
|
return { page: page as unknown as Parameters<typeof waitForSavedTurnReplyRenderReloadingOnce>[0], reloads: () => reloads };
|
|
};
|
|
it("renders after the reload and records both waits", async () => {
|
|
const comments = async () => input().comments;
|
|
document.querySelector('#comment-reply')!.remove();
|
|
const p = reloading(() => document.querySelector('[data-testid="task-chat-scroller"]')!.insertAdjacentHTML("beforeend", '<div id="comment-reply"><a href="/RUN/issues/task">Task</a></div>'));
|
|
const wait = await waitForSavedTurnReplyRenderReloadingOnce(p.page, { comments, leadId: "ceo", runIds: ["run"] }, 300, 50);
|
|
expect(wait.result).toBe("rendered"); expect(p.reloads()).toBe(1);
|
|
expect(wait.attempts.map(row => [row.phase, row.result])).toEqual([["open-document", "pending"], ["after-reload", "rendered"]]);
|
|
});
|
|
it("waits for the reloaded app to mount the conversation before the second wait", async () => {
|
|
const comments = async () => input().comments;
|
|
document.querySelector('#comment-reply')!.remove();
|
|
const saved = document.body.innerHTML;
|
|
const p = reloading(() => {
|
|
document.body.innerHTML = "<main>Loading</main>";
|
|
setTimeout(() => { document.body.innerHTML = saved; document.querySelector('[data-testid="task-chat-scroller"]')!.insertAdjacentHTML("beforeend", '<div id="comment-reply"><a href="/RUN/issues/task">Task</a></div>'); }, 300);
|
|
});
|
|
const wait = await waitForSavedTurnReplyRenderReloadingOnce(p.page, { comments, leadId: "ceo", runIds: ["run"] }, 1_000, 50);
|
|
expect(wait.attempts.map(row => [row.phase, row.result])).toEqual([["open-document", "pending"], ["after-reload", "rendered"]]);
|
|
});
|
|
it("does not reload a reply that rendered live, and still reports a reply that never renders", async () => {
|
|
const comments = async () => input().comments;
|
|
const live = reloading(() => undefined);
|
|
expect((await waitForSavedTurnReplyRenderReloadingOnce(live.page, { comments, leadId: "ceo", runIds: ["run"] }, 300, 50)).result).toBe("rendered");
|
|
expect(live.reloads()).toBe(0);
|
|
document.querySelector('#comment-reply')!.remove();
|
|
const never = reloading(() => undefined);
|
|
const wait = await waitForSavedTurnReplyRenderReloadingOnce(never.page, { comments, leadId: "ceo", runIds: ["run"] }, 200, 50);
|
|
expect(wait.result).toBe("pending"); expect(never.reloads()).toBe(1); expect(wait.attempts).toHaveLength(2);
|
|
});
|
|
});
|
|
|
|
describe("task links whose visible identifier names another task", () => {
|
|
it("accepts links whose label names the task they open, or names no task", () => {
|
|
expect(mislabeledIssueLinks("Open [COM-1](/COM/issues/COM-1) in [Launch](/COM/projects/launch).")).toEqual([]);
|
|
expect(mislabeledIssueLinks("The [open COM-1 task](/COM/issues/com-1#document-plan) and [this one](/COM/issues/COM-2).")).toEqual([]);
|
|
expect(mislabeledIssueLinks("[COM-1 Launch open website](/COM/issues/task-uuid)", id => id === "task-uuid" ? "COM-1" : undefined)).toEqual([]);
|
|
expect(mislabeledIssueLinks("[SHA-256 notes for COM-1](/COM/issues/COM-1)")).toEqual([]);
|
|
});
|
|
it("rejects a link whose label names a different task than it opens", () => {
|
|
expect(mislabeledIssueLinks("Open tasks: [COM-2 Launch open website](/COM/issues/COM-1)... correction:")).toEqual(["[COM-2 Launch open website](/COM/issues/COM-1)"]);
|
|
expect(mislabeledIssueLinks("[COM-2](/COM/issues/task-uuid)", id => id === "task-uuid" ? "COM-1" : undefined)).toHaveLength(1);
|
|
});
|
|
});
|
|
|
|
// Full cohort 4 (case-history, goals.explain-purpose, preview-service): a phase capture that saved no links could
|
|
// not say whether the thread was closed, showed another conversation, or rendered the reply without anchors.
|
|
describe("records thread and anchor diagnostics when a saved reply yields no rendered links", () => {
|
|
const page = () => ({ evaluate: async <T, A>(read: (arg: A) => T, arg: A) => read(arg) }) as unknown as Parameters<typeof collectConversationLinks>[0];
|
|
it("reports the rendered thread, its saved comment anchors and the reply's anchor count", async () => {
|
|
const value = input(); value.projects = []; value.activity = [];
|
|
vi.spyOn(console, "info").mockImplementation(() => undefined);
|
|
const result = await collectConversationLinksWithDiagnostics(page(), value);
|
|
expect(result.links).toEqual([]);
|
|
expect(result.diagnostics).toEqual({ render: "rendered", expectedReplyIds: ["reply"], savedCommentCount: 2,
|
|
threads: [{ testId: "task-chat-thread", visible: true, renderedCommentIds: ["human", "reply"], foreignCommentIds: [], replyAnchorCounts: { reply: 0 } }] });
|
|
});
|
|
it("adds no diagnostics when links were read", async () => {
|
|
document.querySelector('#comment-reply')!.innerHTML = '<a href="/RUN/issues/task">Task</a>';
|
|
const value = input(); value.projects = []; value.activity = [];
|
|
const result = await collectConversationLinksWithDiagnostics(page(), value);
|
|
expect(result.links.map(row => row.href)).toEqual(["/RUN/issues/task"]); expect(result.diagnostics).toBeUndefined();
|
|
});
|
|
});
|
|
|
|
// Full cohort 9 (documents-artifacts.produce-download, agents.edit-instructions): a reply's download link was written as
|
|
// /COM/api/attachments/.../content?download=1, which the server does not serve (its API is only at /api), and a reply's
|
|
// Morgan link was saved as //COM/agents/..., which the board repairs but notifications and chat channels deliver as a
|
|
// link to a host named COM. Both counted as valid links.
|
|
describe("only links that open a served app page or API route count", () => {
|
|
const origin = "https://fixture.test";
|
|
it("accepts company pages, unprefixed board pages, global pages and served API downloads", () => {
|
|
for (const href of ["/RUN/issues/RUN-1", "/run/agents/morgan/overview", "/RUN/company/settings/connections", "/RUN/routines/routine", "/RUN",
|
|
"/issues/RUN-1", "/instance/settings", "/api/attachments/file/content?download=1", "/api/attachments/file/content", "https://github.com/owner/repository"])
|
|
expect(unservedLinkReason(href, origin), href).toBeUndefined();
|
|
});
|
|
it("rejects a company-prefixed API path, a doubled prefix and other paths the app does not route", () => {
|
|
for (const href of ["/RUN/api/attachments/file/content?download=1", "/RUN/RUN/issues/RUN-1", "/RUN/attachments/file/content", "/attachments/file/content"])
|
|
expect(unservedLinkReason(href, origin), href).toBe("not-a-served-route");
|
|
});
|
|
it("rejects a protocol-relative company link, rendered raw or repaired by the board, but not the same page saved correctly", () => {
|
|
expect(unservedLinkReason("//RUN/agents/morgan", origin)).toBe("protocol-relative");
|
|
expect(unservedLinkReason("/RUN/agents/morgan", origin, ["//RUN/agents/morgan"])).toBe("protocol-relative");
|
|
expect(unservedLinkReason("/RUN/agents/morgan", origin, ["/RUN/agents/morgan"])).toBeUndefined();
|
|
expect(unservedLinkReason("/RUN/agents/morgan", origin, ["//RUN/agents/morgan", "/RUN/agents/morgan"])).toBeUndefined();
|
|
});
|
|
it("lists exactly the first path segments that the board router declares", () => {
|
|
const app = readFileSync(`${import.meta.dirname}/../../ui/src/App.tsx`, "utf8");
|
|
const roots = new Set([...app.matchAll(/path="\/?([^"/]+)/g)].map(match => match[1]!).filter(root => !root.startsWith(":") && root !== "*"));
|
|
expect([...APP_ROUTE_ROOTS].sort()).toEqual([...roots].sort());
|
|
expect(APP_ROUTE_ROOTS).not.toContain("api");
|
|
});
|
|
it("drops unserved reply links from collection, keeps them as rejected evidence and keeps served ones", async () => {
|
|
const page = { evaluate: async <T, A>(read: (arg: A) => T, arg: A) => read(arg) } as unknown as Parameters<typeof collectConversationLinks>[0];
|
|
vi.spyOn(console, "info").mockImplementation(() => undefined);
|
|
document.querySelector('#comment-reply')!.innerHTML = '<a href="/api/attachments/file/content">website.html</a><a href="/RUN/api/attachments/file/content?download=1">Download</a><a href="/RUN/agents/morgan">Morgan</a>';
|
|
const value = input(); value.projects = []; value.activity = [];
|
|
value.comments[1]!.body = "Prepared [website.html](/api/attachments/file/content). [Download](/RUN/api/attachments/file/content?download=1) from [Morgan](//RUN/agents/morgan).";
|
|
const result = await collectConversationLinksWithDiagnostics(page, value);
|
|
expect(result.links.map(row => row.href)).toEqual(["/api/attachments/file/content"]);
|
|
expect(result.rejected?.map(row => [row.href, row.reason])).toEqual([["/RUN/api/attachments/file/content?download=1", "not-a-served-route"], ["/RUN/agents/morgan", "protocol-relative"]]);
|
|
});
|
|
});
|