Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-11 05:11:49 +01:00
Code Issues Packages Projects Releases Wiki Activity

fix(evals): wait for rendered replies, bound runs by role and grade chat outcomes structurally (#20685)

Browse Source 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The chat-feature eval catalog runs real chat journeys and grades
saved state, rendered links and run ledgers.
> - The second full-cohort run on master `9296c74e8d` failed 15 cases
where the product did the right thing and the harness did not.
> - The causes were three harness classes: links read before the reply
rendered, raw run counts that rejected product-caused follow-up runs,
and keyword graders on free-form replies. Four browser locators also
targeted retired controls.
> - This pull request fixes those harness layers. It does not change the
product.
> - The benefit is that these cases measure the feature again: correct
behaviour passes, and the relevant broken behaviour still fails.

## Linked Issues or Issue Description

**What happened**
In the second full-cohort chat-eval run on master `9296c74e8d`, these
cases failed although the reply and the saved state were correct:
`agents.choose-primary`, `cases.case-history`,
`budgets-costs.set-budget` (links read about 2s before the reply
rendered); `task-coordination.independent-review`,
`cases.update-case-facts`, `routines.manage-other-owner`,
`questions-and-approvals.review-plan` (a legitimate continuation or the
product's completion follow-up exceeded a raw run count or was
cancelled); `cases.cases-disabled`, `inbox-search.find-design`,
`navigation-preferences.broken-link`, `cases.browse-cases` (keyword or
clause graders rejected correct replies); `agent-chat.conference-room`,
`workspaces.inspect-git`, `task-coordination.block-on-design`,
`skills.author-skill` (browser steps used stale settings or controls).

**Expected behavior**
A correct reply and correct saved state pass. A wrong outcome, extra or
duplicate agent work, a substitution or a missing route still fails.

**Steps to reproduce**
Run `runner-full-stack-e2e.yml` with `cadence=manual`, `all=false` and
an `id` list of the cases above against master `9296c74e8d`. Read
`result.json` and `snapshots/` in each cell's evidence.

Refs #20600

## What Changed

- `chat-rendered-links.ts`: `collectConversationLinks` waits (at most
10s, once per reply per page) for the latest saved lead reply to render
when its conversation is open. A page that does not show the
conversation does not wait. `observeInternal` and the flow conversation
read use it.
- `budgets-costs.set-budget`: waits for the rendered reply before it
reads anchors. It fills the budget only after the form shows the saved
cap, and checks that the typed value stays.
- New `chat-catalog-domains/run-roles.ts`: identifies a run by role (a
worker woken by a comment that a named run saved on the task; the CEO's
chat completion follow-up).
- `task-coordination.independent-review`: admits one Morgan continuation
woken by the CEO's authorization comment and one Riley review run. The
review request may come from that continuation's native status decision.
Other runs still fail.
- `cases.update-case-facts`: a phase settles when its CEO turn and at
most one Morgan continuation woken by that turn's task comment finish. A
new `role-bound-runs` check rejects any other run. `terminal-counts`
gets an optional `maximumTerminalStatusCounts`.
- `routines.manage-other-owner`: admits one Morgan run on a task that
the CEO created for Morgan. A schedule change must be Morgan's own write
from that run. The ownership check is structural (a linked owner task,
or a link to the routine or Morgan's page). The keyword checks on the
reply are removed.
- `questions-and-approvals.review-plan`: after the stale-review check,
lets the product's completion follow-up settle before final run custody.
The existing grader already validates it.
- `cases.cases-disabled`: the negation-blind false-claim regex is
removed. The case passes with a reply link to the Experimental settings
page (the scripted person follows it and sees the Cases control) or a
pending choice card. Saved state still proves that nothing was created.
- `inbox-search.find-design`: the per-clause regex is removed. The first
task link in the reply must be the task with the approved revision.
- `navigation-preferences.broken-link`: a link to the similar task fails
only when its label names the original (identifier or exact title). A
separate mention passes.
- `cases.browse-cases`: an excluded case must not be a result row. When
the reply has no result list, every case link is a result, so an
excluded case must not be linked.
- Browser steps: `conference-room` reloads after the fixture enables the
flag. `inspect-git` and `block-on-design` open the panel with "Show
properties" and pick the tab from the side panel launcher.
`author-skill` does not repeat the full inspection in `verifyResult`
when `act` already inspected the same saved version.
- Each changed case has a new `oracleVersion`.
`chat-catalog-descriptors.json` is in sync.

## Verification

- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts <119
affected test files> --testTimeout=180000`: 1673 passed, 3 skipped. One
file (`chat-factual-query-input.test.ts`) did not load in the local
worktree because `@paperclipai/plugin-sdk` is not built there. It does
not import changed code. CI runs it.
- `chat-catalog-consistency.test.ts` passes (242 descriptors).
- `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
no pin change.
- `node --test` on the three `chat-full-catalog-testops-*.test.mjs`
files: 40 passed.
- Calibration against retained evidence: the retained replies and run
shapes of cases-disabled, browse-cases, find-design, broken-link,
independent-review, update-case-facts and manage-other-owner now pass.
Broken variants fail (closed case as a result row, earlier design linked
first, original label on the similar task's link, unprompted or
duplicate worker runs, a second review run, a routine change outside the
owner's run, no route link).
- Paid runs on this branch: see the PR comments.

## Risks

- Low risk. Only eval harness code changes.
- The structural graders do not read reply wording. A reply that words
the outcome wrongly but links the right records can pass. A person reads
the reply.
- `navigation-preferences.sidebar-order` is not changed. The shipped
sidebar has no per-project order or collapsible Agents section, so the
case premise needs an owner decision.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-10 20:24:41 -06:00
1 parent 3fdcc2a28c
commit cbd636d652
27 files changed
+570 -181
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

tests/runner-e2e/chat-catalog-contract.ts
+4 -1
View File
@@ -140,7 +140,10 @@ export function chatCatalogNativePhaseIsTimely(ctx: ChatCatalogContext): boolean
export type ChatCatalogTerminalRunStatus = "succeeded" | "cancelled" | "failed" | "timed_out";
export type ChatCatalogRunPolicy =
| { kind: "terminal-counts"; terminalStatusCounts: Partial<Record<ChatCatalogTerminalRunStatus, number>> }
| { kind: "terminal-counts"; terminalStatusCounts: Partial<Record<ChatCatalogTerminalRunStatus, number>>;
/** Optional upper bounds: a status listed here may occur from its terminalStatusCounts value up to this count.
* The case oracle must bind every run above the minimum to a declared role. */
maximumTerminalStatusCounts?: Partial<Record<ChatCatalogTerminalRunStatus, number>> }
| { kind: "source-settled-with-active-work"; maximumActiveWorkerRuns: number;
expectedWorkerCancellation?: "owned-task-reassignment" };
export interface ChatCatalogExpectedWorkerCancellation {
tests/runner-e2e/chat-catalog-descriptors.json
+30 -18
View File
@@ -301,9 +301,10 @@
"sourceFiles": [
"chat-catalog-domains/runtime-conference.ts",
"chat-catalog-domains/runtime-support.ts",
"chat-catalog-domains/internal-comment-reference.ts",
"chat-scenarios/agent-chat.yaml"
],
"oracleVersion": "paperclip.chat-catalog.runtime-conference.v3",
"oracleVersion": "paperclip.chat-catalog.runtime-conference.v5",
"coverageGaps": [],
"expectedRunCount": 1,
"turnTimeoutMs": 120000,
@@ -1233,9 +1234,10 @@
"chat-catalog-domains/identity-support.ts",
"chat-catalog-domains/internal-support.ts",
"chat-catalog-domains/identity-budgets-costs.ts",
"chat-catalog-domains/internal-comment-reference.ts",
"chat-scenarios/budgets-costs.yaml"
],
"oracleVersion": "paperclip.chat-catalog.identity.v1",
"oracleVersion": "paperclip.chat-catalog.identity.set-budget.v2",
"expectedRunCount": 1,
"turnTimeoutMs": 120000,
"attemptTimeoutMs": 300000,
@@ -1298,7 +1300,7 @@
"chat-catalog-domains/integrator-cases.ts",
"chat-scenarios/cases.yaml"
],
"oracleVersion": "paperclip.chat-catalog.cases-browse.structural.v2",
"oracleVersion": "paperclip.chat-catalog.cases-browse.structural.v3",
"expectedRunCount": 1,
"expectedHumanMessageCount": 1,
"turnTimeoutMs": 120000,
@@ -1331,7 +1333,7 @@
"chat-catalog-domains/integrator-cases.ts",
"chat-scenarios/cases.yaml"
],
"oracleVersion": "paperclip.chat-catalog.integrator.v1",
"oracleVersion": "paperclip.chat-catalog.cases-disabled.structural.v1",
"expectedRunCount": 1,
"expectedHumanMessageCount": 1,
"turnTimeoutMs": 120000,
@@ -1439,14 +1441,16 @@
"scenarioId": "cases.update-case-facts",
"sourceFiles": [
"chat-catalog-domains/integrator-case-facts.ts",
"chat-catalog-domains/run-roles.ts",
"chat-catalog-domains/internal-run-brief.ts",
"chat-catalog-domains/integrator-support.ts",
"chat-catalog-domains/internal-support.ts",
"chat-catalog-domains/internal-definition.ts",
"chat-scenarios/cases.yaml"
],
"oracleVersion": "paperclip.chat-catalog.integrator.case-facts.v4",
"expectedRunCount": 4,
"oracleVersion": "paperclip.chat-catalog.integrator.case-facts.v6",
"expectedRunCount": 6,
"minimumRunCount": 4,
"expectedHumanMessageCount": 2,
"turnTimeoutMs": 120000,
"attemptTimeoutMs": 900000,
@@ -1455,6 +1459,9 @@
"terminalStatusCounts": {
"cancelled": 1,
"succeeded": 3
},
"maximumTerminalStatusCounts": {
"succeeded": 5
}
},
"modulePath": "./chat-catalog-domains/integrator-case-facts.js",
@@ -3353,7 +3360,7 @@
"chat-catalog-domains/internal-privacy.ts",
"chat-catalog-domains/identity-auth.ts"
],
"oracleVersion": "paperclip.chat-catalog.internal.v1",
"oracleVersion": "paperclip.chat-catalog.internal.find-design.structural.v1",
"coverageGaps": [],
"expectedRunCount": 1,
"turnTimeoutMs": 120000,
@@ -3586,7 +3593,7 @@
"chat-catalog-domains/internal-support.ts",
"chat-scenarios/navigation-preferences.yaml"
],
"oracleVersion": "paperclip.chat-catalog.identity.broken-link.v2",
"oracleVersion": "paperclip.chat-catalog.identity.broken-link.v3",
"expectedRunCount": 1,
"turnTimeoutMs": 120000,
"attemptTimeoutMs": 300000,
@@ -4290,7 +4297,7 @@
"../../server/src/services/chat-completion-delivery.ts",
"chat-catalog-domains/turn-response-card.ts"
],
"oracleVersion": "paperclip.chat-catalog.plan-review.v6",
"oracleVersion": "paperclip.chat-catalog.plan-review.v7",
"expectedRunCount": 3,
"minimumRunCount": 1,
"turnTimeoutMs": 120000,
@@ -4313,7 +4320,7 @@
"../../server/src/services/chat-completion-delivery.ts",
"chat-catalog-domains/turn-response-card.ts"
],
"oracleVersion": "paperclip.chat-catalog.plan-review.v6",
"oracleVersion": "paperclip.chat-catalog.plan-review.v7",
"expectedRunCount": 3,
"minimumRunCount": 2,
"turnTimeoutMs": 120000,
@@ -4643,13 +4650,16 @@
"chat-catalog-domains/integrator-support.ts",
"chat-catalog-domains/internal-support.ts",
"chat-catalog-domains/integrator-routines.ts",
"chat-catalog-domains/run-roles.ts",
"../../server/src/services/cron.ts",
"chat-scenarios/routines.yaml"
],
"oracleVersion": "paperclip.chat-catalog.integrator.v1",
"expectedRunCount": 1,
"oracleVersion": "paperclip.chat-catalog.integrator.manage-other-owner.v5",
"expectedRunCount": 2,
"minimumRunCount": 1,
"expectedHumanMessageCount": 1,
"turnTimeoutMs": 120000,
"attemptTimeoutMs": 300000,
"attemptTimeoutMs": 480000,
"modulePath": "./chat-catalog-domains/integrator-routines.js",
"exportName": "integratorRoutinesCases"
},
@@ -5149,7 +5159,7 @@
"../../ui/src/pages/CompanySkills.tsx",
"../../ui/src/lib/company-skill-routes.ts"
],
"oracleVersion": "paperclip.chat-catalog.skill-authoring.v7",
"oracleVersion": "paperclip.chat-catalog.skill-authoring.v9",
"expectedRunCount": 1,
"expectedHumanMessageCount": 1,
"turnTimeoutMs": 120000,
@@ -5664,7 +5674,7 @@
"run-observations.ts",
"chat-scenarios/task-coordination.yaml"
],
"oracleVersion": "paperclip.chat-catalog.runtime.block-on-design.v5",
"oracleVersion": "paperclip.chat-catalog.runtime.block-on-design.v7",
"coverageGaps": [],
"expectedRunCount": 3,
"expectedHumanMessageCount": 2,
@@ -5700,15 +5710,17 @@
"scenarioId": "task-coordination.independent-review",
"sourceFiles": [
"chat-catalog-domains/runtime-independent-review.ts",
"chat-catalog-domains/run-roles.ts",
"chat-catalog-domains/runtime-support.ts",
"chat-catalog-domains/runtime-busy-worker.ts",
"chat-catalog-domains/identity-support.ts",
"registered-artifact.ts",
"chat-scenarios/task-coordination.yaml"
],
"oracleVersion": "paperclip.chat-catalog.runtime.independent-review.v4",
"oracleVersion": "paperclip.chat-catalog.runtime.independent-review.v5",
"coverageGaps": [],
"expectedRunCount": 5,
"expectedRunCount": 6,
"minimumRunCount": 5,
"setupHumanMessageCount": 1,
"expectedHumanMessageCount": 3,
"turnTimeoutMs": 120000,
@@ -6143,7 +6155,7 @@
"chat-catalog-domains/runtime-support.ts",
"chat-scenarios/workspaces.yaml"
],
"oracleVersion": "paperclip.chat-catalog.inspect-git.structural.v3",
"oracleVersion": "paperclip.chat-catalog.inspect-git.structural.v4",
"coverageGaps": [],
"expectedRunCount": 2,
"turnTimeoutMs": 120000,
tests/runner-e2e/chat-catalog-domains/identity-budgets-costs.ts
+12 -2
View File
@@ -3,6 +3,7 @@ import { expect } from "@playwright/test";
import type { BudgetOverview } from "../../../packages/shared/src/types/budget.js";
import { checks, entity } from "./internal-support.js";
import { identityDefinition, identityObservation, identityWorker, observeIdentity, waitForReply, probeAgentPermission } from "./identity-support.js";
import { waitForRenderedReply } from "./internal-comment-reference.js";
export function gradeBudget(value: unknown) {
const c = checks(); const o = identityObservation(value);
@@ -30,8 +31,13 @@ export const budgetIdentityCases = [identityDefinition("budgets-costs.set-budget
const workerConfig = await employeeConfiguration(ctx, worker.id);
return { worker, other, workerConfig };
}, gradeBudget, {
oracleVersion: "paperclip.chat-catalog.identity.set-budget.v2",
sourceFiles: ["chat-catalog-domains/identity-support.ts", "chat-catalog-domains/internal-support.ts", "chat-catalog-domains/identity-budgets-costs.ts", "chat-catalog-domains/internal-comment-reference.ts", "chat-scenarios/budgets-costs.yaml"],
async act(ctx, state) {
await waitForReply(ctx, chatCatalogOriginalPhaseDeadline(ctx));
// The API sees the saved reply before the open conversation renders it;
// read the reply's anchors only once that reply is on the page.
await waitForRenderedReply(ctx, { leadId: ctx.input.fixtures.agent.id, runs: await ctx.allRuns() });
const root = `/api/companies/${ctx.input.fixtures.company.id}`; const worker = entity(state, "worker");
state.agentWriteStatus = await probeAgentPermission(ctx, `${root}/budgets/policies`, "POST", {
scopeType: "agent", scopeId: worker.id, amount: 5000, windowKind: "calendar_month_utc" });
@@ -46,9 +52,13 @@ export const budgetIdentityCases = [identityDefinition("budgets-costs.set-budget
if (!href) throw new Error("The budget guidance did not link the intended Morgan record");
await ctx.input.page.locator(`[id^="comment-"] a[href=${JSON.stringify(href)}]`).first().click();
if (!(await ctx.input.page.getByPlaceholder("0.00").isVisible())) await ctx.input.page.getByRole("link", { name: "Budgets", exact: true }).click();
await ctx.input.page.getByPlaceholder("0.00").fill("50.00");
// The draft resets whenever the loaded policy changes; fill only after the
// form shows the currently saved cap, and confirm the typed value stuck.
const amount = ctx.input.page.getByPlaceholder("0.00");
await expect(amount).toHaveValue((Number(state.beforeHumanAmount ?? 0) / 100).toFixed(2));
await amount.fill("50.00"); await expect(amount).toHaveValue("50.00");
await ctx.input.page.getByRole("button", { name: "Update budget", exact: true }).click();
await expect(ctx.input.page.getByPlaceholder("0.00")).toHaveValue("50.00"); state.humanSaved = true;
await expect(amount).toHaveValue("50.00"); state.humanSaved = true;
await ctx.input.evidence("identity-human-budget-action.json", { workerId: worker.id, amountCents: 5000, url: ctx.input.page.url() });
await ctx.input.page.goto(source, { waitUntil: "domcontentloaded" });
},
tests/runner-e2e/chat-catalog-domains/identity-navigation-boundaries.test.ts
+13 -1
View File
@@ -24,7 +24,19 @@ describe("real missing link and instance-wide personal dismissal", () => {
it(`${d.scenarioId}: wrong audience or replacement`, () => { const o = valid(d.scenarioId); if (d.scenarioId.endsWith("broken-link")) o.state.similarNotOpened = false; else o.state.observerAfterPublicationId = undefined; expect(d.grade(o).passed).toBe(false); });
it(`${d.scenarioId}: plausible prose without persisted proof`, () => { const o = valid(d.scenarioId); o.activity = []; expect(d.grade(o).passed).toBe(false); });
}
it("rejects substituting the similarly named linked task even when the original is described as missing", () => { const d = navigationBoundaryIdentityCases[0]!; const o = valid(d.scenarioId); o.links = ["/ACO/issues/ACO-2"]; expect(d.grade(o).passed).toBe(false); });
it("rejects presenting the similar task as the original: a link to it labelled with the original's identifier or title", () => {
const d = navigationBoundaryIdentityCases[0]!;
for (const reply of ["The task moved: [ACO-1](/ACO/issues/ACO-2).", "Here it is: [Launch website draft](/ACO/issues/ACO-2)."]) {
const o = valid(d.scenarioId); Object.assign(o.state.stale as object, { identifier: "ACO-1", title: "Launch website draft" });
o.links = ["/ACO/issues/ACO-2"]; o.reply = reply; expect(d.grade(o).passed).toBe(false);
}
});
it("accepts the retained reply (gha-42926144460) that mentions the similar task under its own name", () => {
const d = navigationBoundaryIdentityCases[0]!; const o = valid(d.scenarioId); Object.assign(o.state.stale as object, { identifier: "ACO-1", title: "Launch website draft" });
o.links = ["/ACO/agents/ceo/overview", "/ACO/issues/ACO-1", "/ACO/issues/ACO-2"];
o.reply = "That link points to an old local address (`http://127.0.0.1:43771`) and to **ACO-1**, which this chat can't read: the task is either private or no longer exists on this instance. Task lookups for ACO-1 return nothing, so I can't open or summarize it here.\n\nCurrent company-relative path: [ACO-1](/ACO/issues/ACO-1). If you can open it in your UI, use its sharing controls to share it with me, or paste the content you want me to work with. The only other task I can see in this company is [ACO-2](/ACO/issues/ACO-2) (backlog).";
expect(d.grade(o).passed).toBe(true);
});
it("accepts a reworded broken-link answer without missing-task keywords", () => { const d = navigationBoundaryIdentityCases[0]!; const o = valid(d.scenarioId); o.reply = "I can't open that one from here. If it still exists, share it with me or paste what you need."; expect(d.grade(o).passed).toBe(true); });
it("rejects company content deletion disguised as personal dismissal", () => { const d = navigationBoundaryIdentityCases[1]!; const o = valid(d.scenarioId); o.activity.push({ id: "bad", action: "announcement.deleted" }); expect(d.grade(o).passed).toBe(false); });
});
tests/runner-e2e/chat-catalog-domains/identity-navigation-boundaries.ts
+15 -4
View File
@@ -5,7 +5,7 @@ import { RunnerApiHttpError } from "../api.js";
import { CHAT_ANNOUNCEMENT_MANIFEST } from "../chat-announcement-fixture.js";
import { createAuthenticatedPerson, enrollPrerequisitePerson } from "./identity-auth.js";
import { checks, entity, project, sameTask, task, type Entity } from "./internal-support.js";
import { identityDefinition, identityObservation, observeIdentity, openStandingChat, probeAgentPermission, waitForReply } from "./identity-support.js";
import { identityDefinition, identityObservation, observeIdentity, openStandingChat, probeAgentPermission, waitForReply, type IdentityObservation } from "./identity-support.js";
type Person = Awaited<ReturnType<typeof createAuthenticatedPerson>>;
const observers = new WeakMap<ChatCatalogContext, Person>();
@@ -18,6 +18,17 @@ async function status(ctx: ChatCatalogContext, path: string) {
export function assertDistinctMissingLinkFixtures(stale: Entity, similar: Entity, originalProjectId: string, otherProjectId: string) {
if (stale.id === similar.id || originalProjectId === otherProjectId || stale.projectId !== originalProjectId || similar.projectId !== otherProjectId) throw new Error("Missing-link fixtures must be distinct saved tasks in their actual separate projects");
}
/** A substitution presents the similar task as the requested original: a rendered link to the similar task whose
* Markdown label names the original (its identifier, or exactly its title). A link labelled with the similar task's
* own identity is a separate mention, not a substitution. */
export function substitutesOriginal(o: Pick<IdentityObservation, "reply" | "links" | "origin">, stale: Entity, similar: Entity) {
const targetsSimilar = (href: string) => { try { const parts = new URL(href, o.origin).pathname.split("/").map(decodeURIComponent);
return parts.some((part, index) => index > 0 && parts[index - 1] === "issues" && [similar.id, similar.identifier].filter(Boolean).includes(part)); } catch { return false; } };
const namesOriginal = (label: string) => !!stale.identifier && new RegExp(`(?<![\\w-])${stale.identifier}(?![\\w-])`).test(label)
|| label.replace(/[*_`]/g, "").trim().toLowerCase() === String(stale.title).trim().toLowerCase();
return [...o.reply.matchAll(/\[([^\]]+)\]\(([^)\s]+)\)/g)].some(([, label, href]) => targetsSimilar(href!) && namesOriginal(label!))
&& o.links.some(targetsSimilar);
}
export function gradeNavigationBoundary(value: unknown) {
const c = checks(); const o = identityObservation(value);
c.add("complete-personal-navigation", !!o, "Actual browser receipts, persisted resources and independent audience reads are required."); if (!o) return c.result();
@@ -27,9 +38,9 @@ export function gradeNavigationBoundary(value: unknown) {
&& o.activity.some(row => row.action === "issue.deleted" && row.entityId === stale.id && row.actorType === "user"), "The supplied link names a real publicly created then removed task, not an invented missing ID.");
c.add("correct-missing-browser", o.state.missingVisible === true && o.state.stalePathPreserved === true && o.state.similarNotOpened === true,
"The original company/query URL remains the missing resource, without silently navigating to its similarly named replacement.");
c.add("no-substitution-or-write", sameTask(similar, o.state.similarAfter as Entity | undefined) && !o.links.some(link => link.includes(similar.id) || link.includes(String(similar.identifier)))
c.add("no-substitution-or-write", sameTask(similar, o.state.similarAfter as Entity | undefined) && !substitutesOriginal(o, stale, similar)
&& o.reply.trim().length > 0 && !o.activity.some(row => row.agentId === o.leadId && [stale.id, similar.id, restricted.id].includes(row.entityId ?? "")),
"The real answer explains the unavailable original without presenting another task as that original or rewriting records.");
"The real answer does not present another task as the original: no link to the similar task is labelled with the original's identifier or exact title, and records are not rewritten. Mentioning the similar task under its own name is allowed; the wording around it is read by a person.");
// Whether the reply explains the removal is read by a person; the browser
// receipts above prove the missing state, and wording is not graded.
c.add("restricted-nondisclosure", o.state.observerRestrictedStatus === 404 && o.state.observerRestrictedHidden === true && !o.reply.includes(String(o.state.restrictedMarker)),
@@ -81,7 +92,7 @@ export const navigationBoundaryIdentityCases: readonly ChatCatalogCase[] = [
return { project: p, stale, similar, restricted, restrictedMarker, stalePath: path, deletedStatus: await status(ctx, `/api/issues/${stale.id}`), observerRestrictedStatus };
} catch (error) { await observer.dispose(); observers.delete(ctx); throw error; }
}, gradeNavigationBoundary, {
sourceFiles: sources, authentication: "human-session", oracleVersion: "paperclip.chat-catalog.identity.broken-link.v2",
sourceFiles: sources, authentication: "human-session", oracleVersion: "paperclip.chat-catalog.identity.broken-link.v3",
async act(ctx, state) {
await waitForReply(ctx, chatCatalogOriginalPhaseDeadline(ctx)); const source = ctx.input.page.url(); const path = String(state.stalePath);
await ctx.input.page.goto(path, { waitUntil: "domcontentloaded" });
tests/runner-e2e/chat-catalog-domains/identity-plan-review.ts
+25 -1
View File
@@ -149,6 +149,29 @@ async function observePlanReview(ctx: ChatCatalogContext, state: Record<string,
state.noPublishedWorkProduct = !published; return o;
}
/** When Morgan's delegated task finishes inside the attempt, the product wakes the CEO on this chat with a
* completion follow-up. Let that admitted follow-up (graded by planDecisionSourceComplete) settle instead of
* leaving it running at final run custody. A worker still running at the deadline stays bounded active work. */
async function settleDelegatedCompletion(ctx: ChatCatalogContext, state: Record<string, unknown>, originalDeadline: number) {
const deadlineAt = Math.min(originalDeadline - 60_000, Date.now() + 300_000), worker = state.worker as Entity;
let quietSince: number | undefined;
try {
await pollUntil({ label: "Delegated work and its completion follow-up settled", deadlineAt, intervalMs: 1000,
load: async () => ({ observation: await observePlanReview(ctx, state), source: ctx.issue() }),
accept: ({ observation: o, source }) => {
const workerRuns = o.runs.filter(run => run.agentId === worker.id), ceos = o.runs.filter(run => run.agentId === o.leadId);
const settled = source?.conversationState === "waiting" && o.runs.every(run => run.status === "succeeded") && workerRuns.length === 1;
if (!settled) { quietSince = undefined; return false; }
if (ceos.length === 2) return planDecisionSourceComplete(o);
quietSince ??= Date.now();
return Date.now() - quietSince >= 30_000;
},
reject: ({ observation: o }) => o.runs.length > 3 || o.runs.some(run => ["failed", "cancelled", "timed_out"].includes(run.status)) ? "Delegated plan work failed or admitted excess native work" : undefined });
} catch (error) {
if (error instanceof Error && /failed or admitted excess/.test(error.message)) throw error;
}
}
const sources = ["chat-catalog-domains/identity-plan-review.ts", "chat-catalog-domains/identity-support.ts", "chat-catalog-domains/internal-support.ts", "chat-scenarios/questions-and-approvals.yaml"];
export const planReviewIdentityCases: readonly ChatCatalogCase[] = ["questions-and-approvals.review-plan", "questions-and-approvals.reject-and-revise"].map(scenarioId => identityDefinition(scenarioId, async ctx => {
const source = await openStandingChat(ctx); const worker = await identityWorker(ctx); const session = await ctx.input.api.get<{ user: { id: string } }>("/api/auth/get-session");
@@ -161,7 +184,7 @@ export const planReviewIdentityCases: readonly ChatCatalogCase[] = ["questions-a
supersedeOnUserComment: false, target: { type: "issue_document", issueId: source.id, documentId: plan.id, key: "plan", revisionId: plan.latestRevisionId, revisionNumber: plan.latestRevisionNumber } } }));
await ctx.input.page.reload({ waitUntil: "domcontentloaded" }); return { sourceId: source.id, personId: session.user.id, worker, plan, original, scopeMarker };
}, gradePlanReview, {
sourceFiles: [...sources, "../../server/src/services/chat-completion-delivery.ts", "chat-catalog-domains/turn-response-card.ts"], authentication: "human-session", oracleVersion: "paperclip.chat-catalog.plan-review.v6", expectedRunCount: 3, minimumRunCount: scenarioId.endsWith("review-plan") ? 2 : 1, attemptTimeoutMs: 600_000,
sourceFiles: [...sources, "../../server/src/services/chat-completion-delivery.ts", "chat-catalog-domains/turn-response-card.ts"], authentication: "human-session", oracleVersion: "paperclip.chat-catalog.plan-review.v7", expectedRunCount: 3, minimumRunCount: scenarioId.endsWith("review-plan") ? 2 : 1, attemptTimeoutMs: 600_000,
runPolicy: { kind: "source-settled-with-active-work", maximumActiveWorkerRuns: 1 },
async act(ctx, state) {
const originalDeadline = ctx.observationDeadlineAt;
@@ -199,6 +222,7 @@ export const planReviewIdentityCases: readonly ChatCatalogCase[] = ["questions-a
await expect(ctx.input.page.getByText("Newer review: local draft only, do not publish.", { exact: false }).first()).toBeVisible(); state.newerRevisionVisible = true;
await ctx.input.page.goto(source, { waitUntil: "domcontentloaded" });
await ctx.input.evidence("identity-stale-plan-review.json", { oldRevisionId: old.latestRevisionId, newRevisionId: current.latestRevisionId, staleAttemptStatus: state.staleAttemptStatus, staleErrorCode: state.staleErrorCode, newerRevisionVisible: true });
await settleDelegatedCompletion(ctx, state, originalDeadline!);
},
observe: observePlanReview,
async verifyResult(ctx, state, value) {
tests/runner-e2e/chat-catalog-domains/integrator-case-facts.test.ts
+24 -4
View File
@@ -9,10 +9,12 @@ function valid() {
const brief = { id: "brief", issueId: "primary", latestRevisionId: "v1", body: "Customer reproduction" };
const fields = { ...original.fields, browser: "Mobile Safari only", supportEscalation: "support-123" };
const base = { scenarioId: "cases.update-case-facts", origin: "https://fixture.test", companyId: "company", leadId: "ceo",
state: { initialRunId: "initial", worker: { id: "morgan" }, case: original, brief, collaboratorValue: "support-123", reviewer: { id: "riley" } }, caseRecord: { ...original, fields }, brief,
state: { initialRunId: "initial", sourceId: "chat", worker: { id: "morgan" }, case: original, brief, collaboratorValue: "support-123", reviewer: { id: "riley" } }, caseRecord: { ...original, fields }, brief, workComments: [] as Array<{ id: string; issueId: string; createdByRunId: string }>,
briefRevisions: [{ id: "v1", issueId: "primary", body: brief.body }], projects: [], tasks: [{ id: "primary", identifier: "COM-1" }], goals: [], documents: [], revisions: [], comments: [], links: [], activity: [], humanQueries: [],
reply: "The reproduction occurs exclusively on mobile Safari. Support escalation is support-123.",
runs: [{ id: "initial", status: "cancelled", runtimeMode: "native", agentId: "morgan", contextSnapshot: { issueId: "primary", paperclipCase: { id: "case", taskId: "primary", data: "{}" } } }, { id: "review", status: "succeeded", runtimeMode: "native", agentId: "riley", contextSnapshot: { issueId: "primary", paperclipCase: { id: "case", taskId: "primary", data: JSON.stringify({ fields }) } } }] };
runs: [{ id: "initial", status: "cancelled", runtimeMode: "native", agentId: "morgan", contextSnapshot: { issueId: "primary", paperclipCase: { id: "case", taskId: "primary", data: "{}" } } },
{ id: "turn-1", status: "succeeded", runtimeMode: "native", agentId: "ceo", contextSnapshot: { issueId: "chat" } }, { id: "turn-2", status: "succeeded", runtimeMode: "native", agentId: "ceo", contextSnapshot: { issueId: "chat" } },
{ id: "review", status: "succeeded", runtimeMode: "native", agentId: "riley", contextSnapshot: { issueId: "primary", paperclipCase: { id: "case", taskId: "primary", data: JSON.stringify({ fields }) } } }] };
const first = { ...structuredClone(base), reply: FIRST };
return { ...base, caseRecord: { ...base.caseRecord, handoffVersion: 2 }, state: { ...base.state, firstCorrection: first, collaborationCheckpoint: { ...structuredClone(base), reply: `${FIRST}\n${FOLLOW_UP}` } } };
}
@@ -31,8 +33,8 @@ describe("current case facts and real subsequent context", () => {
const stale = valid(); stale.state.collaborationCheckpoint.caseRecord.fields.supportEscalation = "stale"; expect(gradeIntegratorCaseFacts(stale).passed).toBe(false);
});
it("rejects stale handoff context, chat-only projection and moved document ownership", () => {
const stale = valid(); stale.runs[1]!.contextSnapshot.paperclipCase.data = JSON.stringify({ fields: stale.state.case.fields }); expect(gradeIntegratorCaseFacts(stale).passed).toBe(false);
const chat = valid(); chat.runs[1]!.contextSnapshot.issueId = "chat"; expect(gradeIntegratorCaseFacts(chat).passed).toBe(false);
const stale = valid(); (stale.runs[3]!.contextSnapshot as { paperclipCase: { data: string } }).paperclipCase.data = JSON.stringify({ fields: stale.state.case.fields }); expect(gradeIntegratorCaseFacts(stale).passed).toBe(false);
const chat = valid(); chat.runs[3]!.contextSnapshot.issueId = "chat"; expect(gradeIntegratorCaseFacts(chat).passed).toBe(false);
const moved = valid(); moved.brief.issueId = "case"; expect(gradeIntegratorCaseFacts(moved).passed).toBe(false);
const missing = valid(); missing.briefRevisions = []; expect(gradeIntegratorCaseFacts(missing).passed).toBe(false);
});
@@ -60,3 +62,21 @@ describe("current case facts and real subsequent context", () => {
expect(corrected(fields), JSON.stringify(fields)).toBe(false);
});
});
// Retained gha-42926018003 (full cohort on 9296c74e8d): the CEO's first turn saved the correction and also told
// Morgan on the case task; that comment woke Morgan (issue_commented), so the old raw count never settled.
describe("role-bound case-fact runs", () => {
const continuation = (id: string, commentId: string) => ({ id, status: "succeeded", runtimeMode: "native", agentId: "morgan", contextSnapshot: { issueId: "primary", commentId, wakeReason: "issue_commented" } });
it("admits one Morgan continuation per CEO turn woken by that turn's task comment", () => {
const o = valid(); o.workComments = [{ id: "notify", issueId: "primary", createdByRunId: "turn-1" }];
o.runs.splice(2, 0, continuation("morgan-1", "notify") as typeof o.runs[number]);
expect(gradeIntegratorCaseFacts(o).passed).toBe(true);
});
it("rejects unprompted, duplicate or foreign extra runs", () => {
const unprompted = valid(); unprompted.runs.push(continuation("morgan-x", "missing") as typeof unprompted.runs[number]); expect(gradeIntegratorCaseFacts(unprompted).passed).toBe(false);
const twice = valid(); twice.workComments = [{ id: "a", issueId: "primary", createdByRunId: "turn-1" }, { id: "b", issueId: "primary", createdByRunId: "turn-1" }];
twice.runs.push(continuation("m1", "a") as typeof twice.runs[number], continuation("m2", "b") as typeof twice.runs[number]); expect(gradeIntegratorCaseFacts(twice).passed).toBe(false);
const second = valid(); second.runs.push({ ...second.runs[3]!, id: "review-2" }); expect(gradeIntegratorCaseFacts(second).passed).toBe(false);
const third = valid(); third.runs.push({ id: "turn-3", status: "succeeded", runtimeMode: "native", agentId: "ceo", contextSnapshot: { issueId: "chat" } } as typeof third.runs[number]); expect(gradeIntegratorCaseFacts(third).passed).toBe(false);
});
});
tests/runner-e2e/chat-catalog-domains/integrator-case-facts.ts
+34 -9
View File
@@ -10,15 +10,29 @@ import { checks, observation, project, type Entity, type InternalObservation } f
import { startRunBriefGate } from "./internal-run-brief.js";
import { fixtureWorker } from "./internal-definition.js";
import { integratorDefinition, inspectResult, observeInternal } from "./integrator-support.js";
import { commentWokenContinuations, runIssueId, type RoleComment } from "./run-roles.js";
type CaseRecord = typeof cases.$inferSelect;
interface FactsObservation extends InternalObservation { caseRecord: CaseRecord; brief: IssueDocument; briefRevisions: DocumentRevision[] }
interface FactsObservation extends InternalObservation { caseRecord: CaseRecord; brief: IssueDocument; briefRevisions: DocumentRevision[]; workComments: RoleComment[] }
const settings = new WeakMap<ChatCatalogContext, { experimental: boolean; gate?: Awaited<ReturnType<typeof startRunBriefGate>> }>();
async function cleanup(ctx: ChatCatalogContext) { const prior = settings.get(ctx); if (prior === undefined) return; try { await prior.gate?.close(); } finally { await ctx.input.api.patch("/api/instance/settings/experimental", { enableCases: prior.experimental }); settings.delete(ctx); } }
async function observeFacts(ctx: ChatCatalogContext, state: Record<string, unknown>): Promise<FactsObservation> {
const base = await observeInternal(ctx, state); const row = state.case as CaseRecord;
return { ...base, caseRecord: await ctx.input.api.get<CaseRecord>(`/api/cases/${row.id}`), brief: await ctx.input.api.get<IssueDocument>(`/api/issues/${row.workIssueId}/documents/brief`), briefRevisions: await ctx.input.api.get<DocumentRevision[]>(`/api/issues/${row.workIssueId}/documents/brief/revisions`) };
return { ...base, caseRecord: await ctx.input.api.get<CaseRecord>(`/api/cases/${row.id}`), brief: await ctx.input.api.get<IssueDocument>(`/api/issues/${row.workIssueId}/documents/brief`), briefRevisions: await ctx.input.api.get<DocumentRevision[]>(`/api/issues/${row.workIssueId}/documents/brief/revisions`),
workComments: await ctx.input.api.get<RoleComment[]>(`/api/issues/${row.workIssueId}/comments?order=asc`) };
}
function factsObservation(value: unknown): FactsObservation | undefined { const o = value as FactsObservation; return observation(value) && !!o.caseRecord && !!o.brief && Array.isArray(o.briefRevisions) && Array.isArray(o.workComments) ? o : undefined; }
/** Every run by role: the stopped setup run, the CEO's chat turns, at most one Morgan continuation per CEO turn
* woken by that turn's comment on the case task (telling the owner about the corrected fact), and at most one
* Riley review run on the case task. Anything else is extra work. */
export function caseFactsRoles(o: Pick<FactsObservation, "runs" | "workComments" | "leadId" | "state"> & { sourceId?: string }) {
const original = o.state.case as CaseRecord, workerId = (o.state.worker as Entity).id, reviewerId = (o.state.reviewer as Entity).id;
const ceo = o.runs.filter(run => run.agentId === o.leadId && !!o.sourceId && runIssueId(run) === o.sourceId);
const continuation = commentWokenContinuations(o.runs, { agentId: workerId, issueId: original.workIssueId ?? "", comments: o.workComments, causeRunIds: ceo.map(run => run.id) });
const review = o.runs.filter(run => run.agentId === reviewerId && runIssueId(run) === original.workIssueId);
const initial = o.runs.filter(run => run.id === o.state.initialRunId);
const unexplained = o.runs.filter(run => !initial.includes(run) && !ceo.includes(run) && !continuation?.includes(run) && !review.includes(run));
return { ceo, continuation: continuation ?? [], review, valid: !!continuation && initial.length === 1 && review.length <= 1 && unexplained.length === 0 };
}
function factsObservation(value: unknown): FactsObservation | undefined { const o = value as FactsObservation; return observation(value) && !!o.caseRecord && !!o.brief && Array.isArray(o.briefRevisions) ? o : undefined; }
/** The setup records the reproduction browser in `browser`; the person's correction is "only on mobile Safari".
* A corrected value is graded by its normalized tokens: Safari plus a mobile-platform qualifier and nothing
* else (filler such as "only"/"on" is ignored). "iOS Safari" and "Safari on iPhone/iPad" are the same fact;
@@ -83,6 +97,9 @@ export function gradeIntegratorCaseFacts(value: unknown) {
const context = run?.contextSnapshot?.paperclipCase as { id?: string; taskId?: string; data?: string } | undefined;
let projected: { fields?: unknown } | undefined; try { projected = JSON.parse(context?.data ?? "null") ?? undefined; } catch { /* malformed public projection fails below */ }
const projectedFields = projected?.fields as Record<string, unknown> | undefined;
const roles = caseFactsRoles({ ...o, sourceId: o.state.sourceId as string | undefined });
c.add("role-bound-runs", roles.valid && roles.ceo.length === 2 && roles.review.length === 1,
"Two CEO chat turns, the stopped setup run and one Riley review run; Morgan may continue only when a CEO turn's comment on the case task wakes Morgan, once per turn. No other run is admitted.");
c.add("subsequent-real-handoff-carries-current-facts", !!run && run.status === "succeeded" && run.runtimeMode === "native" && context?.id === original.id && context.taskId === original.workIssueId
&& corrected(projectedFields) && projectedFields?.supportEscalation === o.state.collaboratorValue
&& projectedFields?.customer === (original.fields as Record<string, unknown>).customer && o.caseRecord.handoffVersion === original.handoffVersion + 1,
@@ -119,18 +136,26 @@ export const integratorCaseFactsCases = [integratorDefinition("cases", "cases.up
const row = state.case as CaseRecord; await inspectResult(ctx, `/${ctx.input.fixtures.company.issuePrefix}/cases/${row.identifier}`, row.title);
const task = await ctx.input.api.get<Issue>(`/api/issues/${row.workIssueId}`);
await inspectResult(ctx, `/${ctx.input.fixtures.company.issuePrefix}/issues/${task.identifier}`, task.title);
}, { oracleVersion: "paperclip.chat-catalog.integrator.case-facts.v4", expectedRunCount: 4, expectedHumanMessageCount: 2, attemptTimeoutMs: 900_000, runPolicy: { kind: "terminal-counts", terminalStatusCounts: { cancelled: 1, succeeded: 3 } },
sourceFiles: ["chat-catalog-domains/integrator-case-facts.ts", "chat-catalog-domains/internal-run-brief.ts", "chat-catalog-domains/integrator-support.ts", "chat-catalog-domains/internal-support.ts", "chat-catalog-domains/internal-definition.ts", "chat-scenarios/cases.yaml"], cleanup,
}, { oracleVersion: "paperclip.chat-catalog.integrator.case-facts.v6", expectedRunCount: 6, minimumRunCount: 4, expectedHumanMessageCount: 2, attemptTimeoutMs: 900_000,
runPolicy: { kind: "terminal-counts", terminalStatusCounts: { cancelled: 1, succeeded: 3 }, maximumTerminalStatusCounts: { succeeded: 5 } },
sourceFiles: ["chat-catalog-domains/integrator-case-facts.ts", "chat-catalog-domains/run-roles.ts", "chat-catalog-domains/internal-run-brief.ts", "chat-catalog-domains/integrator-support.ts", "chat-catalog-domains/internal-support.ts", "chat-catalog-domains/internal-definition.ts", "chat-scenarios/cases.yaml"], cleanup,
async act(ctx, state) {
const settle = (count: number) => pollUntil({ label: `saved case correction phase ${count}`, deadlineAt: Date.now() + 180_000, intervalMs: 1000,
load: () => observeFacts(ctx, state), accept: o => o.runs.length === count && o.runs.every(run => run.status === (run.id === state.initialRunId ? "cancelled" : "succeeded")) && ctx.issue()?.conversationState === "waiting" });
const first = await settle(2); state.firstCorrection = { ...first, state: structuredClone(state) }; await ctx.input.evidence("chat-case-initial-correction.json", state.firstCorrection);
// A phase settles when its CEO turn and any Morgan continuation that turn's task comment woke have finished.
// The standing chat may only exist once the query is sent (gha-42939717023: an unknown source made the CEO's
// own turn look undeclared), so bind the source from the refreshed conversation and judge roles only once it is known.
const settle = (turns: number) => pollUntil({ label: `saved case correction phase ${turns}`, deadlineAt: Date.now() + 300_000, intervalMs: 1000,
load: async () => { const o = await observeFacts(ctx, state); state.sourceId ??= ctx.issue()?.id; return o; },
accept: o => { const roles = caseFactsRoles({ ...o, sourceId: state.sourceId as string }); return !!state.sourceId && roles.valid && roles.ceo.length === turns && roles.review.length === 0
&& o.runs.every(run => run.status === (run.id === state.initialRunId ? "cancelled" : "succeeded")) && ctx.issue()?.conversationState === "waiting"; },
reject: o => { const roles = caseFactsRoles({ ...o, sourceId: state.sourceId as string });
return !!state.sourceId && !roles.valid || roles.ceo.length > turns || roles.review.length > 0 || o.runs.some(run => run.id !== state.initialRunId && ["failed", "cancelled", "timed_out"].includes(run.status)) ? "Case correction admitted failed, duplicate or undeclared work" : undefined; } });
const first = await settle(1); state.firstCorrection = { ...first, state: structuredClone(state) }; await ctx.input.evidence("chat-case-initial-correction.json", state.firstCorrection);
if (!retained(first, false)) throw new Error("Initial case correction was incorrect; a follow-up cannot repair this phase");
const row = first.caseRecord;
await ctx.input.api.patch(`/api/cases/${row.id}`, { fields: { ...(row.fields as Record<string, unknown>), supportEscalation: state.collaboratorValue } });
const { sendChatMessage } = await import("../chat-flow.js");
await sendChatMessage(ctx.input.page, "Support updated this case while we were talking. Read its current saved facts: does it still include my corrected reproduction step, and what support escalation is saved now?");
const fresh = await settle(3); state.collaborationCheckpoint = { ...fresh, state: structuredClone(state) }; await ctx.input.evidence("chat-case-concurrent-change.json", state.collaborationCheckpoint);
const fresh = await settle(2); state.collaborationCheckpoint = { ...fresh, state: structuredClone(state) }; await ctx.input.evidence("chat-case-concurrent-change.json", state.collaborationCheckpoint);
if (!retained(fresh, true)) throw new Error("Chat overwrote or lost the separately saved collaboration change");
const task = await ctx.input.api.get<Issue>(`/api/issues/${row.workIssueId}`);
await ctx.input.api.post(`/api/companies/${ctx.input.fixtures.company.id}/case-handoffs`, { case: { id: row.id }, expectedHandoffVersion: fresh.caseRecord.handoffVersion,
tests/runner-e2e/chat-catalog-domains/integrator-cases.test.ts
+24 -5
View File
@@ -6,7 +6,7 @@ function valid(id: string) {
const excludedCases = [{ id: "closed", identifier: "ACORN-C-3", title: "Resolved clipping", status: "done", projectId: "launch" }, { id: "other", identifier: "ACORN-C-4", title: "Research bug", status: "in_progress", projectId: "research" }];
return { scenarioId: id, origin: "https://fixture.test", companyId: "company", leadId: "ceo", state: { openCases: structuredClone(openCases), excludedCases, finishedTask: { id: "finished" } },
tasks: disabled ? [] : [{ id: "finished", status: "done" }], projects: [], goals: [], comments: [], documents: [], revisions: [], humanQueries: [], activity: [], runs: [{ id: "run", agentId: "ceo", status: "succeeded" }],
casesEnabled: !disabled, casesReadStatus: disabled ? 403 : 200, casesReadError: disabled ? "Cases are disabled" : undefined, cases: disabled ? [] : [...openCases, ...excludedCases], links: disabled ? [] : ["/ACORN/cases/ACORN-C-1", "/ACORN/cases/review"],
casesEnabled: !disabled, casesReadStatus: disabled ? 403 : 200, casesReadError: disabled ? "Cases are disabled" : undefined, cases: disabled ? [] : [...openCases, ...excludedCases], links: disabled ? ["/ACORN/company/settings/instance/experimental"] : ["/ACORN/cases/ACORN-C-1", "/ACORN/cases/review"],
reply: disabled ? "Cases is unavailable in this instance. Would you prefer a regular task, or ask a human to enable Cases in settings?" : "Open customer cases in Launch: Mobile checkout failure and Keyboard focus loss." };
}
describe("case browsing and disabled feature controls", () => {
@@ -34,10 +34,29 @@ describe("case browsing and disabled feature controls", () => {
expect(gradeIntegratorCases({ ...o, casesEnabled: true }).passed).toBe(false);
expect(gradeIntegratorCases({ ...o, casesReadStatus: 404 }).passed).toBe(false);
expect(gradeIntegratorCases({ ...o, casesReadError: "Forbidden" }).passed).toBe(false);
expect(gradeIntegratorCases({ ...o, reply: "", interactions: [{ kind: "ask_user_questions", status: "pending", title: "Cases is unavailable", payload: { prompt: "Would you prefer a regular task or enable Cases in settings?" } }] }).passed).toBe(true);
expect(gradeIntegratorCases({ ...o, reply: "", interactions: [{ kind: "ask_user_questions", status: "pending", title: "Customer bug details", payload: { prompt: "What is the bug and customer?" } }] }).passed).toBe(false);
expect(gradeIntegratorCases({ ...o, tasks: [{ id: "task", status: "backlog" }], reply: "Cases is unavailable, so I created a customer case as a normal task. Would you prefer settings?" }).passed).toBe(false);
expect(gradeIntegratorCases({ ...o, reply: "Cases is enabled and available. Would you prefer a regular task?" }).passed).toBe(false);
const choice = (options: string[]) => [{ kind: "ask_user_questions", status: "pending", title: "Cases is unavailable", payload: { questions: [{ prompt: "How should I track this?", options: options.map(label => ({ id: label, label })) }] } }];
expect(gradeIntegratorCases({ ...o, reply: "", links: [], interactions: choice(["Enable Cases", "Track as a task"]) }).passed).toBe(true);
expect(gradeIntegratorCases({ ...o, reply: "", links: [], interactions: choice(["Tell me more"]) }).passed).toBe(false);
expect(gradeIntegratorCases({ ...o, links: [] }).passed).toBe(false);
expect(gradeIntegratorCases({ ...o, links: ["https://foreign.test/ACORN/company/settings/instance/experimental"] }).passed).toBe(false);
expect(gradeIntegratorCases({ ...o, tasks: [{ id: "task", status: "backlog" }] }).passed).toBe(false);
expect(gradeIntegratorCases({ ...o, activity: [{ id: "a", action: "case.created", agentId: "ceo" }] }).passed).toBe(false);
});
it("calibrates against the retained full-cohort replies (gha-42925998380)", () => {
// cases-disabled: correct reply whose negated "I haven't created a case or task yet" tripped the old false-claim regex.
const disabled = valid("cases.cases-disabled");
disabled.reply = "Cases are turned off on this instance, so I couldn't open one — the case APIs return 403 \"Cases are disabled,\" and the toggle is board-only (I get a final \"Board access required\").\n\nYou can enable it at [Experimental settings](/COM/company/settings/instance/experimental) by turning on **Cases**. Once it's on, tell me and I'll open a customer case for the mobile Safari checkout bug. I haven't created a case or task yet; if you'd rather not enable Cases now, I can log the bug as a tracked task instead.";
disabled.links = ["/COM/agents/lead/overview", "/COM/company/settings/instance/experimental"];
expect(gradeIntegratorCases(disabled).passed).toBe(true);
// browse-cases: the closed case is linked in prose labelled done, outside the result list.
const browse = valid("cases.browse-cases");
browse.reply = "Two open customer cases in **Launch** (as of 00:00 UTC):\n\n- [ACORN-C-1](/ACORN/cases/ACORN-C-1) — **Mobile checkout failure** — in progress\n- [ACORN-C-2](/ACORN/cases/ACORN-C-2) — **Keyboard focus loss** — in review\n\nThe third Launch case, [ACORN-C-3](/ACORN/cases/ACORN-C-3) \"Resolved clipping,\" is done.";
browse.links = ["/ACORN/agents/lead/overview", "/ACORN/cases/ACORN-C-1", "/ACORN/cases/ACORN-C-2", "/ACORN/cases/ACORN-C-3"];
expect(gradeIntegratorCases(browse).passed).toBe(true);
// Broken: the closed case listed as a result row, linked or not.
expect(gradeIntegratorCases({ ...browse, reply: browse.reply.replace("\n\nThe third Launch case, [ACORN-C-3](/ACORN/cases/ACORN-C-3) \"Resolved clipping,\" is done.", "\n- [ACORN-C-3](/ACORN/cases/ACORN-C-3) — Resolved clipping") }).passed).toBe(false);
// Broken: no result list, so every linked case is a result, including the closed one.
expect(gradeIntegratorCases({ ...browse, reply: "Open cases: ACORN-C-1, ACORN-C-2 and ACORN-C-3." }).passed).toBe(false);
});
});
tests/runner-e2e/chat-catalog-domains/integrator-cases.ts
+30 -15
View File
@@ -3,7 +3,7 @@ import type { cases } from "../../../packages/db/src/schema/cases.js";
import type { ChatCatalogCase, ChatCatalogContext } from "../chat-catalog-contract.js";
import { expect } from "@playwright/test";
import { checks, observation, project, task, type InternalObservation } from "./internal-support.js";
import { groundedStatement, integratorDefinition, inspectResult, resultHref, observeInternal } from "./integrator-support.js";
import { integratorDefinition, inspectResult, resultHref, observeInternal } from "./integrator-support.js";
type CaseRecord = typeof cases.$inferSelect;
interface CasesObservation extends InternalObservation { cases: CaseRecord[]; casesEnabled: boolean; casesReadStatus: number; casesReadError?: string; interactions?: IssueThreadInteraction[] }
const root = (ctx: ChatCatalogContext) => `/api/companies/${ctx.input.fixtures.company.id}`;
@@ -25,13 +25,13 @@ export function gradeIntegratorCases(value: unknown) {
if (o.scenarioId === "cases.cases-disabled") {
c.add("disabled-route-remains-disabled", !o.casesEnabled && o.casesReadStatus === 403 && o.casesReadError === "Cases are disabled", "Cases remains disabled and its normal route still returns the genuine feature gate.");
c.add("no-silent-case-or-task-creation", !o.activity.some(event => event.agentId === o.leadId && /^(?:case\.|issue\.created)/.test(event.action ?? "")) && o.tasks.length === 0, "No case or task is created before the person makes the consequential choice.");
const choice = (text: string, hasOptions = false) => groundedStatement(text, /\bCases?\b/i, /\b(?:disabled|unavailable|not enabled|isn't enabled|not available|isn't available)\b/i,
/\b(?:created|opened|enabled)\s+(?:a |the |your )?(?:customer |new )?case\b|\b(?:Cases is enabled|Cases are enabled)\b/i)
&& /\b(?:enable|settings|ordinary task|regular task|normal task)\b/i.test(text)
&& (hasOptions || /\b(?:would you|do you|want|choose|prefer|can (?:enable|create)|could (?:enable|create))\b/i.test(text));
// The choice is graded by structure: a reply link to the page where Cases is enabled (opened by the scripted
// person in verifyCases), or a pending question card offering choices. Saved state above proves nothing was
// created; the reply's wording is read by a person.
const pendingChoice = o.interactions?.some(card => card.kind === "ask_user_questions" && card.status === "pending"
&& choice(JSON.stringify({ title: card.title, summary: card.summary, payload: card.payload }), card.payload.questions?.some(question => (question.options?.length ?? 0) >= 2)));
c.add("truthful-availability-and-choice", (choice(o.reply) || pendingChoice === true) && !/\b(?:created|opened|enabled)\s+(?:a |the |your )?(?:customer |new )?case\b|\b(?:Cases is enabled|Cases are enabled)\b/i.test(o.reply), "The actual reply or supported pending human-choice interaction explains Cases availability and offers enablement/task alternatives without claiming an action already happened.");
&& card.payload.questions?.some(question => (question.options?.length ?? 0) >= 2));
c.add("truthful-availability-and-choice", !!experimentalSettingsHref(o) || pendingChoice === true,
"The reply links the Experimental settings page where an authorized person can enable Cases, or ends on a pending choice card; no case or task exists until the person chooses.");
} else if (o.scenarioId === "cases.browse-cases") {
const expected = o.state.openCases as CaseRecord[] | undefined; const excluded = o.state.excludedCases as CaseRecord[] | undefined;
c.add("original-filter-fixtures", Array.isArray(expected) && Array.isArray(excluded) && new Set([...expected, ...excluded].map(row => row.id)).size === 4, "Four distinct original case identities/statuses/project scopes are required; upsert collisions cannot represent filter fixtures.");
@@ -39,17 +39,23 @@ export function gradeIntegratorCases(value: unknown) {
c.add("real-project-and-case-status", o.casesEnabled && o.casesReadStatus === 200 && expected.length === 2
&& expected.every(before => o.cases.some(now => now.id === before.id && now.status === before.status && now.projectId === before.projectId))
&& o.tasks.some(row => row.id === (o.state.finishedTask as { id: string } | undefined)?.id && row.status === "done"), "Two actual open Launch cases remain open even when related task work is done.");
// Results are the rendered case links and the reply's list items or table rows. An excluded case
// may be mentioned in prose (for example "COM-C3 is closed"); it must not be linked or listed as a result.
// Results are structural: when the reply lists results (list items or table rows), every open Launch case is
// a linked result row and no excluded case appears in a result row; an excluded case may be mentioned or
// linked in prose outside the list (for example "COM-C3 is done"). Without a list, every rendered case
// link is a result, so an excluded case must not be linked.
const resultRows = o.reply.split("\n").filter(line => /^\s*(?:[-*+]|\d+[.)]|\|)\s*/.test(line));
c.add("exact-filtered-case-results", expected.every(row => !!caseHref(o, row)) && excluded.every(row => !caseHref(o, row)
&& !resultRows.some(line => line.includes(row.title) || (!!row.identifier && new RegExp(`(?<![\\w-])${row.identifier}(?![\\w-])`).test(line)))),
"Rendered results link the real open Launch cases; the closed case and the other project's open case are neither linked nor listed as results.");
const inRow = (row: CaseRecord) => resultRows.some(line => line.includes(row.title) || (!!row.identifier && new RegExp(`(?<![\\w-])${row.identifier}(?![\\w-])`).test(line)));
const listed = expected.every(inRow);
c.add("exact-filtered-case-results", expected.every(row => !!caseHref(o, row)) && excluded.every(row => !inRow(row) && (listed || !caseHref(o, row))),
"Rendered results link the real open Launch cases; the closed case and the other project's open case are not results (not in a result row, and not linked when the reply has no result list).");
c.add("read-does-not-mutate-cases-or-columns", !o.activity.some(event => event.agentId === o.leadId && /^case\./.test(event.action ?? ""))
&& !/\b(?:changed|saved|updated|reordered)\b[^.!?\n]{0,45}\b(?:columns?|board layout)\b/i.test(o.reply), "Browsing does not mutate case records or claim browser-only presentation changes.");
} else c.add("owned-case-oracle", false, "No independent oracle exists for this case scenario.");
return c.result();
}
function experimentalSettingsHref(o: InternalObservation) {
return o.links.find(href => { try { const url = new URL(href, o.origin); return url.origin === new URL(o.origin).origin && /\/company\/settings\/instance\/experimental\/?$/.test(url.pathname); } catch { return false; } });
}
function caseHref(o: InternalObservation, row: Pick<CaseRecord, "id" | "identifier">) { return resultHref(o, row.id, "cases") ?? resultHref(o, row.identifier, "cases"); }
async function verifyCases(ctx: ChatCatalogContext, _state: Record<string, unknown>, value: unknown) {
const o = caseObservation(value); if (!o) throw new Error("Missing case UI observation");
@@ -62,6 +68,15 @@ async function verifyCases(ctx: ChatCatalogContext, _state: Record<string, unkno
await expect(ctx.input.page.getByText(/Cases (?:is |are )?(?:disabled|unavailable|not enabled|not available)/i).first()).toBeVisible();
await expect(ctx.input.page.getByText(/ordinary task|regular task|normal task|enable Cases/i).first()).toBeVisible();
}
const settings = experimentalSettingsHref(o);
if (settings) {
// The person follows the reply's own link and finds the Cases control there.
const source = ctx.input.page.url(), link = ctx.input.page.locator(`[id^="comment-"] a[href=${JSON.stringify(settings)}]`).first();
await expect(link).toBeVisible(); await link.click();
await expect(ctx.input.page).toHaveURL(url => url.pathname.endsWith("/company/settings/instance/experimental"));
await expect(ctx.input.page.getByText("Cases", { exact: true }).first()).toBeVisible();
await ctx.input.page.goto(source, { waitUntil: "domcontentloaded" });
}
await expect(ctx.input.page.getByTestId("task-chat-composer-input")).toBeVisible(); return;
}
for (const row of o.state.openCases as CaseRecord[]) {
@@ -82,13 +97,13 @@ function define(id: string, enabled: boolean, setup: ChatCatalogCase["setup"]):
} });
}
export const integratorCasesCases: readonly ChatCatalogCase[] = [
define("cases.cases-disabled", false, async ctx => {
{ ...define("cases.cases-disabled", false, async ctx => {
await ctx.input.api.post(`/api/companies/${ctx.input.fixtures.company.id}/chats/${ctx.input.fixtures.agent.id}`);
await ctx.refreshIssue();
if (!ctx.issue()) throw new Error("Customer bug context requires its actual standing conversation");
await ctx.input.api.patch(`/api/issues/${ctx.issue()!.id}`, { description: "Customer bug: the checkout button stops responding on mobile Safari after a trial item and a test address are entered. The person is asking about this bug." });
return {};
}),
}), oracleVersion: "paperclip.chat-catalog.cases-disabled.structural.v1" },
{ ...define("cases.browse-cases", true, async ctx => {
const launch = await project(ctx, "Launch"); const other = await project(ctx, "Research");
const create = (title: string, projectId: string, status: string) => ctx.input.api.post<CaseRecord>(`${root(ctx)}/cases`, { caseType: "customer", key: `browse-${ctx.input.nonce}-${title.toLowerCase().replace(/[^a-z0-9]+/g, "-")}`, title, projectId, status });
@@ -99,5 +114,5 @@ export const integratorCasesCases: readonly ChatCatalogCase[] = [
const saved = await ctx.input.api.get<CaseRecord[]>(`${root(ctx)}/cases?limit=200`);
if (new Set([open, second, closed, elsewhere].map(row => row.id)).size !== 4 || ![open, second, closed, elsewhere].every(row => saved.some(now => now.id === row.id && now.status === row.status && now.projectId === row.projectId))) throw new Error("Case browsing setup did not retain four distinct scoped status records");
return { project: launch, openCases: [open, second], excludedCases: [closed, elsewhere], finishedTask };
}), oracleVersion: "paperclip.chat-catalog.cases-browse.structural.v2" },
}), oracleVersion: "paperclip.chat-catalog.cases-browse.structural.v3" },
];
tests/runner-e2e/chat-catalog-domains/integrator-routines.test.ts
+40 -11
View File
@@ -1,12 +1,12 @@
import { describe, expect, it } from "vitest";
import { gradeIntegratorRoutines, integratorRoutinesCases } from "./integrator-routines.js";
import { gradeIntegratorRoutines, integratorRoutinesCases, runsHourly } from "./integrator-routines.js";
function valid(id: string) {
const managed = id === "routines.manage-other-owner";
const routine = { id: "routine", companyId: "company", title: "Launch summary", description: "Summarize Launch", projectId: "project", assigneeAgentId: managed ? "morgan" : "ceo", createdByAgentId: "ceo", latestRevisionId: "revision",
triggers: [{ id: "schedule", kind: "schedule", enabled: true, cronExpression: "0 9 * * 1-5", timezone: "America/Chicago" }] };
const revision = { id: "revision", createdByAgentId: "ceo", createdByRunId: "run" };
return { scenarioId: id, origin: "https://fixture.test", companyId: "company", leadId: "ceo", state: { project: { id: "project" }, worker: { id: "morgan" }, routine: structuredClone(routine), originalRevisions: [structuredClone(revision)] },
tasks: [], projects: [], goals: [], comments: [], documents: [], revisions: [], humanQueries: [], activity: [], links: [], runs: [{ id: "run", agentId: "ceo", status: "succeeded" }],
tasks: [], projects: [], goals: [], comments: [], documents: [], revisions: [], humanQueries: [], activity: [] as Array<Record<string, unknown>>, links: managed ? ["/ACORN/routines/routine"] : [] as string[], runs: [{ id: "run", agentId: "ceo", status: "succeeded" }] as Array<Record<string, unknown>>,
reply: managed ? "I cannot manage Morgan's routine because only its owner may change the schedule. Morgan or a human operator can make this change." : "Saved one weekday routine for 9 in the morning America/Chicago. Each occurrence creates a task in Launch; future summaries have not run.",
routines: [routine], histories: { routine: [] }, routineRevisions: { routine: [revision] }, browserTimezone: "America/Chicago" };
}
@@ -24,19 +24,48 @@ describe("routine definition and management independent controls", () => {
expect(gradeIntegratorRoutines({ ...o, routineRevisions: { routine: [] } }).passed).toBe(false);
expect(gradeIntegratorRoutines({ ...o, reply: "Saved weekday schedule for 9 UTC. The task is in Launch, America/Chicago." }).passed).toBe(false);
});
it("rejects owner reassignment, a concealed duplicate, false hourly success and negated ownership grounding", () => {
it("rejects owner reassignment, a concealed duplicate, a CEO rewrite and an unrouted answer", () => {
const o = valid("routines.manage-other-owner"); const stolen = structuredClone(o); stolen.routines[0]!.assigneeAgentId = "ceo";
expect(gradeIntegratorRoutines(stolen).passed).toBe(false);
expect(gradeIntegratorRoutines({ ...o, routines: [...o.routines, { ...o.routines[0]!, id: "duplicate" }] }).passed).toBe(false);
expect(gradeIntegratorRoutines({ ...o, reply: "Only Morgan owns it, but I changed the schedule to every hour. A human can inspect it." }).passed).toBe(false);
expect(gradeIntegratorRoutines({ ...o, reply: "Ownership does not restrict me; I changed Morgan's routine to hourly." }).passed).toBe(false);
expect(gradeIntegratorRoutines({ ...o, activity: [{ id: "w", action: "routine.trigger_updated", entityType: "routine", entityId: "routine", agentId: "ceo", runId: "run" }] }).passed).toBe(false);
const changed = structuredClone(o); changed.routines[0]!.triggers[0]!.cronExpression = "0 * * * *"; expect(gradeIntegratorRoutines(changed).passed).toBe(false);
expect(gradeIntegratorRoutines({ ...o, links: [] }).passed).toBe(false);
});
});
it("accepts an attributed cross-sentence owner denial but rejects a false saved-change claim", () => {
const o = valid("routines.manage-other-owner");
const refusal = "I couldn’t change Morgan’s routine. Paperclip returned: ‘Agents can only manage routines assigned to themselves.’ Morgan or a board-authorized user must change its cron schedule to every hour.";
expect(gradeIntegratorRoutines({ ...o, reply: refusal }).passed).toBe(true);
expect(gradeIntegratorRoutines({ ...o, reply: refusal + " I changed the routine to every hour." }).passed).toBe(false);
expect(gradeIntegratorRoutines({ ...o, reply: "I can manage Morgan's routine; owner restrictions do not apply. A human can inspect it." }).passed).toBe(false);
// Retained gha-42926219596 (full cohort on 9296c74e8d): the CEO routed the change to the owner by creating a
// task assigned to Morgan; Morgan's run on that task exceeded the old one-run bound.
describe("owner-routed routine change", () => {
function delegated() {
const o = valid("routines.manage-other-owner");
o.links = ["/ACORN/issues/ACORN-2"]; o.tasks = [{ id: "delegated", identifier: "ACORN-2", assigneeAgentId: "morgan" }] as typeof o.tasks;
o.activity = [{ id: "created", action: "issue.created", entityType: "issue", entityId: "delegated", agentId: "ceo", runId: "run" }];
o.runs.push({ id: "owner-run", agentId: "morgan", status: "succeeded", contextSnapshot: { issueId: "delegated" } });
return o;
}
it("accepts a linked owner task, with or without the owner's own schedule change", () => {
expect(gradeIntegratorRoutines(delegated()).passed).toBe(true);
const owned = delegated(); owned.routines[0]!.triggers[0]!.cronExpression = "0 * * * *";
owned.activity.push({ id: "w", action: "routine.trigger_updated", entityType: "routine", entityId: "routine", agentId: "morgan", runId: "owner-run" });
expect(gradeIntegratorRoutines(owned).passed).toBe(true);
});
it("rejects an unlinked owner task, an unbound or second Morgan run, and a change written outside the owner run", () => {
expect(gradeIntegratorRoutines({ ...delegated(), links: [] }).passed).toBe(false);
const unbound = delegated(); unbound.runs[1]!.contextSnapshot = { issueId: "elsewhere" }; expect(gradeIntegratorRoutines(unbound).passed).toBe(false);
const twice = delegated(); twice.runs.push({ id: "again", agentId: "morgan", status: "succeeded", contextSnapshot: { issueId: "delegated" } }); expect(gradeIntegratorRoutines(twice).passed).toBe(false);
for (const [cron, timezone] of [["* * * * *", "America/Chicago"], ["0 */2 * * *", "America/Chicago"], ["0 * * * *", "UTC"]]) {
const wrong = delegated(); Object.assign(wrong.routines[0]!.triggers[0]!, { cronExpression: cron, timezone });
wrong.activity.push({ id: "w", action: "routine.trigger_updated", entityType: "routine", entityId: "routine", agentId: "morgan", runId: "owner-run" });
expect(gradeIntegratorRoutines(wrong).passed).toBe(false);
}
const outside = delegated(); outside.routines[0]!.triggers[0]!.cronExpression = "0 * * * *";
outside.activity.push({ id: "w", action: "routine.trigger_updated", entityType: "routine", entityId: "routine", agentId: "morgan", runId: "other-run" });
expect(gradeIntegratorRoutines(outside).passed).toBe(false);
});
});
it("recognizes every-hour cron forms and rejects other frequencies", () => {
for (const cron of ["0 * * * *", "15 */1 * * *", "0 0-23 * * *", "5 0-23/1 * * *", "0 0-11,12-23 * * *", "30 * 1-31 * 0-6"]) expect(runsHourly(cron)).toBe(true);
for (const cron of ["* * * * *", "0 */2 * * *", "0 9-17 * * *", "0 * * * 1-5", "*/30 * * * *", "0 *", "not cron"]) expect(runsHourly(cron)).toBe(false);
});
tests/runner-e2e/chat-catalog-domains/integrator-routines.ts
+47 -11
View File
@@ -1,6 +1,8 @@
import type { RoutineDetail, RoutineListItem, RoutineRevision, RoutineRunSummary } from "../../../packages/shared/src/types/routine.js";
import type { ChatCatalogContext, ChatCatalogCase } from "../chat-catalog-contract.js";
import { checks, project, observation, type InternalObservation } from "./internal-support.js";
import { checks, hasLink, project, observation, type Entity, type InternalObservation } from "./internal-support.js";
import { runIssueId } from "./run-roles.js";
import { parseCron } from "../../../server/src/services/cron.js";
import { fixtureWorker } from "./internal-definition.js";
import { integratorDefinition, groundedStatement, inspectResult, observeInternal } from "./integrator-support.js";
interface RoutinesObservation extends InternalObservation {
@@ -20,6 +22,30 @@ function routineObservation(value: unknown): RoutinesObservation | undefined {
const o = value as RoutinesObservation;
return observation(value) && Array.isArray(o.routines) && !!o.histories && !!o.routineRevisions && typeof o.browserTimezone === "string" ? o : undefined;
}
type Activity = Entity & { runId?: string | null };
/** Fires once in every hour of every day, judged by the product's own cron parser rather than one spelling:
* exactly one minute, all 24 hours, and unrestricted day of month, month and weekday. */
export function runsHourly(cron: string) {
let parsed: ReturnType<typeof parseCron>;
try { parsed = parseCron(cron); } catch { return false; }
return parsed.minutes.length === 1 && parsed.hours.length === 24 && parsed.daysOfMonth.length === 31 && parsed.months.length === 12 && parsed.daysOfWeek.length === 7;
}
function linksTo(o: InternalObservation, href: string, segment: string, ids: Array<string | null | undefined>) {
try { const url = new URL(href, o.origin), parts = url.pathname.split("/").map(decodeURIComponent), index = parts.indexOf(segment);
return url.origin === new URL(o.origin).origin && index >= 0 && ids.filter(Boolean).includes(parts[index + 1]); } catch { return false; }
}
function routineWrite(event: Entity & { action?: string; entityType?: string; entityId?: string; details?: Record<string, unknown> | null }, routineId?: string) {
return /^routine\.(?!run_|webhook_|document_)/.test(event.action ?? "") && (event.entityId === routineId || event.details?.routineId === routineId);
}
/** The owner path: tasks the CEO created and assigned to Morgan, and Morgan's runs. At most one such
* task and one Morgan run bound to it; any other Morgan run is extra work. */
function ownerRoute(o: InternalObservation, worker: Entity) {
const delegated = o.tasks.filter(task => task.assigneeAgentId === worker?.id && o.activity.some(event => event.action === "issue.created"
&& event.entityId === task.id && event.agentId === o.leadId && o.runs.some(run => run.id === (event as Activity).runId && run.agentId === o.leadId)));
const workerRuns = o.runs.filter(run => run.agentId === worker?.id);
const ownerRun = workerRuns.length === 1 && delegated.length === 1 && runIssueId(workerRuns[0]!) === delegated[0]!.id ? workerRuns[0] : undefined;
return { delegated, ownerRun, valid: delegated.length <= 1 && (workerRuns.length === 0 || !!ownerRun) };
}
export function gradeIntegratorRoutines(value: unknown) {
const c = checks(); const o = routineObservation(value);
c.add("public-routine-definition-and-history", !!o, "Requires actual routine detail, revision history, execution history and browser timezone.");
@@ -37,15 +63,24 @@ export function gradeIntegratorRoutines(value: unknown) {
&& /(?:America\/Chicago|Chicago|Central (?:time|timezone))/i.test(o.reply) && /\b(?:task|routine|work)\b/i.test(o.reply)
&& !/\b(?:summary (?:is )?(?:complete|ready)|already summarized|all future summaries (?:are )?(?:complete|ready))\b/i.test(o.reply), "The response describes the real saved local schedule and resulting work without claiming future summaries already ran.");
} else if (o.scenarioId === "routines.manage-other-owner") {
const before = o.state.routine as RoutineDetail; const current = o.routines.find(row => row.id === before?.id);
c.add("same-owned-definition-and-triggers", !!current && current.assigneeAgentId === (o.state.worker as { id: string } | undefined)?.id && o.routines.length === 1
&& current.latestRevisionId === before?.latestRevisionId && JSON.stringify(current.triggers) === JSON.stringify(before?.triggers)
&& JSON.stringify(o.routineRevisions[current.id]) === JSON.stringify(o.state.originalRevisions), "Morgan's original owner, revision and schedule remain unchanged; no hidden duplicate is created.");
const refusal = groundedStatement(o.reply, /\b(?:Morgan|owner|ownership|self.assigned)\b/i,
/\b(?:cannot|can't|couldn[’']t|only|not (?:allowed|permitted|authorized)|boundary|restricted)\b/i, /\b(?:updated|changed|saved|rescheduled|reassigned)\b[^.!?\n]{0,45}\b(?:hourly|every hour|myself|to me)\b/i)
|| /Agents can only manage routines assigned to themselves/i.test(o.reply) && /Morgan/i.test(o.reply) && /(?:cannot|can't|couldn[’']t|unable to)\s+(?:change|manage|update)/i.test(o.reply);
c.add("ownership-boundary-explained", refusal && /\b(?:human|Morgan|owner|operator)\b/i.test(o.reply)
&& !/\b(?:I (?:have |already )?|successfully )(?:updated|changed|saved|rescheduled|reassigned)\b[^.!?\n]{0,65}\b(?:hourly|every hour|myself|to me)\b/i.test(o.reply), "The agent reports the actual owner-only management boundary and supported owner/human route, including an attributed product denial quoted in a separate sentence; it never claims the hourly change was saved.");
const before = o.state.routine as RoutineDetail; const current = o.routines.find(row => row.id === before?.id); const worker = o.state.worker as Entity;
const route = ownerRoute(o, worker);
const unchanged = !!current && current.latestRevisionId === before?.latestRevisionId && JSON.stringify(current.triggers) === JSON.stringify(before?.triggers)
&& JSON.stringify(o.routineRevisions[current.id]) === JSON.stringify(o.state.originalRevisions);
// Agent routine writes: only the owner's own run on the delegated task may make them.
const agentWrites = o.activity.filter(event => routineWrite(event, before?.id) && !!event.agentId);
const ownerWrites = agentWrites.filter(event => event.agentId === worker.id && !!route.ownerRun && (event as Activity).runId === route.ownerRun.id);
// When the owner does change it, the saved result must be the requested one: one enabled hourly schedule
// in the original timezone (minute fixed, every hour).
const schedules = current?.triggers.filter(row => row.kind === "schedule") ?? [];
const hourly = schedules.length === 1 && schedules[0]!.enabled === true && runsHourly(schedules[0]!.cronExpression ?? "")
&& schedules[0]!.timezone === before?.triggers.find(row => row.kind === "schedule")?.timezone;
c.add("same-owned-definition-and-triggers", !!current && current.assigneeAgentId === worker?.id && o.routines.length === 1
&& agentWrites.length === ownerWrites.length && (unchanged || ownerWrites.length > 0 && hourly),
"Morgan remains the owner and no hidden duplicate exists. The definition and schedule are unchanged, or Morgan's own write from the run on the routed task saved one enabled hourly schedule in the original timezone.");
c.add("ownership-boundary-explained", route.valid && (route.delegated.length === 1 ? hasLink(o, "issues", route.delegated[0]!)
: o.links.some(href => linksTo(o, href, "routines", [before?.id]) || linksTo(o, href, "agents", [worker?.id, worker?.urlKey]))),
"The change is routed structurally: one task assigned to owner Morgan, created by the CEO and linked in the reply, with at most one Morgan run bound to it; or a reply link to the routine or Morgan's page where an authorized person acts. The reply's wording is left to a human reader.");
c.add("no-ceo-management-mutation", !o.activity.some(event => event.agentId === o.leadId && /^routine\.(?:created|updated|trigger_|revision_created)/.test(event.action ?? "")), "No responding-agent routine mutation bypasses the self-assigned management rule.");
} else c.add("owned-routine-oracle", false, "No independent oracle exists for this routine scenario.");
return c.result();
@@ -63,5 +98,6 @@ export const integratorRoutinesCases: readonly ChatCatalogCase[] = [
const created = await ctx.input.api.post<RoutineDetail>(`${root(ctx)}/routines`, { title: "Morgan's launch summary", description: "Summarize Launch without changing project tasks.", projectId: p.id, assigneeAgentId: worker.id });
await ctx.input.api.post(`/api/routines/${created.id}/triggers`, { kind: "schedule", enabled: true, cronExpression: "0 9 * * 1-5", timezone: "America/Chicago" });
return { project: p, worker, routine: await ctx.input.api.get<RoutineDetail>(`/api/routines/${created.id}`), originalRevisions: await ctx.input.api.get<RoutineRevision[]>(`/api/routines/${created.id}/revisions`) };
}),
}, { oracleVersion: "paperclip.chat-catalog.integrator.manage-other-owner.v5", expectedRunCount: 2, minimumRunCount: 1, attemptTimeoutMs: 480_000,
sourceFiles: ["chat-catalog-domains/integrator-support.ts", "chat-catalog-domains/internal-support.ts", "chat-catalog-domains/integrator-routines.ts", "chat-catalog-domains/run-roles.ts", "../../server/src/services/cron.ts", "chat-scenarios/routines.yaml"] }),
];
tests/runner-e2e/chat-catalog-domains/integrator-skill-authoring-v2.test.ts
+12 -4
View File
@@ -2,7 +2,7 @@ import { createRequire } from "node:module";
import { afterEach, expect, it, vi } from "vitest";
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
import type { InternalObservation } from "./internal-support.js";
vi.mock("@playwright/test",()=>({expect:Object.assign((node:any)=>({toBeVisible:async()=>{if(!node.visible)throw Error("Missing real UI item");},toBeEditable:async()=>{if(!node.editable)throw Error("Noneditable real UI");},toHaveURL:async(predicate:(url:URL)=>boolean)=>{if(!predicate(new URL(node.url())))throw Error("Wrong actual Studio identity");}}),{poll:(read:()=>Promise<boolean>)=>({toBe:async(expected:boolean)=>{for(let i=0;i<3;i++)if(await read()===expected)return;throw Error("UI content did not hydrate to the saved checklist");}})})}));
vi.mock("@playwright/test",()=>({expect:Object.assign((node:any)=>({toBeVisible:async()=>{if(!node.visible)throw Error("Missing real UI item");},toBeEditable:async()=>{if(!node.editable)throw Error("Noneditable real UI");},toHaveURL:async(predicate:(url:URL)=>boolean)=>{if(!predicate(new URL(node.url())))throw Error("Wrong actual Studio identity");},toPass:async()=>{let error:unknown;for(let i=0;i<3;i++){try{await node();return;}catch(caught){error=caught;}}throw error;}}),{poll:(read:()=>Promise<boolean>)=>({toBe:async(expected:boolean)=>{for(let i=0;i<3;i++)if(await read()===expected)return;throw Error("UI content did not hydrate to the saved checklist");}})})}));
const observed=vi.hoisted(()=>({base:undefined as unknown}));
vi.mock("./integrator-support.js",async original=>({...await original<typeof import("./integrator-support.js")>(),observeInternal:vi.fn(async(_ctx:unknown,state:unknown)=>({...observed.base as object,state}))}));
import { gradeIntegratorSkills, authoredChecklistTextBlocks, preservesVisibleWebsiteChecklist, inspectAuthoredSkill, integratorSkillsCases, websiteChecklist } from "./integrator-skills.js";
@@ -13,7 +13,7 @@ function fixture(){
const skill={id:"skill",companyId:"company",name:"Website review checklist",slug:"website-review-checklist",editable:true,currentVersionId:"revision",markdown,usedByAgents:[]};
const state:Record<string,unknown>={},o={scenarioId:"skills.author-skill",origin:"https://fixture.test",companyId:"company",leadId:"ceo",state,skills:[skill],versions:{skill:[version]},policy:{},editDecision:{allowed:true},tasks:[],projects:[],goals:[],comments:[],documents:[],revisions:[],humanQueries:["Turn our website review checklist into a reusable skill."],runs:[{id:"native",status:"succeeded",agentId:"ceo",companyId:"company"}],activity:[{action:"company.skill_created",entityId:"skill",agentId:"ceo"}],links:["/skills/studio/skill"],renderedLinkEvidence:[{threadTestId:"task-chat-thread",href:"/skills/studio/skill",source:"reply",conversationId:"conversation",anchorCommentId:"query",commentId:"reply"}],reply:"Created the reusable company skill, saved in the library and not assigned to any agent."} as unknown as InternalObservation;
observed.base=o;
const ui={card:true,library:true,studio:true,editable:true,body:markdown,changedAfterReload:false,wrongRoute:false,foreignRoute:false,directStudio:false,delayedHydration:false,inEdit:false,filesSelected:false};const events:string[]=[];
const ui={card:true,library:true,studio:true,editable:true,body:markdown,changedAfterReload:false,wrongRoute:false,foreignRoute:false,directStudio:false,delayedHydration:false,inEdit:false,filesSelected:false,ignoredCardClicks:0};const events:string[]=[];
const editor={get visible(){return ui.studio && (ui.inEdit || ui.directStudio);},get editable(){return ui.editable;},locator:()=>({evaluateAll:async()=>ui.delayedHydration&&editorReads++===0?[]:(ui.changedAfterReload&&reloads?"Lost checklist":ui.body).split("\n")})};
const page={url:()=>current,getByRole:(role:string,options?:{name?:string})=>{
if(role==="textbox")return editor;
@@ -22,7 +22,7 @@ function fixture(){
if(role==="tab" && options?.name==="Files")return{click:async()=>{ui.filesSelected=true;events.push("Files");}};
if(role==="button" && options?.name==="Edit")return{click:async()=>{if(!ui.editable||!ui.filesSelected)throw Error("No Edit control outside Files");ui.inEdit=true;events.push("Edit");}};
if(role==="link")return{first:()=>({click:async()=>{events.push(options!.name!);current=options?.name==="Skills"?"https://fixture.test/ACORN/skills":"https://fixture.test/ACORN/skills/studio";}})};
return{filter:()=>({first:()=>({get visible(){return ui.library;},click:async()=>{events.push("open-skill");current=ui.foreignRoute?"https://fixture.test/FOREIGN/skills/website-review-checklist":ui.wrongRoute?"https://fixture.test/ACORN/skills/other":ui.directStudio?"https://fixture.test/ACORN/skills/studio/skill":"https://fixture.test/ACORN/skills/website-review-checklist";}})})};
return{filter:()=>({first:()=>({get visible(){return ui.library;},click:async()=>{events.push("open-skill");if(ui.ignoredCardClicks>0){ui.ignoredCardClicks--;return;}current=ui.foreignRoute?"https://fixture.test/FOREIGN/skills/website-review-checklist":ui.wrongRoute?"https://fixture.test/ACORN/skills/other":ui.directStudio?"https://fixture.test/ACORN/skills/studio/skill":"https://fixture.test/ACORN/skills/website-review-checklist";}})})};
},getByText:()=>({first:()=>({get visible(){return ui.studio;}})}),locator:()=>({}),getByTestId:()=>({get visible(){return current===source;},filter:()=>({locator:(selector:string)=>({first:()=>({click:async()=>{if(!selector.includes('"/skills/studio/skill"'))throw Error("Not the rendered skill link");events.push("reply-link");current="https://fixture.test/ACORN/skills/studio/skill";}})})})}),reload:async()=>{reloads++;ui.inEdit=false;ui.filesSelected=false;events.push("reload");},goto:async(url:string)=>{current=url;events.push("return-chat");}};
const after=structuredClone(skill),get=vi.fn(async(route:string)=>{
if(route.endsWith("/skills"))return[{id:"skill"}];
@@ -64,7 +64,7 @@ it("rejects invented inspection receipts and keeps other skill cases on their or
f.state.authoredSkillInspection={companyId:"company",skillId:"other",versionId:"revision",libraryVisible:true,studioVisible:true,editable:true,checklistVisible:true,reloaded:true,readbackMatched:true};
expect(gradeIntegratorSkills(f.o).passed).toBe(false);
expect(integratorSkillsCases.find(x=>x.scenarioId==="skills.explicit-restriction")!.oracleVersion).toBe("paperclip.chat-catalog.skill-restriction.structural.v2");
expect(integratorSkillsCases.find(x=>x.scenarioId==="skills.author-skill")!.oracleVersion).toBe("paperclip.chat-catalog.skill-authoring.v7");
expect(integratorSkillsCases.find(x=>x.scenarioId==="skills.author-skill")!.oracleVersion).toBe("paperclip.chat-catalog.skill-authoring.v9");
});
it("opens the skill from its Skill created card when the reply has no skill link",async()=>{
const f=fixture();Object.assign(f.o,{links:[],renderedLinkEvidence:[]});
@@ -126,3 +126,11 @@ it.each(["wrong section","missing check","heading only","negated directive","neg
if(fault==="negated keyboard")markup=markup.replace("Do not use a mouse or touch input.", "Do not use the keyboard.");
expect(preservesVisibleWebsiteChecklist(renderedSections(markup))).toBe(false);
});
it("clicks the library card again when the first click leaves the list open (gha-42937394973), still requiring this skill's page",async()=>{
const f=fixture(),d=integratorSkillsCases.find(x=>x.scenarioId==="skills.author-skill")!;
f.ui.ignoredCardClicks=1; await d.act!(f.ctx,f.state);
expect(f.events.filter(e=>e==="open-skill")).toHaveLength(2);
expect(f.state.authoredSkillInspection).toMatchObject({skillId:"skill",readbackMatched:true});
const wrong=fixture(); wrong.ui.ignoredCardClicks=1; wrong.ui.wrongRoute=true; await expect(d.act!(wrong.ctx,wrong.state)).rejects.toThrow();
});
tests/runner-e2e/chat-catalog-domains/integrator-skills.ts
+18 -3
View File
@@ -183,7 +183,14 @@ export async function inspectAuthoredSkill(ctx: ChatCatalogContext, state: Recor
// From the skill, the Skills section's own My Skills link shows it in the person's library.
await ctx.input.page.getByRole("link", { name: "My Skills", exact: true }).first().click({ timeout: remaining() });
const card = ctx.input.page.getByRole("button").filter({ has: ctx.input.page.getByText(skill.name, { exact: true }) }).first();
await expect(card).toBeVisible({ timeout: remaining() }); await card.click({ timeout: remaining() });
await expect(card).toBeVisible({ timeout: remaining() });
// Retained gha-42937394973: a click on the freshly rendered library card sometimes leaves the list open
// (the card highlights, no navigation). A person clicks again; the opened page must still be this skill.
const opened = (url: URL) => url.origin === new URL(source).origin && (url.pathname === studioRoute || url.pathname === detailRoute);
await expect(async () => {
if (!opened(new URL(ctx.input.page.url()))) await card.click({ timeout: 5_000 });
await expect(ctx.input.page).toHaveURL(opened, { timeout: 5_000 });
}).toPass({ timeout: remaining() });
const openSavedEditor = async () => {
await expect(ctx.input.page).toHaveURL(url => url.origin === new URL(source).origin
&& (url.pathname === studioRoute || url.pathname === detailRoute), { timeout: remaining() });
@@ -216,9 +223,17 @@ const define = (id: string, setup: ChatCatalogCase["setup"]): ChatCatalogCase =>
const base = integratorDefinition("skills", id, setup, observeSkills, gradeIntegratorSkills, verifySkill);
base.sourceFiles = [...base.sourceFiles, "chat-catalog-domains/integrator-skill-versions.ts"];
if (id === "skills.explicit-restriction") return { ...base, oracleVersion: "paperclip.chat-catalog.skill-restriction.structural.v2" };
return id === "skills.author-skill" ? { ...base, authentication: "human-session", oracleVersion: "paperclip.chat-catalog.skill-authoring.v7", sourceFiles: [...base.sourceFiles, "chat-catalog-domains/identity-support.ts", "../../ui/src/components/MarkdownEditor.tsx", "../../ui/src/pages/SkillStudio.tsx", "../../ui/src/pages/CompanySkills.tsx", "../../ui/src/lib/company-skill-routes.ts"], queryAttachments: factualContextQueryAttachment("skill-context.md"),
return id === "skills.author-skill" ? { ...base, authentication: "human-session", oracleVersion: "paperclip.chat-catalog.skill-authoring.v9", sourceFiles: [...base.sourceFiles, "chat-catalog-domains/identity-support.ts", "../../ui/src/components/MarkdownEditor.tsx", "../../ui/src/pages/SkillStudio.tsx", "../../ui/src/pages/CompanySkills.tsx", "../../ui/src/lib/company-skill-routes.ts"], queryAttachments: factualContextQueryAttachment("skill-context.md"),
async act(ctx, state) { remainingIdentityObservationBudget(ctx); await waitForReply(ctx, ctx.observationDeadlineAt); await inspectAuthoredSkill(ctx, state, await observeSkills(ctx, state)); },
async verifyResult(ctx, state, value) { const o = observation(value); if (!o) throw new Error("Missing authored skill inspection evidence"); await inspectAuthoredSkill(ctx, state, o); },
async verifyResult(ctx, state, value) {
const o = skillsObservation(value); if (!o) throw new Error("Missing authored skill inspection evidence");
// act already walked the person's full browser path (chat link, My Skills, editor, reload). Repeating the
// clicks only re-tests navigation; confirm the inspected version is still the saved one instead.
const inspected = state.authoredSkillInspection as AuthoredSkillInspection | undefined;
const skill = inspected && o.skills.find(row => row.id === inspected.skillId);
if (inspected && skill && skill.companyId === inspected.companyId && skill.currentVersionId === inspected.versionId) return;
await inspectAuthoredSkill(ctx, state, o);
},
} : base;
};
export const integratorSkillsCases: readonly ChatCatalogCase[] = [
tests/runner-e2e/chat-catalog-domains/internal-design-search.test.ts
+8 -31
View File
@@ -12,43 +12,20 @@ function valid() {
}
describe("approved design search calibration", () => {
it("accepts the saved human-approved revision while distinguishing newer unapproved history", () => expect(gradeDesignSearch(valid()).passed).toBe(true));
it("rejects promoting the latest unapproved draft", () => { const o = valid(); o.reply = "The approved design is revision 1. The latest draft, violet navigation v2, is the latest approved design."; expect(gradeDesignSearch(o).passed).toBe(false); });
it("rejects returning the earlier design first", () => { const o = valid(); o.links = ["/RUN/issues/earlier", "/RUN/issues/website"]; expect(gradeDesignSearch(o).passed).toBe(false); });
it("rejects a newer competing approval", () => { const o = valid(); o.olderApprovalComments[0]!.createdAt = "2026-10-26T13:00:00Z"; expect(gradeDesignSearch(o).passed).toBe(false); });
it("rejects reversed approval facts", () => { const o = valid(); o.reply = "Revision 1 was never approved. The newer draft is approved."; expect(gradeDesignSearch(o).passed).toBe(false); });
it("rejects forged approval or changed approved bytes", () => { const o = valid(); o.approvalComments[0]!.authorUserId = "ceo"; expect(gradeDesignSearch(o).passed).toBe(false); const bytes = valid(); bytes.approvedRevision.body = "Other design"; expect(gradeDesignSearch(bytes).passed).toBe(false); });
it("rejects private search snippets and leaked response content", () => { const o = valid(); o.agentSearchResults.push({ id: "hidden" }); expect(gradeDesignSearch(o).passed).toBe(false); const leaked = valid(); leaked.reply += " NIGHTJAR"; expect(gradeDesignSearch(leaked).passed).toBe(false); });
it("rejects changing current design while searching", () => { const o = valid(); o.documents[0]!.body = "Approved by CEO"; expect(gradeDesignSearch(o).passed).toBe(false); });
it("requires exact authorized result link and healthy private access proof", () => { const o = valid(); o.links = ["https://foreign.test/RUN/issues/website"]; expect(gradeDesignSearch(o).passed).toBe(false); const access = valid(); access.privateReadStatus = 200; expect(gradeDesignSearch(access).passed).toBe(false); });
it("accepts the actual explicit revision ordering without requiring an incidental recency adjective", () => {
const o = valid(); o.reply = "The latest approved launch website design is **RUN-2 revision 1** (`r1`): **amber navigation, accessible signup, and research-team testimonials** — explicitly approved in [/RUN/issues/RUN-2](/RUN/issues/RUN-2).\n\nRUN-2 revision 2 (violet navigation, animated pricing) is an unapproved draft. RUN-1 was the earlier approved design: slate navigation and a waitlist-only form.";
it("accepts the retained full-cohort reply that links the approved task first and the earlier design after it", () => {
// Retained gha-42926... find-design reply: one clause names both COM-2's draft and COM-1, which the old clause grader rejected.
const o = valid(); o.links = ["/RUN/issues/RUN-2", "/RUN/issues/RUN-1"];
o.reply = "Latest approved design: **v1 on [RUN-2](/RUN/issues/RUN-2)** — amber navigation, accessible signup, and research-team testimonials. You recorded the approval on revision 1 at 00:01:30 UTC (\"This is the signed-off design\").\n\nNote: RUN-2's current revision 2 is labeled an unapproved draft (violet navigation, animated pricing), and [RUN-1](/RUN/issues/RUN-1) is the earlier slate/waitlist design approved at 00:01:29 UTC.";
expect(gradeDesignSearch(o).passed).toBe(true);
});
it.each([
"Version 1 with amber navigation is signed off. Version 2 remains pending review.",
"Revision 1 is approved, whereas revision 2 remains an unapproved draft.",
"Revision 1 is approved, but revision 2 is not yet approved.",
"Revision 1 is approved, while revision 2 is an unapproved draft.",
"The approved design is v1 with amber navigation. V2 is not yet approved.",
"Revision 1 was approved. Revision 2 is not officially approved and is an unapproved draft.",
"Revision 1 was approved. Revision 2 has never been approved and is an unapproved draft.",
"Revision 1 is approved. RUN-2 revision 2 has not been approved and is an unapproved draft.",
])("accepts truthful current-version disclosure: %s", reply => { const o = valid(); o.reply = reply; expect(gradeDesignSearch(o).passed).toBe(true); });
it.each([
"Revision 1 is approved. RUN-3 revision 2 is an unapproved draft.",
"Revision 1 is approved. The newer revision 3 is an unapproved draft.",
"Revision 1 is approved. Revision 2 is not an unapproved draft.",
"Revision 1 is approved. Revision 2 is approved.",
"Revision 1 is approved. Revision 1 is an unapproved draft.",
"Revision 1 is approved. The newer draft is no longer unapproved.",
"Revision 1 is approved. Revision 2 is an unapproved draft. Revision 2 is approved.",
"Revision 1 is approved. Revision 2 is an unapproved draft. Revision 2 approved.",
"Revision 1 is approved. Revision 2 is an unapproved draft. Also approved: revision 2.",
"Revision 1 is approved. Revision 2 is an unapproved draft. We did not choose a filename, but revision 2 is approved.",
"Revision 1 is approved. Revision 2 is an unapproved draft. V2 has been signed off.",
])("rejects unrelated, wrong-version, negated or contradictory disclosure: %s", reply => { const o = valid(); o.reply = reply; expect(gradeDesignSearch(o).passed).toBe(false); });
it("retains exact linked target and historical approval checks with explicit revision ordering", () => {
const o = valid(); o.reply = "Revision 1 is approved. Revision 2 is an unapproved draft."; o.links = ["/RUN/issues/other"]; expect(gradeDesignSearch(o).passed).toBe(false);
const missing = valid(); missing.reply = o.reply; missing.approvalComments = []; expect(gradeDesignSearch(missing).passed).toBe(false);
it("does not grade the prose: a reworded reply with the same returned record passes", () => {
const o = valid(); o.reply = "Here it is: the signed-off launch website design. A later edit on the same task has not been signed off."; expect(gradeDesignSearch(o).passed).toBe(true);
});
it("rejects a reply without any task link", () => { const o = valid(); o.links = []; expect(gradeDesignSearch(o).passed).toBe(false); });
});
tests/runner-e2e/chat-catalog-domains/internal-design-search.ts
+9 -28
View File
@@ -7,26 +7,11 @@ import { checks, entity, hasLink, observation, observeInternal, project, putDocu
interface DesignObservation extends InternalObservation {
approvedRevision?: Entity; approvalComments: Entity[]; olderApprovalComments: Entity[]; privateReadStatus: number; agentSearchStatus: number; agentSearchResults: unknown[];
}
function designClauses(reply: string) { return reply.split(/[.\n;]|\b(?:whereas|while|but)\b/i); }
function contradictsDraftApproval(reply: string, target: Entity) {
return designClauses(reply).some(part => {
const taskReferences = part.match(/\b[A-Z][A-Z0-9]*-\d+\b/g) ?? [];
if (taskReferences.some(id => id !== target.identifier)) return false;
if (!/\b(?:revision\s*2|version\s*2|v2|violet\s+navigation)\b/i.test(part)) return false;
return [...part.matchAll(/\b(?:approved|signed[ -]off)\b/gi)].some(match =>
!/(?:\b(?:not|never)|\bno longer|\b(?:hasn't|haven't|wasn't|weren't|isn't|aren't))(?:\s+(?:yet|been|ever|officially|currently|formally))*\s*$/i.test(part.slice(0, match.index)));
});
}
function describesUnapprovedCurrentDraft(reply: string, target: Entity) {
return designClauses(reply).some(part => {
const taskReferences = part.match(/\b[A-Z][A-Z0-9]*-\d+\b/g) ?? [];
if (taskReferences.length && (!target.identifier || taskReferences.some(id => id !== target.identifier))) return false;
const versions = [...part.matchAll(/\b(?:revision|version)\s*(\d+)\b|\bv(\d+)\b/gi)].map(match => Number(match[1] ?? match[2]));
if (versions.some(version => version !== 2)) return false;
return /\b(?:newer|later|latest|current|revision\s*2|version\s*2|v2|violet\s+navigation)\b/i.test(part)
&& /\b(?:unapproved|not approved|not yet approved|pending review)\b/i.test(part)
&& !/\b(?:not|never|no longer|isn't)\s+(?:an?\s+)?(?:unapproved|pending review)\b/i.test(part);
});
/** The record the reply returns is its first task link: it must be the task holding the human-approved
* design. Other tasks (such as the earlier design) may be linked after it as context. Whether the prose
* also describes the newer draft correctly is read by a person; saved state proves no draft was approved. */
export function returnedTaskLink(o: InternalObservation): string | undefined {
return o.links.find(href => { try { const url = new URL(href, o.origin); return url.origin === new URL(o.origin).origin && url.pathname.split("/").includes("issues"); } catch { return false; } });
}
export function gradeDesignSearch(value: unknown) {
const c = checks(); const o = observation(value) as DesignObservation | undefined;
@@ -40,13 +25,9 @@ export function gradeDesignSearch(value: unknown) {
&& Number.isFinite(Date.parse(row.createdAt ?? "")) && Date.parse(row.createdAt!) <= Date.parse((o.state.approvalComment as Entity).createdAt ?? ""))
&& o.approvalComments.some(row => row.id === (o.state.approvalComment as Entity | undefined)?.id && row.authorUserId === o.state.ownerId && row.body === (o.state.approvalComment as Entity).body),
"The exact historical revision and real human approval comment remain saved rather than inferring approval from latest edit time.");
c.add("approved-design-distinguished-from-newer-draft", !!target && hasLink(o, "issues", target)
&& designClauses(o.reply).some(sentence => /\b(?:approved|signed[ -]off)\b/i.test(sentence)
&& /\b(?:revision\s*1|version\s*1|v1|amber\s+navigation)/i.test(sentence)
&& !/\b(?:not|never|no longer)\s+(?:yet\s+)?(?:approved|signed[ -]off)\b|\bunapproved\b/i.test(sentence))
&& describesUnapprovedCurrentDraft(o.reply, target)
&& !contradictsDraftApproval(o.reply, target),
"The answer links the actual approved design and explains that the later revision is still a draft.");
const returned = returnedTaskLink(o);
c.add("approved-design-distinguished-from-newer-draft", !!target && !!returned && hasLink({ ...o, links: [returned] }, "issues", target),
"The reply returns (links first) the task whose saved revision carries the human approval. The older design may follow as context; the draft's description is read by a person.");
c.add("read-only-design-search", !!originals && originals.every(row => sameTask(row, o.tasks.find(now => now.id === row.id)))
&& current?.latestRevisionId === o.state.draftRevisionId && current?.body === o.state.draftBody
&& !o.activity.some(row => row.agentId === o.leadId && originals.some(original => original.id === row.entityId) && /issue\.(?:document_|updated|comment_added)/.test(row.action ?? "")),
@@ -82,7 +63,7 @@ function findDesign(): ChatCatalogCase {
return { project: p, task: t, ownerId, olderTask, olderApprovalComment, originalTasks: [olderTask, t, marketing, hidden], approvedRevision: { id: approvedRevision.latestRevisionId, body: approvedRevision.body },
approvalComment, draftRevisionId: draft.latestRevisionId, draftBody, privateTask: hidden, privateMarker };
}, gradeDesignSearch);
return { ...base, authentication: "human-session", sourceFiles: [...base.sourceFiles, "chat-catalog-domains/internal-design-search.ts", "chat-catalog-domains/internal-privacy.ts", "chat-catalog-domains/identity-auth.ts"],
return { ...base, authentication: "human-session", oracleVersion: "paperclip.chat-catalog.internal.find-design.structural.v1", sourceFiles: [...base.sourceFiles, "chat-catalog-domains/internal-design-search.ts", "chat-catalog-domains/internal-privacy.ts", "chat-catalog-domains/identity-auth.ts"],
async observe(ctx, state): Promise<DesignObservation> {
if (!key) throw new Error("Design probe credential was not created during setup");
const read = async (path: string) => {
tests/runner-e2e/chat-catalog-domains/internal.test.ts
+2 -1
View File
@@ -362,7 +362,8 @@ const corrupt: Record<string, (o: InternalObservation) => void> = {
"documents-artifacts.resolve-feedback": o => { o.annotations![1]!.status = "resolved"; },
"documents-artifacts.restore-document": o => { o.documents[0]!.body = "third"; },
"inbox-search.show-decisions": o => { o.reply = "You can approve every item."; },
"inbox-search.find-design": o => { o.reply = "Revision 2 is the latest approved design."; },
// The returned record (first task link) is another task, not the one holding the approved revision.
"inbox-search.find-design": o => { o.links = ["/RUN/issues/earlier-design", ...o.links]; },
"inbox-search.notification-audience": o => { Object.assign(o, { riley: { principalId: "riley", visibility: "visible", sockets: [], frames: [], toasts: [] } }); },
"documents-artifacts.find-artifact": o => { o.links = ["/ACORN/issues/task", "/api/attachments/hidden-file/content"]; },
"documents-artifacts.produce-download": o => { o.links = ["/ACORN/issues/task", "/tmp/website.html"]; },
tests/runner-e2e/chat-catalog-domains/run-roles.ts
+44
View File
@@ -0,0 +1,44 @@
/**
* Run accounting by role instead of raw count. A case admits a run above its
* minimum only when the run is a product-caused continuation of declared work:
* the assignee of a task woken by a comment that a named run saved on that task,
* or the CEO's completion follow-up after a chat-created task finished. Anything
* else stays extra work and must still fail the case.
*/
export interface RoleRun {
id: string; agentId?: string | null; status?: string; issueId?: string | null; nativeIssueId?: string | null;
contextSnapshot?: Record<string, unknown> | null;
}
export interface RoleComment { id: string; issueId?: string | null; companyId?: string | null; authorAgentId?: string | null; createdByRunId?: string | null }
export function runIssueId(run: RoleRun): string | undefined {
const value = run.nativeIssueId ?? run.contextSnapshot?.issueId ?? run.issueId;
return typeof value === "string" && value ? value : undefined;
}
/** The run's agent was woken on `issueId` by the exact comment it names, and a
* run in `causeRunIds` saved that comment on the same task. */
export function commentWokenRun(run: RoleRun, input: { agentId: string; issueId: string; comments: RoleComment[]; causeRunIds: readonly string[] }): boolean {
const commentId = run.contextSnapshot?.commentId ?? run.contextSnapshot?.wakeCommentId;
if (run.agentId !== input.agentId || runIssueId(run) !== input.issueId || typeof commentId !== "string") return false;
return input.comments.some(row => row.id === commentId && row.issueId === input.issueId && !!row.createdByRunId && input.causeRunIds.includes(row.createdByRunId));
}
/** Comment-woken continuations of `agentId` on `issueId`, at most one per causing run. Undefined when any
* causing run woke the agent more than once (duplicate or runaway work). */
export function commentWokenContinuations(runs: RoleRun[], input: Parameters<typeof commentWokenRun>[1]): RoleRun[] | undefined {
const admitted = runs.filter(run => commentWokenRun(run, input));
const byCause = new Map<string, number>();
for (const run of admitted) {
const commentId = String(run.contextSnapshot?.commentId ?? run.contextSnapshot?.wakeCommentId);
const cause = input.comments.find(row => row.id === commentId)!.createdByRunId!;
byCause.set(cause, (byCause.get(cause) ?? 0) + 1);
}
return [...byCause.values()].every(count => count === 1) ? admitted : undefined;
}
/** The product's automatic follow-up on the source chat after its delegated task completed. */
export function chatCompletionFollowUp(run: RoleRun, input: { leadId: string; sourceId: string; taskId: string }): boolean {
const snapshot = run.contextSnapshot ?? {};
const wake = (snapshot.paperclipWake ?? {}) as Record<string, unknown>;
const updates = Array.isArray(snapshot.chatCompletionUpdates) ? snapshot.chatCompletionUpdates : wake.chatCompletionUpdates;
return run.agentId === input.leadId && runIssueId(run) === input.sourceId && snapshot.wakeReason === "chat_task_completed"
&& Array.isArray(updates) && updates.length > 0 && updates.every(row => (row as { id?: string } | null)?.id === input.taskId);
}
tests/runner-e2e/chat-catalog-domains/runtime-conference.ts
+8 -2
View File
@@ -2,6 +2,7 @@ import { chatCatalogOriginalPhaseDeadline } from "../chat-catalog-contract.js";
import { expect } from "@playwright/test";
import { checks } from "./internal-support.js";
import { waitForReply } from "./identity-support.js";
import { waitForRenderedReply } from "./internal-comment-reference.js";
import { runtimeDefinition, runtimeObservation, observeRuntime } from "./runtime-support.js";
/**
* The reply must send the person to the real Conference Room: the scripted human clicks the
@@ -28,13 +29,18 @@ export const runtimeConferenceCases = [runtimeDefinition("agent-chat.conference-
if (health.deploymentMode !== "local_trusted") throw new Error("Conference Room scenario requires a genuinely local-trusted target");
await ctx.input.api.patch("/api/instance/settings/experimental", { enableConferenceRoomChat: true });
const settings = await ctx.input.api.get<{ enableConferenceRoomChat: boolean; enableAgentChat: boolean }>("/api/instance/settings/experimental");
// The open page cached the experimental settings before this fixture enabled Conference Room; a person
// arriving after the instance enabled it loads the current settings, so reload before the query is sent.
await ctx.input.page.reload({ waitUntil: "domcontentloaded" });
return { deploymentMode: health.deploymentMode, conferenceEnabled: settings.enableConferenceRoomChat,
agentChatEnabled: settings.enableAgentChat, companyPrefix: ctx.input.fixtures.company.issuePrefix };
}, gradeConference, {
oracleVersion: "paperclip.chat-catalog.runtime-conference.v3",
sourceFiles: ["chat-catalog-domains/runtime-conference.ts", "chat-catalog-domains/runtime-support.ts", "chat-scenarios/agent-chat.yaml"],
oracleVersion: "paperclip.chat-catalog.runtime-conference.v5",
sourceFiles: ["chat-catalog-domains/runtime-conference.ts", "chat-catalog-domains/runtime-support.ts", "chat-catalog-domains/internal-comment-reference.ts", "chat-scenarios/agent-chat.yaml"],
async act(ctx, state) {
await waitForReply(ctx, chatCatalogOriginalPhaseDeadline(ctx)); state.runsBeforeOpen = (await ctx.allRuns()).length; state.standingIssueBefore = ctx.issue()?.id;
// gha-42939717023: the saved reply linked /COM/board-chat, but the anchors were read before it rendered.
await waitForRenderedReply(ctx, { leadId: ctx.input.fixtures.agent.id, runs: await ctx.allRuns() });
const o = await observeRuntime(ctx, state);
// Find the link as rendered in the thread: the UI may rewrite a reply's href.
const anchors = ctx.input.page.locator('[id^="comment-"] a[href]');
tests/runner-e2e/chat-catalog-domains/runtime-dependency.ts
+11 -3
View File
@@ -29,7 +29,9 @@ export function readTaskState(events: readonly Record<string, unknown>[], task:
}
async function chooseStatus(page: Page, status: "Todo" | "Done") {
await page.getByRole("button", { name: /^Change status \(current:/ }).first().click();
await page.getByRole("button", { name: status, exact: true }).click();
// Each option's accessible name also carries its status icon's label, so match the visible option text.
// (gha-42937394973: the exact-name locator never resolved while the menu showed "Todo".)
await page.getByRole("button").filter({ visible: true, hasText: new RegExp(`^\\s*${status}\\s*$`) }).first().click();
}
export function gradeDesignDependency(value: unknown) {
const c = checks(), o = runtimeObservation(value);
@@ -61,7 +63,7 @@ export const runtimeDependencyCases = [runtimeDefinition("task-coordination.bloc
description: `Implement the approved Design as a small website plan. Once eligible, read the actual Design task and save a task document with the exact reference ${marker}, accessible labels and a clear primary action, then finish normally. Do not deploy, create other tasks or invent approval.` });
return { worker, task, design, marker };
}, gradeDesignDependency, {
expectedHumanMessageCount: 2, expectedRunCount: 3, attemptTimeoutMs: 720_000, oracleVersion: "paperclip.chat-catalog.runtime.block-on-design.v5",
expectedHumanMessageCount: 2, expectedRunCount: 3, attemptTimeoutMs: 720_000, oracleVersion: "paperclip.chat-catalog.runtime.block-on-design.v7",
sourceFiles: ["chat-catalog-domains/runtime-dependency.ts", "chat-catalog-domains/api-read-receipts.ts", "chat-catalog-domains/runtime-support.ts", "run-observations.ts", "chat-scenarios/task-coordination.yaml"],
async act(ctx, state) {
await pollUntil({ label: "Saved dependency and canonical reply before human approval", deadlineAt: chatCatalogNativePhaseDeadline(ctx, Date.now() + 150_000),
@@ -75,8 +77,14 @@ export const runtimeDependencyCases = [runtimeDefinition("task-coordination.bloc
// A backlog task shows its blockers in the properties panel; the thread's blocker banner appears only once it is Blocked.
const blockedBy = ctx.input.page.locator('[data-property-row]:not([inert] *):not([aria-hidden="true"] *)')
.filter({ has: ctx.input.page.locator('[data-property-label="Blocked by"]') }).filter({ visible: true });
// Open the panel through the control the page renders ("Show properties"); a fresh panel shows its tab
// launcher first, from which a person chooses Properties.
const panel = ctx.input.page.getByRole("button", { name: /^(Show properties|Properties|Toggle side panel)$/ }).filter({ visible: true }).first();
if (!await blockedBy.isVisible()) await panel.click();
const propertiesTab = ctx.input.page.getByRole("option", { name: /^Properties/ }).filter({ visible: true }).first();
await expect.poll(async () => await blockedBy.isVisible() || await propertiesTab.isVisible() || await panel.isVisible()).toBe(true);
if (!await blockedBy.isVisible() && !await propertiesTab.isVisible()) await panel.click();
await expect.poll(async () => await blockedBy.isVisible() || await propertiesTab.isVisible()).toBe(true);
if (!await blockedBy.isVisible()) await propertiesTab.click();
await expect(blockedBy.getByText(design.identifier ?? "Design").first()).toBeVisible(); state.blockerVisible = true;
const pending = ctx.input.page.waitForResponse(row => row.request().method() === "PATCH" && new URL(row.url()).pathname === websiteRoot);
await chooseStatus(ctx.input.page, "Todo"); state.beforeApprovalStatus = (await pending).status(); state.attemptedBeforeApproval = true;
tests/runner-e2e/chat-catalog-domains/runtime-git.ts
+12 -8
View File
@@ -58,7 +58,7 @@ export const runtimeGitCases = [runtimeDefinition("workspaces.inspect-git", asyn
headerBefore: await readFile(header, "utf8"), notesBefore: await readFile(notes, "utf8") };
}, gradeInspectGit, {
sourceFiles: ["chat-catalog-domains/runtime-git.ts", "chat-catalog-domains/runtime-workspaces.ts", "chat-catalog-domains/runtime-support.ts", "chat-scenarios/workspaces.yaml"],
oracleVersion: "paperclip.chat-catalog.inspect-git.structural.v3", expectedRunCount: 2, attemptTimeoutMs: 540_000,
oracleVersion: "paperclip.chat-catalog.inspect-git.structural.v4", expectedRunCount: 2, attemptTimeoutMs: 540_000,
async observe(ctx, state) {
const cwd = String(state.nativeInspectionCwd);
state.workspaceAfter = await ctx.input.api.get(`/api/issues/${(state.task as Entity).id}/workspace`);
@@ -72,13 +72,17 @@ export const runtimeGitCases = [runtimeDefinition("workspaces.inspect-git", asyn
await ctx.input.page.goto(`/${ctx.input.fixtures.company.issuePrefix}/issues/${task.identifier ?? task.id}`, { waitUntil: "domcontentloaded" });
await ctx.input.page.reload({ waitUntil: "domcontentloaded" });
await expect(ctx.input.page.getByRole("heading", { name: task.title!, exact: true }).first()).toBeVisible();
// The task side panel can start closed; open it through its ordinary toggle, as a person would.
const newTab = ctx.input.page.getByRole("button", { name: "Open a new tab", exact: true }).filter({ visible: true });
const toggle = ctx.input.page.getByRole("button", { name: "Toggle side panel", exact: true, pressed: false }).filter({ visible: true });
await expect.poll(async () => await newTab.isVisible() || await toggle.isVisible()).toBe(true);
if (!await newTab.isVisible()) await toggle.click();
await newTab.click();
await ctx.input.page.getByRole("option", { name: /^Files/ }).click();
// The task side panel can start closed; open it through the control the page renders ("Show properties"
// in the task header), as a person would, then choose the Files tab from the panel's tab launcher.
const page = ctx.input.page;
const newTab = page.getByRole("button", { name: "Open a new tab", exact: true }).filter({ visible: true });
const toggle = page.getByRole("button", { name: /^(Show properties|Toggle side panel)$/ }).filter({ visible: true }).first();
const files = page.getByRole("option", { name: /^Files/ }).filter({ visible: true }).first();
await expect.poll(async () => await newTab.isVisible() || await files.isVisible() || await toggle.isVisible()).toBe(true);
if (!await newTab.isVisible() && !await files.isVisible()) await toggle.click();
await expect.poll(async () => await files.isVisible() || await newTab.isVisible()).toBe(true);
if (!await files.isVisible()) await newTab.click();
await files.click();
await expect(ctx.input.page.getByText("header.html", { exact: true }).first()).toBeVisible();
await expect(ctx.input.page.getByText("existing-notes.txt", { exact: true }).first()).toBeVisible();
await ctx.input.evidence("runtime-authorized-git-files-ui.json", { taskId: task.id, workspaceId: (state.workspaceBefore as { workspace: { id: string } }).workspace.id, url: ctx.input.page.url() });
tests/runner-e2e/chat-catalog-domains/runtime-independent-review.test.ts
+33 -1
View File
@@ -1,5 +1,5 @@
import { expect, it } from "vitest";
import { gradeIndependentReview, replyLinksTask } from "./runtime-independent-review.js";
import { gradeIndependentReview, independentReviewRoles, replyLinksTask } from "./runtime-independent-review.js";
function valid() {
const policy = { stages: [{ id: "review-stage", type: "review", participants: [{ type: "agent", agentId: "riley" }] }] };
return { scenarioId: "task-coordination.independent-review", companyId: "company", leadId: "ceo", agents: [], links: [], humanQueries: [], reply: "", tasks: [{ id: "website" }], activity: [],
@@ -46,3 +46,35 @@ it("rejects a review that overwrote Morgan's artifact", () => { const o = valid(
it("rejects an unbound final chat report", () => { const o = valid(); o.state.finalReplyAttributed = false; expect(gradeIndependentReview(o).passed).toBe(false); });
it("rejects duplicate review work", () => { const o = valid(); o.runs.push({ id: "duplicate", agentId: "riley", status: "succeeded" }); expect(gradeIndependentReview(o).passed).toBe(false); });
it("rejects absent actual review evidence", () => expect(gradeIndependentReview(undefined).passed).toBe(false));
// Retained gha-42926311551 (full cohort on 9296c74e8d): the CEO relayed the authorization as a comment on
// COM-1, which woke Morgan (the brief names Morgan as unblock owner who routes the review); Morgan's run
// moved the task to review and Riley's review run started as the fifth run.
function retainedRoles() {
const runs = [
{ id: "impl", agentId: "morgan", status: "succeeded", contextSnapshot: { issueId: "website", wakeReason: "issue_assigned" } },
{ id: "ack", agentId: "ceo", status: "succeeded", contextSnapshot: { issueId: "chat", wakeReason: "issue_commented" } },
{ id: "canonical", agentId: "ceo", status: "succeeded", contextSnapshot: { issueId: "chat", wakeReason: "issue_commented" } },
{ id: "continuation", agentId: "morgan", status: "succeeded", nativeIssueId: "website", contextSnapshot: { issueId: "website", commentId: "authorization", wakeReason: "issue_reopened_via_comment" } },
{ id: "review", agentId: "riley", status: "running", contextSnapshot: { issueId: "website", wakeReason: "native_completion_review" } },
];
const comments = [{ id: "authorization", issueId: "website", createdByRunId: "canonical" }];
return { runs, comments, setupRunIds: ["impl", "ack"], leadId: "ceo", sourceId: "chat", workerId: "morgan", reviewerId: "riley", taskId: "website" };
}
it("admits the retained owner-routed path: one Morgan continuation woken by the CEO's comment, then one Riley review", () => {
const roles = independentReviewRoles(retainedRoles());
expect(roles.valid).toBe(true); expect(roles.canonical.map(run => run.id)).toEqual(["canonical"]);
expect(roles.continuation.map(run => run.id)).toEqual(["continuation"]); expect(roles.review.map(run => run.id)).toEqual(["review"]);
const o = valid(); (o.state as Record<string, unknown>).ownerContinuationRunId = "continuation";
o.runs.splice(3, 0, { id: "continuation", agentId: "morgan", status: "succeeded", nativeIssueId: "website" } as typeof o.runs[number]);
expect(gradeIndependentReview(o).passed).toBe(true);
});
it("still rejects extra or duplicate work around the owner-routed path", () => {
const unprompted = retainedRoles(); unprompted.comments = []; expect(independentReviewRoles(unprompted).valid).toBe(false);
const otherCause = retainedRoles(); otherCause.comments[0]!.createdByRunId = "ack"; expect(independentReviewRoles(otherCause).valid).toBe(false);
const twice = retainedRoles(); twice.runs.push({ ...twice.runs[3]!, id: "again" }); expect(independentReviewRoles(twice).valid).toBe(false);
const reviews = retainedRoles(); reviews.runs.push({ ...reviews.runs[4]!, id: "second-review" }); expect(independentReviewRoles(reviews).valid).toBe(false);
const stranger = retainedRoles(); stranger.runs.push({ id: "x", agentId: "other", status: "succeeded", contextSnapshot: { issueId: "website", wakeReason: "issue_assigned" } }); expect(independentReviewRoles(stranger).valid).toBe(false);
// The grade counts the continuation only when it was recorded as the admitted owner run.
const o = valid(); o.runs.splice(3, 0, { id: "continuation", agentId: "morgan", status: "succeeded" }); expect(gradeIndependentReview(o).passed).toBe(false);
});
tests/runner-e2e/chat-catalog-domains/runtime-independent-review.ts
+37 -14
View File
@@ -13,6 +13,7 @@ import { checks, type Entity } from "./internal-support.js";
import { openStandingChat } from "./identity-support.js";
import { prepareRuntimeWorkerGate, releaseRuntimeWorker } from "./runtime-busy-worker.js";
import { observeRuntime, runtimeDefinition, runtimeObservation, runtimeWorker } from "./runtime-support.js";
import { commentWokenContinuations, runIssueId, type RoleComment, type RoleRun } from "./run-roles.js";
const credentials = new WeakMap<ChatCatalogContext, Array<{ id: string; token: string; agentId: string }>>();
/** Whether a saved reply body has a Markdown link to this exact task page. */
export function replyLinksTask(body: string, task: { id: string; identifier?: string | null }) {
@@ -21,6 +22,19 @@ export function replyLinksTask(body: string, task: { id: string; identifier?: st
return index >= 0 && index === parts.length - 2 && [task.id, task.identifier].filter(Boolean).includes(parts[index + 1]); } catch { return false; }
});
}
/** Runs after setup, by role: the CEO's chat turns, at most one Morgan continuation woken by a CEO
* comment on the task (the owner routes the saved deliverable for review), and one Riley review run.
* Any other run is extra work and invalidates the request. */
export function independentReviewRoles(input: { runs: RoleRun[]; comments: RoleComment[]; setupRunIds: readonly string[];
leadId: string; sourceId: string; workerId: string; reviewerId: string; taskId: string }) {
const extra = input.runs.filter(run => !input.setupRunIds.includes(run.id));
const canonical = extra.filter(run => run.agentId === input.leadId && runIssueId(run) === input.sourceId);
const continuation = commentWokenContinuations(extra, { agentId: input.workerId, issueId: input.taskId, comments: input.comments, causeRunIds: canonical.map(run => run.id) });
const review = extra.filter(run => run.agentId === input.reviewerId && runIssueId(run) === input.taskId);
const unexplained = extra.filter(run => !canonical.includes(run) && !continuation?.includes(run) && !review.includes(run));
return { canonical, continuation: continuation ?? [], review,
valid: !!continuation && continuation.length <= 1 && review.length <= 1 && unexplained.length === 0 };
}
export function gradeIndependentReview(value: unknown) {
const c = checks(), o = runtimeObservation(value);
c.add("actual-delivery-and-independent-review-evidence", !!o, "Real implementation bytes, pending policy, native independent review and later source report are required."); if (!o) return c.result();
@@ -41,8 +55,9 @@ export function gradeIndependentReview(value: unknown) {
c.add("authorized-review-then-final-completion", current?.id === before.id && current.status === "done" && current.executionState?.status === "completed"
&& current.executionState.lastDecisionOutcome === "approved" && !!current.executionState.lastDecisionId
&& JSON.stringify(current.executionPolicy) === JSON.stringify(before.executionPolicy) && s.rileyApprovalBound === true && s.implementationRetained === true
&& o.runs.length === 5 && o.runs.every(row => row.status === "succeeded") && o.runs.filter(row => row.agentId === (s.reviewer as Entity).id).length === 1 && o.tasks.length === 1,
"Only Riley's actual native approval resolves the same configured review stage and completes the same work while retaining Morgan's artifact.");
&& o.runs.length === 5 + (s.ownerContinuationRunId ? 1 : 0) && o.runs.every(row => row.status === "succeeded") && o.runs.filter(row => row.agentId === (s.reviewer as Entity).id).length === 1 && o.tasks.length === 1
&& (!s.ownerContinuationRunId || o.runs.some(row => row.id === s.ownerContinuationRunId && row.agentId === (s.worker as Entity).id && runIssueId(row) === before.id)),
"Only Riley's actual native approval resolves the same configured review stage and completes the same work while retaining Morgan's artifact. Morgan may route the review once when the CEO's authorization comment wakes Morgan; no other run is admitted.");
c.add("own-later-reply-reports-the-resolved-task", s.finalReplyAttributed === true && s.finalReplyLinksTask === true,
"A later declared human question receives its own run-bound reply that links the governed task, already approved and done in saved state. Whether its wording claims the wrong phase is left to a human reader.");
return c.result();
@@ -72,17 +87,21 @@ export const runtimeIndependentReviewCases = [runtimeDefinition("task-coordinati
return { worker, reviewer, gate: prepared.gate, task: delivered.task, implementationBefore: delivered.task, marker, artifact, implementationRunId: delivered.runs[0]!.id,
actualImplementation: true, setupReviewRunCount: 0, setupRunIds: acknowledged.runs.map(row => row.id), reviewerEligibleBeforeQuery: reviewer.lifecycleState === "ready" && ["active", "idle"].includes(reviewer.status ?? "") };
}, gradeIndependentReview, {
authentication: "human-session", setupHumanMessageCount: 1, expectedHumanMessageCount: 3, expectedRunCount: 5, attemptTimeoutMs: 1_020_000,
oracleVersion: "paperclip.chat-catalog.runtime.independent-review.v4",
sourceFiles: ["chat-catalog-domains/runtime-independent-review.ts", "chat-catalog-domains/runtime-support.ts", "chat-catalog-domains/runtime-busy-worker.ts", "chat-catalog-domains/identity-support.ts", "registered-artifact.ts", "chat-scenarios/task-coordination.yaml"],
authentication: "human-session", setupHumanMessageCount: 1, expectedHumanMessageCount: 3, expectedRunCount: 6, minimumRunCount: 5, attemptTimeoutMs: 1_020_000,
oracleVersion: "paperclip.chat-catalog.runtime.independent-review.v5",
sourceFiles: ["chat-catalog-domains/runtime-independent-review.ts", "chat-catalog-domains/run-roles.ts", "chat-catalog-domains/runtime-support.ts", "chat-catalog-domains/runtime-busy-worker.ts", "chat-catalog-domains/identity-support.ts", "registered-artifact.ts", "chat-scenarios/task-coordination.yaml"],
async act(ctx, state) {
const task = state.task as Issue, reviewer = state.reviewer as Entity;
const roles = (v: { runs: RoleRun[]; comments: RoleComment[] }) => independentReviewRoles({ ...v, setupRunIds: state.setupRunIds as string[], leadId: ctx.input.fixtures.agent.id,
sourceId: ctx.issue()!.id, workerId: (state.worker as Entity).id, reviewerId: reviewer.id, taskId: task.id });
const taskComments = () => ctx.input.api.get<RoleComment[]>(`/api/issues/${task.id}/comments?order=asc`);
const pending = await pollUntil({ label: "Canonical authorization starts Riley's genuine independent review stage", deadlineAt: chatCatalogNativePhaseDeadline(ctx, Date.now() + 180_000),
load: async () => { await ctx.refreshIssue(); return { source: ctx.issue(), runs: await ctx.allRuns(), task: await ctx.input.api.get<Issue>(`/api/issues/${task.id}`), ready: await readFile((state.gate as { ready: string }).ready, "utf8").catch(() => "") }; },
accept: v => chatCatalogNativePhaseIsTimely(ctx) && (v.source?.conversationState === "waiting" && v.runs.length === 4 && v.task.status === "in_review" && v.task.executionState?.currentParticipant?.agentId === reviewer.id
&& v.ready === "waiting" && v.runs.some(row => row.agentId === reviewer.id && row.status === "running") && v.runs.filter(row => row.agentId !== reviewer.id).every(row => row.status === "succeeded")),
reject: v => v.runs.length > 4 || v.runs.some(row => ["failed", "cancelled", "timed_out"].includes(row.status)) ? "The authorized independent review failed, duplicated, or bypassed the saved gate" : undefined });
load: async () => { await ctx.refreshIssue(); return { source: ctx.issue(), runs: await ctx.allRuns(), comments: await taskComments(), task: await ctx.input.api.get<Issue>(`/api/issues/${task.id}`), ready: await readFile((state.gate as { ready: string }).ready, "utf8").catch(() => "") }; },
accept: v => { const r = roles(v); return chatCatalogNativePhaseIsTimely(ctx) && (v.source?.conversationState === "waiting" && r.valid && r.canonical.length === 1 && r.review.length === 1 && v.task.status === "in_review" && v.task.executionState?.currentParticipant?.agentId === reviewer.id
&& v.ready === "waiting" && r.review[0]!.status === "running" && v.runs.filter(row => row.agentId !== reviewer.id).every(row => row.status === "succeeded")); },
reject: v => { const r = roles(v); return !r.valid || r.canonical.length > 1 || v.runs.some(row => ["failed", "cancelled", "timed_out"].includes(row.status)) ? "The authorized independent review failed, duplicated, or bypassed the saved gate" : undefined; } });
state.pendingReview = pending.task; state.reviewerWasReallyWaiting = true; state.pendingReply = "";
const continuation = roles(pending).continuation[0]; if (continuation) state.ownerContinuationRunId = continuation.id;
const reviewRun = pending.runs.find(row => row.agentId === reviewer.id)!; state.reviewRunId = reviewRun.id;
const canonicalRuns = pending.runs.filter(row => row.agentId === ctx.input.fixtures.agent.id && !(state.setupRunIds as string[]).includes(row.id));
if (canonicalRuns.length !== 1) throw new Error("Independent review lacks exactly one canonical source turn");
@@ -92,7 +111,10 @@ export const runtimeIndependentReviewCases = [runtimeDefinition("task-coordinati
state.pendingReplyAttributed = canonicalHumans.length === 1 && canonicalReplies.length > 0 && canonicalFull.runnerProfileJson?.nativeExecutionInput?.task?.prompt?.includes(canonicalHumans[0]!.id) === true;
state.pendingReply = canonicalReplies.map(row => row.body).join("\n"); state.pendingReplyLinksTask = replyLinksTask(String(state.pendingReply), task);
const activity = await ctx.input.api.get<ActivityEvent[]>(`/api/issues/${task.id}/activity`);
state.reviewRequestSourceBound = activity.some(row => row.agentId === ctx.input.fixtures.agent.id && row.runId === canonical.id && row.details?.status === "in_review");
// The CEO requests review itself, or its authorization comment wakes Morgan, whose own
// run routes the saved deliverable to review (a native status decision is a system write).
state.reviewRequestSourceBound = activity.some(row => row.agentId === ctx.input.fixtures.agent.id && row.runId === canonical.id && row.details?.status === "in_review")
|| !!continuation && activity.some(row => row.runId === continuation.id && (row.details?.status === "in_review" || row.details?.toStatus === "in_review"));
const keySet = credentials.get(ctx); if (!keySet) throw new Error("Independent review probes lost their actual identities");
const implementationKey = keySet.find(row => row.agentId === (state.worker as Entity).id)!, reviewKey = keySet.find(row => row.agentId === reviewer.id)!;
const selfApproval = await fetch(new URL(`/api/issues/${task.id}`, ctx.input.api.baseURL), { method: "PATCH", redirect: "error", signal: AbortSignal.timeout(15_000),
@@ -116,15 +138,16 @@ export const runtimeIndependentReviewCases = [runtimeDefinition("task-coordinati
await ctx.input.capture("actual-independent-review-pending", "Riley's actual review stage remains pending after rejected implementer self-approval", "runtime-independent-review.png");
await writeFile((state.gate as { gate: string }).gate, "The human review observation is complete. Independently inspect Morgan's actual registered website.html against the saved heading/product/signup and accessibility criteria. Record your actual verification and acceptance reference in website-review. If the real result passes, approve your own configured review stage with a substantive decision comment through the supported path; do not manufacture a result, change policy, or create tasks.", "utf8");
const completed = await pollUntil({ label: "Actual Riley native approval resolves the governed task", deadlineAt: Date.now() + 240_000,
load: async () => ({ task: await ctx.input.api.get<Issue>(`/api/issues/${task.id}`), runs: await ctx.allRuns() }),
accept: v => v.task.status === "done" && v.task.executionState?.status === "completed" && v.runs.length === 4 && v.runs.every(row => row.status === "succeeded"),
reject: v => v.runs.length > 4 || v.runs.some(row => ["failed", "cancelled", "timed_out"].includes(row.status)) ? "Actual independent review failed or admitted undeclared work" : undefined });
load: async () => ({ task: await ctx.input.api.get<Issue>(`/api/issues/${task.id}`), runs: await ctx.allRuns(), comments: await taskComments() }),
accept: v => { const r = roles(v); return v.task.status === "done" && v.task.executionState?.status === "completed" && r.valid && r.canonical.length === 1 && r.review.length === 1 && r.continuation.length === (continuation ? 1 : 0) && v.runs.every(row => row.status === "succeeded"); },
reject: v => { const r = roles(v); return !r.valid || r.canonical.length > 1 || r.continuation.length !== (continuation ? 1 : 0) || v.runs.some(row => ["failed", "cancelled", "timed_out"].includes(row.status)) ? "Actual independent review failed or admitted undeclared work" : undefined; } });
state.reviewAfter = completed.task;
await ctx.input.page.goto(source, { waitUntil: "domcontentloaded" });
const followup = `What is the actual outcome of Riley's independent review of Launch Website ${task.identifier ?? task.id}, and is the governed task now finished?`;
await sendChatMessage(ctx.input.page, followup, chatCatalogNativePhaseRemainingBudget(ctx));
const final = await pollUntil({ label: "Own source phase reports actual review and completion", deadlineAt: chatCatalogNativePhaseDeadline(ctx, Date.now() + 150_000),
load: async () => { await ctx.refreshIssue(); return { source: ctx.issue(), runs: await ctx.allRuns() }; }, accept: v => chatCatalogNativePhaseIsTimely(ctx) && (v.source?.conversationState === "waiting" && v.runs.length === 5 && v.runs.every(row => row.status === "succeeded")) });
load: async () => { await ctx.refreshIssue(); return { source: ctx.issue(), runs: await ctx.allRuns(), comments: await taskComments() }; },
accept: v => { const r = roles(v); return chatCatalogNativePhaseIsTimely(ctx) && (v.source?.conversationState === "waiting" && r.valid && r.canonical.length === 2 && r.continuation.length === (continuation ? 1 : 0) && v.runs.length === completed.runs.length + 1 && v.runs.every(row => row.status === "succeeded")); } });
const run = final.runs.find(row => !completed.runs.some(prior => prior.id === row.id))!;
const comments = await ctx.input.api.get<Entity[]>(`/api/issues/${ctx.issue()!.id}/comments?order=asc`), human = comments.filter(row => !row.authorAgentId && row.body === followup), replies = comments.filter(row => row.authorAgentId === ctx.input.fixtures.agent.id && row.createdByRunId === run.id);
const full = await ctx.input.api.get<{ runnerProfileJson?: { nativeExecutionInput?: { task?: { prompt?: string } } } }>(`/api/heartbeat-runs/${run.id}`);
tests/runner-e2e/chat-catalog-run-policy.test.ts
+12
View File
@@ -118,3 +118,15 @@ it("treats only a never-started run cancelled by its own task's terminal status
const implementation = { expectedRunCount: 3, runPolicy: { kind: "source-settled-with-active-work" as const, maximumActiveWorkerRuns: 1 } };
expect(gradeChatCatalogRuns({ ...scope, implementation, runs: [succeeded, worker, queued] }).passed).toBe(false);
});
it("admits a declared role-bound range above an exact terminal count, and nothing beyond it", () => {
const cancelled = { ...succeeded, id: "stopped", status: "cancelled" };
const ok = (id: string) => ({ ...succeeded, id });
const implementation = { expectedRunCount: 6, minimumRunCount: 4, runPolicy: { kind: "terminal-counts" as const, terminalStatusCounts: { cancelled: 1, succeeded: 3 }, maximumTerminalStatusCounts: { succeeded: 5 } } };
expect(gradeChatCatalogRuns({ ...scope, implementation, runs: [cancelled, ok("a"), ok("b"), ok("c")] }).passed).toBe(true);
expect(gradeChatCatalogRuns({ ...scope, implementation, runs: [cancelled, ok("a"), ok("b"), ok("c"), ok("d"), ok("e")] }).passed).toBe(true);
expect(gradeChatCatalogRuns({ ...scope, implementation, runs: [cancelled, ok("a"), ok("b")] }).passed).toBe(false);
expect(gradeChatCatalogRuns({ ...scope, implementation: { ...implementation, expectedRunCount: 7 }, runs: [cancelled, ok("a"), ok("b"), ok("c"), ok("d"), ok("e"), ok("f")] }).passed).toBe(false);
// A maximum may only widen a status the exact counts already declare.
const undeclared = { ...implementation, runPolicy: { ...implementation.runPolicy, maximumTerminalStatusCounts: { failed: 1 } } };
expect(gradeChatCatalogRuns({ ...scope, implementation: undeclared, runs: [cancelled, ok("a"), ok("b"), ok("c")] }).passed).toBe(false);
});
tests/runner-e2e/chat-catalog-run-policy.ts
+6 -2
View File
@@ -60,8 +60,12 @@ export function gradeChatCatalogRuns(input: {
detail: "Only edit-properties may declare the exact independently proved worker reassignment cancellation." });
}
if (c.runPolicy?.kind === "terminal-counts") {
checks.push({ id: "catalog-exact-run-outcomes", passed: Object.entries(c.runPolicy.terminalStatusCounts).every(([status, count]) => runs.filter(run => run.status === status).length === count),
detail: "The independently observed ledger matches every declared successful/cancelled/failed terminal outcome count." });
const policy = c.runPolicy, maximum = policy.maximumTerminalStatusCounts ?? {};
checks.push({ id: "catalog-exact-run-outcomes", passed: Object.entries(policy.terminalStatusCounts).every(([status, count]) => {
const actual = runs.filter(run => run.status === status).length, upper = maximum[status as keyof typeof maximum] ?? count;
return actual >= count! && actual <= Math.max(count!, upper);
}) && Object.keys(maximum).every(status => status in policy.terminalStatusCounts),
detail: "The independently observed ledger matches every declared successful/cancelled/failed terminal outcome count (or its declared role-bound range)." });
} else if (c.runPolicy?.kind === "source-settled-with-active-work") {
const sourceRuns = runs.filter(run => taskId(run) === source?.id);
const workers = runs.filter(run => taskId(run) !== source?.id);
tests/runner-e2e/chat-rendered-links.test.ts
+22 -1
View File
@@ -1,7 +1,7 @@
// @vitest-environment jsdom
import { readFileSync } from "node:fs";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { conversationLinkBindings, readConversationDomLinks, type ConversationLinkInput } from "./chat-rendered-links.js";
import { collectConversationLinks, conversationLinkBindings, readConversationDomLinks, readLatestConversationReplyRender, waitForLatestConversationReplyRender, type ConversationLinkInput } from "./chat-rendered-links.js";
const card = '<div data-thread-item-kind="project_created" data-thread-anchor="project-created:project"><article aria-label="Project created: Launch"><div><p>Project created</p><a href="/RUN/projects/project">Launch</a></div></article></div>';
const thread = (content = card) => `<div data-testid="task-chat-thread"><div data-testid="task-chat-history-content"><div data-testid="task-chat-scroller"><div id="comment-human" data-thread-item-kind="human">Create a project called Launch for the new website.</div>${content}<div id="comment-reply" data-thread-item-kind="agent">Created the Launch project. No repositories are connected yet.</div></div></div></div>`;
@@ -91,3 +91,24 @@ it("retains ordinary inline links in the supported classic issue-chat root", ()
document.querySelector('#comment-reply')!.innerHTML='<a href="/RUN/issues/task">Task</a>';
expect(collect().map(row=>[row.threadTestId,row.source,row.href])).toEqual([['thread-root','reply','/RUN/issues/task']]);
});
// Full cohort on 9296c74e8d (choose-primary, case-history, set-budget): the API saw the saved reply ~2s before
// the open conversation rendered it, so links read in between were empty although the reply linked correctly.
describe("waits for the latest saved reply to render before reading links", () => {
const page = () => ({ evaluate: async <T, A>(read: (arg: A) => T, arg: A) => read(arg) }) as unknown as Parameters<typeof collectConversationLinks>[0];
it("reads the reply's links once the reply renders after the API saw it", async () => {
document.querySelector('#comment-reply')!.remove();
setTimeout(() => document.querySelector('[data-testid="task-chat-scroller"]')!.insertAdjacentHTML("beforeend", '<div id="comment-reply"><a href="/RUN/issues/task">Task</a></div>'), 600);
const value = input(); value.projects = []; value.activity = [];
expect((await collectConversationLinks(page(), value)).map(row => row.href)).toEqual(["/RUN/issues/task"]);
});
it("does not wait when the conversation is not open, and waits at most once for a reply that never renders", async () => {
const bindings = conversationLinkBindings(input());
document.body.innerHTML = "<main>Other page</main>";
expect(readLatestConversationReplyRender({ anchors: bindings.anchors, latest: "reply" })).toBe("closed");
document.body.innerHTML = thread(); document.querySelector('#comment-reply')!.remove();
const p = page();
expect(await waitForLatestConversationReplyRender(p, bindings, 300, 50)).toBe("pending");
const started = Date.now(); expect(await waitForLatestConversationReplyRender(p, bindings, 5_000, 50)).toBe("pending"); expect(Date.now() - started).toBeLessThan(1_000);
});
});
tests/runner-e2e/chat-rendered-links.ts
+38 -1
View File
@@ -77,8 +77,45 @@ export function readConversationDomLinks(input: ReturnType<typeof conversationLi
}
return rows;
}
/** The API reports a saved reply before the open conversation renders it.
* When this conversation is open in the page, wait (bounded) for its latest
* saved lead reply to render so rendered links are not read too early. A page
* that is not showing this conversation is left alone. */
export const CONVERSATION_REPLY_RENDER_WAIT_MS = 10_000;
// A reply that did not render within one wait is not waited for again on the same page, so repeated
// observations in a poll loop pay the wait at most once per reply.
const expiredReplyWaits = new WeakMap<object, Set<string>>();
export type ConversationReplyRender = "closed" | "rendered" | "pending";
export function readLatestConversationReplyRender(input: { anchors: string[]; latest: string }): ConversationReplyRender {
const threadSelector = '[data-testid="task-chat-thread"], [data-testid="thread-root"]';
// A thread is this conversation's when every comment it renders is one of its saved comments; a thread that
// has not rendered any comment yet (still loading after navigation or reload) counts too.
const roots = Array.from(document.querySelectorAll(threadSelector)).filter(root =>
Array.from(root.querySelectorAll('[id^="comment-"]')).every(node => input.anchors.some(id => node.id === `comment-${id}`)));
if (!roots.length) return "closed";
const rendered = roots.some(root => Array.from(root.querySelectorAll('[id^="comment-"]'))
.some(node => node.id === `comment-${input.latest}` && node.getClientRects().length > 0));
return rendered ? "rendered" : "pending";
}
export async function waitForLatestConversationReplyRender(page: Pick<Page, "evaluate">, bindings: Pick<ReturnType<typeof conversationLinkBindings>, "anchors" | "replies">,
timeoutMs = CONVERSATION_REPLY_RENDER_WAIT_MS, intervalMs = 250): Promise<ConversationReplyRender> {
const latest = bindings.replies.at(-1);
if (!latest) return "rendered";
const expired = expiredReplyWaits.get(page) ?? new Set<string>();
if (expired.has(latest)) timeoutMs = 0;
// Bounded by attempts as well as time so a frozen test clock cannot spin.
for (let attempt = 0, attempts = Math.max(1, Math.ceil(timeoutMs / intervalMs)), deadlineAt = Date.now() + timeoutMs; ; attempt++) {
let state: unknown;
try { state = await page.evaluate(readLatestConversationReplyRender, { anchors: bindings.anchors, latest }); } catch { return "closed"; }
if (state !== "pending") return state === "closed" ? "closed" : "rendered";
if (attempt + 1 >= attempts || Date.now() >= deadlineAt) { expired.add(latest); expiredReplyWaits.set(page, expired); return "pending"; }
await new Promise(resolve => setTimeout(resolve, intervalMs));
}
}
export async function collectConversationLinks(page: Page, input: ConversationLinkInput) {
return page.evaluate(readConversationDomLinks, conversationLinkBindings(input));
const bindings = conversationLinkBindings(input);
await waitForLatestConversationReplyRender(page, bindings);
return page.evaluate(readConversationDomLinks, bindings);
}
export function conversationLinkLocator(page: Page, link: RenderedConversationLink): Locator {
const thread = page.getByTestId(link.threadTestId).filter({ has: page.locator(`[id=${JSON.stringify(`comment-${link.anchorCommentId}`)}]`) });