Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-10 20:46:28 +01:00
Code Issues Packages Projects Releases Wiki Activity

test: isolate chat case modules and fingerprint executable dependencies

Browse Source 
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip committed 2026-10-26 16:58:34 -05:00
1 parent bc7b66912b
commit de73104b34
25 files changed
+1631 -505
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

doc/plans/2026-10-26-chat-catalog-eval-progress.md
+21 -4
View File
@@ -4,15 +4,32 @@ Updated: 2026-10-26. This is the single campaign progress record. Exact per-scen
- Baseline: `8373d7867963674ff960d2345d0d28086f799bc7`, branch `codex/chat-catalog-evals`. Integrator confirmed PR #17794 merged via `2ee36a4`.
- Frozen denominator: **244 scenarios / 35 areas / 191 capabilities**; **196 passes required**. Catalog digest: `9dc9cf6a5426cf1b56bc30a3571851f233f9dd5e93a1292eaee414d33b52df77`. Full query/context/outcome and YAML hashes are preserved. Defaults are hashed as well.
- Initial state: 2 implemented, 242 unimplemented, 0 attempted, 0 passes; first/final rates **0/244 (0%)**. Unrun, blocked and unsupported stay in denominator. Registration is not implementation.
- Domain implementation update: first 10 internal primary flows are written and calibrated by `eval_domains`; registry/shared integration and full conditional review are pending. `goals.align-existing-work` and `goals.explain-purpose` are explicitly partial because cross-company rejection and missing/inaccessible variants are absent. Partial cases cannot reserve an acceptance attempt. They are not yet counted as campaign-ready. The worker owns 45 catalog scenarios across tasks/projects/goals/documents-artifacts/inbox-search/activity; two legacy task cases remain integrator-owned.
- Campaign state: **13 implemented, 2 partial, 229 unimplemented; 0 attempted, 0 passes; first/final rates 0/244 (0%)**. The two original task journeys and eleven complete domain definitions are implemented; `tasks.edit-properties` retains an execution-ownership-denial gap and `documents-artifacts.restore-document` retains a concurrent-edit gap. Registration and calibration remain separate from live acceptance.
- Domain implementation update: 13 domain definitions are integrated across projects, tasks, goals, documents and activity. Goal retry, cross-company rejection, missing/inaccessible purpose chains, task clarification, document annotations and scoped company overview now have explicit phases and case-specific checks. The worker owns 45 catalog scenarios across tasks/projects/goals/documents-artifacts/inbox-search/activity; two legacy task implementations retain their original shared driver.
- Owners: manager `/root`; baseline/shared framework `eval_integrator`; initial domain implementations `eval_domains`; sole live execution/ledger/report owner `eval_campaign`. Area ownership is reconciled with domain assignments in the manifest before campaigns.
- Boundaries: one live instance at a time initially; zero automatic retries; every selected-case attempt counts, including setup/infrastructure failures. Maximum one initial and one coherent fix/rerun per initially failing scenario. Existing measurements are historical, not new campaign attempts.
- Host readiness: node/pnpm/python/docker/codex and dependencies present. Provider credential variables and `.env.runner-e2e.local` absent. Existing private Codex auth file present; local subscription may be loaded directly into a child environment without printing or persisting its value. Daytona credentials and immutable image are absent. No external-connected integration is qualified by simulations.
- Credential-free manifest generation passed using Node's tsx loader. The tsx CLI's IPC socket is denied by the sandbox; this was tooling inspection, before any selected-case attempt. It is not model execution or a free selected-case retry.
- Preflight: native runner binary is executable; discovery lists exactly the two legacy cells (`gpt-5.6-sol`, subscription). Read-only host IPC measurement found 86,723/87,381 semaphores consumed and 16/32 shared-memory slots used. One isolated instance is plausible; recheck immediately before launch and do not start if exhausted. No IPC cleanup performed. Ledger policy calibration passed 6/6 focused tests. Direct E2E TypeScript checking reported domain `plan` nullability (owner notified) and existing server/plugin-built-export errors; no campaign-helper errors. Full local suites were not launched.
- Next: wait for the integrator definition closure barrier, create an immutable integration snapshot, reserve the backlog initial attempt, run the smallest registered case, and retain the standard sanitized evidence, native grading and cleanup result. No live databases or arbitrary host IPC cleanup are authorized.
- Preflight: native runner binary is executable; original discovery listed the two legacy cells (`gpt-5.6-sol`, subscription). Read-only host IPC measurement found 86,723/87,381 semaphores consumed and 16/32 shared-memory slots used. One isolated instance is plausible; recheck immediately before launch and do not start if exhausted. No IPC cleanup performed. Ledger policy calibration passed 6/6 focused tests. Targeted plugin SDK and Runner TypeScript builds repaired stale generated declarations without production source changes. E2E TypeScript checking then passed after the identity owner corrected its in-progress nullability error. Full local suites were not launched.
- Integration snapshot: `bc7b66912ba0c84e076aa201dd3bb6b7b7dffc95` contains the first ten executable domain definitions plus the two legacy cells. The initial snapshot passed 48 targeted checks; the lazy/policy/domain integration now passes 71 focused checks and E2E TypeScript checking. Discovery lists 15 cells; discovery and calibration do not establish complete conditions or live acceptance. No canonical attempt is reserved.
- Architecture decision: use genuinely lazy domain dispatch and per-case fingerprints for incremental acceptance. Pure descriptors support discovery; only the selected domain executes. Fingerprints bind its actual static local-import closure, shared runtime/grader/fixture inputs, normalized selected execution/descriptor, exact query and defaults. Whole source revision and global suite hash remain separate provenance. The actual selected projects closure contains 493 source files and no unresolved dynamic sites. Six known dynamic sites use reviewed source-hash-pinned resolver declarations: actual worker closure, pinned SDK package identities, and explicitly inactive external launch branches. A changed site falls back conservatively; activating an installed/external target refuses the reviewed local-chat fingerprint. No global loader hook is used. Shared-policy work must preserve scenario-specific funding, paused/readiness and cancellation/denial outcomes with explicit independently checked policies.
- Next: finish lazy closure/typed policies and their targeted calibration, commit an immutable slice, then hand the exact source/fingerprint to the sole campaign coordinator. No acceptance run may read mutable source or carry a pass across changed executable/grader dependencies.
## Results and delivery
No campaign execution or PR yet. Per-scenario qualifying definition/grader digests remain unset until the immutable integration candidate is declared. Final pass counts require matching both per-scenario fingerprints; a later source-closure change invalidates earlier passes for that numerator while preserving their original attempts. Evidence is retained under ignored `tests/runner-e2e/results/`; no private artifact publication. Core fixes, if demonstrated, require separate focused PRs that remain unmerged; their dependent passes do not count toward the eval PR threshold. No eval merge is eligible until all 244 scenarios are genuinely implemented and processed and at least 196 pass on the declared production baseline with current definition provenance.
## Complete implementation ownership across waves
The six bounded assignments cover all 244 scenarios. Only three worker slots run concurrently with the manager; future wave owners below are reserved assignments, not running agents.
| Owner | Scenarios | Areas | State |
| --- | ---: | --- | --- |
| `eval_domains` | 45 | tasks, projects, goals, documents-artifacts, inbox-search, activity | Active; includes two unchanged legacy task cases whose shared integration remains with integrator |
| `eval_integrator` | 27 | cases, status-cards, routines, skills | After shared harness handoff |
| `eval_identity` | 44 | access, navigation-preferences, onboarding, questions-and-approvals, companies, budgets-costs | Active; owns authenticated identity helpers |
| `eval_runtime` | 60 | agents, agent-chat, runs-adapters, execution-environments, recovery, steering, task-coordination, workspaces | Future wave |
| `eval_external` | 52 | connection-setup, app-permissions, chat-channels, assistant-connections, gateways-profiles, plugins, teams, secrets | Future wave |
| `eval_campaign` | 16 | instance-operations, cli-operations, developer-labs | Implement before sole-coordinator campaign execution |
Baseline is unchanged at `8373d7867963674ff960d2345d0d28086f799bc7`; integration work is on `codex/chat-catalog-evals`. All 35 areas retain their frozen requests, contexts, outcomes and denominator membership. No core-fix PR or eval PR has been created.
tests/runner-e2e/CHAT-FEATURES.md
+24
View File
@@ -22,6 +22,30 @@ must be real or explicitly labeled simulations. The frozen campaign ledger owns
implementation readiness and all attempts separately from catalog discovery.
No model judge supplies a generic pass.
Discovery reads pure `chat-catalog-descriptors.json` metadata and does not import
any domain. The runtime loads the selected area's module and rejects descriptor
bounds/oracle/policy drift. `chat-catalog-fingerprint.ts` compiles imports to find
real local dependencies, including transitive helpers and the selected domain,
and binds the normalized execution, exact scenario/defaults, package versions and
lockfile. Whole source revision and global suite hash remain separate provenance.
The six reviewed dynamic resolutions are pinned to their callsite source hashes;
source drift falls back to conservative source coverage. Installed or external
journey overrides refuse the reviewed local-chat lane. No global loader hook is
installed. This permits unchanged cases to retain a measurement when an unrelated,
unloaded domain changes; a changed executable dependency invalidates that case.
Standard funding/readiness and successful native runs are defaults. A scenario
that preserves a special funding or paused state declares a setup policy and
must independently verify its named prerequisites before the query. Declared
terminal counts distinguish cancellation and denial from successful execution.
Creation-only journeys may explicitly allow bounded separate worker execution
while requiring a settled source and their own exact task/assignee oracle.
Authenticated identity cases select the existing human-session fixture path;
private secondary-user credentials join the attempt's redaction sink. Scenarios
whose stated context has no working chat require a genuine boundary journey;
these cannot receive a native chat pass from seeing an onboarding or login gate.
| Scenario | Actual user request | Independent outcome |
| --- | --- | --- |
| `tasks.create-backlog` | Add a task for Morgan to draft the launch announcement in Launch, but don't start it yet. | Exactly one task, Launch project, Morgan assignee, backlog at creation, no worker execution, usable chat link after reload. |
tests/runner-e2e/chat-campaign-manifest.json
+229 -229
View File
File diff suppressed because it is too large. Load diff
tests/runner-e2e/chat-catalog-contract.ts
+17
View File
@@ -10,9 +10,24 @@ export interface ChatCatalogContext {
allRuns(): Promise<ChatRun[]>;
refreshIssue(): Promise<void>;
}
export type ChatCatalogTerminalRunStatus = "succeeded" | "cancelled" | "failed" | "timed_out";
export type ChatCatalogRunPolicy =
| { kind: "terminal-counts"; terminalStatusCounts: Partial<Record<ChatCatalogTerminalRunStatus, number>> }
| { kind: "source-settled-with-active-work"; maximumActiveWorkerRuns: number };
export interface ChatCatalogSetupPolicy {
funding?: "standard" | "preserve-scenario";
readiness?: "all-ready" | "preserve-scenario";
/** Required independent prerequisite checks when preserving a scenario-specific policy. */
invariantCheckIds: readonly string[];
}
/** One concrete scenario, with production API setup and a case-specific independent oracle. */
export interface ChatCatalogCase {
scenarioId: string;
journeyKind?: "agent-chat";
authentication?: "local-trusted" | "human-session";
setupPolicy?: ChatCatalogSetupPolicy;
runPolicy?: ChatCatalogRunPolicy;
queryOutcome?: "persisted" | "rejected";
/** All case/grader helpers must be listed; the suite also fingerprints its complete source tree. */
sourceFiles: readonly string[];
oracleVersion: string;
@@ -24,6 +39,8 @@ export interface ChatCatalogCase {
minimumRunCount?: number;
turnTimeoutMs: number;
attemptTimeoutMs: number;
verifySetup?(context: ChatCatalogContext, state: Record<string, unknown>): Promise<ChatCatalogGrade>;
verifyQueryRejection?(context: ChatCatalogContext, state: Record<string, unknown>): Promise<ChatCatalogGrade>;
setup(context: ChatCatalogContext): Promise<Record<string, unknown>>;
/** Only explicit scenario user followups/approvals through visible production UI. */
act?(context: ChatCatalogContext, state: Record<string, unknown>): Promise<void>;
tests/runner-e2e/chat-catalog-descriptors.json
+219
View File
@@ -0,0 +1,219 @@
[
{
"scenarioId": "projects.create-project",
"coverageGaps": [],
"sourceFiles": [
"chat-catalog-domains/internal-definition.ts",
"chat-catalog-domains/internal-support.ts",
"chat-catalog-domains/internal-projects.ts",
"chat-scenarios/projects.yaml"
],
"oracleVersion": "paperclip.chat-catalog.internal.v1",
"expectedRunCount": 2,
"turnTimeoutMs": 120000,
"attemptTimeoutMs": 420000,
"expectedHumanMessageCount": 2,
"modulePath": "./chat-catalog-domains/internal-projects.js",
"exportName": "projectsCases"
},
{
"scenarioId": "projects.archive-project",
"coverageGaps": [],
"sourceFiles": [
"chat-catalog-domains/internal-definition.ts",
"chat-catalog-domains/internal-support.ts",
"chat-catalog-domains/internal-projects.ts",
"chat-scenarios/projects.yaml"
],
"oracleVersion": "paperclip.chat-catalog.internal.v1",
"expectedRunCount": 1,
"turnTimeoutMs": 120000,
"attemptTimeoutMs": 300000,
"modulePath": "./chat-catalog-domains/internal-projects.js",
"exportName": "projectsCases"
},
{
"scenarioId": "tasks.ambiguous-task",
"coverageGaps": [],
"sourceFiles": [
"chat-catalog-domains/internal-definition.ts",
"chat-catalog-domains/internal-support.ts",
"chat-catalog-domains/internal-tasks.ts",
"chat-scenarios/tasks.yaml"
],
"oracleVersion": "paperclip.chat-catalog.internal.v1",
"expectedRunCount": 2,
"turnTimeoutMs": 120000,
"attemptTimeoutMs": 420000,
"expectedHumanMessageCount": 2,
"modulePath": "./chat-catalog-domains/internal-tasks.js",
"exportName": "tasksCases"
},
{
"scenarioId": "tasks.edit-properties",
"coverageGaps": [
"execution-ownership-denied"
],
"sourceFiles": [
"chat-catalog-domains/internal-definition.ts",
"chat-catalog-domains/internal-support.ts",
"chat-catalog-domains/internal-tasks.ts",
"chat-scenarios/tasks.yaml"
],
"oracleVersion": "paperclip.chat-catalog.internal.v1",
"expectedRunCount": 1,
"turnTimeoutMs": 120000,
"attemptTimeoutMs": 300000,
"modulePath": "./chat-catalog-domains/internal-tasks.js",
"exportName": "tasksCases"
},
{
"scenarioId": "tasks.comment-with-reference",
"coverageGaps": [],
"sourceFiles": [
"chat-catalog-domains/internal-definition.ts",
"chat-catalog-domains/internal-support.ts",
"chat-catalog-domains/internal-tasks.ts",
"chat-scenarios/tasks.yaml"
],
"oracleVersion": "paperclip.chat-catalog.internal.v1",
"expectedRunCount": 1,
"turnTimeoutMs": 120000,
"attemptTimeoutMs": 300000,
"modulePath": "./chat-catalog-domains/internal-tasks.js",
"exportName": "tasksCases"
},
{
"scenarioId": "goals.create-goal-tree",
"coverageGaps": [],
"sourceFiles": [
"chat-catalog-domains/internal-definition.ts",
"chat-catalog-domains/internal-support.ts",
"chat-catalog-domains/internal-goals.ts",
"chat-scenarios/goals.yaml"
],
"oracleVersion": "paperclip.chat-catalog.internal.v1",
"expectedRunCount": 2,
"turnTimeoutMs": 120000,
"attemptTimeoutMs": 420000,
"expectedHumanMessageCount": 2,
"modulePath": "./chat-catalog-domains/internal-goals.js",
"exportName": "goalsCases"
},
{
"scenarioId": "goals.align-existing-work",
"coverageGaps": [],
"sourceFiles": [
"chat-catalog-domains/internal-definition.ts",
"chat-catalog-domains/internal-support.ts",
"chat-catalog-domains/internal-goals.ts",
"chat-scenarios/goals.yaml"
],
"oracleVersion": "paperclip.chat-catalog.internal.v1",
"expectedRunCount": 2,
"turnTimeoutMs": 120000,
"attemptTimeoutMs": 420000,
"expectedHumanMessageCount": 2,
"modulePath": "./chat-catalog-domains/internal-goals.js",
"exportName": "goalsCases"
},
{
"scenarioId": "goals.explain-purpose",
"coverageGaps": [],
"sourceFiles": [
"chat-catalog-domains/internal-definition.ts",
"chat-catalog-domains/internal-support.ts",
"chat-catalog-domains/internal-goals.ts",
"chat-scenarios/goals.yaml"
],
"oracleVersion": "paperclip.chat-catalog.internal.v1",
"expectedRunCount": 3,
"turnTimeoutMs": 120000,
"attemptTimeoutMs": 600000,
"expectedHumanMessageCount": 3,
"modulePath": "./chat-catalog-domains/internal-goals.js",
"exportName": "goalsCases"
},
{
"scenarioId": "documents-artifacts.resolve-feedback",
"coverageGaps": [],
"sourceFiles": [
"chat-catalog-domains/internal-definition.ts",
"chat-catalog-domains/internal-support.ts",
"chat-catalog-domains/internal-documents-artifacts.ts",
"chat-scenarios/documents-artifacts.yaml"
],
"oracleVersion": "paperclip.chat-catalog.internal.v1",
"expectedRunCount": 1,
"turnTimeoutMs": 120000,
"attemptTimeoutMs": 300000,
"modulePath": "./chat-catalog-domains/internal-documents-artifacts.js",
"exportName": "documents_artifactsCases"
},
{
"scenarioId": "documents-artifacts.write-plan",
"coverageGaps": [],
"sourceFiles": [
"chat-catalog-domains/internal-definition.ts",
"chat-catalog-domains/internal-support.ts",
"chat-catalog-domains/internal-documents-artifacts.ts",
"chat-scenarios/documents-artifacts.yaml"
],
"oracleVersion": "paperclip.chat-catalog.internal.v1",
"expectedRunCount": 1,
"turnTimeoutMs": 120000,
"attemptTimeoutMs": 300000,
"modulePath": "./chat-catalog-domains/internal-documents-artifacts.js",
"exportName": "documents_artifactsCases"
},
{
"scenarioId": "documents-artifacts.restore-document",
"coverageGaps": [
"concurrent-edit-conflict"
],
"sourceFiles": [
"chat-catalog-domains/internal-definition.ts",
"chat-catalog-domains/internal-support.ts",
"chat-catalog-domains/internal-documents-artifacts.ts",
"chat-scenarios/documents-artifacts.yaml"
],
"oracleVersion": "paperclip.chat-catalog.internal.v1",
"expectedRunCount": 1,
"turnTimeoutMs": 120000,
"attemptTimeoutMs": 300000,
"modulePath": "./chat-catalog-domains/internal-documents-artifacts.js",
"exportName": "documents_artifactsCases"
},
{
"scenarioId": "activity.company-overview",
"coverageGaps": [],
"sourceFiles": [
"chat-catalog-domains/internal-definition.ts",
"chat-catalog-domains/internal-support.ts",
"chat-catalog-domains/internal-activity.ts",
"chat-scenarios/activity.yaml"
],
"oracleVersion": "paperclip.chat-catalog.internal.v1",
"expectedRunCount": 1,
"turnTimeoutMs": 120000,
"attemptTimeoutMs": 300000,
"modulePath": "./chat-catalog-domains/internal-activity.js",
"exportName": "activityCases"
},
{
"scenarioId": "activity.audit-change",
"coverageGaps": [],
"sourceFiles": [
"chat-catalog-domains/internal-definition.ts",
"chat-catalog-domains/internal-support.ts",
"chat-catalog-domains/internal-activity.ts",
"chat-scenarios/activity.yaml"
],
"oracleVersion": "paperclip.chat-catalog.internal.v1",
"expectedRunCount": 1,
"turnTimeoutMs": 120000,
"attemptTimeoutMs": 300000,
"modulePath": "./chat-catalog-domains/internal-activity.js",
"exportName": "activityCases"
}
]
tests/runner-e2e/chat-catalog-domains/internal-activity.ts
+111
View File
@@ -0,0 +1,111 @@
import type { ChatCatalogCase } from "../chat-catalog-contract.js";
import { project, task, goal, putDocument, entity, observation, checks, hasLink, sameTask, type Entity } from "./internal-support.js";
import { makeDefinition, retryDefinition, fixtureWorker } from "./internal-definition.js";
export function grade_activity(value: unknown) {
const c = checks(); const o = observation(value);
c.add("complete-independent-observation", !!o, "Missing API, document, audit or rendered-link evidence cannot pass.");
if (!o) return c.result();
const before = o.state.task ? entity(o.state, "task") : undefined;
const current = before && o.tasks.find(row => row.id === before.id);
const savedProject = o.state.project ? entity(o.state, "project") : undefined;
const projectNow = savedProject && o.projects.find(row => row.id === savedProject.id);
const authored = (action: string, id: string) => o.activity.some(row => row.action === action && row.entityId === id && row.agentId === o.leadId);
const linkedTask = () => !!current && hasLink(o, "issues", current);
switch (o.scenarioId) {
case "activity.company-overview": {
const blocked = entity(o.state, "blockedTask"); const done = entity(o.state, "doneTask"); const worker = entity(o.state, "worker");
const expected = [before, blocked, done].filter((row): row is Entity => !!row);
c.add("actual-work-summary", expected.length === 3 && expected.every(row => sameTask(row, o.tasks.find(now => now.id === row.id)) && hasLink(o, "issues", row))
&& /\b(?:active|in.progress|underway)\b/i.test(o.reply) && /\bblocked\b/i.test(o.reply) && /\b(?:completed|done|finished)\b/i.test(o.reply),
"The overview links the actual active, blocked and completed work without changing its state.");
c.add("grounded-blocker", o.reply.includes(String(o.state.blocker)), "The summary describes the fixture's actual recorded blocker.");
c.add("idle-is-not-broken", o.agents?.some(row => row.id === worker.id && row.status === "idle") === true
&& /\bMorgan\b[^.\n]*\b(?:idle|available|not currently working)\b/i.test(o.reply)
&& !/\bMorgan\b\s+(?:is|looks|appears)\s+(?:broken|failed|stuck)/i.test(o.reply),
"The actual idle engineer is described as available, without claiming a runtime failure.");
const asserted = [...o.reply.matchAll(/\b(\d+)\s+(active|blocked|completed|done)\s+(?:tasks?|items?)\b/gi)];
c.add("scoped-counts", asserted.every(match => Number(match[1]) === 1) && !o.reply.includes(String(o.state.hiddenMarker)),
"Any explicit state counts match the three visible tasks, and hidden-company work is absent.");
c.add("observation-context", /\b(?:as of|right now|currently|at present)\b/i.test(o.reply), "The overview identifies its current observation context.");
break;
}
case "activity.audit-change": {
const priorityEvents = (o.state.priorityEvents as Entity[] | undefined) ?? [];
c.add("real-two-change-history", priorityEvents.length === 2 && priorityEvents.every(event => o.activity.some(row => row.id === event.id)), "Both fixture priority changes remain visible on the exact task.");
c.add("attribution-and-values", priorityEvents.length === 2 && priorityEvents.every(event => !!event.actorId
&& (o.reply.includes(event.actorId) || event.actorId === "local-board" && /\b(?:local board|board)\b/i.test(o.reply))
&& o.reply.toLowerCase().includes(String(event.details?.priority).toLowerCase())),
"Editor identity is the recorded actor ID or the canonical Local board label, never a generic operator inference.");
c.add("recorded-event-times", priorityEvents.length === 2 && priorityEvents.every(event => mentionsRecordedTime(o.reply, event.createdAt,
String(o.state.observedAt ?? ""), String(o.state.timezone ?? "UTC"), priorityEvents.map(row => row.createdAt))),
"Reported calendar and clock times match the actual retained events in UTC or the browser's declared timezone.");
c.add("read-only-history", !!before && sameTask(before, current) && linkedTask() && !o.activity.some(row => row.entityId === before.id && row.agentId === o.leadId), "Reading task audit does not mutate the task or manufacture audit records.");
break;
}
default: c.add("implemented-scenario", false, "No independent oracle is implemented for this scenario.");
}
return c.result();
}
const define = (id: string, setup: ChatCatalogCase["setup"]) => makeDefinition(id, setup, grade_activity);
export const activityCases: readonly ChatCatalogCase[] = [
companyOverview(),
define("activity.audit-change", async ctx => {
const p = await project(ctx); let t = await task(ctx, "Website task", p.id);
await ctx.input.api.patch(`/api/issues/${t.id}`, { priority: "high" });
t = await ctx.input.api.patch(`/api/issues/${t.id}`, { priority: "low" });
const activity = await ctx.input.api.get<Entity[]>(`/api/issues/${t.id}/activity`);
return { project: p, task: t, observedAt: new Date().toISOString(), timezone: await ctx.input.page.evaluate(() => Intl.DateTimeFormat().resolvedOptions().timeZone), priorityEvents: activity.filter(row => row.action === "issue.updated" && row.details?.priority) };
}),
];
export function mentionsRecordedTime(reply: string, recordedAt: string | undefined, observedAt: string, timezone: string, allRecordedAt: Array<string | undefined> = []): boolean {
const date = new Date(recordedAt ?? ""); const observed = new Date(observedAt);
if (!Number.isFinite(date.getTime()) || !Number.isFinite(observed.getTime())) return false;
const zones = [...new Set(["UTC", timezone])];
try {
const dates = zones.map(timeZone => new Intl.DateTimeFormat("en-CA", { timeZone, year: "numeric", month: "2-digit", day: "2-digit" }).format(date));
const times = zones.map(timeZone => new Intl.DateTimeFormat("en-GB", { timeZone, hour: "2-digit", minute: "2-digit", hourCycle: "h23" }).format(date));
const explicitDates = [...reply.matchAll(/\b\d{4}-\d{2}-\d{2}\b/g)].map(match => match[0]);
const explicitTimes = [...reply.matchAll(/\b(\d{1,2}):(\d{2})(?::\d{2})?\s*(AM|PM)?\b/gi)].map(match => {
let hour = Number(match[1]); if (match[3]) hour = hour % 12 + (match[3].toUpperCase() === "PM" ? 12 : 0);
return `${String(hour).padStart(2, "0")}:${match[2]}`;
});
const allDates = allRecordedAt.length ? allRecordedAt.flatMap(value => zones.map(timeZone => new Intl.DateTimeFormat("en-CA", {
timeZone, year: "numeric", month: "2-digit", day: "2-digit",
}).format(new Date(value ?? "")))) : dates;
const allTimes = allRecordedAt.length ? allRecordedAt.flatMap(value => zones.map(timeZone => new Intl.DateTimeFormat("en-GB", {
timeZone, hour: "2-digit", minute: "2-digit", hourCycle: "h23",
}).format(new Date(value ?? "")))) : times;
if (!explicitTimes.some(time => times.includes(time)) || explicitTimes.some(time => !allTimes.includes(time))) return false;
if (explicitDates.length) return explicitDates.some(day => dates.includes(day)) && explicitDates.every(day => allDates.includes(day));
if (/\btoday\b/i.test(reply)) return zones.some((timeZone, index) => dates[index] === new Intl.DateTimeFormat("en-CA", {
timeZone, year: "numeric", month: "2-digit", day: "2-digit",
}).format(observed));
return zones.some(timeZone => {
const month = new Intl.DateTimeFormat("en-US", { timeZone, month: "long" }).format(date);
const day = new Intl.DateTimeFormat("en-US", { timeZone, day: "numeric" }).format(date);
return new RegExp(`\\b${month}\\s+${day}\\b`, "i").test(reply);
});
} catch { return false; }
}
function companyOverview(): ChatCatalogCase {
const base = define("activity.company-overview", async ctx => {
const p = await project(ctx); const worker = await fixtureWorker(ctx);
const active = await task(ctx, "Website draft", p.id, { status: "in_progress" });
const blocker = `approved launch colors ${ctx.input.nonce}`;
const blockedTask = await task(ctx, "Accessibility review", p.id, { status: "blocked", description: `Waiting for ${blocker}.` });
const doneTask = await task(ctx, "Launch brief delivered", p.id, { status: "done" });
const hiddenCompany = await ctx.input.api.post<Entity>("/api/companies", { name: `Hidden company ${ctx.input.nonce}` });
const hiddenMarker = `Hidden acquisition ${ctx.input.nonce}`;
await ctx.input.api.post(`/api/companies/${hiddenCompany.id}/issues`, { title: hiddenMarker, status: "blocked" });
return { project: p, task: active, blockedTask, doneTask, worker, blocker, hiddenCompany, hiddenMarker };
});
return { ...base, async observe(ctx, state) {
const o = await observeInternal(ctx, state);
return { ...o, agents: await ctx.input.api.get<Entity[]>(`/api/companies/${ctx.input.fixtures.company.id}/agents`) };
}, async cleanup(ctx, state) { await ctx.input.api.delete(`/api/companies/${entity(state, "hiddenCompany").id}`, { allowNotFound: true }); } };
}
import { observeInternal } from "./internal-support.js";
tests/runner-e2e/chat-catalog-domains/internal-definition.ts
+47
View File
@@ -0,0 +1,47 @@
import type { ChatCatalogCase, ChatCatalogContext } from "../chat-catalog-contract.js";
import { observeInternal, verifyInternalResult, type Entity, type InternalObservation } from "./internal-support.js";
import { codexSubscriptionAgentConfig } from "../codex-subscription-fixture.js";
import { pollUntil } from "../api.js";
export function makeDefinition(scenarioId: string, setup: ChatCatalogCase["setup"], grade: ChatCatalogCase["grade"]): ChatCatalogCase {
const coverageGaps = scenarioId === "goals.align-existing-work" ? ["cross-company-goal-rejection"]
: scenarioId === "goals.explain-purpose" ? ["missing-association", "inaccessible-goal-nondisclosure"]
: scenarioId === "documents-artifacts.restore-document" ? ["concurrent-edit-conflict"]
: scenarioId === "tasks.edit-properties" ? ["execution-ownership-denied"] : [];
return { scenarioId, setup, coverageGaps, sourceFiles: ["chat-catalog-domains/internal-definition.ts", "chat-catalog-domains/internal-support.ts", `chat-catalog-domains/internal-${scenarioId.split(".")[0]}.ts`, `chat-scenarios/${scenarioId.split(".")[0]}.yaml`],
oracleVersion: "paperclip.chat-catalog.internal.v1", expectedRunCount: 1, verifyResult: verifyInternalResult, turnTimeoutMs: 120_000, attemptTimeoutMs: 300_000, observe: observeInternal, grade };
}
export async function fixtureWorker(ctx: ChatCatalogContext) {
const { execution, fixtures, workspacePath, nonce, api } = ctx.input;
const base = execution.profile.buildAgent({ environmentId: fixtures.environment.id, environmentFixtureId: execution.environment.id,
workspacePath, secretRefs: fixtures.secretRefs, executionId: nonce });
const payload = { ...base, name: "Morgan", role: "engineer", title: "Frontend engineer", reportsTo: fixtures.agent.id,
capabilities: "Builds accessible websites.", runtimeConfig: { ...(base.runtimeConfig as Record<string, unknown>), heartbeat: { enabled: false, maxConcurrentRuns: 1 } } };
const worker = await api.post<Entity>(`/api/companies/${fixtures.company.id}/agents`, execution.profile.credential === "CODEX_AUTH_JSON"
? codexSubscriptionAgentConfig(payload, process.env, fixtures.company.id) : payload);
await pollUntil({ label: "Morgan runtime readiness", deadlineAt: Date.now() + 45_000,
load: () => api.get<{ lifecycleState: string; lifecycleError?: string }>(`/api/agents/${worker.id}`),
accept: row => row.lifecycleState === "ready", reject: row => row.lifecycleError ? "Morgan runtime setup failed" : undefined });
return worker;
}
export function retryDefinition(scenarioId: string, grade: ChatCatalogCase["grade"], initialOutcome: (observation: InternalObservation) => boolean): ChatCatalogCase {
const definition = makeDefinition(scenarioId, async ctx => ({ retryQuery: ctx.scenario.query }), grade);
return { ...definition, expectedRunCount: 2, expectedHumanMessageCount: 2, attemptTimeoutMs: 420_000,
async act(ctx, state) {
const first = await pollUntil({ label: "Initial creation before explicit user retry", deadlineAt: Date.now() + 150_000,
load: () => observeInternal(ctx, state),
accept: o => {
const rows = scenarioId === "projects.create-project" ? o.projects.filter(row => row.name === "Launch") : o.goals;
return rows.length === (scenarioId === "projects.create-project" ? 1 : 2) && initialOutcome(o) && ctx.issue()?.conversationState === "waiting"
&& o.runs.length === 1 && o.runs.every(row => row.status === "succeeded");
},
});
state.firstCreation = { ...first, state: structuredClone(state) };
state.firstIds = (scenarioId === "projects.create-project" ? first.projects.filter(row => row.name === "Launch") : first.goals).map(row => row.id);
await ctx.input.evidence("chat-catalog-before-explicit-retry.json", { firstIds: state.firstIds, observation: first });
const { sendChatMessage } = await import("../chat-flow.js");
await sendChatMessage(ctx.input.page, ctx.scenario.query);
},
};
}
tests/runner-e2e/chat-catalog-domains/internal-documents-artifacts.ts
+108
View File
@@ -0,0 +1,108 @@
import type { ChatCatalogCase } from "../chat-catalog-contract.js";
import { project, task, goal, putDocument, entity, observation, checks, hasLink, sameTask, type Entity } from "./internal-support.js";
import { makeDefinition, retryDefinition, fixtureWorker } from "./internal-definition.js";
export function grade_documents_artifacts(value: unknown) {
const c = checks(); const o = observation(value);
c.add("complete-independent-observation", !!o, "Missing API, document, audit or rendered-link evidence cannot pass.");
if (!o) return c.result();
const before = o.state.task ? entity(o.state, "task") : undefined;
const current = before && o.tasks.find(row => row.id === before.id);
const savedProject = o.state.project ? entity(o.state, "project") : undefined;
const projectNow = savedProject && o.projects.find(row => row.id === savedProject.id);
const authored = (action: string, id: string) => o.activity.some(row => row.action === action && row.entityId === id && row.agentId === o.leadId);
const linkedTask = () => !!current && hasLink(o, "issues", current);
switch (o.scenarioId) {
case "documents-artifacts.write-plan": {
const plans = o.documents.filter(row => row.key === "plan"); const plan = plans[0];
c.add("saved-implementation-plan", plans.length === 1 && !!plan && !!plan.latestRevisionId && /\b(?:implement|build|develop)\b/i.test(plan.body ?? "")
&& /\b(?:test|verify|validation)\b/i.test(plan.body ?? "") && (plan.body ?? "").includes(String(o.state.requirement)), "A durable plan reflects the website's actual prerequisite brief and verification, rather than merely chat prose.");
c.add("versioned-author", !!plan && o.revisions.some(row => row.id === plan.latestRevisionId && row.body === plan.body && row.createdByAgentId === o.leadId
&& o.runs.some(run => run.id === row.createdByRunId)), "The current document revision is attributed to this real agent run.");
c.add("plan-audit", !!before && authored("issue.document_created", before.id), "The API audit records the CEO creating the plan after the request.");
c.add("task-context-and-link", !!before && sameTask(before, current) && linkedTask(), "Saving a plan preserves the task and makes the saved plan reachable through its task.");
break;
}
case "documents-artifacts.resolve-feedback": {
const target = entity(o.state, "pricingThread"); const sibling = entity(o.state, "siblingThread");
const annotation = o.annotations?.find(row => row.id === target.id); const unrelated = o.annotations?.find(row => row.id === sibling.id);
const plan = o.documents.find(row => row.key === "plan");
c.add("substantive-pricing-fix", !!plan && /\$19/.test(plan.body ?? "") && /\b(?:monthly|month)\b/i.test(plan.body ?? "")
&& /\bteam\b/i.test(plan.body ?? "") && /\b(?:total|entire|whole|per team|for the team)\b/i.test(plan.body ?? "")
&& !/\b(?:per|each)\s+(?:user|person|seat)\b/i.test(plan.body ?? "") && (plan.body ?? "").includes(String(o.state.supportText)),
"The actual current document unambiguously preserves $19 monthly total for the team and unrelated support content.");
c.add("correct-thread-resolved", annotation?.status === "resolved" && annotation.resolvedByAgentId === o.leadId
&& unrelated?.status === "open" && annotation.documentId === plan?.id && annotation.originalRevisionId === o.state.originalRevisionId,
"Only the anchored pricing annotation is resolved by the CEO, preserving its original document and revision relationship.");
const edited = o.activity.find(row => row.action === "issue.document_updated" && row.agentId === o.leadId);
const resolved = o.activity.find(row => row.action === "issue.document_annotation_thread_resolved" && row.agentId === o.leadId && row.details?.threadId === target.id);
c.add("edit-before-resolution", !!edited && !!resolved && Date.parse(edited.createdAt ?? "") <= Date.parse(resolved.createdAt ?? "")
&& !!plan && o.revisions.some(row => row.id === plan.latestRevisionId && row.createdByAgentId === o.leadId && row.body === plan.body),
"Saved revision and independent audit events establish the substantive edit before resolution.");
c.add("task-and-document-access", !!before && sameTask(before, current) && linkedTask(), "Chat links the changed document's task without modifying task execution.");
break;
}
case "documents-artifacts.restore-document": {
const plan = o.documents.find(row => row.key === "plan"); const revisions = o.state.originalRevisions as Entity[];
const previous = revisions?.[1];
c.add("restored-exact-previous-bytes", !!previous && !!plan && plan.body === previous.body && plan.latestRevisionId !== previous.id
&& plan.latestRevisionId !== revisions[2]?.id, "Restoration creates a new current revision with the exact second revision's contents.");
c.add("retained-history", Array.isArray(revisions) && revisions.length === 3 && o.revisions.length === 4
&& revisions.every(original => o.revisions.some(row => row.id === original.id && row.body === original.body)), "All three historical revisions remain unmodified alongside the restoration.");
c.add("restore-path-attribution", !!before && o.activity.some(row => row.action === "issue.document_restored" && row.entityId === before.id
&& row.agentId === o.leadId && row.details?.restoredFromRevisionId === previous?.id), "An independently observed restore audit identifies the exact source revision.");
c.add("no-execution-rewind", !!before && sameTask(before, current) && linkedTask() && /\b(?:revision|version)\b/i.test(o.reply), "Chat identifies a document revision and leaves task execution unchanged.");
break;
}
default: c.add("implemented-scenario", false, "No independent oracle is implemented for this scenario.");
}
return c.result();
}
const define = (id: string, setup: ChatCatalogCase["setup"]) => makeDefinition(id, setup, grade_documents_artifacts);
export const documents_artifactsCases: readonly ChatCatalogCase[] = [
resolveFeedback(),
define("documents-artifacts.write-plan", async ctx => {
const p = await project(ctx); const requirement = `Orchid ${ctx.input.nonce}`;
const t = await task(ctx, "Website task", p.id, { assigneeAgentId: ctx.input.fixtures.agent.id,
description: `Build a single-page website for ${requirement}, with accessible signup and responsive layout. Verify keyboard navigation and mobile presentation.` });
return { project: p, task: t, requirement };
}),
define("documents-artifacts.restore-document", async ctx => {
const p = await project(ctx); const t = await task(ctx, "Website task", p.id, { description: "The website task's plan has three saved revisions. The previous version before the last edit is the desired content." });
const revisions: Entity[] = [];
for (const body of [`# Plan\nBuild a responsive website for ${ctx.input.nonce}.`, `# Plan\nBuild accessible signup for ${ctx.input.nonce}.\nVerify keyboard operation.`, `# Plan\nReplace signup with an unrelated pricing experiment for ${ctx.input.nonce}.`]) {
const saved = await putDocument(ctx, t.id, body, revisions.at(-1)?.id ?? null);
revisions.push({ id: saved.latestRevisionId!, body });
}
return { project: p, task: t, originalRevisions: revisions };
}),
];
function resolveFeedback(): ChatCatalogCase {
const base = define("documents-artifacts.resolve-feedback", async ctx => {
const p = await project(ctx); const t = await task(ctx, "Website task", p.id);
const pricing = "Pricing is $19 every month and $19 per user every month.";
const supportText = `Support is provided by the launch team ${ctx.input.nonce}.`;
const body = `${pricing}\n\n${supportText}`;
const doc = await putDocument(ctx, t.id, body);
const createAnnotation = (quote: string, feedback: string) => {
const start = body.indexOf(quote);
return ctx.input.api.post<Entity>(`/api/issues/${t.id}/documents/plan/annotations`, {
baseRevisionId: doc.latestRevisionId, baseRevisionNumber: doc.latestRevisionNumber,
selector: { quote: { exact: quote, prefix: "", suffix: "" },
position: { normalizedStart: start, normalizedEnd: start + quote.length, markdownStart: start, markdownEnd: start + quote.length } },
body: feedback,
});
};
const pricingThread = await createAnnotation(pricing, "Clarify that $19 is the monthly total for the team, not per person. Keep the amount and billing cadence.");
const siblingThread = await createAnnotation(supportText, "Please add a support contact later; leave this unrelated comment open for now.");
return { project: p, task: t, pricingThread, siblingThread, supportText, originalRevisionId: doc.latestRevisionId };
});
return { ...base, async observe(ctx, state) {
const o = await observeInternal(ctx, state);
return { ...o, annotations: await ctx.input.api.get<Entity[]>(`/api/issues/${entity(state, "task").id}/documents/plan/annotations?status=all&includeComments=true`) };
} };
}
import { observeInternal } from "./internal-support.js";
tests/runner-e2e/chat-catalog-domains/internal-goals.ts
+176
View File
@@ -0,0 +1,176 @@
import type { ChatCatalogCase } from "../chat-catalog-contract.js";
import { project, task, goal, putDocument, entity, observation, checks, hasLink, sameTask, type Entity, type InternalObservation } from "./internal-support.js";
import { makeDefinition, retryDefinition, fixtureWorker } from "./internal-definition.js";
export function grade_goals(value: unknown) {
const c = checks(); const o = observation(value);
c.add("complete-independent-observation", !!o, "Missing API, document, audit or rendered-link evidence cannot pass.");
if (!o) return c.result();
const before = o.state.task ? entity(o.state, "task") : undefined;
const current = before && o.tasks.find(row => row.id === before.id);
const savedProject = o.state.project ? entity(o.state, "project") : undefined;
const projectNow = savedProject && o.projects.find(row => row.id === savedProject.id);
const authored = (action: string, id: string) => o.activity.some(row => row.action === action && row.entityId === id && row.agentId === o.leadId);
const linkedTask = () => !!current && hasLink(o, "issues", current);
switch (o.scenarioId) {
case "goals.create-goal-tree": {
const companyGoals = o.goals.filter(row => row.level === "company" && /(?:1[,.]?000|thousand)\s+signups?/i.test(`${row.title} ${row.description ?? ""}`));
const teamGoals = o.goals.filter(row => row.level === "team" && /\b(?:launch|launching)\b.*\bwebsite\b|\bwebsite\b.*\blaunch\b/i.test(`${row.title} ${row.description ?? ""}`));
c.add("one-requested-tree", o.goals.length === 2 && companyGoals.length === 1 && teamGoals.length === 1
&& teamGoals[0]?.parentId === companyGoals[0]?.id && companyGoals[0]?.parentId == null
&& o.goals.every(row => row.companyId === o.companyId), "The two saved goals have exact levels, a valid parent and company scope.");
c.add("attributed-goal-creation", o.goals.length === 2 && o.goals.every(row => authored("goal.created", row.id)), "Both requested goals were created by the CEO through the production path.");
c.add("ownership-or-question", o.goals.every(row => row.ownerAgentId === o.leadId) || /\b(?:who|which)\b[^\n]*\b(?:own|owner|responsib)/i.test(o.reply), "Goal ownership is valid or its consequential missing input is explicit.");
c.add("hierarchy-link", o.goals.length === 2 && o.goals.every(row => hasLink(o, "goals", row)), "The reply links both saved levels for inspection.");
c.add("retry-reuses-first-tree", initialGoalTreeCreation(observation(o.state.firstCreation)) && Array.isArray(o.state.firstIds) && o.state.firstIds.length === 2
&& o.goals.every(row => (o.state.firstIds as string[]).includes(row.id))
&& o.humanQueries.filter(query => query === o.state.retryQuery).length === 2, "An explicit unchanged request reuses the initial goal tree.");
c.add("no-invented-progress", !/\b\d+\s*%\s*(?:complete|done|progress)|\balready\s+\d+\s+signups?/i.test(o.reply), "New goals do not claim fabricated progress metrics.");
break;
}
case "goals.align-existing-work": {
const growth = entity(o.state, "growth");
c.add("saved-associations", !!current && current.goalId === growth.id && !!projectNow
&& (projectNow.goalId === growth.id || projectNow.goalIds?.includes(growth.id) === true), "The intended Launch and website task both reference the saved growth goal.");
c.add("cross-company-request-denied", !!o.state.authorizedAlignment
&& typeof o.state.foreignRequest === "string" && o.humanQueries.includes(o.state.foreignRequest)
&& /\b(?:cannot|can't|unable|not authorized|not accessible|different company|another company|cross-company|switch compan)\b/i.test(o.reply)
&& !o.reply.includes(String(o.state.protectedText))
&& !o.tasks.some(row => row.goalId === entity(o.state, "foreignGoal").id)
&& !o.projects.some(row => row.goalId === entity(o.state, "foreignGoal").id || row.goalIds?.includes(entity(o.state, "foreignGoal").id)),
"The explicit foreign-goal request is denied with no foreign contents or cross-company association.");
c.add("context-only", !!before && sameTask(before, current, ["goalId"]) && o.tasks.length === 1 && o.goals.length === 1,
"Alignment does not alter priority, ownership, execution state or duplicate records.");
c.add("attributed-alignment", !!current && authored("issue.updated", current.id) && !!projectNow && authored("project.updated", projectNow.id), "Both association writes are independently attributed.");
c.add("linked-work-and-goal", linkedTask() && !!projectNow && hasLink(o, "projects", projectNow) && hasLink(o, "goals", growth), "Chat gives inspectable task, project and strategic goal references.");
break;
}
case "goals.explain-purpose": {
const growth = entity(o.state, "growth"); const team = entity(o.state, "team");
c.add("actual-chain", current?.goalId === team.id && team.parentId === growth.id && !!projectNow
&& (projectNow.goalId === team.id || projectNow.goalIds?.includes(team.id) === true), "The observed task/project/team/company references form the fixture's real purpose chain.");
c.add("grounded-rationale", o.reply.includes(String(o.state.rationale)) && /\bwebsite\b/i.test(o.reply)
&& [growth, team].every(row => hasLink(o, "goals", row)) && linkedTask(), "The explanation uses the unusual saved business rationale and links the actual hierarchy.");
const missing = entity(o.state, "missingTask");
c.add("missing-association-is-explicit", !!o.tasks.find(row => row.id === missing.id && row.goalId == null)
&& typeof o.state.missingReply === "string"
&& /(?:no|missing|not|unlinked|hasn't|doesn't|without)[^.!?]*?(?:goal|strateg|purpose|associat)|cannot[^.!?]*?determine/i.test(o.state.missingReply)
&& !(o.state.missingReply as string).includes(String(o.state.rationale)),
"The unaligned-task phase describes the actual missing relationship instead of importing the first task's rationale.");
c.add("inaccessible-rationale-not-leaked", typeof o.state.foreignRequest === "string" && o.humanQueries.includes(o.state.foreignRequest)
&& /\b(?:cannot|can't|unable|not authorized|not accessible|different company|another company|cross-company)\b/i.test(o.reply)
&& !o.reply.includes(String(o.state.protectedText)) && !o.links.some(link => link.includes(entity(o.state, "foreignTask").id)),
"A separate inaccessible task/goal produces a scoped limitation without the foreign rationale or resource links.");
c.add("read-only-explanation", !!before && sameTask(before, current) && !o.activity.some(row => row.entityId !== o.state.sourceId && [before.id, growth.id, team.id, savedProject?.id].includes(row.entityId) && row.agentId === o.leadId && /(?:updated|created|deleted)$/.test(row.action ?? "")),
"An explanation does not rewrite the work's strategic context.");
break;
}
default: c.add("implemented-scenario", false, "No independent oracle is implemented for this scenario.");
}
return c.result();
}
const define = (id: string, setup: ChatCatalogCase["setup"]) => makeDefinition(id, setup, grade_goals);
export const goalsCases: readonly ChatCatalogCase[] = [
retryDefinition("goals.create-goal-tree", grade_goals, initialGoalTreeCreation),
crossCompanyAlignment(),
purposeVariants(),
];
/** The requested cross-company condition is a real second human request, with no imported foreign content. */
function crossCompanyAlignment(): ChatCatalogCase {
const base = define("goals.align-existing-work", async ctx => {
const growth = await goal(ctx, "Growth: reach 1,000 signups");
const p = await project(ctx); const t = await task(ctx, "Website task", p.id);
const foreignCompany = await ctx.input.api.post<Entity>("/api/companies", { name: `Other company ${ctx.input.nonce}` });
const protectedText = `Protected foreign strategy ${ctx.input.nonce}`;
const foreignGoal = await ctx.input.api.post<Entity>(`/api/companies/${foreignCompany.id}/goals`, {
title: "Other company growth", description: protectedText, level: "company",
});
return { growth, project: p, task: t, foreignCompany, foreignGoal, protectedText,
foreignRequest: `Use the growth goal ${foreignGoal.id} from our other company instead.` };
});
return { ...base, coverageGaps: [], expectedRunCount: 2, expectedHumanMessageCount: 2, attemptTimeoutMs: 420_000,
async act(ctx, state) {
const first = await pollUntil({ label: "Initial authorized goal alignment", deadlineAt: Date.now() + 150_000,
load: () => observeInternal(ctx, state),
accept: o => {
const g = entity(state, "growth"); const p = o.projects.find(row => row.id === entity(state, "project").id);
return o.tasks.find(row => row.id === entity(state, "task").id)?.goalId === g.id
&& (p?.goalId === g.id || p?.goalIds?.includes(g.id) === true)
&& o.runs.length === 1 && o.runs.every(row => row.status === "succeeded") && ctx.issue()?.conversationState === "waiting";
},
});
state.authorizedAlignment = { taskGoalId: first.tasks.find(row => row.id === entity(state, "task").id)?.goalId,
projectGoalIds: first.projects.find(row => row.id === entity(state, "project").id)?.goalIds };
await ctx.input.evidence("chat-catalog-before-cross-company-request.json", { authorizedAlignment: state.authorizedAlignment });
const { sendChatMessage } = await import("../chat-flow.js");
await sendChatMessage(ctx.input.page, String(state.foreignRequest));
},
async cleanup(ctx, state) { await ctx.input.api.delete(`/api/companies/${entity(state, "foreignCompany").id}`, { allowNotFound: true }); },
};
}
import { observeInternal } from "./internal-support.js";
import { pollUntil } from "../api.js";
function purposeVariants(): ChatCatalogCase {
const base = define("goals.explain-purpose", async ctx => {
const rationale = `quiet research teams ${ctx.input.nonce}`;
const growth = await goal(ctx, "Growth: reach 1,000 signups", { description: `We need a thousand signups to support ${rationale}.` });
const team = await goal(ctx, "Launch the website", { level: "team", parentId: growth.id });
const p = await project(ctx, "Launch", { goalIds: [team.id] });
const t = await task(ctx, "Website task", p.id, { goalId: team.id });
const missingProject = await project(ctx, "Exploration");
const missingTask = await task(ctx, "Unaligned website work", missingProject.id);
const foreignCompany = await ctx.input.api.post<Entity>("/api/companies", { name: `Inaccessible company ${ctx.input.nonce}` });
const protectedText = `Protected purpose ${ctx.input.nonce}`;
const foreignGoal = await ctx.input.api.post<Entity>(`/api/companies/${foreignCompany.id}/goals`, {
title: "Private strategic rationale", description: protectedText, level: "company",
});
const foreignTask = await ctx.input.api.post<Entity>(`/api/companies/${foreignCompany.id}/issues`, {
title: "Inaccessible website work", goalId: foreignGoal.id, status: "backlog",
});
return { growth, team, project: p, task: t, rationale, missingTask, missingProject, foreignTask, foreignCompany, protectedText,
missingRequest: `Why are we doing Unaligned website work (${missingTask.identifier ?? missingTask.id})?`,
foreignRequest: `Why are we doing the website task ${foreignTask.identifier ?? foreignTask.id} in our other company?` };
});
return { ...base, coverageGaps: [], expectedRunCount: 3, expectedHumanMessageCount: 3, attemptTimeoutMs: 600_000,
async act(ctx, state) {
const first = await waitForPurposePhase(ctx, state, 1);
state.initialPurpose = { reply: first.reply, taskGoalId: first.tasks.find(row => row.id === entity(state, "task").id)?.goalId };
const previous = await sourceReplies(ctx);
const { sendChatMessage } = await import("../chat-flow.js");
await sendChatMessage(ctx.input.page, String(state.missingRequest));
await waitForPurposePhase(ctx, state, 2);
const missing = (await sourceReplies(ctx)).filter(row => !previous.some(before => before.id === row.id));
state.missingReply = missing.map(row => row.body ?? "").join("\n");
await ctx.input.evidence("chat-catalog-purpose-conditional-phases.json", { initialPurpose: state.initialPurpose, missingReplies: missing });
await sendChatMessage(ctx.input.page, String(state.foreignRequest));
},
async cleanup(ctx, state) { await ctx.input.api.delete(`/api/companies/${entity(state, "foreignCompany").id}`, { allowNotFound: true }); },
};
}
async function waitForPurposePhase(ctx: ChatCatalogContext, state: Record<string, unknown>, count: number) {
return pollUntil({ label: `Purpose explanation phase ${count}`, deadlineAt: Date.now() + 150_000,
load: () => observeInternal(ctx, state), accept: o => o.runs.length === count && o.runs.every(row => row.status === "succeeded")
&& ctx.issue()?.conversationState === "waiting" });
}
async function sourceReplies(ctx: ChatCatalogContext) {
const source = ctx.issue(); if (!source) throw new Error("Purpose phase lost its source conversation");
const comments = await ctx.input.api.get<Entity[]>(`/api/issues/${source.id}/comments?order=asc`);
const runs = await ctx.allRuns();
return comments.filter(row => row.authorAgentId === ctx.input.fixtures.agent.id && runs.some(run => run.id === row.createdByRunId));
}
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
function initialGoalTreeCreation(o: InternalObservation | undefined): boolean {
if (!o || o.goals.length !== 2 || o.goals.some(row => row.companyId !== o.companyId || !hasLink(o, "goals", row))) return false;
const root = o.goals.find(row => row.level === "company" && /(?:1[,.]?000|thousand)\s+signups?/i.test(`${row.title} ${row.description ?? ""}`));
const team = o.goals.find(row => row.level === "team" && /\b(?:launch|launching)\b.*\bwebsite\b|\bwebsite\b.*\blaunch\b/i.test(`${row.title} ${row.description ?? ""}`));
return !!root && !!team && root.parentId == null && team.parentId === root.id
&& o.goals.every(goal => o.activity.some(row => row.action === "goal.created" && row.entityId === goal.id && row.agentId === o.leadId))
&& (o.goals.every(row => row.ownerAgentId === o.leadId) || /\b(?:who|which)\b[^\n]*\b(?:own|owner|responsib)/i.test(o.reply))
&& !/\b\d+\s*%\s*(?:complete|done|progress)|\balready\s+\d+\s+signups?/i.test(o.reply);
}
tests/runner-e2e/chat-catalog-domains/internal-projects.ts
+59
View File
@@ -0,0 +1,59 @@
import type { ChatCatalogCase } from "../chat-catalog-contract.js";
import { project, task, goal, putDocument, entity, observation, checks, hasLink, sameTask, type Entity, type InternalObservation } from "./internal-support.js";
import { makeDefinition, retryDefinition, fixtureWorker } from "./internal-definition.js";
export function grade_projects(value: unknown) {
const c = checks(); const o = observation(value);
c.add("complete-independent-observation", !!o, "Missing API, document, audit or rendered-link evidence cannot pass.");
if (!o) return c.result();
const before = o.state.task ? entity(o.state, "task") : undefined;
const current = before && o.tasks.find(row => row.id === before.id);
const savedProject = o.state.project ? entity(o.state, "project") : undefined;
const projectNow = savedProject && o.projects.find(row => row.id === savedProject.id);
const authored = (action: string, id: string) => o.activity.some(row => row.action === action && row.entityId === id && row.agentId === o.leadId);
const linkedTask = () => !!current && hasLink(o, "issues", current);
switch (o.scenarioId) {
case "projects.create-project": {
const created = o.projects.filter(row => row.name === "Launch");
c.add("one-company-project", created.length === 1 && created[0]?.companyId === o.companyId, "Exactly one Launch is saved in the active company.");
c.add("agent-created-project", created.length === 1 && authored("project.created", created[0]!.id), "Creation is attributed to the acting CEO rather than fixture setup.");
c.add("usable-project-link", created.length === 1 && hasLink(o, "projects", created[0]!), "Chat renders the exact saved project link.");
c.add("retry-reuses-first-project", initialProjectCreation(observation(o.state.firstCreation)) && Array.isArray(o.state.firstIds) && o.state.firstIds.length === 1 && created[0]?.id === o.state.firstIds[0]
&& o.humanQueries.filter(query => query === o.state.retryQuery).length === 2, "The exact user retry preserves the first saved project rather than silently creating and replacing it.");
c.add("no-invented-ready-work", o.tasks.length === 0 && !/\b(?:deployed|website is ready|repository is ready)\b/i.test(o.reply), "Creation does not invent execution, repository readiness or a deployment.");
break;
}
case "projects.archive-project": {
c.add("archived-not-deleted", !!projectNow && Number.isFinite(Date.parse(projectNow.archivedAt ?? "")), "Launch remains retrievable in the supported archived state.");
c.add("history-retained", !!before && sameTask(before, current) && o.documents.some(row => row.body === o.state.originalBody)
&& o.activity.some(row => row.id === o.state.originalActivityId), "The original task, document bytes and creation audit survive archiving.");
c.add("attributed-project-update", !!projectNow && authored("project.updated", projectNow.id), "The saved lifecycle change belongs to the acting agent.");
c.add("historical-link", !!projectNow && hasLink(o, "projects", projectNow), "Chat supplies a reachable historical project.");
break;
}
default: c.add("implemented-scenario", false, "No independent oracle is implemented for this scenario.");
}
return c.result();
}
const define = (id: string, setup: ChatCatalogCase["setup"]) => makeDefinition(id, setup, grade_projects);
export const projectsCases: readonly ChatCatalogCase[] = [
retryDefinition("projects.create-project", grade_projects, initialProjectCreation),
define("projects.archive-project", async ctx => {
const p = await project(ctx, "Launch", { status: "completed" });
const t = await task(ctx, "Website task", p.id, { status: "done" });
const body = `# Delivered website\nHistorical receipt ${ctx.input.nonce}.`;
await putDocument(ctx, t.id, body);
const activity = await ctx.input.api.get<Entity[]>(`/api/issues/${t.id}/activity`);
return { project: p, task: t, originalBody: body, originalActivityId: activity.find(row => row.action === "issue.created")?.id };
}),
];
function initialProjectCreation(o: InternalObservation | undefined): boolean {
if (!o) return false;
const projects = o.projects.filter(row => row.name === "Launch"); const p = projects[0];
return projects.length === 1 && !!p && p.companyId === o.companyId && o.tasks.length === 0 && hasLink(o, "projects", p)
&& o.activity.some(row => row.action === "project.created" && row.entityId === p.id && row.agentId === o.leadId)
&& !/\b(?:deployed|website is ready|repository is ready)\b/i.test(o.reply);
}
tests/runner-e2e/chat-catalog-domains/internal-support.ts
+10 -10
View File
@@ -4,19 +4,19 @@ import type { ChatRun } from "../chat-flow.js";
export interface Entity {
id: string; companyId?: string; name?: string; title?: string; description?: string | null;
identifier?: string; status?: string; priority?: string; assigneeAgentId?: string | null;
identifier?: string; urlKey?: string; status?: string; priority?: string; assigneeAgentId?: string | null;
projectId?: string | null; parentId?: string | null; goalId?: string | null; goalIds?: string[];
level?: string; ownerAgentId?: string | null; archivedAt?: string | null;
key?: string; body?: string; latestRevisionId?: string; latestRevisionNumber?: number;
createdByAgentId?: string | null; updatedByAgentId?: string | null; createdByRunId?: string | null;
createdAt?: string; updatedAt?: string; authorAgentId?: string | null; authorUserId?: string | null;
issueId?: string; action?: string; agentId?: string | null; actorId?: string; actorType?: string;
entityId?: string; details?: Record<string, unknown>;
entityId?: string; documentId?: string; originalRevisionId?: string | null; currentRevisionId?: string | null; resolvedByAgentId?: string | null; selectedText?: string; resolvedAt?: string | null; details?: Record<string, unknown>;
}
export interface InternalObservation {
scenarioId: string; companyId: string; leadId: string; state: Record<string, unknown>;
scenarioId: string; origin: string; companyId: string; leadId: string; state: Record<string, unknown>;
tasks: Entity[]; projects: Entity[]; goals: Entity[]; comments: Entity[];
activity: Entity[]; documents: Entity[]; revisions: Entity[]; reply: string; links: string[]; humanQueries: string[]; runs: ChatRun[];
activity: Entity[]; annotations?: Entity[]; agents?: Entity[]; documents: Entity[]; revisions: Entity[]; reply: string; links: string[]; humanQueries: string[]; runs: ChatRun[];
}
export function entity(state: Record<string, unknown>, key: string): Entity {
const value = state[key];
@@ -67,17 +67,17 @@ export async function observeInternal(ctx: ChatCatalogContext, state: Record<str
revisions.push(...await api.get<Entity[]>(`${endpoint}/revisions`));
}
}
return { scenarioId: ctx.scenario.id, companyId: fixtures.company.id, leadId: fixtures.agent.id,
return { scenarioId: ctx.scenario.id, origin: api.baseURL, companyId: fixtures.company.id, leadId: fixtures.agent.id,
state, tasks: tasks.filter(row => row.id !== source?.id), projects, goals, runs, comments, activity, documents, revisions,
humanQueries: chat.filter(row => !row.authorAgentId).map(row => row.body ?? ""), reply: replies.map(row => row.body ?? "").join("\n"), links };
}
export function hasLink(o: InternalObservation, kind: "issues" | "projects" | "goals", row: Entity): boolean {
return o.links.some(href => {
try {
const url = new URL(href, "http://fixture.invalid");
if (url.origin !== "http://fixture.invalid") return false;
const url = new URL(href, o.origin);
if (url.origin !== new URL(o.origin).origin) return false;
const parts = url.pathname.split("/").map(decodeURIComponent); const index = parts.indexOf(kind);
return index >= 0 && [row.id, row.identifier].includes(parts[index + 1]);
return index >= 0 && [row.id, row.identifier, row.urlKey].includes(parts[index + 1]);
} catch { return false; }
});
}
@@ -85,7 +85,7 @@ export function observation(value: unknown): InternalObservation | undefined {
if (!value || typeof value !== "object") return;
const o = value as InternalObservation;
if (!o.state || ![o.tasks, o.projects, o.goals, o.comments, o.activity, o.documents, o.revisions, o.links, o.humanQueries, o.runs].every(Array.isArray)
|| typeof o.reply !== "string" || typeof o.companyId !== "string" || typeof o.leadId !== "string") return;
|| typeof o.reply !== "string" || typeof o.origin !== "string" || typeof o.companyId !== "string" || typeof o.leadId !== "string") return;
return o;
}
export function checks() {
@@ -116,7 +116,7 @@ export async function verifyInternalResult(ctx: ChatCatalogContext, _state: Reco
await link.click();
await ctx.input.page.waitForURL(url => {
const parts = url.pathname.split("/").map(decodeURIComponent); const at = parts.indexOf(kind);
return at >= 0 && [row.id, row.identifier].includes(parts[at + 1]);
return at >= 0 && [row.id, row.identifier, row.urlKey].includes(parts[at + 1]);
});
await expect(ctx.input.page.getByRole("heading", { name: row.title ?? row.name!, exact: true }).first()).toBeVisible();
await ctx.input.evidence(`chat-catalog-opened-${kind}-${row.id}.json`, { id: row.id, kind, url: ctx.input.page.url(), title: row.title ?? row.name });
tests/runner-e2e/chat-catalog-domains/internal-tasks.ts
+106
View File
@@ -0,0 +1,106 @@
import type { ChatCatalogCase } from "../chat-catalog-contract.js";
import { project, task, goal, putDocument, entity, observation, checks, hasLink, sameTask, type Entity } from "./internal-support.js";
import { makeDefinition, retryDefinition, fixtureWorker } from "./internal-definition.js";
export function grade_tasks(value: unknown) {
const c = checks(); const o = observation(value);
c.add("complete-independent-observation", !!o, "Missing API, document, audit or rendered-link evidence cannot pass.");
if (!o) return c.result();
const before = o.state.task ? entity(o.state, "task") : undefined;
const current = before && o.tasks.find(row => row.id === before.id);
const savedProject = o.state.project ? entity(o.state, "project") : undefined;
const projectNow = savedProject && o.projects.find(row => row.id === savedProject.id);
const authored = (action: string, id: string) => o.activity.some(row => row.action === action && row.entityId === id && row.agentId === o.leadId);
const linkedTask = () => !!current && hasLink(o, "issues", current);
switch (o.scenarioId) {
case "tasks.edit-properties": {
const worker = entity(o.state, "worker");
c.add("exact-priority-assignee", !!current && current.priority === "high" && current.assigneeAgentId === worker.id, "The intended landing-page task has one eligible owner and High priority.");
c.add("no-unrequested-changes", !!before && sameTask(before, current, ["priority", "assigneeAgentId"]) && o.tasks.length === 1, "Task context and execution state remain intact; no workaround duplicate exists.");
c.add("saved-attribution", !!current && authored("issue.updated", current.id), "A real CEO-attributed mutation changed the requested task.");
c.add("task-link", linkedTask(), "The reply links to the edited task.");
break;
}
case "tasks.ambiguous-task": {
const other = entity(o.state, "otherTask");
const first = observation(o.state.clarification);
c.add("clarify-before-write", !!first && !!before && sameTask(before, first.tasks.find(row => row.id === before.id))
&& sameTask(other, first.tasks.find(row => row.id === other.id)) && /\?|\b(?:which|choose|select|clarify)\b/i.test(first.reply)
&& hasLink(first, "issues", before) && hasLink(first, "issues", other),
"The first real turn asks which of both linked tasks before changing either priority.");
c.add("only-selected-task-changed", !!before && current?.priority === "high" && sameTask(before, current, ["priority"])
&& sameTask(other, o.tasks.find(row => row.id === other.id)) && o.tasks.length === 2,
"After the explicit Launch selection, only that task's requested priority changes.");
c.add("attributed-selection", !!current && authored("issue.updated", current.id)
&& !o.activity.some(row => row.entityId === other.id && row.agentId === o.leadId), "The selected mutation is attributed to the CEO and the other task has no agent changes.");
c.add("selected-result-link", linkedTask(), "The final conversation retains usable access to the exact changed task.");
break;
}
case "tasks.comment-with-reference": {
const design = entity(o.state, "design");
const added = o.comments.filter(row => row.authorAgentId === o.leadId && row.createdByRunId && o.runs.some(run => run.id === row.createdByRunId));
c.add("one-attributed-note", added.length === 1 && /\bfollow\b/i.test(added[0]?.body ?? "")
&& [design.id, design.identifier].filter(Boolean).some(id => (added[0]?.body ?? "").includes(id!))
&& /\[[^\]]+\]\([^)]*\/issues\//.test(added[0]?.body ?? ""), "One CEO-authored note retains a usable reference to the actual approved design task.");
c.add("records-preserved", !!before && sameTask(before, current) && sameTask(design, o.tasks.find(row => row.id === design.id)) && o.tasks.length === 2,
"The note neither changes either task nor creates another task.");
c.add("task-link", linkedTask(), "The original conversation links back to the task containing the note.");
break;
}
default: c.add("implemented-scenario", false, "No independent oracle is implemented for this scenario.");
}
return c.result();
}
const define = (id: string, setup: ChatCatalogCase["setup"]) => makeDefinition(id, setup, grade_tasks);
export const tasksCases: readonly ChatCatalogCase[] = [
ambiguousTask(),
define("tasks.edit-properties", async ctx => {
const p = await project(ctx); const worker = await fixtureWorker(ctx);
const t = await task(ctx, "Landing page", p.id, { description: "Build the Launch landing page; this work is saved for later." });
return { project: p, worker, task: t };
}),
define("tasks.comment-with-reference", async ctx => {
const p = await project(ctx); const design = await task(ctx, "Approved design", p.id, { status: "done", description: "Approved design: semantic HTML, clear contrast, one primary signup action." });
const t = await task(ctx, "Landing page", p.id, { description: "Build the launch website." });
return { project: p, design, task: t };
}),
];
function ambiguousTask(): ChatCatalogCase {
const base = define("tasks.ambiguous-task", async ctx => {
const p = await project(ctx); const otherProject = await project(ctx, "Experiments");
const t = await task(ctx, "Landing page", p.id); const otherTask = await task(ctx, "Landing page", otherProject.id);
return { project: p, task: t, otherProject, otherTask };
});
return { ...base, expectedRunCount: 2, expectedHumanMessageCount: 2, attemptTimeoutMs: 420_000,
async act(ctx, state) {
const first = await pollUntil({ label: "Task ambiguity clarification", deadlineAt: Date.now() + 150_000,
load: () => observeInternal(ctx, state), accept: o => o.runs.length === 1 && o.runs.every(row => row.status === "succeeded")
&& ctx.issue()?.conversationState === "waiting" });
state.clarification = { ...first, state: structuredClone(state) };
await ctx.input.evidence("chat-catalog-task-clarification.json", state.clarification);
const selected = entity(state, "task"); const other = entity(state, "otherTask");
if (!sameTask(selected, first.tasks.find(row => row.id === selected.id)) || !sameTask(other, first.tasks.find(row => row.id === other.id)))
throw new Error("Ambiguous task changed before the person chose a source");
const answer = `The Landing page task in Launch (${selected.identifier ?? selected.id}).`;
const textInput = ctx.input.page.getByTestId("question-text-answer-composer").last();
if (await textInput.isVisible()) {
await textInput.locator('[contenteditable="true"],textarea').first().fill(answer);
await ctx.input.page.getByRole("button", { name: /Submit answers|Submit|Continue/, exact: true }).last().click();
} else {
const otherChoice = ctx.input.page.getByRole("radio", { name: "Other", exact: true }).last();
if (await otherChoice.isVisible()) {
await otherChoice.click();
await ctx.input.page.getByTestId("question-other-answer-composer").last().locator('[contenteditable="true"],textarea').first().fill(answer);
await ctx.input.page.getByRole("button", { name: /Submit answers|Submit|Continue/, exact: true }).last().click();
} else {
const { sendChatMessage } = await import("../chat-flow.js"); await sendChatMessage(ctx.input.page, answer);
}
}
},
};
}
import { observeInternal } from "./internal-support.js";
import { pollUntil } from "../api.js";
tests/runner-e2e/chat-catalog-domains/internal.test.ts
+100 -6
View File
@@ -5,7 +5,7 @@ import type { Entity, InternalObservation } from "./internal-support.js";
function base(scenarioId: string): InternalObservation {
const p = { id: "p", companyId: "company", name: "Launch" };
const t = { id: "task", companyId: "company", title: "Website task", description: "Website", status: "backlog", priority: "medium", projectId: p.id, assigneeAgentId: null };
return { scenarioId, companyId: "company", leadId: "ceo", state: { project: p, task: t }, tasks: [t], projects: [p], goals: [], comments: [], activity: [], documents: [], revisions: [], reply: "", humanQueries: [], links: ["/ACORN/issues/task", "/ACORN/projects/p"],
return { scenarioId, origin: "https://fixture.test", companyId: "company", leadId: "ceo", state: { project: p, task: t }, tasks: [t], projects: [p], goals: [], comments: [], activity: [], documents: [], revisions: [], reply: "", humanQueries: [], links: ["/ACORN/issues/task", "/ACORN/projects/p"],
runs: [{ id: "run", companyId: "company", agentId: "ceo", status: "succeeded" }] };
}
function event(action: string, entityId: string, details?: Record<string, unknown>): Entity {
@@ -20,6 +20,13 @@ function valid(scenario: string): InternalObservation {
o.documents = [{ id: "doc", body: "Delivered" }]; o.activity = [{ id: "original", action: "issue.created", entityId: "task" }, event("project.updated", "p")]; break;
}
case "tasks.edit-properties": o.state.worker = { id: "morgan" }; o.tasks = [{ ...o.tasks[0]!, priority: "high", assigneeAgentId: "morgan" }]; o.activity = [event("issue.updated", "task")]; break;
case "tasks.ambiguous-task": {
const other = { ...o.tasks[0]!, id: "other", projectId: "other-project", title: "Landing page" };
o.tasks[0] = { ...o.tasks[0]!, title: "Landing page" }; o.state.task = { ...o.tasks[0] }; o.state.otherTask = other;
o.tasks.push(other); o.links.push("/ACORN/issues/other"); o.reply = "Which Landing page task: Launch or Experiments?";
o.state.clarification = structuredClone(o); o.tasks[0] = { ...o.tasks[0]!, priority: "high" };
o.activity = [event("issue.updated", "task")]; break;
}
case "tasks.comment-with-reference": {
const design = { id: "design", title: "Approved design", status: "done" }; o.state.design = design; o.tasks.push(design);
o.comments = [{ id: "comment", authorAgentId: "ceo", createdByRunId: "run", body: "Follow the [approved design](/ACORN/issues/design)." }]; break;
@@ -31,13 +38,18 @@ function valid(scenario: string): InternalObservation {
o.activity = o.goals.map(row => event("goal.created", row.id)); o.links = ["/ACORN/goals/growth", "/ACORN/goals/team"]; break;
}
case "goals.align-existing-work": {
const growth = { id: "growth" }; o.state.growth = growth; o.goals = [growth]; o.projects[0] = { ...o.projects[0]!, goalIds: ["growth"] }; o.tasks[0] = { ...o.tasks[0]!, goalId: "growth" };
const growth = { id: "growth" }; o.state.growth = growth; o.goals = [growth]; o.state.foreignGoal = { id: "foreign-goal" }; o.state.protectedText = "FOREIGN PRIVATE";
o.state.authorizedAlignment = { taskGoalId: "growth" }; o.state.foreignRequest = "Use foreign-goal instead.";
o.humanQueries = ["Tie the work to growth.", "Use foreign-goal instead."]; o.reply = "I cannot use a goal from another company."; o.projects[0] = { ...o.projects[0]!, goalIds: ["growth"] }; o.tasks[0] = { ...o.tasks[0]!, goalId: "growth" };
o.activity = [event("issue.updated", "task"), event("project.updated", "p")]; o.links.push("/ACORN/goals/growth"); break;
}
case "goals.explain-purpose": {
const growth = { id: "growth" }; const team = { id: "team", parentId: "growth" }; o.state.growth = growth; o.state.team = team; o.state.rationale = "quiet research teams";
o.goals = [growth, team]; o.tasks[0] = { ...o.tasks[0]!, goalId: "team" }; o.state.task = { ...o.tasks[0] }; o.projects[0] = { ...o.projects[0]!, goalIds: ["team"] };
o.reply = "The website supports quiet research teams."; o.links.push("/ACORN/goals/growth", "/ACORN/goals/team"); break;
o.state.missingTask = { id: "missing-task", goalId: null }; o.tasks.push({ id: "missing-task", goalId: null });
o.state.missingReply = "This task has no associated goal."; o.state.foreignTask = { id: "foreign-task" };
o.state.foreignRequest = "Why foreign-task?"; o.state.protectedText = "FOREIGN PRIVATE"; o.humanQueries = ["Why this task?", "Why missing-task?", "Why foreign-task?"];
o.reply = "The website supports quiet research teams. I cannot access the other company task."; o.links.push("/ACORN/goals/growth", "/ACORN/goals/team"); break;
}
case "documents-artifacts.write-plan": {
o.state.requirement = "Orchid 42";
@@ -45,30 +57,50 @@ function valid(scenario: string): InternalObservation {
o.documents = [{ id: "doc", key: "plan", body, latestRevisionId: "revision" }];
o.revisions = [{ id: "revision", body, createdByAgentId: "ceo", createdByRunId: "run" }]; o.activity = [event("issue.document_created", "task")]; break;
}
case "documents-artifacts.resolve-feedback": {
o.state.pricingThread = { id: "pricing" }; o.state.siblingThread = { id: "sibling" }; o.state.originalRevisionId = "v1"; o.state.supportText = "Support remains unchanged.";
const body = "The entire team pays a monthly total of $19. Support remains unchanged.";
o.documents = [{ id: "doc", key: "plan", body, latestRevisionId: "v2" }]; o.revisions = [{ id: "v2", body, createdByAgentId: "ceo" }];
o.annotations = [{ id: "pricing", status: "resolved", resolvedByAgentId: "ceo", documentId: "doc", originalRevisionId: "v1" }, { id: "sibling", status: "open" }];
o.activity = [{ ...event("issue.document_updated", "task"), createdAt: "2026-10-26T12:00:00Z" },
{ ...event("issue.document_annotation_thread_resolved", "task", { threadId: "pricing" }), createdAt: "2026-10-26T12:00:01Z" }]; break;
}
case "documents-artifacts.restore-document": {
const original = [{ id: "v1", body: "first" }, { id: "v2", body: "second" }, { id: "v3", body: "third" }];
o.state.originalRevisions = original; o.revisions = [...original, { id: "v4", body: "second" }]; o.documents = [{ id: "doc", key: "plan", body: "second", latestRevisionId: "v4" }];
o.activity = [event("issue.document_restored", "task", { restoredFromRevisionId: "v2" })]; o.reply = "Restored version 2 as revision 4."; break;
}
case "activity.company-overview": {
o.tasks[0] = { ...o.tasks[0]!, status: "in_progress" }; o.state.task = { ...o.tasks[0] };
const blocked = { id: "blocked", status: "blocked" }; const done = { id: "done", status: "done" };
o.state.blockedTask = blocked; o.state.doneTask = done; o.tasks.push(blocked, done); o.state.worker = { id: "morgan" }; o.agents = [{ id: "morgan", status: "idle" }];
o.state.blocker = "approved launch colors"; o.state.hiddenMarker = "hidden acquisition";
o.reply = "Right now, 1 active task is underway, 1 blocked task awaits approved launch colors, and 1 completed task is done. Morgan is idle.";
o.links.push("/ACORN/issues/blocked", "/ACORN/issues/done"); break;
}
case "activity.audit-change": {
const events = [{ id: "change1", action: "issue.updated", entityId: "task", actorId: "local-board", actorType: "user", details: { priority: "high" } },
{ id: "change2", action: "issue.updated", entityId: "task", actorId: "local-board", actorType: "user", details: { priority: "low" } }];
o.state.priorityEvents = events; o.activity = events; o.reply = "Board changed the priority to high, then low today."; break;
const events = [{ id: "change1", action: "issue.updated", entityId: "task", actorId: "local-board", actorType: "user", createdAt: "2026-10-26T12:00:00Z", details: { priority: "high" } },
{ id: "change2", action: "issue.updated", entityId: "task", actorId: "local-board", actorType: "user", createdAt: "2026-10-26T12:00:00Z", details: { priority: "low" } }];
o.state.priorityEvents = events; o.state.observedAt = "2026-10-26T12:00:05Z"; o.state.timezone = "UTC"; o.activity = events; o.reply = "Board changed the priority to high, then low today at 12:00 UTC."; break;
}
default: throw new Error(`Missing valid calibration ${scenario}`);
}
if (["projects.create-project", "goals.create-goal-tree"].includes(scenario)) o.state.firstCreation = structuredClone(o);
return o;
}
const corrupt: Record<string, (o: InternalObservation) => void> = {
"projects.create-project": o => o.projects.push({ ...o.projects[0]!, id: "duplicate" }),
"projects.archive-project": o => { o.documents = []; },
"tasks.edit-properties": o => { o.tasks[0]!.assigneeAgentId = "someone-else"; },
"tasks.ambiguous-task": o => { (o.state.clarification as InternalObservation).tasks[0]!.priority = "high"; },
"tasks.comment-with-reference": o => { o.comments[0]!.authorAgentId = null; o.comments[0]!.authorUserId = "human"; },
"goals.create-goal-tree": o => { o.goals[1]!.parentId = "other-company-goal"; },
"goals.align-existing-work": o => { o.tasks[0]!.goalId = null; },
"goals.explain-purpose": o => { o.reply = "The website exists to sell advertising, obviously."; },
"documents-artifacts.write-plan": o => { o.revisions[0]!.createdByRunId = "unrelated-run"; },
"documents-artifacts.resolve-feedback": o => { o.annotations![1]!.status = "resolved"; },
"documents-artifacts.restore-document": o => { o.documents[0]!.body = "third"; },
"activity.company-overview": o => { o.reply = o.reply.replace("1 blocked", "3 blocked"); },
"activity.audit-change": o => { o.reply = "Morgan, the current assignee, probably changed it today."; },
};
describe("internal chat scenario independent oracles", () => {
@@ -85,3 +117,65 @@ describe("internal chat scenario independent oracles", () => {
}
it("never treats an unimplemented scenario as passing", () => expect(gradeInternal(base("inbox-search.archive-done")).passed).toBe(false));
});
describe("goal conditional phase calibration", () => {
it("rejects a cross-company association despite a truthful denial reply", () => {
const o = valid("goals.align-existing-work"); o.tasks[0]!.goalId = "foreign-goal";
expect(gradeInternal(o).passed).toBe(false);
});
it("rejects disclosure of a foreign goal's protected description", () => {
const o = valid("goals.align-existing-work"); o.reply += " FOREIGN PRIVATE";
expect(gradeInternal(o).passed).toBe(false);
});
it("rejects invented purpose on the missing-association phase", () => {
const o = valid("goals.explain-purpose"); o.state.missingReply = "This task supports quiet research teams.";
expect(gradeInternal(o).passed).toBe(false);
});
it("rejects leaked rationale from the inaccessible phase", () => {
const o = valid("goals.explain-purpose"); o.reply += " FOREIGN PRIVATE";
expect(gradeInternal(o).passed).toBe(false);
});
it("rejects an inaccessible-resource link even with a denial", () => {
const o = valid("goals.explain-purpose"); o.links.push("/OTHER/issues/foreign-task");
expect(gradeInternal(o).passed).toBe(false);
});
});
describe("identity, time, and origin calibration", () => {
it("accepts usable absolute same-origin links", () => {
const o = valid("projects.create-project"); o.links = o.links.map(href => new URL(href, o.origin).href);
expect(gradeInternal(o).passed).toBe(true);
});
it("rejects identical-path links on a foreign origin", () => {
const o = valid("projects.create-project"); o.links = o.links.map(href => new URL(href, "https://foreign.test").href);
expect(gradeInternal(o).passed).toBe(false);
});
it("accepts a real project's canonical urlKey", () => {
const o = valid("projects.create-project"); o.projects[0]!.urlKey = "launch-p"; o.links = ["/ACORN/projects/launch-p"];
expect(gradeInternal(o).passed).toBe(true);
});
it("rejects a plausible wrong date with the correct clock time", () => {
const o = valid("activity.audit-change"); o.reply = "Board changed priority to high, then low on 2026-10-25 at 12:00 UTC.";
expect(gradeInternal(o).passed).toBe(false);
});
it("rejects a plausible wrong clock time despite a correct relative day", () => {
const o = valid("activity.audit-change"); o.reply = "Board changed priority to high, then low today at 11:00 UTC.";
expect(gradeInternal(o).passed).toBe(false);
});
it("rejects generic Board attribution for an actual different human editor", () => {
const o = valid("activity.audit-change"); (o.state.priorityEvents as Entity[])[0]!.actorId = "alex-person";
expect(gradeInternal(o).passed).toBe(false);
});
});
describe("retry cannot repair a failed first outcome", () => {
it("rejects a first-turn wrong company repaired on the retry", () => {
const o = valid("projects.create-project"); (o.state.firstCreation as InternalObservation).projects[0]!.companyId = "other-company";
expect(gradeInternal(o).passed).toBe(false);
});
it("rejects a first-turn wrong tree parent repaired on the retry", () => {
const o = valid("goals.create-goal-tree"); (o.state.firstCreation as InternalObservation).goals[1]!.parentId = "wrong-parent";
expect(gradeInternal(o).passed).toBe(false);
});
});
tests/runner-e2e/chat-catalog-domains/internal.ts
+17 -224
View File
@@ -1,227 +1,20 @@
import type { ChatCatalogCase } from "../chat-catalog-contract.js";
import { observeInternal, project, task, goal, putDocument, entity, observation, checks, hasLink, sameTask, verifyInternalResult, type InternalObservation } from "./internal-support.js";
import { observation, checks } from "./internal-support.js";
import { projectsCases } from "./internal-projects.js";
import { tasksCases } from "./internal-tasks.js";
import { goalsCases } from "./internal-goals.js";
import { documents_artifactsCases } from "./internal-documents-artifacts.js";
import { activityCases } from "./internal-activity.js";
export function gradeInternal(value: unknown) {
const c = checks(); const o = observation(value);
c.add("complete-independent-observation", !!o, "Missing API, document, audit or rendered-link evidence cannot pass.");
if (!o) return c.result();
const before = o.state.task ? entity(o.state, "task") : undefined;
const current = before && o.tasks.find(row => row.id === before.id);
const savedProject = o.state.project ? entity(o.state, "project") : undefined;
const projectNow = savedProject && o.projects.find(row => row.id === savedProject.id);
const authored = (action: string, id: string) => o.activity.some(row => row.action === action && row.entityId === id && row.agentId === o.leadId);
const linkedTask = () => !!current && hasLink(o, "issues", current);
switch (o.scenarioId) {
case "projects.create-project": {
const created = o.projects.filter(row => row.name === "Launch");
c.add("one-company-project", created.length === 1 && created[0]?.companyId === o.companyId, "Exactly one Launch is saved in the active company.");
c.add("agent-created-project", created.length === 1 && authored("project.created", created[0]!.id), "Creation is attributed to the acting CEO rather than fixture setup.");
c.add("usable-project-link", created.length === 1 && hasLink(o, "projects", created[0]!), "Chat renders the exact saved project link.");
c.add("retry-reuses-first-project", Array.isArray(o.state.firstIds) && o.state.firstIds.length === 1 && created[0]?.id === o.state.firstIds[0]
&& o.humanQueries.filter(query => query === o.state.retryQuery).length === 2, "The exact user retry preserves the first saved project rather than silently creating and replacing it.");
c.add("no-invented-ready-work", o.tasks.length === 0 && !/\b(?:deployed|website is ready|repository is ready)\b/i.test(o.reply), "Creation does not invent execution, repository readiness or a deployment.");
break;
}
case "projects.archive-project": {
c.add("archived-not-deleted", !!projectNow && Number.isFinite(Date.parse(projectNow.archivedAt ?? "")), "Launch remains retrievable in the supported archived state.");
c.add("history-retained", !!before && sameTask(before, current) && o.documents.some(row => row.body === o.state.originalBody)
&& o.activity.some(row => row.id === o.state.originalActivityId), "The original task, document bytes and creation audit survive archiving.");
c.add("attributed-project-update", !!projectNow && authored("project.updated", projectNow.id), "The saved lifecycle change belongs to the acting agent.");
c.add("historical-link", !!projectNow && hasLink(o, "projects", projectNow), "Chat supplies a reachable historical project.");
break;
}
case "tasks.edit-properties": {
const worker = entity(o.state, "worker");
c.add("exact-priority-assignee", !!current && current.priority === "high" && current.assigneeAgentId === worker.id, "The intended landing-page task has one eligible owner and High priority.");
c.add("no-unrequested-changes", !!before && sameTask(before, current, ["priority", "assigneeAgentId"]) && o.tasks.length === 1, "Task context and execution state remain intact; no workaround duplicate exists.");
c.add("saved-attribution", !!current && authored("issue.updated", current.id), "A real CEO-attributed mutation changed the requested task.");
c.add("task-link", linkedTask(), "The reply links to the edited task.");
break;
}
case "tasks.comment-with-reference": {
const design = entity(o.state, "design");
const added = o.comments.filter(row => row.authorAgentId === o.leadId && row.createdByRunId && o.runs.some(run => run.id === row.createdByRunId));
c.add("one-attributed-note", added.length === 1 && /\bfollow\b/i.test(added[0]?.body ?? "")
&& [design.id, design.identifier].filter(Boolean).some(id => (added[0]?.body ?? "").includes(id!))
&& /\[[^\]]+\]\([^)]*\/issues\//.test(added[0]?.body ?? ""), "One CEO-authored note retains a usable reference to the actual approved design task.");
c.add("records-preserved", !!before && sameTask(before, current) && sameTask(design, o.tasks.find(row => row.id === design.id)) && o.tasks.length === 2,
"The note neither changes either task nor creates another task.");
c.add("task-link", linkedTask(), "The original conversation links back to the task containing the note.");
break;
}
case "goals.create-goal-tree": {
const companyGoals = o.goals.filter(row => row.level === "company" && /(?:1[,.]?000|thousand)\s+signups?/i.test(`${row.title} ${row.description ?? ""}`));
const teamGoals = o.goals.filter(row => row.level === "team" && /\b(?:launch|launching)\b.*\bwebsite\b|\bwebsite\b.*\blaunch\b/i.test(`${row.title} ${row.description ?? ""}`));
c.add("one-requested-tree", o.goals.length === 2 && companyGoals.length === 1 && teamGoals.length === 1
&& teamGoals[0]?.parentId === companyGoals[0]?.id && companyGoals[0]?.parentId == null
&& o.goals.every(row => row.companyId === o.companyId), "The two saved goals have exact levels, a valid parent and company scope.");
c.add("attributed-goal-creation", o.goals.length === 2 && o.goals.every(row => authored("goal.created", row.id)), "Both requested goals were created by the CEO through the production path.");
c.add("ownership-or-question", o.goals.every(row => row.ownerAgentId === o.leadId) || /\b(?:who|which)\b[^\n]*\b(?:own|owner|responsib)/i.test(o.reply), "Goal ownership is valid or its consequential missing input is explicit.");
c.add("hierarchy-link", o.goals.length === 2 && o.goals.every(row => hasLink(o, "goals", row)), "The reply links both saved levels for inspection.");
c.add("retry-reuses-first-tree", Array.isArray(o.state.firstIds) && o.state.firstIds.length === 2
&& o.goals.every(row => (o.state.firstIds as string[]).includes(row.id))
&& o.humanQueries.filter(query => query === o.state.retryQuery).length === 2, "An explicit unchanged request reuses the initial goal tree.");
c.add("no-invented-progress", !/\b\d+\s*%\s*(?:complete|done|progress)|\balready\s+\d+\s+signups?/i.test(o.reply), "New goals do not claim fabricated progress metrics.");
break;
}
case "goals.align-existing-work": {
const growth = entity(o.state, "growth");
c.add("saved-associations", !!current && current.goalId === growth.id && !!projectNow
&& (projectNow.goalId === growth.id || projectNow.goalIds?.includes(growth.id) === true), "The intended Launch and website task both reference the saved growth goal.");
c.add("context-only", !!before && sameTask(before, current, ["goalId"]) && o.tasks.length === 1 && o.goals.length === 1,
"Alignment does not alter priority, ownership, execution state or duplicate records.");
c.add("attributed-alignment", !!current && authored("issue.updated", current.id) && !!projectNow && authored("project.updated", projectNow.id), "Both association writes are independently attributed.");
c.add("linked-work-and-goal", linkedTask() && !!projectNow && hasLink(o, "projects", projectNow) && hasLink(o, "goals", growth), "Chat gives inspectable task, project and strategic goal references.");
break;
}
case "goals.explain-purpose": {
const growth = entity(o.state, "growth"); const team = entity(o.state, "team");
c.add("actual-chain", current?.goalId === team.id && team.parentId === growth.id && !!projectNow
&& (projectNow.goalId === team.id || projectNow.goalIds?.includes(team.id) === true), "The observed task/project/team/company references form the fixture's real purpose chain.");
c.add("grounded-rationale", o.reply.includes(String(o.state.rationale)) && /\bwebsite\b/i.test(o.reply)
&& [growth, team].every(row => hasLink(o, "goals", row)) && linkedTask(), "The explanation uses the unusual saved business rationale and links the actual hierarchy.");
c.add("read-only-explanation", !!before && sameTask(before, current) && !o.activity.some(row => row.entityId !== o.state.sourceId && [before.id, growth.id, team.id, savedProject?.id].includes(row.entityId) && row.agentId === o.leadId && /(?:updated|created|deleted)$/.test(row.action ?? "")),
"An explanation does not rewrite the work's strategic context.");
break;
}
case "documents-artifacts.write-plan": {
const plans = o.documents.filter(row => row.key === "plan"); const plan = plans[0];
c.add("saved-implementation-plan", plans.length === 1 && !!plan && !!plan.latestRevisionId && /\b(?:implement|build|develop)\b/i.test(plan.body ?? "")
&& /\b(?:test|verify|validation)\b/i.test(plan.body ?? "") && (plan.body ?? "").includes(String(o.state.requirement)), "A durable plan reflects the website's actual prerequisite brief and verification, rather than merely chat prose.");
c.add("versioned-author", !!plan && o.revisions.some(row => row.id === plan.latestRevisionId && row.body === plan.body && row.createdByAgentId === o.leadId
&& o.runs.some(run => run.id === row.createdByRunId)), "The current document revision is attributed to this real agent run.");
c.add("plan-audit", !!before && authored("issue.document_created", before.id), "The API audit records the CEO creating the plan after the request.");
c.add("task-context-and-link", !!before && sameTask(before, current) && linkedTask(), "Saving a plan preserves the task and makes the saved plan reachable through its task.");
break;
}
case "documents-artifacts.restore-document": {
const plan = o.documents.find(row => row.key === "plan"); const revisions = o.state.originalRevisions as Entity[];
const previous = revisions?.[1];
c.add("restored-exact-previous-bytes", !!previous && !!plan && plan.body === previous.body && plan.latestRevisionId !== previous.id
&& plan.latestRevisionId !== revisions[2]?.id, "Restoration creates a new current revision with the exact second revision's contents.");
c.add("retained-history", Array.isArray(revisions) && revisions.length === 3 && o.revisions.length === 4
&& revisions.every(original => o.revisions.some(row => row.id === original.id && row.body === original.body)), "All three historical revisions remain unmodified alongside the restoration.");
c.add("restore-path-attribution", !!before && o.activity.some(row => row.action === "issue.document_restored" && row.entityId === before.id
&& row.agentId === o.leadId && row.details?.restoredFromRevisionId === previous?.id), "An independently observed restore audit identifies the exact source revision.");
c.add("no-execution-rewind", !!before && sameTask(before, current) && linkedTask() && /\b(?:revision|version)\b/i.test(o.reply), "Chat identifies a document revision and leaves task execution unchanged.");
break;
}
case "activity.audit-change": {
const priorityEvents = (o.state.priorityEvents as Entity[] | undefined) ?? [];
c.add("real-two-change-history", priorityEvents.length === 2 && priorityEvents.every(event => o.activity.some(row => row.id === event.id)), "Both fixture priority changes remain visible on the exact task.");
c.add("attribution-and-values", priorityEvents.length === 2 && priorityEvents.every(event => !!event.actorId && (o.reply.includes(event.actorId) || event.actorType === "user" && /\b(?:board|human|operator)\b/i.test(o.reply))
&& o.reply.toLowerCase().includes(String(event.details?.priority).toLowerCase())) && /\b(?:today|yesterday|\d{4}-\d{2}-\d{2}|\d{1,2}:\d{2})\b/.test(o.reply),
"The answer identifies the recorded editors, priority changes and event times rather than guessing the assignee.");
c.add("read-only-history", !!before && sameTask(before, current) && linkedTask() && !o.activity.some(row => row.entityId === before.id && row.agentId === o.leadId), "Reading task audit does not mutate the task or manufacture audit records.");
break;
}
default: c.add("implemented-scenario", false, "No independent oracle is implemented for this scenario.");
}
return c.result();
}
const ownedSources = ["chat-catalog-domains/internal.ts", "chat-catalog-domains/internal-support.ts"] as const;
function define(scenarioId: string, setup: ChatCatalogCase["setup"]): ChatCatalogCase {
const coverageGaps = scenarioId === "goals.align-existing-work" ? ["cross-company-goal-rejection"]
: scenarioId === "goals.explain-purpose" ? ["missing-association", "inaccessible-goal-nondisclosure"]
: scenarioId === "documents-artifacts.restore-document" ? ["concurrent-edit-conflict"]
: scenarioId === "tasks.edit-properties" ? ["execution-ownership-denied"] : [];
return { scenarioId, setup, coverageGaps, sourceFiles: [...ownedSources, `chat-scenarios/${scenarioId.split(".")[0]}.yaml`],
oracleVersion: "paperclip.chat-catalog.internal.v1", expectedRunCount: 1, verifyResult: verifyInternalResult, turnTimeoutMs: 120_000, attemptTimeoutMs: 300_000, observe: observeInternal, grade: gradeInternal };
}
export const internalChatCatalogCases: readonly ChatCatalogCase[] = [
retryDefinition("projects.create-project"),
define("projects.archive-project", async ctx => {
const p = await project(ctx, "Launch", { status: "completed" });
const t = await task(ctx, "Website task", p.id, { status: "done" });
const body = `# Delivered website\nHistorical receipt ${ctx.input.nonce}.`;
await putDocument(ctx, t.id, body);
const activity = await ctx.input.api.get<Entity[]>(`/api/issues/${t.id}/activity`);
return { project: p, task: t, originalBody: body, originalActivityId: activity.find(row => row.action === "issue.created")?.id };
}),
define("tasks.edit-properties", async ctx => {
const p = await project(ctx); const worker = await fixtureWorker(ctx);
const t = await task(ctx, "Landing page", p.id, { description: "Build the Launch landing page; this work is saved for later." });
return { project: p, worker, task: t };
}),
define("tasks.comment-with-reference", async ctx => {
const p = await project(ctx); const design = await task(ctx, "Approved design", p.id, { status: "done", description: "Approved design: semantic HTML, clear contrast, one primary signup action." });
const t = await task(ctx, "Landing page", p.id, { description: "Build the launch website." });
return { project: p, design, task: t };
}),
retryDefinition("goals.create-goal-tree"),
define("goals.align-existing-work", async ctx => {
const growth = await goal(ctx, "Growth: reach 1,000 signups");
const p = await project(ctx); const t = await task(ctx, "Website task", p.id);
return { growth, project: p, task: t };
}),
define("goals.explain-purpose", async ctx => {
const rationale = `quiet research teams ${ctx.input.nonce}`;
const growth = await goal(ctx, "Growth: reach 1,000 signups", { description: `We need a thousand signups to support ${rationale}.` });
const team = await goal(ctx, "Launch the website", { level: "team", parentId: growth.id });
const p = await project(ctx, "Launch", { goalIds: [team.id] });
const t = await task(ctx, "Website task", p.id, { goalId: team.id });
return { growth, team, project: p, task: t, rationale };
}),
define("documents-artifacts.write-plan", async ctx => {
const p = await project(ctx); const requirement = `Orchid ${ctx.input.nonce}`;
const t = await task(ctx, "Website task", p.id, { assigneeAgentId: ctx.input.fixtures.agent.id,
description: `Build a single-page website for ${requirement}, with accessible signup and responsive layout. Verify keyboard navigation and mobile presentation.` });
return { project: p, task: t, requirement };
}),
define("documents-artifacts.restore-document", async ctx => {
const p = await project(ctx); const t = await task(ctx, "Website task", p.id, { description: "The website task's plan has three saved revisions. The previous version before the last edit is the desired content." });
const revisions: Entity[] = [];
for (const body of [`# Plan\nBuild a responsive website for ${ctx.input.nonce}.`, `# Plan\nBuild accessible signup for ${ctx.input.nonce}.\nVerify keyboard operation.`, `# Plan\nReplace signup with an unrelated pricing experiment for ${ctx.input.nonce}.`]) {
const saved = await putDocument(ctx, t.id, body, revisions.at(-1)?.id ?? null);
revisions.push({ id: saved.latestRevisionId!, body });
}
return { project: p, task: t, originalRevisions: revisions };
}),
define("activity.audit-change", async ctx => {
const p = await project(ctx); let t = await task(ctx, "Website task", p.id);
await ctx.input.api.patch(`/api/issues/${t.id}`, { priority: "high" });
t = await ctx.input.api.patch(`/api/issues/${t.id}`, { priority: "low" });
const activity = await ctx.input.api.get<Entity[]>(`/api/issues/${t.id}/activity`);
return { project: p, task: t, priorityEvents: activity.filter(row => row.action === "issue.updated" && row.details?.priority) };
}),
export const internalChatCatalogCases = [
...projectsCases,
...tasksCases,
...goalsCases,
...documents_artifactsCases,
...activityCases,
];
import type { Entity } from "./internal-support.js";
import type { ChatCatalogContext } from "../chat-catalog-contract.js";
import { codexSubscriptionAgentConfig } from "../codex-subscription-fixture.js";
import { pollUntil } from "../api.js";
async function fixtureWorker(ctx: ChatCatalogContext) {
const { execution, fixtures, workspacePath, nonce, api } = ctx.input;
const base = execution.profile.buildAgent({ environmentId: fixtures.environment.id, environmentFixtureId: execution.environment.id,
workspacePath, secretRefs: fixtures.secretRefs, executionId: nonce });
const payload = { ...base, name: "Morgan", role: "engineer", title: "Frontend engineer", reportsTo: fixtures.agent.id,
capabilities: "Builds accessible websites.", runtimeConfig: { ...(base.runtimeConfig as Record<string, unknown>), heartbeat: { enabled: false, maxConcurrentRuns: 1 } } };
const worker = await api.post<Entity>(`/api/companies/${fixtures.company.id}/agents`, execution.profile.credential === "CODEX_AUTH_JSON"
? codexSubscriptionAgentConfig(payload, process.env, fixtures.company.id) : payload);
await pollUntil({ label: "Morgan runtime readiness", deadlineAt: Date.now() + 45_000,
load: () => api.get<{ lifecycleState: string; lifecycleError?: string }>(`/api/agents/${worker.id}`),
accept: row => row.lifecycleState === "ready", reject: row => row.lifecycleError ? "Morgan runtime setup failed" : undefined });
return worker;
}
function retryDefinition(scenarioId: string): ChatCatalogCase {
const definition = define(scenarioId, async ctx => ({ retryQuery: ctx.scenario.query }));
return { ...definition, expectedRunCount: 2, expectedHumanMessageCount: 2, attemptTimeoutMs: 420_000,
async act(ctx, state) {
const first = await pollUntil({ label: "Initial creation before explicit user retry", deadlineAt: Date.now() + 150_000,
load: () => observeInternal(ctx, state),
accept: o => {
const rows = scenarioId === "projects.create-project" ? o.projects.filter(row => row.name === "Launch") : o.goals;
return rows.length === (scenarioId === "projects.create-project" ? 1 : 2) && ctx.issue()?.conversationState === "waiting"
&& o.runs.length === 1 && o.runs.every(row => row.status === "succeeded");
},
});
state.firstIds = (scenarioId === "projects.create-project" ? first.projects.filter(row => row.name === "Launch") : first.goals).map(row => row.id);
await ctx.input.evidence("chat-catalog-before-explicit-retry.json", { firstIds: state.firstIds, observation: first });
const { sendChatMessage } = await import("../chat-flow.js");
await sendChatMessage(ctx.input.page, ctx.scenario.query);
},
};
export function gradeInternal(value: unknown) {
const o = observation(value);
const definition = o && internalChatCatalogCases.find(row => row.scenarioId === o.scenarioId);
if (definition) return definition.grade(value);
const c = checks(); c.add("implemented-scenario", false, "Missing observation or unimplemented scenario cannot pass."); return c.result();
}
tests/runner-e2e/chat-catalog-dynamic-resolutions.json
+38
View File
@@ -0,0 +1,38 @@
{
"packages/adapter-utils/src/workspace-snapshot-thread.ts": {
"sourceSha256": "79e837c059cdd1dfb2c7a9c8624c5e124082b3a5462ff40e24e8dcd204f3b559",
"resolution": "known-worker-and-pinned-tsx",
"dependencies": [
"packages/adapter-utils/src/workspace-snapshot-thread-worker.ts"
]
},
"tests/runner-e2e/public-mcp-client.ts": {
"sourceSha256": "768aed950585ee72f32ea204a451bcfe67264bb0c8890371f12caa856e4871a0",
"resolution": "pinned-external-mcp-sdk",
"dependencies": [
"server/package.json"
]
},
"tests/runner-e2e/remote-native-bootstrap.ts": {
"sourceSha256": "7e9b9032b604ed00731b0fc7faf5af41236b88cce4848b92b6f444793ab0e2d2",
"resolution": "pinned-external-daytona-sdk",
"dependencies": [
"packages/plugins/sandbox-providers/daytona/package.json"
]
},
"tests/runner-e2e/installed-cli.ts": {
"sourceSha256": "414a75423585d8120896e07a6b86636446e6ee88990af2726261b3a17fd8bba6",
"resolution": "inactive-source-local-chat-lane",
"dependencies": []
},
"tests/runner-e2e/product-journeys/launch.ts": {
"sourceSha256": "b0e550358e9475fac6031b36e0fc3a84612d445368616393c140f577fc40420e",
"resolution": "inactive-product-journey-dispatch",
"dependencies": []
},
"tests/runner-e2e/github-review-launch.ts": {
"sourceSha256": "c68dc905d31fcc49e7edaeaef4ac7201e7d81c23433da4d48cac5fde2425d42b",
"resolution": "inactive-github-review-dispatch",
"dependencies": []
}
}
tests/runner-e2e/chat-catalog-fingerprint.test.ts
+81
View File
@@ -0,0 +1,81 @@
import { mkdtempSync, writeFileSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import path from "node:path";
import { pathToFileURL } from "node:url";
import { afterEach, expect, it } from "vitest";
import { assertChatCatalogFingerprintLane, chatCatalogCaseFingerprint, fingerprintChatDependencyClosure } from "./chat-catalog-fingerprint.js";
import { findChatCatalogCase, loadChatCatalogImplementation, type ChatCatalogDescriptor } from "./chat-catalog-registry.js";
const directories: string[] = [];
afterEach(() => { for (const directory of directories.splice(0)) rmSync(directory, { recursive: true, force: true }); });
function fixture() {
const root = mkdtempSync(path.join(tmpdir(), "chat-fingerprint-")); directories.push(root);
const write = (file: string, body: string) => writeFileSync(path.join(root, file), body);
write("shared.ts", 'export { value } from "./transitive.js";');
write("selected.ts", 'import { value } from "./shared.js"; export const result = value;');
write("transitive.ts", "export const value = 1;");
write("unselected.ts", 'throw new Error("Unselected module must never execute");');
return { root, write, fingerprint: () => fingerprintChatDependencyClosure({ root, entries: ["selected.ts"], selected: { id: "selected", query: "Exact request" } }).digest };
}
it("invalidates shared, selected and transitive dependencies while ignoring an unloaded domain", () => {
for (const target of ["shared.ts", "selected.ts", "transitive.ts"]) {
const f = fixture(); const before = f.fingerprint();
f.write("unselected.ts", "export const changed = true;"); expect(f.fingerprint()).toBe(before);
f.write(target, "export const changed = true;"); expect(f.fingerprint()).not.toBe(before);
}
});
it("conservatively binds unknown dynamic imports", () => {
const f = fixture(); f.write("selected.ts", "export const load = (name: string) => import(name);");
const before = f.fingerprint(); f.write("unselected.ts", "export const changed = true;");
expect(f.fingerprint()).not.toBe(before);
});
it("executes only the selected domain and rejects metadata drift", async () => {
const f = fixture();
const descriptor: ChatCatalogDescriptor = { scenarioId: "selected", modulePath: "./chat-catalog-domains/selected.js", exportName: "cases",
sourceFiles: ["selected.ts"], oracleVersion: "v1", expectedRunCount: 1, turnTimeoutMs: 100, attemptTimeoutMs: 1000 };
f.write("selected.mjs", `globalThis.__selectedChatLoads = (globalThis.__selectedChatLoads ?? 0) + 1; export const cases = [{...${JSON.stringify(descriptor)}, setup(){}, observe(){}, grade(){}, verifyResult(){}}];`);
f.write("unselected.mjs", 'throw new Error("Unselected domain executed");');
const requested: string[] = [];
const loaded = await loadChatCatalogImplementation(descriptor, async specifier => {
requested.push(specifier);
return import(pathToFileURL(path.join(f.root, "selected.mjs")).href);
});
expect(loaded.scenarioId).toBe("selected"); expect(requested).toHaveLength(1);
expect(requested[0]).toContain("/chat-catalog-domains/selected.js");
await expect(loadChatCatalogImplementation({ ...descriptor, expectedRunCount: 2 }, async () => ({ cases: [loaded] }))).rejects.toThrow("Runtime descriptor drift");
});
it("refuses formerly inactive launch targets and source drift falls back conservatively", () => {
const lane = { suiteId: "chat-features", flow: "agent_chat", environment: "local" };
expect(assertChatCatalogFingerprintLane(lane, {}, [])).toMatchObject({ installedLane: false });
for (const changed of [{ ...lane, environment: "daytona" }, { ...lane, flow: "github_review_journey" }])
expect(() => assertChatCatalogFingerprintLane(changed, {}, [])).toThrow();
expect(() => assertChatCatalogFingerprintLane(lane, { PAPERCLIP_RUNNER_E2E_INSTALLED_CLI: "/candidate" }, [])).toThrow();
expect(() => assertChatCatalogFingerprintLane(lane, {}, ["--target-config"])).toThrow();
const f = fixture(); f.write("selected.ts", "export const load=(name:string)=>import(name);");
const before = fingerprintChatDependencyClosure({ root: f.root, entries: ["selected.ts"], selected: lane,
reviewedDynamicResolutions: { "selected.ts": { sourceSha256: "outdated", resolution: "reviewed-inactive", dependencies: [] } } });
expect(before.unknownDynamicImports).toEqual(["selected.ts"]);
f.write("unselected.ts", "export const changed=true;");
expect(fingerprintChatDependencyClosure({ root: f.root, entries: ["selected.ts"], selected: lane }).digest).not.toBe(before.digest);
});
it("binds the real selected area and reviewed worker closure without importing unrelated domain areas", () => {
const descriptor = findChatCatalogCase("projects.create-project")!;
const result = chatCatalogCaseFingerprint(descriptor, { suiteId: "chat-features", flow: "agent_chat", environment: "local" });
const files = result.sources.map(source => source.file);
expect(files).toContain("tests/runner-e2e/chat-catalog-domains/internal-projects.ts");
expect(files).toContain("tests/runner-e2e/chat-catalog-domains/internal-support.ts");
expect(files).toContain("packages/adapter-utils/src/workspace-snapshot-thread-worker.ts");
expect(files).not.toContain("tests/runner-e2e/chat-catalog-domains/internal-goals.ts");
expect(files).not.toContain("tests/runner-e2e/chat-catalog-domains/identity-auth.ts");
expect(result.unknownDynamicImports).toEqual([]);
});
it("fails closed when preserved prerequisites or a rejected query lack their independent verifier", async () => {
const base: ChatCatalogDescriptor = { scenarioId: "selected", modulePath: "./chat-catalog-domains/selected.js", exportName: "cases",
sourceFiles: ["selected.ts"], oracleVersion: "v1", expectedRunCount: 1, turnTimeoutMs: 100, attemptTimeoutMs: 1000 };
const callbacks = { setup() {}, observe() {}, grade() {}, verifyResult() {} };
const setup = { ...base, setupPolicy: { funding: "preserve-scenario" as const, invariantCheckIds: ["saved-budget"] } };
await expect(loadChatCatalogImplementation(setup, async () => ({ cases: [{ ...setup, ...callbacks }] }))).rejects.toThrow("independent prerequisite assertions");
const rejected = { ...base, queryOutcome: "rejected" as const };
await expect(loadChatCatalogImplementation(rejected, async () => ({ cases: [{ ...rejected, ...callbacks }] }))).rejects.toThrow("independent rejection evidence");
});
tests/runner-e2e/chat-catalog-fingerprint.ts
+122
View File
@@ -0,0 +1,122 @@
import { createHash } from "node:crypto";
import { existsSync, readFileSync, readdirSync, statSync } from "node:fs";
import path from "node:path";
import { fileURLToPath } from "node:url";
import { buildSync } from "esbuild";
import type { MatrixExecution } from "./types.js";
import type { ChatCatalogDescriptor } from "./chat-catalog-registry.js";
const hash = (value: string) => createHash("sha256").update(value).digest("hex");
function resolveSource(specifier: string, parent: string): string {
const absolute = path.resolve(path.dirname(parent), specifier);
const candidates = [absolute.replace(/\.js$/, ".ts"), absolute.replace(/\.js$/, ".tsx"), absolute,
`${absolute}.ts`, path.join(absolute, "index.ts")];
const found = candidates.find(candidate => existsSync(candidate) && statSync(candidate).isFile());
if (!found) throw new Error(`Unresolved fingerprint dependency: ${specifier} from ${parent}`);
return found;
}
function conservativeSources(directory: string): string[] {
return readdirSync(directory, { withFileTypes: true }).flatMap(entry => {
if (entry.name.startsWith(".") || ["node_modules", "dist", "target", "results", "test-results", "playwright-report"].includes(entry.name)) return [];
const child = path.join(directory, entry.name);
return entry.isDirectory() ? conservativeSources(child) : /\.(ts|tsx|js|mjs|json|yaml)$/.test(entry.name) ? [child] : [];
});
}
/** Compile and inspect actual executable imports, rather than trusting a hand-maintained sourceFiles list. */
export function fingerprintChatDependencyClosure(input: {
root: string; entries: readonly string[]; selected: unknown;
/** Only pure metadata may be normalized; executable modules may never use this escape. */
normalizedMetadata?: Readonly<Record<string, unknown>>;
/** The registry's one variable import is bound to selected.modulePath above. */
selectedDispatchFile?: string;
reviewedDynamicResolutions?: Readonly<Record<string, { sourceSha256: string; resolution: string; dependencies: readonly string[] }>>;
}) {
const files = new Map<string, string>();
const unknownDynamicImports: string[] = [];
const resolvedDynamicImports: Array<{ file: string; resolution: string }> = [];
const visit = (absolute: string) => {
const relative = path.relative(input.root, absolute).split(path.sep).join("/");
if (relative.startsWith("../")) throw new Error(`Fingerprint dependency escaped repository: ${relative}`);
if (files.has(relative)) return;
const normalized = input.normalizedMetadata?.[relative];
if (normalized !== undefined) { files.set(relative, JSON.stringify(normalized)); return; }
const body = readFileSync(absolute, "utf8");
files.set(relative, body);
if (!/\.[cm]?[jt]sx?$/.test(absolute)) return;
const parsed = buildSync({ stdin: { contents: body, sourcefile: absolute,
loader: absolute.endsWith(".tsx") ? "tsx" : /\.ts$/.test(absolute) ? "ts" : "js" },
bundle: false, metafile: true, write: false, format: "esm", platform: "node", minifyWhitespace: true,
legalComments: "none", logLevel: "silent" });
const dependencies = Object.values(parsed.metafile!.outputs).flatMap(output => output.imports.map(item => item.path));
if (/\bimport\s*\(\s*(?!["'])/.test(parsed.outputFiles[0]!.text) && relative !== input.selectedDispatchFile) {
const reviewed = input.reviewedDynamicResolutions?.[relative];
if (reviewed && reviewed.sourceSha256 === hash(body)) {
resolvedDynamicImports.push({ file: relative, resolution: reviewed.resolution });
for (const dependency of reviewed.dependencies) visit(path.resolve(input.root, dependency));
} else unknownDynamicImports.push(relative);
}
for (const dependency of dependencies.filter(value => value.startsWith("."))) visit(resolveSource(dependency, absolute));
};
for (const entry of input.entries) visit(resolveSource(`./${entry}`, path.join(input.root, "entry.ts")));
// Unknown imports conservatively bind all source files.
if (unknownDynamicImports.length) for (const absolute of conservativeSources(input.root)) {
const relative = path.relative(input.root, absolute).split(path.sep).join("/");
if (!files.has(relative)) files.set(relative, input.normalizedMetadata?.[relative] !== undefined
? JSON.stringify(input.normalizedMetadata[relative]) : readFileSync(absolute, "utf8"));
}
for (const name of ["package.json", "pnpm-lock.yaml"]) if (existsSync(path.join(input.root, name))) files.set(name, readFileSync(path.join(input.root, name), "utf8"));
const sources = [...files].sort(([a], [b]) => a.localeCompare(b)).map(([file, body]) => ({ file, sha256: hash(body) }));
const digest = hash(JSON.stringify({ selected: input.selected, sources }));
return { digest, definitionDigest: digest, graderDigest: digest, fingerprintKind: "complete-execution-and-oracle-closure",
sources, resolvedDynamicImports, unknownDynamicImports: [...new Set(unknownDynamicImports)].sort() };
}
export function assertChatCatalogFingerprintLane(selected: unknown, env: NodeJS.ProcessEnv = process.env, args: readonly string[] = process.argv) {
const execution = selected as { suiteId?: string; flow?: string; environment?: string };
const installedKeys = ["PAPERCLIP_RUNNER_E2E_INSTALLED_CLI", "PAPERCLIP_RUNNER_E2E_INSTALLED_CLI_SHA256",
"PAPERCLIP_RUNNER_E2E_INSTALLED_SERVER_ROOT", "PAPERCLIP_RUNNER_E2E_INSTALLED_SERVER_SHA256", "PAPERCLIP_RUNNER_E2E_INSTALLED_STARTUP_ONLY"];
const forbiddenArguments = ["--target-config", "--github-review-config", "--connection-config", "--installed-startup-only", "product-journeys"];
if (execution.suiteId !== "chat-features" || execution.flow !== "agent_chat" || execution.environment !== "local"
|| installedKeys.some(key => env[key] && env[key] !== "0") || args.some(argument => forbiddenArguments.includes(argument)))
throw new Error("Reviewed dynamic resolutions require source-local Agent Chat without external dispatch overrides");
return { suiteId: execution.suiteId, flow: execution.flow, environment: execution.environment,
installedLane: false, externalJourneyDispatch: false };
}
export function chatCatalogCaseFingerprint(descriptor: ChatCatalogDescriptor, selectedExecution: unknown) {
const root = fileURLToPath(new URL("../../", import.meta.url));
const lane = assertChatCatalogFingerprintLane(selectedExecution);
const reviewedDynamicResolutions = JSON.parse(readFileSync(new URL("./chat-catalog-dynamic-resolutions.json", import.meta.url), "utf8"));
const selectedModule = `tests/runner-e2e/${descriptor.modulePath.slice(2)}`;
const scenario = JSON.parse(readFileSync(new URL("./chat-campaign-manifest.json", import.meta.url), "utf8")).scenarios
.find((row: { id: string }) => row.id === descriptor.scenarioId);
if (!scenario) throw new Error(`Missing frozen scenario: ${descriptor.scenarioId}`);
const { id, capabilities, context, query, expected_outcome } = scenario;
return fingerprintChatDependencyClosure({ root,
entries: [selectedModule, "tests/runner-e2e/chat-catalog-flow.ts", "tests/runner-e2e/chat-catalog-fingerprint.ts", "tests/runner-e2e/chat-catalog-dynamic-resolutions.json",
"tests/runner-e2e/chat-feature-flow.ts", "tests/runner-e2e/runner.spec.ts", "tests/runner-e2e/launch.ts",
"tests/runner-e2e/chat-scenarios/defaults.yaml", ...descriptor.sourceFiles.map(file => `tests/runner-e2e/${file}`)],
selected: { descriptor, selectedExecution, lane, scenario: { id, capabilities, context, query, expected_outcome } },
reviewedDynamicResolutions,
selectedDispatchFile: "tests/runner-e2e/chat-catalog-registry.ts",
normalizedMetadata: { "tests/runner-e2e/chat-catalog-descriptors.json": descriptor,
"tests/runner-e2e/chat-campaign-manifest.json": { id, capabilities, context, query, expected_outcome } },
});
}
export function normalizedChatCatalogExecution(execution: MatrixExecution) {
return { executionId: execution.id, suiteId: execution.suite.id, flow: execution.task.flow, journeyKind: "agent-chat",
profile: { id: execution.profile.id, provider: execution.profile.provider, model: execution.profile.model, credential: execution.profile.credential },
environment: execution.environment.id, task: { expectedRunCount: execution.task.expectedRunCount,
minimumExpectedRunCount: execution.task.minimumExpectedRunCount, turnTimeoutMs: execution.task.turnTimeoutMs,
attemptTimeoutMs: execution.task.attemptTimeoutMs } };
}
export function legacyChatFeatureFingerprint(execution: MatrixExecution) {
if (!["create-backlog", "delegate-announcement"].includes(execution.task.id)) throw new Error("Unknown legacy chat feature case");
return chatCatalogCaseFingerprint({ scenarioId: `tasks.${execution.task.id}`, modulePath: "./chat-feature-flow.js", exportName: "runChatFeatureFlow",
sourceFiles: ["chat-feature-flow.ts", "chat-feature-scoring.ts", "chat-scenarios/tasks.yaml", "completion-updates.ts"],
oracleVersion: "paperclip.chat-features.v1", expectedRunCount: execution.task.expectedRunCount,
expectedHumanMessageCount: 1, turnTimeoutMs: execution.task.turnTimeoutMs!, attemptTimeoutMs: execution.task.attemptTimeoutMs.local,
}, normalizedChatCatalogExecution(execution));
}
tests/runner-e2e/chat-catalog-flow.ts
+31 -16
View File
@@ -1,15 +1,19 @@
import { chatCatalogAllowedRunStatuses, gradeChatCatalogRuns } from "./chat-catalog-run-policy.js";
import { chatCatalogCaseFingerprint, normalizedChatCatalogExecution } from "./chat-catalog-fingerprint.js";
import { expect } from "@playwright/test";
import { pollUntil } from "./api.js";
import { collectChatRunEvidence, sendChatMessage } from "./chat-flow.js";
import { validateChatCatalogGrade, type ChatCatalogContext, type ChatCatalogGrade } from "./chat-catalog-contract.js";
import { findChatCatalogCase } from "./chat-catalog-registry.js";
import { findChatCatalogCase, loadChatCatalogImplementation } from "./chat-catalog-registry.js";
import { chatCatalogInventoryDigest } from "./chat-catalog-inventory.js";
/** The case owns product semantics; this flow owns the actual chat entry point and evidence lifecycle. */
export async function runChatCatalogFlow(context: ChatCatalogContext) {
const { input, scenario } = context;
const implementation = findChatCatalogCase(scenario.id);
if (!implementation) throw new Error(`No executable chat scenario: ${scenario.id}`);
const descriptor = findChatCatalogCase(scenario.id);
if (!descriptor) throw new Error(`No executable chat scenario: ${scenario.id}`);
const fingerprint = chatCatalogCaseFingerprint(descriptor, normalizedChatCatalogExecution(input.execution));
const implementation = await loadChatCatalogImplementation(descriptor);
const deadlineAt = Date.now() + implementation.attemptTimeoutMs - 30_000;
let state: Record<string, unknown> | undefined;
let observation: unknown;
@@ -40,13 +44,11 @@ export async function runChatCatalogFlow(context: ChatCatalogContext) {
const checks = [...grade.checks,
{ id: "catalog-complete-conditions", passed: !implementation!.coverageGaps?.length,
detail: implementation!.coverageGaps?.length ? `Missing conditions: ${implementation!.coverageGaps.join(", ")}` : "No known catalog condition is omitted." },
{ id: "catalog-original-human-query", passed: human[0]?.body.trim() === scenario.query.trim()
&& human.length === (implementation!.expectedHumanMessageCount ?? 1),
{ id: "catalog-original-human-query", passed: (implementation!.queryOutcome === "rejected" ? human.length === 0 : human[0]?.body.trim() === scenario.query.trim()
&& human.length === (implementation!.expectedHumanMessageCount ?? 1)),
detail: "The first persisted human message is the unchanged catalog request and all human turns were declared." },
{ id: "catalog-bounded-native-runs", passed: evidence.runs.length >= (implementation!.minimumRunCount ?? implementation!.expectedRunCount)
&& evidence.runs.length <= implementation!.expectedRunCount && evidence.runs.every(run => run.companyId === input.fixtures.company.id
&& run.runtimeMode === "native" && run.status === "succeeded"),
detail: "All observed company provider runs satisfy the declared count, identity and successful native execution bounds." },
...gradeChatCatalogRuns({ implementation: implementation!, companyId: input.fixtures.company.id,
leadId: input.fixtures.agent.id, source: evidence.issue, runs: evidence.runs }).checks,
];
grade = { passed: checks.every(check => check.passed), checks };
await input.evidence("chat-catalog-observation.json", { scenarioId: scenario.id, inventoryDigest: chatCatalogInventoryDigest,
@@ -54,10 +56,12 @@ export async function runChatCatalogFlow(context: ChatCatalogContext) {
return { grade, runs: evidence.runs, issue: evidence.issue };
}
try {
await input.evidence("chat-catalog-definition.json", { scenarioId: scenario.id, ...fingerprint,
suiteDefinitionHash: input.execution.suiteDefinitionHash, journeyKind: "agent-chat" });
state = await implementation.setup(context);
const company = `/api/companies/${input.fixtures.company.id}`;
const agents = await input.api.get<Array<{ id: string }>>(`${company}/agents`);
for (const agent of agents) await pollUntil({ label: `catalog agent readiness ${agent.id}`,
if (implementation.setupPolicy?.readiness !== "preserve-scenario") for (const agent of agents) await pollUntil({ label: `catalog agent readiness ${agent.id}`,
deadlineAt: Math.min(deadlineAt, Date.now() + 45_000), intervalMs: 500,
load: () => input.api.get<{ lifecycleState: string; status: string; lifecycleError?: string }>(`/api/agents/${agent.id}`),
accept: value => value.lifecycleState === "ready" && ["idle", "active"].includes(value.status),
@@ -65,18 +69,29 @@ export async function runChatCatalogFlow(context: ChatCatalogContext) {
});
const scopes = [{ scopeType: "company", scopeId: input.fixtures.company.id },
...agents.map(agent => ({ scopeType: "agent", scopeId: agent.id }))];
for (const scope of scopes) await input.api.post(`${company}/budgets/policies`, { ...scope, amount: 500,
if (implementation.setupPolicy?.funding !== "preserve-scenario") for (const scope of scopes) await input.api.post(`${company}/budgets/policies`, { ...scope, amount: 500,
hardStopEnabled: true, isActive: true, unpricedUsagePolicy: "allow" });
const funding = await input.api.get<{ policies: Array<{ scopeType: string; scopeId: string; amount: number; hardStopEnabled: boolean; isActive: boolean; unpricedUsagePolicy: string }> }>(`${company}/budgets/overview`);
expect(scopes.every(scope => funding.policies.some(policy => policy.scopeType === scope.scopeType && policy.scopeId === scope.scopeId
if (implementation.setupPolicy?.funding !== "preserve-scenario") expect(scopes.every(scope => funding.policies.some(policy => policy.scopeType === scope.scopeType && policy.scopeId === scope.scopeId
&& policy.amount === 500 && policy.hardStopEnabled && policy.isActive && policy.unpricedUsagePolicy === "allow")), "Public funding policy persisted").toBe(true);
await input.evidence("chat-catalog-funding.json", { scopes, amountCents: 500, hardStopEnabled: true,
unpricedUsagePolicy: "allow", setupProbeUsage: "unknown; production hello probes are outside heartbeat accounting" });
await input.evidence("chat-catalog-funding.json", { scopes, mode: implementation.setupPolicy?.funding ?? "standard",
policies: funding.policies.filter(policy => scopes.some(scope => scope.scopeType === policy.scopeType && scope.scopeId === policy.scopeId)), setupProbeUsage: "unknown; production hello probes are outside heartbeat accounting" });
await input.evidence("chat-catalog-setup.json", { scenario, state, inventoryDigest: chatCatalogInventoryDigest,
oracleVersion: implementation.oracleVersion, expectedRunCount: implementation.expectedRunCount,
minimumRunCount: implementation.minimumRunCount, turnTimeoutMs: implementation.turnTimeoutMs,
attemptTimeoutMs: implementation.attemptTimeoutMs });
if (implementation.verifySetup) {
const setupGrade = validateChatCatalogGrade(await implementation.verifySetup(context, state));
await input.evidence("chat-catalog-prepared-state.json", setupGrade);
expect(setupGrade.passed && (implementation.setupPolicy?.invariantCheckIds ?? []).every(id => setupGrade.checks.some(check => check.id === id && check.passed)), "Independent scenario prerequisites").toBe(true);
}
await input.evidence("chat-catalog-query.json", { query: scenario.query, expectedOutcome: implementation.queryOutcome ?? "persisted", journeyKind: "agent-chat" });
await sendChatMessage(input.page, scenario.query);
if (implementation.queryOutcome === "rejected") {
const rejection = validateChatCatalogGrade(await implementation.verifyQueryRejection!(context, state));
await input.evidence("chat-catalog-query-rejection.json", rejection);
expect(rejection.passed, "Real browser/API query rejection").toBe(true);
}
if (implementation.act) await implementation.act(context, state);
let settledAt: number | undefined;
let passedAt: number | undefined;
@@ -88,8 +103,8 @@ export async function runChatCatalogFlow(context: ChatCatalogContext) {
},
reject: result => {
if (result.runs.length > implementation.expectedRunCount) return "Scenario exceeded its declared run bound";
if (result.runs.some(run => ["failed", "cancelled", "timed_out"].includes(run.status))) return "Observed provider run failed";
if (result.runs.length && result.runs.every(run => run.status === "succeeded") && result.issue?.conversationState === "waiting") settledAt ??= Date.now();
if (result.runs.some(run => ["failed", "cancelled", "timed_out"].includes(run.status) && !chatCatalogAllowedRunStatuses(implementation).includes(run.status))) return "Observed provider run failed";
if (result.runs.length && result.runs.every(run => !["queued", "running"].includes(run.status)) && result.issue?.conversationState === "waiting") settledAt ??= Date.now();
else settledAt = undefined;
return settledAt && Date.now() - settledAt >= 15_000
? `Settled scenario failed: ${result.grade.checks.filter(check => !check.passed).map(check => check.id).join(", ")}` : undefined;
tests/runner-e2e/chat-catalog-registry.ts
+43 -13
View File
@@ -1,18 +1,48 @@
import { internalChatCatalogCases } from "./chat-catalog-domains/internal.js";
import { readFileSync } from "node:fs";
import type { ChatCatalogCase } from "./chat-catalog-contract.js";
import { chatCatalogScenarios } from "./chat-catalog-inventory.js";
// Import domain arrays here only after they supply executable setup/action/oracle.
export const chatCatalogCases: readonly ChatCatalogCase[] = [...internalChatCatalogCases];
export type ChatCatalogDescriptor = Omit<ChatCatalogCase, "setup" | "act" | "observe" | "grade" | "cleanup" | "verifyResult" | "verifySetup" | "verifyQueryRejection"> & {
modulePath: string;
exportName: string;
};
/** Pure discovery metadata. Importing the catalog never evaluates a domain module. */
export const chatCatalogCases: readonly ChatCatalogDescriptor[] = JSON.parse(
readFileSync(new URL("./chat-catalog-descriptors.json", import.meta.url), "utf8"),
);
const known = new Set(chatCatalogScenarios.map(row => row.id));
const registered = new Map<string, ChatCatalogCase>();
for (const implementation of chatCatalogCases) {
if (!known.has(implementation.scenarioId) || registered.has(implementation.scenarioId))
throw new Error(`Unknown or duplicate chat implementation: ${implementation.scenarioId}`);
if (typeof implementation.verifyResult !== "function" || !implementation.oracleVersion || !implementation.sourceFiles.length
|| implementation.expectedRunCount < 1 || implementation.turnTimeoutMs <= 0 || implementation.attemptTimeoutMs <= 0
|| (implementation.minimumRunCount ?? implementation.expectedRunCount) > implementation.expectedRunCount)
throw new Error(`Incomplete bounds/provenance for ${implementation.scenarioId}`);
registered.set(implementation.scenarioId, implementation);
export function findChatCatalogCase(scenarioId: string): ChatCatalogDescriptor | undefined {
const matches = chatCatalogCases.filter(row => row.scenarioId === scenarioId);
if (matches.length > 1) throw new Error(`Duplicate chat implementation: ${scenarioId}`);
const descriptor = matches[0];
if (!descriptor) return;
if (!known.has(scenarioId) || !descriptor.oracleVersion || !descriptor.sourceFiles.length
|| !/^\.\/chat-catalog-domains\/[a-z0-9-]+\.js$/.test(descriptor.modulePath)
|| !descriptor.exportName || descriptor.expectedRunCount < 0 || descriptor.turnTimeoutMs <= 0 || descriptor.attemptTimeoutMs <= 0
|| (descriptor.minimumRunCount ?? descriptor.expectedRunCount) > descriptor.expectedRunCount)
throw new Error(`Incomplete bounds/provenance for ${scenarioId}`);
return descriptor;
}
export async function loadChatCatalogImplementation(
descriptor: ChatCatalogDescriptor,
importer: (specifier: string) => Promise<Record<string, unknown>> = specifier => import(specifier),
): Promise<ChatCatalogCase> {
// The sole dynamic import boundary: only the selected descriptor's module executes.
const module = await importer(new URL(descriptor.modulePath, import.meta.url).href);
const implementations = module[descriptor.exportName];
if (!Array.isArray(implementations)) throw new Error(`Missing domain export: ${descriptor.exportName}`);
const selected = implementations.filter((row: ChatCatalogCase) => row.scenarioId === descriptor.scenarioId);
if (selected.length !== 1) throw new Error(`Missing or duplicate runtime case: ${descriptor.scenarioId}`);
const implementation = selected[0] as ChatCatalogCase;
for (const key of ["sourceFiles", "oracleVersion", "expectedRunCount", "minimumRunCount", "expectedHumanMessageCount", "turnTimeoutMs", "attemptTimeoutMs", "coverageGaps", "journeyKind", "authentication", "setupPolicy", "runPolicy", "queryOutcome"] as const) {
if (JSON.stringify(implementation[key]) !== JSON.stringify(descriptor[key])) throw new Error(`Runtime descriptor drift: ${descriptor.scenarioId}.${key}`);
}
if ((descriptor.setupPolicy?.funding === "preserve-scenario" || descriptor.setupPolicy?.readiness === "preserve-scenario")
&& (!descriptor.setupPolicy.invariantCheckIds.length || typeof implementation.verifySetup !== "function"))
throw new Error("Scenario-specific setup policy requires independent prerequisite assertions");
if (descriptor.queryOutcome === "rejected" && typeof implementation.verifyQueryRejection !== "function")
throw new Error("Rejected chat query requires independent rejection evidence");
if ([implementation.setup, implementation.observe, implementation.grade, implementation.verifyResult].some(value => typeof value !== "function"))
throw new Error(`Incomplete executable case: ${descriptor.scenarioId}`);
return implementation;
}
export function findChatCatalogCase(scenarioId: string): ChatCatalogCase | undefined { return registered.get(scenarioId); }
tests/runner-e2e/chat-catalog-run-policy.test.ts
+21
View File
@@ -0,0 +1,21 @@
import { expect, it } from "vitest";
import { gradeChatCatalogRuns } from "./chat-catalog-run-policy.js";
import type { ChatRun } from "./chat-flow.js";
const succeeded: ChatRun = { id: "source-run", companyId: "company", agentId: "ceo", status: "succeeded", runtimeMode: "native", contextSnapshot: { issueId: "source" } };
const worker: ChatRun = { id: "worker-run", companyId: "company", agentId: "worker", status: "running", runtimeMode: "native", contextSnapshot: { issueId: "task" } };
const scope = { companyId: "company", leadId: "ceo", source: { id: "source", companyId: "company", title: "Chat", status: "in_review", conversationState: "waiting" } };
it("requires successful native work by default and exact declared cancellation counts", () => {
expect(gradeChatCatalogRuns({ ...scope, implementation: { expectedRunCount: 1 }, runs: [succeeded] }).passed).toBe(true);
const cancelled = { ...succeeded, status: "cancelled" };
expect(gradeChatCatalogRuns({ ...scope, implementation: { expectedRunCount: 1 }, runs: [cancelled] }).passed).toBe(false);
const implementation = { expectedRunCount: 1, runPolicy: { kind: "terminal-counts" as const, terminalStatusCounts: { cancelled: 1 } } };
expect(gradeChatCatalogRuns({ ...scope, implementation, runs: [cancelled] }).passed).toBe(true);
expect(gradeChatCatalogRuns({ ...scope, implementation, runs: [succeeded] }).passed).toBe(false);
});
it("admits declared separate active work while rejecting active CEO, wrong scope, unbound and excess work", () => {
const implementation = { expectedRunCount: 2, runPolicy: { kind: "source-settled-with-active-work" as const, maximumActiveWorkerRuns: 1 } };
expect(gradeChatCatalogRuns({ ...scope, implementation, runs: [succeeded, worker] }).passed).toBe(true);
for (const bad of [{ ...succeeded, status: "running" }, { ...worker, companyId: "other" }, { ...worker, contextSnapshot: {} }, { ...worker, agentId: "ceo" }])
expect(gradeChatCatalogRuns({ ...scope, implementation, runs: [succeeded, bad] }).passed).toBe(false);
expect(gradeChatCatalogRuns({ ...scope, implementation, runs: [succeeded, worker, { ...worker, id: "extra" }] }).passed).toBe(false);
});
tests/runner-e2e/chat-catalog-run-policy.ts
+32
View File
@@ -0,0 +1,32 @@
import type { ChatCatalogCase, ChatCatalogGrade } from "./chat-catalog-contract.js";
import type { ChatIssue, ChatRun } from "./chat-flow.js";
export function chatCatalogAllowedRunStatuses(implementation: Pick<ChatCatalogCase, "runPolicy">): string[] {
const policy = implementation.runPolicy;
return policy?.kind === "terminal-counts" ? Object.keys(policy.terminalStatusCounts)
: policy?.kind === "source-settled-with-active-work" ? ["queued", "running", "succeeded"] : ["succeeded"];
}
export function gradeChatCatalogRuns(input: {
implementation: Pick<ChatCatalogCase, "expectedRunCount" | "minimumRunCount" | "runPolicy">;
companyId: string; leadId: string; source?: ChatIssue; runs: ChatRun[];
}): ChatCatalogGrade {
const { implementation: c, runs, source } = input;
const checks = [
{ id: "catalog-bounded-native-runs", passed: runs.length >= (c.minimumRunCount ?? c.expectedRunCount) && runs.length <= c.expectedRunCount
&& runs.every(run => run.companyId === input.companyId && run.runtimeMode === "native" && chatCatalogAllowedRunStatuses(c).includes(run.status)),
detail: "Observed provider runs satisfy declared count, company, native runtime and allowed outcome bounds." },
];
if (c.runPolicy?.kind === "terminal-counts") {
checks.push({ id: "catalog-exact-run-outcomes", passed: Object.entries(c.runPolicy.terminalStatusCounts).every(([status, count]) => runs.filter(run => run.status === status).length === count),
detail: "The independently observed ledger matches every declared successful/cancelled/failed terminal outcome count." });
} else if (c.runPolicy?.kind === "source-settled-with-active-work") {
const taskId = (run: ChatRun) => (run as ChatRun & { nativeIssueId?: string }).nativeIssueId ?? run.contextSnapshot?.issueId;
const sourceRuns = runs.filter(run => taskId(run) === source?.id);
const workers = runs.filter(run => taskId(run) !== source?.id);
checks.push({ id: "catalog-settled-source-separate-work", passed: !!source && source.companyId === input.companyId
&& source.conversationState === "waiting" && sourceRuns.length > 0 && sourceRuns.every(run => run.agentId === input.leadId && run.status === "succeeded")
&& workers.every(run => typeof taskId(run) === "string" && run.agentId !== input.leadId)
&& workers.filter(run => ["queued", "running"].includes(run.status)).length <= c.runPolicy.maximumActiveWorkerRuns,
detail: "The CEO source is settled; unfinished worker runs are separately task-bound and bounded. The case oracle must additionally bind their exact task and assignee." });
}
return { passed: checks.every(check => check.passed), checks };
}
tests/runner-e2e/chat-feature-flow.ts
+3
View File
@@ -1,3 +1,4 @@
import { legacyChatFeatureFingerprint } from "./chat-catalog-fingerprint.js";
import { runChatCatalogFlow } from "./chat-catalog-flow.js";
import { chatCatalogScenarios } from "./chat-catalog-inventory.js";
import { expect } from "@playwright/test";
@@ -19,6 +20,8 @@ export async function runChatFeatureFlow(context: {
return runChatCatalogFlow({ ...context, scenario });
}
const caseId = execution.task.id as ChatFeatureCase;
await input.evidence("chat-catalog-definition.json", { scenarioId: `tasks.${caseId}`,
...legacyChatFeatureFingerprint(execution), suiteDefinitionHash: execution.suiteDefinitionHash, journeyKind: "agent-chat" });
const scenario = chatFeatureScenarios.find(row => row.id === `tasks.${caseId}`);
if (!scenario) throw new Error(`Unregistered chat feature scenario: ${caseId}`);
const marker = execution.task.buildVisibleMarker(nonce);
tests/runner-e2e/chat-flow.ts
+11 -1
View File
@@ -1,3 +1,4 @@
import { chatCatalogAllowedRunStatuses } from "./chat-catalog-run-policy.js";
import { runHiringTemplateFlow } from "./hiring-template-flow.js";
import { gradeHiringTemplateTurns } from "./hiring-template-turn-accounting.js";
import type { HiringTemplateEvidence } from "./hiring-template-scoring.js";
@@ -18,6 +19,7 @@ import { enableChatThroughSettings, runChatInterruption, runChatSettingsLifecycl
import { runActiveReassignment, runWorkerCrash, runAnswerQuality } from "./chat-qualification.js";
import { matchesRunCount, minimumRunCount } from "./run-count.js";
import { runChatCompletionUpdate } from "./completion-update-flow.js";
import { findChatCatalogCase } from "./chat-catalog-registry.js";
import { CHAT_FEATURE_SUITE } from "./chat-feature-cases.js";
import { runChatFeatureFlow } from "./chat-feature-flow.js";
@@ -361,6 +363,8 @@ export interface ChatFlowInput {
execution: MatrixExecution;
nonce: string;
workspacePath: string;
/** Attempt-private redaction sink for synthetic secondary-user credentials. Never evidence. */
privateSecrets?: string[];
restart: () => Promise<void>;
observe: (issue: ChatIssue, runs: ChatRun[]) => void;
capture: (id: string, label: string, file: string) => Promise<void>;
@@ -374,6 +378,7 @@ export async function runChatFlow(input: ChatFlowInput) {
const marker = execution.task.buildVisibleMarker(nonce);
const draftMarker = chatMarker("DRAFT", nonce);
const caseId = execution.task.id;
const catalogDescriptor = execution.suite.id === CHAT_FEATURE_SUITE ? findChatCatalogCase(caseId) : undefined;
const stopCase = ["stop-new-resume", "stop-startup-new-resume"].includes(caseId);
const interruptionCase = ["followup-while-running", "revise-while-running"].includes(caseId);
const expectedStops = new Map<string, string>();
@@ -1007,7 +1012,8 @@ export async function runChatFlow(input: ChatFlowInput) {
projects,
});
}
await idle(minimumRunCount(execution.task));
if (catalogDescriptor) runs = await allRuns();
else await idle(minimumRunCount(execution.task));
if (execution.suite.id === "hiring-templates") {
// Refresh the whole consistent evidence generation for the final guard.
hiringEvidence = await refreshHiringEvidence!();
@@ -1021,6 +1027,10 @@ export async function runChatFlow(input: ChatFlowInput) {
});
for (const run of runs.filter((run) => !isResetRun(run))) {
expect(run.runtimeMode).toBe(execution.profile.expectedRuntimeMode);
if (catalogDescriptor?.runPolicy) {
expect(chatCatalogAllowedRunStatuses(catalogDescriptor)).toContain(run.status);
continue;
}
expect(run.status).toBe(
expectedStops.get(run.id) ?? (stopCase && run.status === "cancelled"
? "cancelled"
tests/runner-e2e/launch.ts
+2 -1
View File
@@ -1,3 +1,4 @@
import { findChatCatalogCase } from "./chat-catalog-registry.js";
import { recoverWorkerTiming } from "./worker-timing.js";
import { openEvalOperations, type EvalOperations } from "./operations.js";
import { readRecoveredFeedback } from "./feedback-recovery.js";
@@ -564,7 +565,7 @@ async function runAttempt(input: {
PAPERCLIP_E2E_ACCOUNTING_CAMPAIGN_ID: accountingCampaignId,
PAPERCLIP_E2E_ACCOUNTING_ATTEMPT_ID: accountingAttemptId,
PAPERCLIP_RUNNER_E2E_INSTALLED_STARTUP_ONLY: options.installedStartupOnly ? "1" : undefined,
PAPERCLIP_RUNNER_E2E_PUBLIC_MCP: executions.some(candidate => candidate.task.flow === "public_mcp") ? "1" : "0",
PAPERCLIP_RUNNER_E2E_PUBLIC_MCP: executions.some(candidate => candidate.task.flow === "public_mcp" || candidate.suite.id === "chat-features" && findChatCatalogCase(candidate.task.id)?.authentication === "human-session") ? "1" : "0",
PAPERCLIP_RUNNER_E2E_PORT: String(port),
PAPERCLIP_RUNNER_E2E_TEMP_ROOT: temporaryRoot,
PAPERCLIP_RUNNER_E2E_PRIVATE_DIR: privateDir,
tests/runner-e2e/runner.spec.ts
+3 -1
View File
@@ -1,3 +1,4 @@
import { findChatCatalogCase } from "./chat-catalog-registry.js";
import { createWorkerTiming } from "./worker-timing.js";
import { FEEDBACK_PERSISTENCE_SUITE, gradeFeedbackPersistence } from "./feedback-persistence.js";
import { EvalAttemptTrace, evalBraintrustConfig, EVAL_OBSERVABILITY_SECRET_NAMES } from "./braintrust.js";
@@ -918,7 +919,7 @@ for (const execution of executions) {
try {
trace?.phase("setup");
await saveTrace();
if (execution.task.flow === "public_mcp") publicMcpUserId = await establishPublicMcpSession(api, page, secrets);
if (execution.task.flow === "public_mcp" || execution.suite.id === "chat-features" && findChatCatalogCase(execution.task.id)?.authentication === "human-session") publicMcpUserId = await establishPublicMcpSession(api, page, secrets);
if (execution.suite.id === "stock-harness") {
const receipt = verifyStockHarnessPreflight(process.env[STOCK_PREFLIGHT_ENV]);
await writeSanitizedJson(snapshotsDir, "stock-harness-preflight.json", receipt, secrets);
@@ -1272,6 +1273,7 @@ for (const execution of executions) {
selectedRuns = contextIntegrity.runs as unknown as RunRecord[];
} else if (execution.task.flow === "agent_chat") {
const chat = await runChatFlow({
privateSecrets: secrets,
page, api, fixtures, execution, nonce, workspacePath,
restart: () => restartChatServer(page, () => restartIsolatedPaperclipServer({ api, requestId: `chat-${nonce}`, deadlineAt: startedAtMs + deadlineMs })),
observe: (chatIssue, chatRuns) => { issue = chatIssue; selectedRuns = chatRuns; },