mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-14 00:26:28 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog runs real chat journeys and grades saved
product state
> - Three cases failed in runs 43088288923 and 43101125481. In two of
them the product did the right thing
> - `cases.update-case-facts` used a word allow-list on a saved field.
`cases.create-case` read a field the product never fills for a new task
> - `task-coordination.block-on-design` timed out in the browser on a
missing row, which hid the real cause
> - This pull request grades what the user asked for, binds records by
structure, and stops with the real cause
> - The benefit is that each case fails only when the product fails, and
says why
## Linked Issues or Issue Description
**What happened**
- `cases.update-case-facts`: the CEO saved `browser: "Mobile Safari only
(bug reproduces only on mobile Safari)"` and kept the other fields. The
grader allowed only the words "safari" and mobile qualifiers, so "bug
reproduces" failed the first phase. The earlier rewording of the
scripted follow-up was not the cause: it is sent only after this phase.
- `cases.create-case`: the CEO created the case, the `workflow` and
`customer_brief` documents and Morgan's kickoff in one run, before
Morgan started. The grader required the kickoff's `sourceRunId` to be a
CEO run. The product sets `sourceRunId` to the run that holds the task
being handed off, so it is always null when the kickoff creates the
task. The grade never passed, so the harness kept waiting. Morgan then
found no app to test, handed the case back to the CEO, and the extra CEO
run broke the run bound.
- `task-coordination.block-on-design`: the CEO saved no Design blocker.
The scripted human then waited 30 s for the "Blocked by" row and failed
with a locator timeout.
**Expected behavior**
Each case accepts every valid product behavior and rejects the failures
that matter. A failure names its cause.
**Steps to reproduce**
1. Regrade the retained observation of run 43101125481 for
`cases.create-case` with the old grader: `real-first-morgan-kickoff` and
`required-documents-persist-before-investigation` fail although the
records are correct.
2. Call `corrected({ browser: "Mobile Safari only (bug reproduces only
on mobile Safari)" })` with the old grader: it returns false.
3. Run `task-coordination.block-on-design` when the CEO saves no
blocker: it fails on a UI locator instead of on the missing blocker.
## What Changed
- `cases.update-case-facts` (oracle v8): grade the saved correction
structurally, with no word lists. The browser fact differs from the
case's own setup value, and no field keeps the setup's stale literal
"not yet confirmed", in the saved case and in the reviewer's handoff
context. A person reads whether the wording says mobile Safari.
- `cases.create-case` (oracle case-create.v2): bind the kickoff and its
documents to the CEO chat run that was active when the kickoff was
saved, not to `sourceRunId`. Admit one CEO run on the primary task woken
by Morgan's own case handoff back to the CEO, and add a check that binds
it to that handoff. Run bound is now 2 to 3.
- Shared run policy (`chat-catalog-run-policy.ts`,
`chat-catalog-contract.ts`): new optional `maximumLeadCaseHandbackRuns`
for `source-settled-with-active-work`. The default is 0, so other cases
grade exactly as before. No other oracle version changes.
- `task-coordination.block-on-design` (oracle v9): after the first turn,
stop with "The CEO did not save Design as a blocker on Website" when the
edge is missing.
- Descriptors updated for the three cases.
## Verification
- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts` on
`integrator-case-create`, `integrator-case-facts`, `runtime-dependency`,
`chat-catalog-run-policy`, `chat-catalog-readiness`,
`chat-catalog-failure-retirement`, `chat-catalog-consistency` and
`chat-ordinary-phase-callers` tests: all pass.
- Calibration in both directions:
- case facts: the retained values of runs 43088288923 and 43107042937
and four rewordings pass. The unchanged setup value, a kept stale
literal (in any field), empty facts and a first turn that left the value
unchanged fail.
- create-case: the retained run 43101125481 regrades to pass. Documents
from another run, a kickoff outside the CEO run, a kickoff by Morgan, a
handback with no handoff, a handback woken another way and two handbacks
fail.
- run policy: an undeclared handback, a lead run woken another way and a
second handback fail.
- `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
unchanged.
- End-to-end: chat eval run linked in a comment below.
## Risks
- Low risk. Only eval code changes. The shared run-policy option is
opt-in, and its default keeps the old result.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
89 lines
9.2 KiB
TypeScript
89 lines
9.2 KiB
TypeScript
import { describe, expect, it, vi } from "vitest";
|
|
import { uploadFactualQueryAttachment } from "../chat-query-attachments.js";
|
|
vi.mock("../chat-query-attachments.js", async importOriginal => ({ ...await importOriginal<typeof import("../chat-query-attachments.js")>(), uploadFactualQueryAttachment: vi.fn(async () => ({ id: "customer-brief-upload" })) }));
|
|
import { customerBugFacts, preservesCustomerBugFacts, gradeIntegratorCaseCreation, integratorCaseCreationCases } from "./integrator-case-create.js";
|
|
function valid() {
|
|
const brief = { id: "brief", issueId: "primary", companyId: "company", key: "brief", body: Object.values(customerBugFacts).join("\n") };
|
|
const workflow = { id: "workflow", issueId: "primary", companyId: "company", key: "workflow", body: "Morgan investigates, saves findings, then hands this same primary task to its next owner for review." };
|
|
const revision = (doc: typeof brief) => ({ id: `${doc.id}-revision`, documentId: doc.id, issueId: "primary", key: doc.key, body: doc.body, createdByAgentId: "ceo", createdByRunId: "source-run", createdAt: "2026-10-26T12:00:00Z" });
|
|
return { scenarioId: "cases.create-case", origin: "https://fixture.test", companyId: "company", leadId: "ceo", state: { project: { id: "project" }, task: { id: "source-brief" }, worker: { id: "morgan" } },
|
|
tasks: [{ id: "source-brief", title: "Customer bug brief" }, { id: "primary", companyId: "company", title: "Customer checkout bug", assigneeAgentId: "morgan" }], projects: [], goals: [], comments: [], documents: [], revisions: [], humanQueries: [], activity: [], reply: "Created the case and started Morgan's investigation on its primary task.", links: ["/ACORN/cases/case", "/ACORN/issues/primary"],
|
|
cases: [{ id: "case", companyId: "company", projectId: "project", workIssueId: "primary", fields: structuredClone(customerBugFacts) }],
|
|
handoffs: { case: [{ id: "kickoff", caseId: "case", version: 1, issueId: "primary", toAgentId: "morgan", actorId: "ceo", sourceRunId: null as string | null, state: "delivered", createdAt: "2026-10-26T12:00:30Z" }] as Array<Record<string, unknown>> }, taskDocuments: { primary: [workflow, brief] },
|
|
documentVersions: { "primary:workflow": [revision(workflow)], "primary:brief": [revision(brief)] },
|
|
runs: [{ id: "source-run", agentId: "ceo", status: "succeeded", contextSnapshot: { issueId: "source-brief" } as Record<string, unknown>, startedAt: "2026-10-26T11:59:00Z", finishedAt: "2026-10-26T12:00:40Z" },
|
|
{ id: "investigation", agentId: "morgan", status: "running", contextSnapshot: { issueId: "primary" } as Record<string, unknown>, startedAt: "2026-10-26T12:01:00Z" } as Record<string, unknown>] };
|
|
}
|
|
describe("real case kickoff independent controls", () => {
|
|
it("requires the real primary task, first Morgan dispatch and original saved documents", () => {
|
|
expect(gradeIntegratorCaseCreation(valid()).passed).toBe(true);
|
|
expect(gradeIntegratorCaseCreation(undefined).passed).toBe(false);
|
|
const passive = valid(); passive.cases[0]!.workIssueId = "";
|
|
expect(gradeIntegratorCaseCreation(passive).passed).toBe(false);
|
|
const late = valid(); late.documentVersions["primary:brief"][0]!.createdAt = "2026-10-26T12:02:00Z";
|
|
expect(gradeIntegratorCaseCreation(late).passed).toBe(false);
|
|
const otherOwner = valid(); otherOwner.taskDocuments.primary[1]!.issueId = "source-brief";
|
|
expect(gradeIntegratorCaseCreation(otherOwner).passed).toBe(false);
|
|
const wrongWorker = valid(); wrongWorker.handoffs.case[0]!.toAgentId = "riley";
|
|
expect(gradeIntegratorCaseCreation(wrongWorker).passed).toBe(false);
|
|
const repaired = valid(); repaired.documentVersions["primary:brief"][0]!.body = "Investigate a bug.";
|
|
expect(gradeIntegratorCaseCreation(repaired).passed).toBe(false);
|
|
expect(gradeIntegratorCaseCreation({ ...valid(), links: ["/ACORN/cases/case"] }).passed).toBe(false);
|
|
});
|
|
// gha-43101125481: a kickoff that creates its primary task has no sourceRunId (that field names the run holding the
|
|
// task being handed off). The kickoff and its documents are bound to the CEO chat run active when it was saved.
|
|
it("binds the kickoff and its documents to the CEO chat run that saved them, not to sourceRunId", () => {
|
|
const failed = (value: unknown) => gradeIntegratorCaseCreation(value).checks.filter(check => !check.passed).map(check => check.id);
|
|
expect(failed(valid())).toEqual([]);
|
|
const otherRun = valid(); for (const key of Object.keys(otherRun.documentVersions)) otherRun.documentVersions[key as keyof typeof otherRun.documentVersions][0]!.createdByRunId = "elsewhere";
|
|
expect(failed(otherRun)).toContain("required-documents-persist-before-investigation");
|
|
const outside = valid(); outside.handoffs.case[0]!.createdAt = "2026-10-26T12:05:00Z";
|
|
expect(failed(outside)).toEqual(expect.arrayContaining(["real-first-morgan-kickoff", "required-documents-persist-before-investigation"]));
|
|
const byWorker = valid(); byWorker.handoffs.case[0]!.actorId = "morgan";
|
|
expect(failed(byWorker)).toContain("real-first-morgan-kickoff");
|
|
const otherChat = valid(); otherChat.runs[0]!.contextSnapshot = { issueId: "primary" };
|
|
expect(failed(otherChat)).toContain("real-first-morgan-kickoff");
|
|
});
|
|
it("accepts one CEO run woken by Morgan's own case handoff back, and nothing else", () => {
|
|
const failed = (value: unknown) => gradeIntegratorCaseCreation(value).checks.filter(check => !check.passed).map(check => check.id);
|
|
const handback = (o: ReturnType<typeof valid>, wakeReason = "case_handoff", issueId = "primary") => {
|
|
o.runs.push({ id: "handback", agentId: "ceo", status: "running", contextSnapshot: { issueId, wakeReason }, startedAt: "2026-10-26T12:03:00Z" });
|
|
return o;
|
|
};
|
|
const returned = (o: ReturnType<typeof valid>) => { o.handoffs.case.push({ id: "back", caseId: "case", version: 2, issueId: "primary", toAgentId: "ceo", actorId: "morgan", state: "delivered" }); return o; };
|
|
expect(failed(returned(handback(valid())))).toEqual([]);
|
|
expect(failed(handback(valid()))).toContain("only-declared-case-handback-to-ceo");
|
|
expect(failed(returned(handback(valid(), "issue_commented")))).toContain("only-declared-case-handback-to-ceo");
|
|
expect(failed(returned(handback(valid(), "case_handoff", "elsewhere")))).toContain("only-declared-case-handback-to-ceo");
|
|
expect(failed(returned(handback(handback(valid()))))).toContain("only-declared-case-handback-to-ceo");
|
|
});
|
|
it("accepts changed wording of factual fields but rejects swapped expected/observed behavior", () => {
|
|
const paraphrase = { customer: "Acorn Test Shop", issue: "Checkout stops responding", browser: "Safari 18 on iPhone 15", steps: ["Add the trial item", "Fill the test address", "Press Pay"], expectedBehavior: "An order confirmation should be shown", observedBehavior: "Without confirmation; customer is not charged" };
|
|
expect(preservesCustomerBugFacts(paraphrase)).toBe(true);
|
|
expect(preservesCustomerBugFacts({ ...paraphrase, expectedBehavior: "No order confirmation appears" })).toBe(false);
|
|
expect(preservesCustomerBugFacts({ ...paraphrase, browser: "Chrome on Android" })).toBe(false);
|
|
expect(preservesCustomerBugFacts({ ...paraphrase, observedBehavior: "Order confirmation appears and customer is charged" })).toBe(false);
|
|
});
|
|
});
|
|
|
|
it("places the genuine customer brief on the canonical conversation's normal document surface", async () => {
|
|
const source = { id: "standing-chat", title: "Chat with CEO" }; let saved: Record<string, unknown> | undefined;
|
|
const created: string[] = []; const patches: Array<{ path: string; body: unknown }> = [];
|
|
const api = {
|
|
get: async (path: string) => path.endsWith("/documents/brief") ? saved : path.startsWith("/api/agents/") ? { id: "morgan", companyId: "company", lifecycleState: "ready" } : { enableCases: false },
|
|
patch: async (path: string, body: unknown) => { patches.push({ path, body }); return {}; },
|
|
post: async (path: string, body: Record<string, unknown>) => { created.push(path); return { ...body, id: path.endsWith("/projects") ? "project" : "morgan" }; },
|
|
putSensitive: async (path: string, body: Record<string, unknown>) => { expect(path).toBe("/api/issues/standing-chat/documents/brief"); saved = { ...body, issueId: source.id }; return saved; },
|
|
};
|
|
const ctx = { input: { nonce: "offline", api, evidence: async () => undefined, fixtures: { company: { id: "company" }, agent: { id: "ceo" }, environment: { id: "environment" }, secretRefs: {} }, execution: { environment: { id: "local" }, profile: { credential: "ANTHROPIC_API_KEY", buildAgent: () => ({}) } } },
|
|
refreshIssue: async () => {}, issue: () => source } as unknown as Parameters<typeof integratorCaseCreationCases[0]["setup"]>[0];
|
|
const state = await integratorCaseCreationCases[0]!.setup(ctx);
|
|
expect((state.task as { id: string }).id).toBe(source.id);
|
|
expect(saved?.body).toContain(customerBugFacts.customer);
|
|
expect(uploadFactualQueryAttachment).toHaveBeenLastCalledWith(ctx, { filename: "customer-bug-brief.md", body: saved?.body });
|
|
expect((state.factualContextAttachment as { id: string }).id).toBe("customer-brief-upload");
|
|
expect(created.some(path => path.endsWith("/issues"))).toBe(false);
|
|
expect(patches).toContainEqual({ path: "/api/issues/standing-chat", body: { projectId: "project" } });
|
|
await integratorCaseCreationCases[0]!.cleanup!(ctx, state);
|
|
});
|