diff --git a/doc/evals.md b/doc/evals.md index 77dca7c299f..5acd205811b 100644 --- a/doc/evals.md +++ b/doc/evals.md @@ -61,7 +61,9 @@ tests do not establish a completed live pass or unattended CI qualification. The deployment-targeted `question-answer-resume` journey reuses the classic two-question scenario and native continuation oracle on an attached target. It checks the actual board answers, one to three bound work runs, an agent-authored -task document and browser readback after reload. Its provider owns target setup, +task document and browser readback after reload. It accepts provider questions and +durable questions proven by an applied semantic receipt or an immutable completed +native API mutation receipt. It does not qualify named-tool selection. Its provider owns target setup, usage and cleanup. Registration and local calibration do not establish a fresh Cloud pass or inclusion in a recurring selection. @@ -551,7 +553,7 @@ any qualification claim; catalog presence alone is not a pass. The explicit-only [native question/resume qualification](../tests/runner-e2e/README.md#task-continuation) separates a completed two-answer user journey from semantic-tool documentation -qualification. It verifies the exact provider-pause or semantic-response-wake +qualification. It verifies the exact provider-pause or native response-wake binding for each answer and preserves prior grades when the definition changes. ## Lifecycle behavior baseline diff --git a/tests/runner-e2e/PRODUCT-JOURNEYS.md b/tests/runner-e2e/PRODUCT-JOURNEYS.md index ee715a8936d..c8c3c00bcfa 100644 --- a/tests/runner-e2e/PRODUCT-JOURNEYS.md +++ b/tests/runner-e2e/PRODUCT-JOURNEYS.md @@ -36,8 +36,11 @@ never start a server or depend on a tenant filesystem. The question-resumption journey uses the same natural-language scenario, browser answer helper and behavioral grader as the classic `question-resume` suite. Native -provider questions can stay in one run; semantic questions create one exactly bound -continuation each. A target declares a three-run maximum and positive model allowance, +provider questions can stay in one run; durable native questions create one exactly bound +continuation each. Creation must have an applied semantic receipt or a completed native +API mutation receipt with the original card, exact scope, form and idempotency key. +An API receipt must retain its creation identity unchanged after the answer. A form or +run count alone is never provenance. A target declares a three-run maximum and positive model allowance, and a passing receipt must contain one to three attributed native runs. Other journeys retain their exact run-count requirements. Both real board answers, no premature output, the sole document's agent/company/task ownership and its reloaded revision diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index beb3ffe5c3b..df5a2cc39f4 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -192,8 +192,8 @@ The explicit-only `necessary-question` suite has four development cells: the recorded Codex and Claude models, on local and Daytona environments. It requests one essential booking reference that only the test user knows. The browser must show the actual free-text prompt and its explanation, then submit the real answer. -The oracle checks the unchanged question, its server-owned provider or semantic -receipt, the exact stored user answer, and one bound continuation. The saved +The oracle checks the unchanged question, its server-owned provider, semantic or +native API receipt, the exact stored user answer, and one bound continuation. The saved confirmation must contain that answer exactly. The final reply's link must open that same task document and display its confirmation. A title, success claim, replacement card, duplicate wake, or uncommitted result @@ -1705,13 +1705,18 @@ placement; it is not a statistical comparison with the former prompt arrangement The separate explicit-only `question-resume` suite uses the same neutral two-question request on native Codex and ACPX Claude. It qualifies the user journey through -either the provider's built-in question or Paperclip's semantic question tool. +the provider's built-in question, Paperclip's semantic question tool, or its native API fallback. It does not replace or regrade `question-tool-documentation`. The oracle classifies each wait from its exact task/agent/run identity. Provider questions require the server-generated request key and retain the paused run. -Semantic questions require an applied `request_human_input` receipt and one new -response wake bound to that interaction and source run. Only a verified provider +Durable questions require an applied `request_human_input` receipt or a completed +server-owned native `POST /api/issues/{id}/interactions` receipt. An API creation +receipt must report HTTP 201 and preserve the original pending card's exact task, +company, agent, source run, idempotency key and complete normalized form. It must +remain unchanged after the answer. Both durable paths require one new response +wake bound to that interaction and source run. Missing, pending, foreign or +mismatched receipts fail; form shape and run counts cannot replace them. Only a verified provider choice may carry the adapter's optional Other field. A second substantive question is still rejected; the subsequent reference question must be text-only. Both real board answers must remain bound to unchanged forms, and one saved document must diff --git a/tests/runner-e2e/fixtures/native-api-question.json b/tests/runner-e2e/fixtures/native-api-question.json new file mode 100644 index 00000000000..48084858be0 --- /dev/null +++ b/tests/runner-e2e/fixtures/native-api-question.json @@ -0,0 +1,152 @@ +{ + "phase": "initial", + "issue": { + "id": "00000000-0000-4000-8000-000000000002", + "companyId": "00000000-0000-4000-8000-000000000001", + "assigneeAgentId": "00000000-0000-4000-8000-000000000003", + "status": "in_review" + }, + "runs": [ + { + "id": "00000000-0000-4000-8000-000000000004", + "companyId": "00000000-0000-4000-8000-000000000001", + "issueId": "00000000-0000-4000-8000-000000000002", + "agentId": "00000000-0000-4000-8000-000000000003", + "status": "succeeded", + "runtimeMode": "native", + "resultJson": { + "apiToolReceipts": { + "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb": { + "state": "completed", + "digest": "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc", + "operationId": "POST /api/issues/{id}/interactions", + "result": { + "ok": true, + "status": 201, + "contentType": "application/json; charset=utf-8", + "operationId": "POST /api/issues/{id}/interactions", + "data": { + "kind": "ask_user_questions", + "status": "pending", + "continuationPolicy": "wake_assignee", + "payload": { + "version": 1, + "supersedeOnUserComment": false, + "questions": [ + { + "id": "time_choice", + "prompt": "Should the club meetup welcome note reference Morning or Afternoon?", + "selectionMode": "single", + "required": true, + "allowOther": false, + "options": [ + { + "id": "morning", + "label": "Morning" + }, + { + "id": "afternoon", + "label": "Afternoon" + } + ] + } + ], + "questionSet": { + "schema": "paperclip.question_set.v1", + "questions": [ + { + "id": "time_choice", + "prompt": "Should the club meetup welcome note reference Morning or Afternoon?", + "required": true, + "answerMode": "single_select", + "options": [ + { + "id": "morning", + "label": "Morning" + }, + { + "id": "afternoon", + "label": "Afternoon" + } + ] + } + ] + } + }, + "id": "00000000-0000-4000-8000-000000000005", + "sourceRunId": "00000000-0000-4000-8000-000000000004", + "companyId": "00000000-0000-4000-8000-000000000001", + "issueId": "00000000-0000-4000-8000-000000000002", + "createdByAgentId": "00000000-0000-4000-8000-000000000003", + "createdByUserId": null, + "idempotencyKey": "meetup-time-choice-api-v1" + } + } + } + } + } + } + ], + "interactions": [ + { + "kind": "ask_user_questions", + "status": "pending", + "continuationPolicy": "wake_assignee", + "payload": { + "version": 1, + "supersedeOnUserComment": false, + "questions": [ + { + "id": "time_choice", + "prompt": "Should the club meetup welcome note reference Morning or Afternoon?", + "selectionMode": "single", + "required": true, + "allowOther": false, + "options": [ + { + "id": "morning", + "label": "Morning" + }, + { + "id": "afternoon", + "label": "Afternoon" + } + ] + } + ], + "questionSet": { + "schema": "paperclip.question_set.v1", + "questions": [ + { + "id": "time_choice", + "prompt": "Should the club meetup welcome note reference Morning or Afternoon?", + "required": true, + "answerMode": "single_select", + "options": [ + { + "id": "morning", + "label": "Morning" + }, + { + "id": "afternoon", + "label": "Afternoon" + } + ] + } + ] + } + }, + "id": "00000000-0000-4000-8000-000000000005", + "sourceRunId": "00000000-0000-4000-8000-000000000004", + "companyId": "00000000-0000-4000-8000-000000000001", + "issueId": "00000000-0000-4000-8000-000000000002", + "createdByAgentId": "00000000-0000-4000-8000-000000000003", + "createdByUserId": null, + "idempotencyKey": "meetup-time-choice-api-v1" + } + ], + "documents": [], + "attachments": [], + "children": [], + "comments": [] +} diff --git a/tests/runner-e2e/product-journeys/clarification-evidence.test.ts b/tests/runner-e2e/product-journeys/clarification-evidence.test.ts index d80af59931b..88c0ccb47cf 100644 --- a/tests/runner-e2e/product-journeys/clarification-evidence.test.ts +++ b/tests/runner-e2e/product-journeys/clarification-evidence.test.ts @@ -28,7 +28,7 @@ import type { JourneyContext } from "./types.js"; type Row = Record; const context = { company: { id: "company" }, user: { id: "board" }, environmentId: "daytona", agent: { id: "agent", provider: "opencode", model: "model", runtimeMode: "native" } } as JourneyContext; -function recording(paths: Array<"provider" | "semantic">) { +function recording(paths: Array<"provider" | "semantic" | "api">) { const runs: Row[] = [], cards: Row[] = [], checkpoints: any[] = []; const createRun = (previous?: Row) => ({ id: `run-${runs.length}`, status: "running", runtimeMode: "native", companyId: "company", issueId: "task", agentId: "agent", contextSnapshot: { paperclipEnvironment: { id: "daytona" }, ...(previous ? { issueId: "task", sourceRunId: previous.sourceRunId, @@ -42,12 +42,18 @@ function recording(paths: Array<"provider" | "semantic">) { for (const [index, path] of paths.entries()) { const run = runs.at(-1)!; const field = index === 0 ? { id: "choice", answerMode: "single_select", options: [{ id: "am", label: "Morning" }, { id: "pm", label: "Afternoon" }] } : { id: "reference", answerMode: "text" }; - const request = `request-${index}`, semantic = path === "semantic"; + const request = `request-${index}`, semantic = path !== "provider"; const card: Row = { id: `question-${index}`, kind: "ask_user_questions", status: "pending", companyId: "company", issueId: "task", createdByAgentId: "agent", sourceRunId: run.id, - idempotencyKey: semantic ? `semantic-${index}` : `paperclip-runner-question:${run.id}:${request}`, continuationPolicy: semantic ? "wake_assignee" : "none", + idempotencyKey: semantic ? `semantic-${index}` : `paperclip-runner-question:${run.id}:${request}`, continuationPolicy: semantic ? "wake_assignee" : "none", createdByUserId: null, payload: { ...(semantic ? {} : { runtimeRequestId: request }), questionSet: { questions: [field] } } }; cards.push(card); - if (semantic) { run.status = "succeeded"; run.resultJson.semanticToolReceipts[card.idempotencyKey] = { operationId: "request_human_input", result: { disposition: "applied", interaction: structuredClone(card) } }; } + if (semantic) { + run.status = "succeeded"; + if (path === "api") run.resultJson.apiToolReceipts = { ["a".repeat(64)]: { state: "completed", digest: "b".repeat(64), + operationId: "POST /api/issues/{id}/interactions", result: { ok: true, status: 201, + operationId: "POST /api/issues/{id}/interactions", data: structuredClone(card) } } }; + else run.resultJson.semanticToolReceipts[card.idempotencyKey] = { operationId: "request_human_input", result: { disposition: "applied", interaction: structuredClone(card) } }; + } snapshot(index === 0 ? "initial" : "answered"); card.status = "answered"; card.resolvedByUserId = "board"; card.result = { answers: [{ questionId: field.id, optionIds: index === 0 ? ["pm"] : [], ...(index === 1 ? { otherText: "Use AMBERnonce." } : {}) }] }; @@ -59,7 +65,7 @@ function recording(paths: Array<"provider" | "semantic">) { const failures = (f: ReturnType) => gradeClarificationEvidence(context, "task", "AMBERnonce", f.checkpoints, f.readback).filter(c => !c.passed); describe("attached clarification evidence", () => { - it.each([true, false])("reports API fallback without accepting it as native question evidence (choice=%s)", choice => { + it.each([true, false])("rejects an incomplete API receipt instead of inferring native question evidence (choice=%s)", choice => { const f = recording(["semantic", "semantic"]), point = f.checkpoints[choice ? 0 : 1]; const question = point.interactions.find((i: Row) => i.status === "pending"), source = point.runs.at(-1); source.resultJson = { apiToolReceipts: { call: { state: "completed", operationId: "POST /api/issues/{id}/interactions", @@ -76,25 +82,28 @@ describe("attached clarification evidence", () => { point.runs[0].resultJson.semanticToolReceipts[question.idempotencyKey].result.interaction.payload = structuredClone(question.payload); expect(clarificationQuestionFailure(point, question, true)).toBe("clarification_choice_form_invalid"); }); - it.each([["provider", "provider"], ["provider", "semantic"], ["semantic", "provider"], ["semantic", "semantic"]] as const)("accepts the real %s/%s paths with native binding and reload proof", (...paths) => { + it.each([["provider", "provider"], ["provider", "semantic"], ["semantic", "provider"], ["semantic", "semantic"], + ["api", "api"], ["api", "semantic"], ["semantic", "api"], ["provider", "api"], ["api", "provider"]] as const)("accepts the real %s/%s paths with native binding and reload proof", (...paths) => { expect(failures(recording([...paths]))).toEqual([]); }); it.each(["foreign-user", "foreign-agent-document", "foreign-task-document", "lost-revision", "wrong-visible-answer", "changed-readback-body", "premature-document", "extra-child", "foreign-model", "foreign-environment", "missing-native-input", "duplicate-question", "wrong-answer"])("rejects a plausible false pass: %s", kind => { - const f = recording(["semantic", "semantic"]), first = f.checkpoints[0], final = f.checkpoints[2]; - if (kind === "foreign-user") final.interactions[1].resolvedByUserId = "other-board"; - if (kind === "foreign-agent-document") final.documents[0].updatedByAgentId = "other-agent"; - if (kind === "foreign-task-document") final.documents[0].issueId = "other-task"; - if (kind === "lost-revision") f.readback.revisionId = "old-revision"; - if (kind === "wrong-visible-answer") f.readback.visibleBody = "Welcome to the morning meetup."; - if (kind === "changed-readback-body") f.readback.body += " changed"; - if (kind === "premature-document") first.documents = [final.documents[0]]; - if (kind === "extra-child") final.children.push({ id: "extra-child" }); - if (kind === "foreign-model") final.runs[0].runnerProfileJson.nativeExecutionInput.provider.model = "other-model"; - if (kind === "foreign-environment") final.runs[0].contextSnapshot.paperclipEnvironment.id = "local"; - if (kind === "missing-native-input") delete final.runs[0].runnerProfileJson.nativeExecutionInput.task.prompt; - if (kind === "duplicate-question") final.interactions.push(structuredClone(final.interactions[1])); - if (kind === "wrong-answer") final.interactions[1].result.answers[0].otherText = "Use the old reference."; - expect(failures(f).length).toBeGreaterThan(0); + for (const path of ["semantic", "api"] as const) { + const f = recording([path, path]), first = f.checkpoints[0], final = f.checkpoints[2]; + if (kind === "foreign-user") final.interactions[1].resolvedByUserId = "other-board"; + if (kind === "foreign-agent-document") final.documents[0].updatedByAgentId = "other-agent"; + if (kind === "foreign-task-document") final.documents[0].issueId = "other-task"; + if (kind === "lost-revision") f.readback.revisionId = "old-revision"; + if (kind === "wrong-visible-answer") f.readback.visibleBody = "Welcome to the morning meetup."; + if (kind === "changed-readback-body") f.readback.body += " changed"; + if (kind === "premature-document") first.documents = [final.documents[0]]; + if (kind === "extra-child") final.children.push({ id: "extra-child" }); + if (kind === "foreign-model") final.runs[0].runnerProfileJson.nativeExecutionInput.provider.model = "other-model"; + if (kind === "foreign-environment") final.runs[0].contextSnapshot.paperclipEnvironment.id = "local"; + if (kind === "missing-native-input") delete final.runs[0].runnerProfileJson.nativeExecutionInput.task.prompt; + if (kind === "duplicate-question") final.interactions.push(structuredClone(final.interactions[1])); + if (kind === "wrong-answer") final.interactions[1].result.answers[0].otherText = "Use the old reference."; + expect(failures(f).length).toBeGreaterThan(0); + } }); it("requires reload evidence even for a correct API document", () => { const f = recording(["provider", "provider"]); diff --git a/tests/runner-e2e/question-resume-scoring.ts b/tests/runner-e2e/question-resume-scoring.ts index 10e3f81e6bc..85f8902d95e 100644 --- a/tests/runner-e2e/question-resume-scoring.ts +++ b/tests/runner-e2e/question-resume-scoring.ts @@ -3,14 +3,33 @@ import type { ContinuationCheckpoint } from "./continuation-scoring.js"; import { isSingleClaudeQuestion } from "./runtime-question-readiness.js"; type Row = Record; -export type QuestionPath = "provider" | "semantic"; +export type QuestionPath = "provider" | "semantic" | "api"; const nonempty = (v: unknown): v is string => typeof v === "string" && v.trim().length > 0; const questions = (c?: ContinuationCheckpoint): Row[] => (c?.interactions ?? []).filter((i: any) => i.kind === "ask_user_questions") as Row[]; export const resumeQuestionFields = (i?: Row): Row[] => Array.isArray(i?.payload?.questionSet?.questions) ? i.payload.questionSet.questions : []; -/** Classify by server-owned identities and the applied semantic receipt. A +function nativeApiQuestionReceipt(run: Row | undefined, question: Row) { + const hash = (value: unknown) => typeof value === "string" && /^[a-f0-9]{64}$/.test(value); + return Object.entries(run?.resultJson?.apiToolReceipts ?? {}).find(([key, value]) => { + const receipt = value as Row, original = receipt?.result?.data; + return hash(key) && hash(receipt?.digest) && receipt?.state === "completed" + && receipt.operationId === "POST /api/issues/{id}/interactions" && receipt.result?.operationId === receipt.operationId + && receipt.result.ok === true && receipt.result.status === 201 && original?.status === "pending" + && original.kind === "ask_user_questions" && original.createdByUserId === null && question.createdByUserId === null + && sameQuestion(original, question); + }); +} + +function sameNativeApiQuestionReceipt(before: ContinuationCheckpoint | undefined, after: ContinuationCheckpoint | undefined, question?: Row) { + if (!question) return false; + const receipt = nativeApiQuestionReceipt(before?.runs.find(run => run.id === question.sourceRunId), question); + return !!receipt && isDeepStrictEqual(receipt, + nativeApiQuestionReceipt(after?.runs.find(run => run.id === question.sourceRunId), question)); +} + +/** Classify by server-owned identities and an applied native mutation receipt. A * missing receipt is unknown, never inferred from form shape or run count. */ export function questionPath(checkpoint: ContinuationCheckpoint | undefined, i?: Row): QuestionPath | null { if (!checkpoint || !i || !nonempty(i.id) || !nonempty(i.sourceRunId) || !nonempty(i.idempotencyKey)) return null; @@ -29,11 +48,13 @@ export function questionPath(checkpoint: ContinuationCheckpoint | undefined, i?: } const receipt = run.resultJson?.semanticToolReceipts?.[i.idempotencyKey]; const created = receipt?.result?.interaction; - return run.status === "succeeded" && i.continuationPolicy === "wake_assignee" && + if (run.status !== "succeeded" || i.continuationPolicy !== "wake_assignee") return null; + if ( receipt?.operationId === "request_human_input" && receipt.result?.disposition === "applied" && created?.id === i.id && created?.sourceRunId === run.id && created?.issueId === i.issueId && created?.companyId === i.companyId && created?.kind === "ask_user_questions" && - isDeepStrictEqual(created?.payload?.questionSet, i.payload?.questionSet) ? "semantic" : null; + isDeepStrictEqual(created?.payload?.questionSet, i.payload?.questionSet)) return "semantic"; + return nativeApiQuestionReceipt(run, i) ? "api" : null; } export function validResumeQuestionForm(checkpoint: ContinuationCheckpoint, i: Row, choice: boolean): boolean { @@ -62,14 +83,15 @@ export function questionResumedOnce(before: ContinuationCheckpoint | undefined, before.runs.some(r => !after.runs.some(n => n.id === r.id))) return false; if (path === "provider") return next.length === 0 && after.runs.some(r => r.id === i.sourceRunId && ["running", "succeeded"].includes(r.status)); - if (path !== "semantic" || next.length !== 1) return false; + if ((path !== "semantic" && path !== "api") || next.length !== 1) return false; + if (path === "api" && !sameNativeApiQuestionReceipt(before, after, i)) return false; const context = next[0].contextSnapshot; return context?.issueId === i.issueId && context?.sourceRunId === i.sourceRunId && context?.interactionId === i.id && context?.interactionKind === "ask_user_questions" && context?.interactionStatus === "answered"; } -/** Behavioral qualification supports either valid native path. It does not +/** Behavioral qualification supports each verified native path. It does not * replace the separate documentation-placement/semantic-tool qualification. */ export function gradeQuestionResume(checkpoints: ContinuationCheckpoint[], marker: string) { const initial = checkpoints.find(c => c.phase === "initial"); @@ -112,8 +134,9 @@ export function gradeQuestionResume(checkpoints: ContinuationCheckpoint[], marke detail: "The real board answer must remain bound to the unchanged question, with no fabricated or substituted response.", }, { id: `question-resume.${answer}.path`, - passed: questionResumedOnce(before, after, i), - detail: `Observed path: ${questionPath(before, i) ?? "unknown"}; provider answers resume the same run; semantic answers require one exactly bound response wake.`, + passed: questionResumedOnce(before, after, i) && + (questionPath(before, i) !== "api" || sameNativeApiQuestionReceipt(before, final, i)), + detail: `Observed path: ${questionPath(before, i) ?? "unknown"}; provider answers resume the same run; durable native questions require one exactly bound response wake.`, }); } checks.push({ diff --git a/tests/runner-e2e/question-resume.test.ts b/tests/runner-e2e/question-resume.test.ts index 525bce0c199..bded4d1c8d2 100644 --- a/tests/runner-e2e/question-resume.test.ts +++ b/tests/runner-e2e/question-resume.test.ts @@ -1,4 +1,5 @@ import { describe, expect, it } from "vitest"; +import { readFileSync } from "node:fs"; import { gradeQuestionResume, questionPath, validResumeQuestionForm, type QuestionPath } from "./question-resume-scoring.js"; import { gradeContinuation } from "./continuation-scoring.js"; import { gradeLifecycleBaseline } from "./lifecycle-baseline.js"; @@ -26,7 +27,7 @@ function recording(paths: QuestionPath[] = ["provider", "semantic"]) { }; for (let n = 0; n < 2; n++) { const run = runs.at(-1)!; - const provider = paths[n] === "provider"; + const provider = paths[n] === "provider", api = paths[n] === "api"; const request = `request-${n}`; const idempotencyKey = provider ? `paperclip-runner-question:${run.id}:${request}` : `question-${n}`; const field = n === 0 ? { id: "field-1-question_0-hash", answerMode: "single_select", options: [ @@ -36,12 +37,15 @@ function recording(paths: QuestionPath[] = ["provider", "semantic"]) { if (provider && n === 0) questions.push({ id: "field-2-question_0_custom-hash", answerMode: "text", header: "Other", required: false }); const card: Row = { id: `card-${n}`, kind: "ask_user_questions", status: "pending", companyId: "company", issueId: "task", createdByAgentId: "agent", sourceRunId: run.id, idempotencyKey, - continuationPolicy: provider ? "none" : "wake_assignee", + continuationPolicy: provider ? "none" : "wake_assignee", createdByUserId: null, payload: { ...(provider ? { runtimeRequestId: request } : {}), questionSet: { questions } } }; cards.push(card); if (!provider) { run.status = "succeeded"; - run.resultJson.semanticToolReceipts[idempotencyKey] = { operationId: "request_human_input", + if (api) run.resultJson.apiToolReceipts = { ["a".repeat(64)]: { state: "completed", digest: "b".repeat(64), + operationId: "POST /api/issues/{id}/interactions", result: { ok: true, status: 201, + operationId: "POST /api/issues/{id}/interactions", data: structuredClone(card) } } }; + else run.resultJson.semanticToolReceipts[idempotencyKey] = { operationId: "request_human_input", result: { disposition: "applied", interaction: structuredClone(card) } }; } snap(n === 0 ? "initial" : "answered"); @@ -58,6 +62,58 @@ const failed = (r: any[]) => gradeQuestionResume(r, "AMBERnonce").filter(c => !c const question = (r: any[], n = 0) => r[n].interactions[n]; describe("native question/resume path qualification", () => { + it("accepts a native API creation receipt with the retained server response shape", () => { + const point = JSON.parse(readFileSync(new URL("./fixtures/native-api-question.json", import.meta.url), "utf8")); + expect(questionPath(point, point.interactions[0])).toBe("api"); + expect(validResumeQuestionForm(point, point.interactions[0], true)).toBe(true); + }); + it.each(["missing", "pending", "wrong-operation", "wrong-result-operation", "unknown-outcome", "wrong-status", "missing-digest", "wrong-key", + "foreign-card", "foreign-company", "foreign-issue", "foreign-agent", "foreign-source", "wrong-idempotency", "changed-form", "changed-compatibility-form", + "board-created", "resolved-creation", "wrong-policy", "runtime-request", "legacy-run", "active-source", "foreign-run", "duplicate-source", "spoofed-store"])("rejects unverified native API provenance: %s", kind => { + const point = JSON.parse(readFileSync(new URL("./fixtures/native-api-question.json", import.meta.url), "utf8")); + const card = point.interactions[0], run = point.runs[0], receipts = run.resultJson.apiToolReceipts; + const receipt = Object.values(receipts)[0] as Row, original = receipt.result.data; + if (kind === "missing") delete run.resultJson.apiToolReceipts; + if (kind === "pending") receipt.state = "pending"; + if (kind === "wrong-operation") receipt.operationId = "POST /api/issues/{id}/comments"; + if (kind === "wrong-result-operation") receipt.result.operationId = "POST /api/issues/{id}/comments"; + if (kind === "unknown-outcome") receipt.result.ok = false; + if (kind === "wrong-status") receipt.result.status = 200; + if (kind === "missing-digest") delete receipt.digest; + if (kind === "wrong-key") run.resultJson.apiToolReceipts = { "model-authored-claim": receipt }; + if (kind === "foreign-card") original.id = "other"; + if (kind === "foreign-company") original.companyId = "other"; + if (kind === "foreign-issue") original.issueId = "other"; + if (kind === "foreign-agent") original.createdByAgentId = "other"; + if (kind === "foreign-source") original.sourceRunId = "other"; + if (kind === "wrong-idempotency") original.idempotencyKey = "other"; + if (kind === "changed-form") original.payload.questionSet.questions[0].options[1].label = "Other"; + if (kind === "changed-compatibility-form") original.payload.questions[0].prompt = "Other"; + if (kind === "board-created") original.createdByUserId = "other"; + if (kind === "resolved-creation") original.status = "answered"; + if (kind === "wrong-policy") card.continuationPolicy = "none"; + if (kind === "runtime-request") card.payload.runtimeRequestId = "fabricated-provider-request"; + if (kind === "legacy-run") run.runtimeMode = "legacy"; + if (kind === "active-source") run.status = "running"; + if (kind === "foreign-run") run.agentId = "other"; + if (kind === "duplicate-source") point.runs.push(structuredClone(run)); + if (kind === "spoofed-store") { run.resultJson.modelClaim = receipts; delete run.resultJson.apiToolReceipts; } + expect(questionPath(point, card)).toBeNull(); + }); + it.each(["lost-receipt", "changed-receipt", "interactionId", "sourceRunId", "issueId", "interactionStatus", "duplicate-continuation"])("retains one immutable API receipt and one bound answer wake: %s", kind => { + const r = recording(["api", "api"]), source = r[1].runs[0], next = r[1].runs[1]; + if (kind === "lost-receipt") delete source.resultJson.apiToolReceipts; + else if (kind === "changed-receipt") source.resultJson.apiToolReceipts["a".repeat(64)].digest = "c".repeat(64); + else if (kind === "duplicate-continuation") r[1].runs.push({ ...structuredClone(next), id: "extra" }); + else next.contextSnapshot[kind] = "other"; + expect(failed(r)).toContain("question-resume.choice.path"); + }); + it.each(["lost", "changed"])("rejects the first API receipt %s after the second answer", kind => { + const r = recording(["api", "api"]), source = r[2].runs[0]; + if (kind === "lost") delete source.resultJson.apiToolReceipts; + else source.resultJson.apiToolReceipts["a".repeat(64)].digest = "c".repeat(64); + expect(failed(r)).toContain("question-resume.choice.path"); + }); it("keeps the new journey explicit-only and the original documentation cells intact", () => { const cells = runnerMatrix.filter(c => c.suite.id === "question-resume"); expect(cells.map(c => c.profile.id).sort()).toEqual(["runner-acpx-claude", "runner-codex"]); @@ -70,6 +126,7 @@ describe("native question/resume path qualification", () => { }); it.each([ ["provider", "provider"], ["provider", "semantic"], ["semantic", "provider"], ["semantic", "semantic"], + ["api", "api"], ["api", "semantic"], ["semantic", "api"], ["provider", "api"], ["api", "provider"], ] as QuestionPath[][])("accepts exactly bound %s then %s journeys", (a, b) => { const r = recording([a,b]); expect(failed(r)).toEqual([]);