From 3937215a058d18555e80a2bc87397ffeffe5d898 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 31 Oct 2026 13:16:40 -0500 Subject: [PATCH] Qualify explicit native written references in the basic question journey (#18459) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Basic task journeys must prove that people can answer an agent and receive saved work. > - The question journey asks for a choice, then a typed reference, then a saved note. > - A native choice card can offer a real written-answer control as well as predefined options. > - The old observer rejected that form before a person could type the reference. > - This pull request accepts that explicit native control and keeps exact answer and work evidence. > - The separate text-only and named-tool qualifications keep their stricter contracts. ## Linked Issues or Issue Description Refs #18346 and #18456. The integration inherits the reviewed fixture prerequisite #18387 and Agent Chat recovery #18465 through their actual merged parent. The producer fix in #18456 is separate from this basic journey contract correction. Public duplicate PR and issue searches found no other correction for this form equivalence. **What happened?** The basic question journey requires the reference card to have `answerMode: text`. It rejects a native single-select card even when its canonical form explicitly enables a written answer. This makes the observer test a widget shape in addition to the requested behavior. **Expected behavior** The person must make a real choice and type the complete reference. A native card with an explicit written-answer control can satisfy the second step. Selecting a predefined option cannot satisfy it. The agent must resume the bound work and save both actual answers in its own task document. The document must survive reload. **Steps to reproduce** Use the focused recording and real QuestionForm browser test in this PR. Keep the reference as a single-select native question with one predefined option and `customAnswer.enabled: true`. The previous grader rejects `question-resume.forms`. The previous browser helper rejects `single_select` before it uses the written-answer control. **Paperclip version / commit** Original source base `f2db909393de124caf6a5380a52d4e0e1d621661`. Current integration `a417fd827347ed6f7998774a9de6aee70b76d867` merges exactly `12b0cc630471ee151ec5bc8e41a34f09236d6723` into reviewed question source `faec4ce6d775a34d303d200987e4e84d6dc26caf`. The scheduled failure on `82d15946c5844d3ae41c6ade62fa224053f20a9a` remains a failure. Its saved card had no canonical written-answer capability and would still fail this change. The retained normalized events do not prove whether the original provider flag was omitted or false. **Deployment mode** Attached native task journeys, including QA Cloud. This PR changes the observer and grader. It does not change the app, provider prompt, model, budget, timeouts, cleanup, or runtime deployment. ## What Changed - Accept one provider-bound single-select reference card with explicit canonical custom answers and valid, distinct options. Keep pure-text reference support. - Make the browser select the actual written-answer control and fill the real rich input. Opt in only for the basic question journey. - Keep the exact no-option typed response, immutable card, board actor, native request and continuation, saved agent document, reload, model, environment, checkpoints, and existing one-to-three-run checks. - Reject missing or disabled custom capability, inferred legacy capability, malformed or extra questions, foreign native identities, changed cards, substituted options, and wrong or missing reference text. - Add a real production QuestionForm and TaskChatRichInput Chromium probe. It first selects a predefined option, then proves the helper clears it and submits only the exact custom text. - Document the contract correction and preserve the historical failed attempt and its separate cleanup outcome. Keep necessary-question and named-tool documentation qualification unchanged. ## Verification - Before the observer change, the focused grader regression failed only the new valid-written-reference case; the other 108 cases passed. - Before the browser helper change, real Chromium passed the text control and four rejection controls but failed the valid native written-answer control at the old pure-text assertion. - Initial focused Runner E2E unit selection at `af07f4de`: **256 passed across 9 files**. This includes question/resumption, clarification evidence/flow/collection/deadline, necessary question, documentation, contracts, and report catalog tests. - Initial real Chromium selection at `af07f4de`: **12 passed**. Six new production QuestionForm/control checks and all six existing document-readiness checks pass. No form or rich-input component is mocked. - Historical test-only head `faec4ce6d775a34d303d200987e4e84d6dc26caf` adds a direct canonical-absent legacy `freeText` grader negative and renders the actual legacy presentation in the browser. All four production helper blobs remain identical to the reviewed head. A temporary legacy inference mutant makes the new negative fail. After restoring production source, **179 focused unit tests and all 6 real QuestionForm checks pass**. The six document-readiness specs and their helper are unchanged. - Original [CI run 41205748589](https://github.com/paperclipai/paperclip/actions/runs/41205748589) failed the staged-listen Runner handoff test; that failure is preserved and was not retried. Its Runner source closure was identical to an earlier passing canonical lane. The symptom also recurred on other heads and is tracked independently; no infrastructure attribution or live pass is claimed. - Current integration `a417fd827347ed6f7998774a9de6aee70b76d867` preserves **all 12 question files byte-for-byte** from faec. Every other tracked path is identical to exact merged parent `12b0cc630471ee151ec5bc8e41a34f09236d6723`, which contains actual prerequisite `baf651c8d0122efc93f3ef03e891919eeb666215`. There is no manual conflict resolution or additional production edit. - On that integrated head: **257 focused Vitest tests across 9 files, 206 required Node source/reporter tests, and 12 real Chromium tests pass**. Map/module-boundary checks and catalog discovery pass. The actual `--preview` command passes with a synthetic question target and an intentionally unavailable adapter; it performs no credential read, adapter loading, provisioning or model call. It retains maxRuns3, a 600000ms campaign bound and zero retries. - Exact-head [CI run 41210806950](https://github.com/paperclipai/paperclip/actions/runs/41210806950) passes, including official typecheck, build, all eight browser shards, both Runner Vitest lanes and aggregate verify. All 59 current PR contexts are terminal with no failures. Fresh [Greptile summary](https://github.com/paperclipai/paperclip/pull/18459#issuecomment-6543878408) explicitly reviews `a417fd827347ed6f7998774a9de6aee70b76d867` at 5/5 with no findings; its exact-head check succeeds and all threads are resolved. No fresh deployed journey pass is claimed yet. - Catalog discovery lists the existing question journey. Map checks and module-boundary checks pass. Diff checks pass. - `tsc -p tests/runner-e2e/tsconfig.json` was attempted. The fresh integrated checkout still lacks compiled Runner, plugin SDK, and adapter declarations. The attempted local check reports 344 diagnostic lines from dependency/type closure and zero diagnostics in all 12 question-owned paths. Official exact-head CI establishes the full typecheck/build gates. Full repository build, typecheck, and test suites were not repeated locally for this observer change. - The first broad local report selection lacked the CLI loader and then hit sandbox IPC restrictions. Locked dependency installation and an authorized local IPC run resolved those setup errors; the full selected 256 tests pass. The initial localhost probe also needed the normal local-listen permission. - No fresh QA pass is claimed. The coordinator must qualify the final reviewed runtime and harness together. Prior failed results remain preserved. ## Risks The basic journey accepts one additional form with explicit native provenance. It still rejects the original closed card without a written-answer capability. The separate producer default fix must reach the selected remote provider pack before it can affect a deployed run. Local browser and grader tests do not prove that artifact served or that a model completed the live journey. Existing cleanup and physical-stop obligations remain separate. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, source inspection, browser testing, and code execution. The host does not expose the exact underlying model version or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used with available version and capability details - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked related work above - [x] I have described the issue in-PR following the bug template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket or instance-derived details - [x] I have run the selected tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --- doc/evals.md | 4 +- tests/runner-e2e/PRODUCT-JOURNEYS.md | 7 +- tests/runner-e2e/README.md | 14 +++- tests/runner-e2e/continuation-flow.ts | 9 +- tests/runner-e2e/playwright-support.config.ts | 2 +- .../clarification-evidence.test.ts | 15 +++- .../clarification-evidence.ts | 5 +- .../product-journeys/clarification-flow.ts | 2 +- tests/runner-e2e/question-resume-scoring.ts | 13 ++- tests/runner-e2e/question-resume-ui.spec.ts | 83 +++++++++++++++++++ tests/runner-e2e/question-resume-ui.ts | 12 ++- tests/runner-e2e/question-resume.test.ts | 47 +++++++++++ 12 files changed, 194 insertions(+), 19 deletions(-) create mode 100644 tests/runner-e2e/question-resume-ui.spec.ts diff --git a/doc/evals.md b/doc/evals.md index e396d676907..517eb0b4ecd 100644 --- a/doc/evals.md +++ b/doc/evals.md @@ -71,7 +71,9 @@ tests do not establish a completed live pass or unattended CI qualification. The deployment-targeted `question-answer-resume` journey reuses the classic two-question scenario and native continuation oracle on an attached target. It checks the actual board answers, one to three bound work runs, an agent-authored -task document and browser readback after reload. It accepts provider questions and +task document and browser readback after reload. A typed reference may use a text +field or the native card's explicit written-answer control; a predefined option +cannot replace the exact typed answer. It accepts provider questions and durable questions proven by an applied semantic receipt or an immutable completed native API mutation receipt. It does not qualify named-tool selection. Its provider owns target setup, usage and cleanup. Registration and local calibration do not establish a fresh diff --git a/tests/runner-e2e/PRODUCT-JOURNEYS.md b/tests/runner-e2e/PRODUCT-JOURNEYS.md index 34cdd31a158..f0ace922eeb 100644 --- a/tests/runner-e2e/PRODUCT-JOURNEYS.md +++ b/tests/runner-e2e/PRODUCT-JOURNEYS.md @@ -40,7 +40,12 @@ never start a server or depend on a tenant filesystem. The question-resumption journey uses the same natural-language scenario, browser answer helper and behavioral grader as the classic `question-resume` suite. Native -provider questions can stay in one run; durable native questions create one exactly bound +reference cards may expose a text field or an explicit canonical written-answer +control on one single-select question. The browser must type the complete reference +and submit no predefined option. An absent or disabled custom-answer capability is +not an equivalent form. The user still answers exactly two distinct native questions; +the document must retain the actual choice and typed reference after reload. +Provider questions can stay in one run; durable native questions create one exactly bound continuation each. Creation must have an applied semantic receipt or a completed native API mutation receipt with the original card, exact scope, form and idempotency key. An API receipt must retain its creation identity unchanged after the answer. A form or diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index fbfbe27169c..33231603802 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -1771,7 +1771,13 @@ remain unchanged after the answer. Both durable paths require one new response wake bound to that interaction and source run. Missing, pending, foreign or mismatched receipts fail; form shape and run counts cannot replace them. Only a verified provider choice may carry the adapter's optional Other field. A second substantive question -is still rejected; the subsequent reference question must be text-only. Both real +is still rejected. The subsequent reference must use a text field or one native +provider single-select question with an explicit canonical `customAnswer.enabled=true`. +The browser selects its written-answer control and submits the exact typed reference +with no predefined option. Missing or disabled custom answers, inferred legacy +capabilities, extra questions and substituted choices fail. This behavioral form +equivalence does not change the text-only necessary-question or named-tool +documentation qualifications. Both real board answers must remain bound to unchanged forms, and one saved document must include both answers. Existing lifecycle, ownership, premature-output, terminal state and no-extra-task checks apply. Three checkpoints are required even when @@ -1785,6 +1791,12 @@ Other field stopped the driver before any answer remains a FAIL. A new behavior pass does not establish semantic-tool selection, documentation placement, default hiring behavior or general reliability. +The original scheduled attempt that stopped at a closed reference card remains a +FAIL, including its separate cleanup failure. Supporting a server-bound written +answer does not regrade that card, infer a missing provider flag, or establish a +fresh Cloud pass. The general OpenCode omitted-default normalization fix requires +its own reviewed runtime artifact; browser/grader calibration cannot prove it served. + Continuation screenshots wait for the correct task heading and fully revealed conversation before capture. A loading screen or wrong task fails capture. Browser-only regressions exercise delayed rendering without provider calls: diff --git a/tests/runner-e2e/continuation-flow.ts b/tests/runner-e2e/continuation-flow.ts index 89072dc005e..3da0668ebae 100644 --- a/tests/runner-e2e/continuation-flow.ts +++ b/tests/runner-e2e/continuation-flow.ts @@ -1,6 +1,6 @@ import { gradeNecessaryQuestion } from "./necessary-question.js"; import { necessaryQuestionContract } from "./continuation-cases.js"; -import { validResumeQuestionForm } from "./question-resume-scoring.js"; +import { validResumeQuestionForm, validResumeReferenceForm } from "./question-resume-scoring.js"; import { answerResumeQuestionThroughUi } from "./question-resume-ui.js"; import { gradeLifecycleBaseline, type LifecycleCheckpoint } from "./lifecycle-baseline.js"; import { lifecycleLiveCase, lifecycleLiveContinuation, gradeLifecycleNarrative } from "./lifecycle-live-cases.js"; @@ -220,12 +220,15 @@ export async function runContinuationFlow(input: { if (scenario.id === "necessary-free-text") { expect(validResumeQuestionForm(checkpoints[0]!, questions[0], false), "a bound text-only question").toBe(true); } else if (scenario.id === "question-answer-resume") { - expect(validResumeQuestionForm(checkpoints.at(-1)!, questions[0], !!choice), "a verified native question path and the requested form").toBe(true); + expect(choice ? validResumeQuestionForm(checkpoints.at(-1)!, questions[0], true) + : validResumeReferenceForm(checkpoints.at(-1)!, questions[0]), "a verified native question path and the requested form").toBe(true); } else if (scenario.id === "provider-question-bridge") { expect(isSingleClaudeQuestion(set.questions), "one choice question with only the optional provider Other field").toBe(true); } else expect(set.questions, "ask only the requested next question").toHaveLength(1); const before = new Set(runs.map((r) => r.id)); - await answerResumeQuestionThroughUi(page, questions[0], choice ? { choice } : { text: scenario.answer }); + await answerResumeQuestionThroughUi(page, questions[0], choice ? { choice } : { + text: scenario.answer, allowProviderCustomAnswer: scenario.id === "question-answer-resume", + }); await settle(before, false, questions[0].id); } async function reply(body: string) { diff --git a/tests/runner-e2e/playwright-support.config.ts b/tests/runner-e2e/playwright-support.config.ts index 000c7d502c6..72cc2bfac65 100644 --- a/tests/runner-e2e/playwright-support.config.ts +++ b/tests/runner-e2e/playwright-support.config.ts @@ -3,7 +3,7 @@ import { defineConfig } from "@playwright/test"; /** Browser-only harness regressions: no Paperclip instance or provider credentials. */ export default defineConfig({ testDir: ".", - testMatch: ["clarification-readback.spec.ts", "project-context-readback.spec.ts", "skill-studio-flow.spec.ts", "native-task-reply.spec.ts", "screenshot-readiness.spec.ts", "service-worker-reload.spec.ts", "lost-send.spec.ts", "chat-restart.spec.ts", "settings-toggle.spec.ts", "browser-bootstrap-diagnostics.spec.ts", "service-worker-module-loading.spec.ts", "browser-bootstrap-recovery.spec.ts", "blocker-input.spec.ts", "native-blocker-visible.spec.ts", "copilot-permission-card.spec.ts", "kimi-permission-card.spec.ts", "github-review-check.spec.ts", "antigravity-journey-browser.spec.ts", "task-search-filter.spec.ts"], + testMatch: ["question-resume-ui.spec.ts", "clarification-readback.spec.ts", "project-context-readback.spec.ts", "skill-studio-flow.spec.ts", "native-task-reply.spec.ts", "screenshot-readiness.spec.ts", "service-worker-reload.spec.ts", "lost-send.spec.ts", "chat-restart.spec.ts", "settings-toggle.spec.ts", "browser-bootstrap-diagnostics.spec.ts", "service-worker-module-loading.spec.ts", "browser-bootstrap-recovery.spec.ts", "blocker-input.spec.ts", "native-blocker-visible.spec.ts", "copilot-permission-card.spec.ts", "kimi-permission-card.spec.ts", "github-review-check.spec.ts", "antigravity-journey-browser.spec.ts", "task-search-filter.spec.ts"], workers: 1, retries: 0, timeout: 10_000, diff --git a/tests/runner-e2e/product-journeys/clarification-evidence.test.ts b/tests/runner-e2e/product-journeys/clarification-evidence.test.ts index 88c0ccb47cf..2a0287cc8d3 100644 --- a/tests/runner-e2e/product-journeys/clarification-evidence.test.ts +++ b/tests/runner-e2e/product-journeys/clarification-evidence.test.ts @@ -28,7 +28,7 @@ import type { JourneyContext } from "./types.js"; type Row = Record; const context = { company: { id: "company" }, user: { id: "board" }, environmentId: "daytona", agent: { id: "agent", provider: "opencode", model: "model", runtimeMode: "native" } } as JourneyContext; -function recording(paths: Array<"provider" | "semantic" | "api">) { +function recording(paths: Array<"provider" | "semantic" | "api">, writtenReference = false) { const runs: Row[] = [], cards: Row[] = [], checkpoints: any[] = []; const createRun = (previous?: Row) => ({ id: `run-${runs.length}`, status: "running", runtimeMode: "native", companyId: "company", issueId: "task", agentId: "agent", contextSnapshot: { paperclipEnvironment: { id: "daytona" }, ...(previous ? { issueId: "task", sourceRunId: previous.sourceRunId, @@ -41,7 +41,9 @@ function recording(paths: Array<"provider" | "semantic" | "api">) { runs.push(createRun()); for (const [index, path] of paths.entries()) { const run = runs.at(-1)!; - const field = index === 0 ? { id: "choice", answerMode: "single_select", options: [{ id: "am", label: "Morning" }, { id: "pm", label: "Afternoon" }] } : { id: "reference", answerMode: "text" }; + const field = index === 0 ? { id: "choice", answerMode: "single_select", options: [{ id: "am", label: "Morning" }, { id: "pm", label: "Afternoon" }] } + : writtenReference ? { id: "reference", answerMode: "single_select", options: [{ id: "none", label: "No reference" }], customAnswer: { enabled: true } } + : { id: "reference", answerMode: "text" }; const request = `request-${index}`, semantic = path !== "provider"; const card: Row = { id: `question-${index}`, kind: "ask_user_questions", status: "pending", companyId: "company", issueId: "task", createdByAgentId: "agent", sourceRunId: run.id, idempotencyKey: semantic ? `semantic-${index}` : `paperclip-runner-question:${run.id}:${request}`, continuationPolicy: semantic ? "wake_assignee" : "none", createdByUserId: null, @@ -65,6 +67,11 @@ function recording(paths: Array<"provider" | "semantic" | "api">) { const failures = (f: ReturnType) => gradeClarificationEvidence(context, "task", "AMBERnonce", f.checkpoints, f.readback).filter(c => !c.passed); describe("attached clarification evidence", () => { + it("accepts an explicit provider written reference with both exact board answers and saved native work", () => { + const f = recording(["provider", "provider"], true), point = f.checkpoints[1]; + expect(clarificationQuestionFailure(point, point.interactions[1], false)).toBeUndefined(); + expect(failures(f)).toEqual([]); + }); it.each([true, false])("rejects an incomplete API receipt instead of inferring native question evidence (choice=%s)", choice => { const f = recording(["semantic", "semantic"]), point = f.checkpoints[choice ? 0 : 1]; const question = point.interactions.find((i: Row) => i.status === "pending"), source = point.runs.at(-1); @@ -87,8 +94,8 @@ describe("attached clarification evidence", () => { expect(failures(recording([...paths]))).toEqual([]); }); it.each(["foreign-user", "foreign-agent-document", "foreign-task-document", "lost-revision", "wrong-visible-answer", "changed-readback-body", "premature-document", "extra-child", "foreign-model", "foreign-environment", "missing-native-input", "duplicate-question", "wrong-answer"])("rejects a plausible false pass: %s", kind => { - for (const path of ["semantic", "api"] as const) { - const f = recording([path, path]), first = f.checkpoints[0], final = f.checkpoints[2]; + for (const path of ["semantic", "api", "provider"] as const) { + const f = recording([path, path], path === "provider"), first = f.checkpoints[0], final = f.checkpoints[2]; if (kind === "foreign-user") final.interactions[1].resolvedByUserId = "other-board"; if (kind === "foreign-agent-document") final.documents[0].updatedByAgentId = "other-agent"; if (kind === "foreign-task-document") final.documents[0].issueId = "other-task"; diff --git a/tests/runner-e2e/product-journeys/clarification-evidence.ts b/tests/runner-e2e/product-journeys/clarification-evidence.ts index 69e39b7fa62..4d3ab02e70a 100644 --- a/tests/runner-e2e/product-journeys/clarification-evidence.ts +++ b/tests/runner-e2e/product-journeys/clarification-evidence.ts @@ -1,4 +1,4 @@ -import { gradeQuestionResume, questionPath, validResumeQuestionForm } from "../question-resume-scoring.js"; +import { gradeQuestionResume, questionPath, validResumeQuestionForm, validResumeReferenceForm } from "../question-resume-scoring.js"; import type { ContinuationCheckpoint } from "../continuation-scoring.js"; import { validateAgentTaskState } from "./agent-flows.js"; import type { JourneyContext } from "./types.js"; @@ -19,7 +19,8 @@ export function clarificationQuestionFailure(point: ContinuationCheckpoint, ques }); return apiCreated ? `clarification_${stage}_api_fallback` : `clarification_${stage}_native_path_invalid`; } - return validResumeQuestionForm(point, question, choice) ? undefined : `clarification_${stage}_form_invalid`; + return (choice ? validResumeQuestionForm(point, question, true) : validResumeReferenceForm(point, question)) + ? undefined : `clarification_${stage}_form_invalid`; } export function gradeClarificationEvidence(context: JourneyContext, issueId: string, marker: string, diff --git a/tests/runner-e2e/product-journeys/clarification-flow.ts b/tests/runner-e2e/product-journeys/clarification-flow.ts index 800a747284d..8edb160c5e0 100644 --- a/tests/runner-e2e/product-journeys/clarification-flow.ts +++ b/tests/runner-e2e/product-journeys/clarification-flow.ts @@ -91,7 +91,7 @@ export async function runClarificationJourney(context: JourneyContext, session: await session.own({ kind: "issue", name: title, id: created.issueId, journeyId: "question-answer-resume" }); issue = await context.api.get(`/api/issues/${created.issueId}`); let state = await settle(); - for (const [index, input] of [{ choice: "Afternoon" }, { text: scenario.answer }].entries()) { + for (const [index, input] of [{ choice: "Afternoon" }, { text: scenario.answer, allowProviderCustomAnswer: true }].entries()) { const point = await snapshot(index === 0 ? "initial" : "answered", state); const question = state.interactions.find(i => i.kind === "ask_user_questions" && i.status === "pending")!; const questionFailure = clarificationQuestionFailure(point, question, index === 0); diff --git a/tests/runner-e2e/question-resume-scoring.ts b/tests/runner-e2e/question-resume-scoring.ts index 85f8902d95e..30f1e46c350 100644 --- a/tests/runner-e2e/question-resume-scoring.ts +++ b/tests/runner-e2e/question-resume-scoring.ts @@ -67,6 +67,15 @@ export function validResumeQuestionForm(checkpoint: ContinuationCheckpoint, i: R (fields.length === 1 || (path === "provider" && isSingleClaudeQuestion(fields))); } +export function validResumeReferenceForm(checkpoint: ContinuationCheckpoint, i: Row): boolean { + if (validResumeQuestionForm(checkpoint, i, false)) return true; + const fields = resumeQuestionFields(i), field = fields[0], options: Row[] = Array.isArray(field?.options) ? field.options : []; + return questionPath(checkpoint, i) === "provider" && fields.length === 1 && nonempty(field.id) + && field.answerMode === "single_select" && field.customAnswer?.enabled === true + && options.length > 0 && options.every(option => nonempty(option.id) && nonempty(option.label)) + && new Set(options.map(option => option.id)).size === options.length; +} + export function sameQuestion(a: Row, b?: Row) { return !!b && ["id", "kind", "sourceRunId", "issueId", "companyId", "createdByAgentId", "idempotencyKey", "continuationPolicy"] .every(k => a[k] === b[k]) && isDeepStrictEqual(a.payload, b.payload); @@ -110,8 +119,8 @@ export function gradeQuestionResume(checkpoints: ContinuationCheckpoint[], marke options.every(o => nonempty(o.id)) && new Set(options.map(o => o.id)).size === options.length && options.some(o => /^morning\b/i.test(String(o.label).trim())) && !!afternoon && questions(middle).length === 2 && questions(middle).filter(i => i.status === "pending").length === 1 && - nonempty(second?.id) && second?.id !== first?.id && validResumeQuestionForm(middle, second, false), - detail: "One choice plus only a verified provider Other companion, then a distinct text-only question.", + nonempty(second?.id) && second?.id !== first?.id && validResumeReferenceForm(middle, second), + detail: "One choice plus only a verified provider Other companion, then a distinct typed reference with an explicit provider written-answer control or a text field.", }]; for (const [before, after, i, field, answer] of [ [initial, middle, first, choice, "choice"], [middle, final, second, text, "text"], diff --git a/tests/runner-e2e/question-resume-ui.spec.ts b/tests/runner-e2e/question-resume-ui.spec.ts new file mode 100644 index 00000000000..e1510b3aee7 --- /dev/null +++ b/tests/runner-e2e/question-resume-ui.spec.ts @@ -0,0 +1,83 @@ +import path from "node:path"; +import type { IncomingMessage, ServerResponse } from "node:http"; +import { expect, test } from "@playwright/test"; +import { createServer, type ViteDevServer } from "../../ui/node_modules/vite/dist/node/index.js"; +import { answerResumeQuestionThroughUi } from "./question-resume-ui.js"; +import { chatQuestionPresentation } from "./chat-flow.js"; + +let server: ViteDevServer; +let origin: string; +const uiRoot = path.resolve(import.meta.dirname, "../../ui"); +const probe = path.join(uiRoot, "src/__question_resume_probe.tsx"); + +test.beforeAll(async () => { + server = await createServer({ + root: uiRoot, configFile: false, + resolve: { alias: { "@": path.join(uiRoot, "src") } }, + oxc: { jsx: { runtime: "automatic" } }, + server: { host: "127.0.0.1", port: 0 }, + plugins: [{ + name: "question-resume-controls", + resolveId(id: string) { if (id === "/src/__question_resume_probe.tsx") return probe; }, + load(id: string) { + if (id !== probe) return; + return `import React from "react"; + import { createRoot } from "react-dom/client"; + import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; + import { QuestionForm } from "/src/components/task-chat/QuestionForm.tsx"; + const root = createRoot(document.getElementById("root")); + const client = new QueryClient(); + window.showQuestion = questionSet => root.render( + { window.submitted = response; }} /> + );`; + }, + configureServer(vite: ViteDevServer) { + vite.middlewares.use((req: IncomingMessage, res: ServerResponse, next: () => void) => { + if (req.url !== "/") return next(); + res.setHeader("Content-Type", "text/html"); + res.end('
'); + }); + }, + }], + }); + await server.listen(); + const address = server.httpServer!.address(); + if (!address || typeof address === "string") throw new Error("Missing test server address"); + origin = `http://127.0.0.1:${address.port}`; +}); +test.afterAll(async () => { await server?.close(); }); + +for (const mode of ["text", "single_select"] as const) { + test(`submits an exact typed reference through the real ${mode} QuestionForm control`, async ({ page }) => { + const field = { id: "reference", prompt: "What reference should I include?", required: true, answerMode: mode, + ...(mode === "single_select" ? { options: [{ id: "none", label: "No reference" }], customAnswer: { enabled: true, label: "Write reference" } } : {}) }; + const card = { payload: { questionSet: { schema: "paperclip.question_set.v1", questions: [field] } } }; + await page.goto(origin); + await page.waitForFunction(() => typeof (window as any).showQuestion === "function"); + await page.evaluate(set => (window as any).showQuestion(set), card.payload.questionSet); + if (mode === "single_select") await page.getByRole("radio", { name: "No reference", exact: true }).click(); + await answerResumeQuestionThroughUi(page, card, { text: "Use AMBERwritten.", allowProviderCustomAnswer: true }); + expect(await page.evaluate(() => (window as any).submitted)).toEqual({ + schema: "paperclip.question_response.v1", answers: { reference: mode === "text" + ? { text: "Use AMBERwritten." } : { selectedOptionIds: [], customText: "Use AMBERwritten." } }, + }); + if (mode === "single_select") await expect(page.getByRole("radio", { name: "No reference", exact: true })).toHaveAttribute("aria-checked", "false"); + }); +} + +for (const invalid of ["missing", "disabled", "legacy", "strict"] as const) { + test(`does not infer a written-answer capability from ${invalid} metadata`, async ({ page }) => { + const field = { id: "reference", prompt: "Reference?", answerMode: "single_select", options: [{ id: "none", label: "No reference" }], + ...(invalid === "missing" ? {} : { customAnswer: { enabled: invalid !== "disabled" } }) }; + const card = { payload: invalid === "legacy" ? { questions: [{ ...field, options: [{ id: "none", label: "No reference", freeText: true }] }] } + : { questionSet: { schema: "paperclip.question_set.v1", questions: [field] } } }; + await page.goto(origin); + await page.waitForFunction(() => typeof (window as any).showQuestion === "function"); + const set = invalid === "legacy" ? chatQuestionPresentation(card.payload as any) : card.payload.questionSet; + await page.evaluate(set => (window as any).showQuestion(set), set); + await expect(page.getByRole("radio", { name: "No reference", exact: true })).toBeVisible(); + await expect(page.getByRole("radio", { name: "Other", exact: true })).toHaveCount(invalid === "strict" ? 1 : 0); + await expect(answerResumeQuestionThroughUi(page, card, { text: "Use AMBERwritten.", ...(invalid === "strict" ? {} : { allowProviderCustomAnswer: true }) })).rejects.toThrow(); + expect(await page.evaluate(() => (window as any).submitted)).toBeUndefined(); + }); +} diff --git a/tests/runner-e2e/question-resume-ui.ts b/tests/runner-e2e/question-resume-ui.ts index ac6fa3f9de8..9e325337e93 100644 --- a/tests/runner-e2e/question-resume-ui.ts +++ b/tests/runner-e2e/question-resume-ui.ts @@ -3,7 +3,7 @@ import { chatQuestionPresentation } from "./chat-flow.js"; type Row = Record; -export async function answerResumeQuestionThroughUi(page: Page, interaction: Row, input: { choice: string } | { text: string }) { +export async function answerResumeQuestionThroughUi(page: Page, interaction: Row, input: { choice: string } | { text: string; allowProviderCustomAnswer?: boolean }) { const set = chatQuestionPresentation(interaction.payload); if ("choice" in input) { expect(set.questions[0]?.answerMode, "choices must use radio controls").toBe("single_select"); @@ -12,8 +12,14 @@ export async function answerResumeQuestionThroughUi(page: Page, interaction: Row const choice = input.choice.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); await page.getByRole("radio", { name: new RegExp(`^${choice}\\b`, "i") }).last().click(); } else { - expect(set.questions[0]?.answerMode, "open answers must render a text field, not a choice question").toBe("text"); - await page.getByTestId("question-text-answer-composer").last() + const field = set.questions[0]; + const custom = field?.answerMode !== "text"; + if (custom) { + expect(input.allowProviderCustomAnswer && interaction.payload.questionSet && field?.answerMode === "single_select" + && field.customAnswer?.enabled === true, "the canonical provider form must explicitly permit a written reference").toBeTruthy(); + await page.getByRole("radio", { name: field!.customAnswer?.label ?? "Other", exact: true }).last().click(); + } + await page.getByTestId(custom ? "question-other-answer-composer" : "question-text-answer-composer").last() .locator('[contenteditable="true"],textarea').first().fill(input.text); } for (let index = 1; index < set.questions.length; index += 1) { diff --git a/tests/runner-e2e/question-resume.test.ts b/tests/runner-e2e/question-resume.test.ts index bded4d1c8d2..c312d4af580 100644 --- a/tests/runner-e2e/question-resume.test.ts +++ b/tests/runner-e2e/question-resume.test.ts @@ -60,8 +60,55 @@ function recording(paths: QuestionPath[] = ["provider", "semantic"]) { } const failed = (r: any[]) => gradeQuestionResume(r, "AMBERnonce").filter(c => !c.passed).map(c => c.id); const question = (r: any[], n = 0) => r[n].interactions[n]; +function writtenReference(r: any[]) { + for (const point of r.slice(1)) Object.assign(point.interactions[1].payload.questionSet.questions[0], { + answerMode: "single_select", options: [{ id: "none", label: "No reference" }], customAnswer: { enabled: true }, + }); + return r; +} describe("native question/resume path qualification", () => { + it("accepts an explicit provider written answer without selecting a predefined option", () => { + const r = writtenReference(recording(["provider", "provider"])); + expect(failed(r)).toEqual([]); + expect(validResumeQuestionForm(r[1], question(r, 1), false)).toBe(false); + }); + it("rejects an inferred legacy freeText reference without a canonical question set", () => { + const r = writtenReference(recording(["provider", "provider"])); + for (const point of r.slice(1)) { + const payload = point.interactions[1].payload; + payload.questions = [{ id: "reference", prompt: "What reference should I include?", selectionMode: "single", + options: [{ id: "write", label: "Write reference", freeText: true }] }]; + delete payload.questionSet; + } + expect(failed(r)).toContain("question-resume.forms"); + }); + it.each(["missing-custom", "disabled-custom", "missing-options", "duplicate-options", "missing-option-id", "multi-select", "extra-question", + "wrong-request", "foreign-source", "foreign-agent", "foreign-company", "semantic", "api"])("rejects an unverified written-reference form: %s", kind => { + const r = writtenReference(recording(["provider", kind === "semantic" || kind === "api" ? kind : "provider"])); + const card = question(r, 1), field = card.payload.questionSet.questions[0]; + if (kind === "missing-custom") delete field.customAnswer; + if (kind === "disabled-custom") field.customAnswer.enabled = false; + if (kind === "missing-options") field.options = []; + if (kind === "duplicate-options") field.options.push(structuredClone(field.options[0])); + if (kind === "missing-option-id") delete field.options[0].id; + if (kind === "multi-select") field.answerMode = "multi_select"; + if (kind === "extra-question") card.payload.questionSet.questions.push({ id: "extra", answerMode: "text" }); + if (kind === "wrong-request") card.payload.runtimeRequestId = "other"; + if (kind === "foreign-source") card.sourceRunId = "other"; + if (kind === "foreign-agent") card.createdByAgentId = "other"; + if (kind === "foreign-company") card.companyId = "other"; + expect(failed(r)).toContain("question-resume.forms"); + }); + it.each(["predefined-option", "wrong-text", "missing-text", "foreign-user", "changed-form"])("rejects a corrupt written-reference answer: %s", kind => { + const r = writtenReference(recording(["provider", "provider"])), card = r[2].interactions[1], answer = card.result.answers[0]; + if (kind === "predefined-option") answer.optionIds = ["none"]; + if (kind === "wrong-text") answer.otherText = "Use OLD."; + if (kind === "missing-text") delete answer.otherText; + if (kind === "foreign-user") { card.resolvedByUserId = null; card.resolvedByAgentId = "agent"; } + if (kind === "changed-form") card.payload.questionSet.questions[0].customAnswer.label = "Changed"; + expect(failed(r)).toContain("question-resume.text.answer"); + }); it("accepts a native API creation receipt with the retained server response shape", () => { const point = JSON.parse(readFileSync(new URL("./fixtures/native-api-question.json", import.meta.url), "utf8")); expect(questionPath(point, point.interactions[0])).toBe("api");