From d9354bbfc35ef74333e90e596ce21dc8e59149b9 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 30 Oct 2026 16:18:21 -0500 Subject: [PATCH] Add a portable agent question and answer resumption journey (#18346) ## Thinking Path > - Paperclip helps people manage agent work and keep human decisions attached to tasks. > - Product E2E journeys verify the browser, native agent, saved state, and cleanup. > - The classic suite already checks a choice question followed by a text question. > - Deployment-targeted journeys need the same behavior on an attached instance. > - Production questions can use provider input, the dedicated semantic tool, or the authenticated native API fallback. > - The basic journey must verify the original card and native continuation for each supported path. > - This pull request shares the scenario and browser actions, adds strict API receipt evidence, and verifies the saved document after reload. > - Separate tool-documentation qualification and the original failed measurements keep their existing scope and outcomes. ## Linked Issues or Issue Description **What existing behavior does this improve?** It adds portable coverage for answering agent questions and resuming the same task on an attached deployment. **Current behavior** The classic suite has the two-question scenario. The portable catalog does not expose it. Its shared native-path grader recognizes provider and dedicated semantic receipts, but rejects a valid completed native API creation receipt. **Proposed behavior** Expose `question-answer-resume` as a deployment-targeted journey. Answer one choice question and one text question in the browser. Require server-owned creation evidence, unchanged forms, exact board answers, bound continuation, an agent-authored task document, and readback after reload. **Reason and benefit** The same story can qualify local and hosted targets through their normal provider setup and cleanup. It catches lost answers, duplicate or foreign work, premature output, and stale documents. ## What Changed - Register the journey with a three-run maximum and positive model allowance. Accept one to three attributed work runs at both collection and retained-result boundaries. - Share the existing two-question scenario, browser answer actions, and behavioral grader. Keep separate tool-documentation grading unchanged. - Recognize a completed native API creation receipt only when its HTTP 201 result retains the original pending card, exact task/company/agent/source run, idempotency key, and full normalized form. Require the receipt to remain unchanged after the answer and through the final checkpoint. Missing, pending, foreign, or altered proof still fails. - Check signed-in board ownership, each native continuation, no premature or extra output, document authorship, and the saved revision after reload. - Wait for the authenticated product region before screenshots. Recheck origin and path after readiness. Bound readiness and capture by the campaign deadline. - Add closed stage-specific errors, catalog coverage, sanitized receipt calibration, and documentation. Include the provider-form and browser-question presentation helpers in the definition fingerprint. ## Verification - Fresh QA staging qualification passed at `7f7de8be286a436010a48af89aba1727b4451493`: all 11 required checks passed, cleanup passed, and all three native runs used the configured QA target and model. The [sanitized run report](https://pages.paperclip.ing/evals-qa-smoke-41121966520-1/) records the measured pass. This is one successful qualification, not a reliability claim. - Final integration head `ae0d9d289831c770a413658d3520fb1dc8ad60e3` contains the one main merge through `fd008979f5794fda3d5f54151d36acd31d0de140` and an exact cherry-pick of the four-file qualified search fix `4edd10804e2b42941aa2038e90c362c13a0b98dc` from #18370. Patch identity matches. It preserves the project and search registrations and atomic search readback. The question live qualification ran at the earlier SHA above; the final integration head has separate local and CI verification. - Eighteen question journey, helper, and test file blobs are identical between the qualified and integration SHAs. This includes clarification flow and evidence, question grader and browser helper, the sanitized receipt fixture, screenshot helper, provider-form helper, browser question presentation, continuation scenario/readiness/flow, shared agent validation, and user actions. The question run-count contract and dispatch branch are also unchanged. Reviewers can compare those paths directly between the two commits. - All 622 selected Product E2E, catalog, question-resumption, necessary-question, report, and readiness tests pass after the main merge. After the exact search delta, all 145 narrow question/search/project/contract tests and all nine real Chromium search regressions pass. Map checks, combined credential-free preview, source fingerprints, and the earlier workspace-link preflight pass. The local TypeScript attempt reports stale Runner declaration errors for `onAccountingEvent` in unchanged native-session code; fresh official CI provides the clean-build type gate. The three search source/test blobs match the qualified search merge, and the three project source blobs are unchanged. - An in-memory probe executes the actual launcher fingerprint expression. Each project, search, and question implementation input changes the digest. Separate red probes first reproduced the omitted provider-form and browser presentation helpers; both pass after their source inputs are added. - The sanitized retained native API response-shape regression fails before the grading correction and passes afterward. Negative controls reject missing, pending, foreign, changed, spoofed, or wrongly bound receipts, answers, continuation, model, environment, and document evidence. All nine provider/semantic/API path combinations pass calibration. - The real local Chromium screenshot probe fails before the readiness repair and passes afterward. Login redirects and sign-in pages at task URLs remain excluded with no screenshot files. - The two original attached failures remain retained: the first stopped at early screenshot capture; the second stopped before answering at the former API provenance gate. An earlier sandbox run also retained one report subprocess failure; the full selected tests then passed with local subprocess IPC access. - The prior integration CI had an unchanged canary mock `EADDRINUSE` failure. That history is retained; its documented retry is owned by the coordinator. No application API, schema, production UI, model, runtime policy, credential, or deployment configuration change is included. Final integration head ae0d9d289831c770a413658d3520fb1dc8ad60e3 passed all 58 CI checks, the external Snyk status, and fresh Greptile 5/5 with no unresolved findings. This clean CI includes the official TypeScript and build gates. Recurring selection follows merge and a separate normal TestOps job verification. ## Risks An API result must prove native creation; form shape and run count cannot establish it. The grader checks the server receipt, exact scope, unchanged card and receipt, real board answers, and bound native continuation. The shared answer helper still serves classic journeys. The screenshot wait affects attached journeys and remains limited to the authenticated product region and campaign deadline. One live QA pass establishes the measured journey outcome at the qualified SHA. It does not establish reliability across models or deployments. The integration SHA preserves its journey and helper blobs, while shared registry changes receive fresh CI and review. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, source inspection, and code execution. The host does not expose the exact underlying model version or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/evals.md | 11 +- tests/runner-e2e/PRODUCT-JOURNEYS.md | 14 ++ tests/runner-e2e/README.md | 15 +- tests/runner-e2e/catalog.test.ts | 2 +- tests/runner-e2e/continuation-flow.ts | 25 +-- .../fixtures/native-api-question.json | 152 ++++++++++++++++++ .../product-journeys/agent-flows.ts | 2 +- .../product-journeys/campaign.test.ts | 2 +- tests/runner-e2e/product-journeys/catalog.ts | 1 + .../clarification-campaign.test.ts | 59 +++++++ .../clarification-evidence.test.ts | 122 ++++++++++++++ .../clarification-evidence.ts | 57 +++++++ .../clarification-flow.test.ts | 35 ++++ .../product-journeys/clarification-flow.ts | 110 +++++++++++++ tests/runner-e2e/product-journeys/launch.ts | 21 ++- .../product-screenshot.test.ts | 69 ++++++-- .../product-journeys/product-screenshot.ts | 20 ++- .../product-journeys/success-contract.ts | 12 +- tests/runner-e2e/product-journeys/types.ts | 3 +- tests/runner-e2e/question-resume-scoring.ts | 39 ++++- tests/runner-e2e/question-resume-ui.ts | 23 +++ tests/runner-e2e/question-resume.test.ts | 63 +++++++- tests/runner-e2e/result-validation.ts | 4 +- 23 files changed, 789 insertions(+), 72 deletions(-) create mode 100644 tests/runner-e2e/fixtures/native-api-question.json create mode 100644 tests/runner-e2e/product-journeys/clarification-campaign.test.ts create mode 100644 tests/runner-e2e/product-journeys/clarification-evidence.test.ts create mode 100644 tests/runner-e2e/product-journeys/clarification-evidence.ts create mode 100644 tests/runner-e2e/product-journeys/clarification-flow.test.ts create mode 100644 tests/runner-e2e/product-journeys/clarification-flow.ts create mode 100644 tests/runner-e2e/question-resume-ui.ts diff --git a/doc/evals.md b/doc/evals.md index 2ce6878ae74..78c9c3ce467 100644 --- a/doc/evals.md +++ b/doc/evals.md @@ -61,6 +61,15 @@ self-hosted/Cloud targets and separately selected local/Daytona agents. Account and credential bindings stay in the private eval repo; registration and oracle tests do not establish a completed live pass or unattended CI qualification. +The deployment-targeted `question-answer-resume` journey reuses the classic +two-question scenario and native continuation oracle on an attached target. +It checks the actual board answers, one to three bound work runs, an agent-authored +task document and browser readback after reload. It accepts provider questions and +durable questions proven by an applied semantic receipt or an immutable completed +native API mutation receipt. It does not qualify named-tool selection. Its provider owns target setup, +usage and cleanup. Registration and local calibration do not establish a fresh +Cloud pass or inclusion in a recurring selection. + The explicit-only [necessary free-text question journey](../tests/runner-e2e/README.md#necessary-free-text-question) checks one essential input, the rendered prompt and context, the real answer, one bound continuation, and an exact saved confirmation. Its four development @@ -547,7 +556,7 @@ any qualification claim; catalog presence alone is not a pass. The explicit-only [native question/resume qualification](../tests/runner-e2e/README.md#task-continuation) separates a completed two-answer user journey from semantic-tool documentation -qualification. It verifies the exact provider-pause or semantic-response-wake +qualification. It verifies the exact provider-pause or native response-wake binding for each answer and preserves prior grades when the definition changes. ## Lifecycle behavior baseline diff --git a/tests/runner-e2e/PRODUCT-JOURNEYS.md b/tests/runner-e2e/PRODUCT-JOURNEYS.md index d92ef26d490..895005cd15a 100644 --- a/tests/runner-e2e/PRODUCT-JOURNEYS.md +++ b/tests/runner-e2e/PRODUCT-JOURNEYS.md @@ -26,6 +26,7 @@ never start a server or depend on a tenant filesystem. | `task-create` | Company and signed-in board user | Create a task through the shared production composer, assign it to yourself, reopen and reload it, verify the saved prompt and no agent work | | `project-task-context` | Company, signed-in board user, task-write capability; no model runs | Create a project and a self-assigned task with the project selected through the production dialogs. Reopen the project, open its task, reload both, and verify exact saved company, project, prompt, assignment, standard work mode and zero runs | | `ask-question` | Company, configured agent capability, positive model allowance and exactly one permitted work run | Reuse the existing cross-model Ask question prompt; require one successful run with matching company/task/agent/model/environment, one saved exact answer, and visible readback after reload | +| `question-answer-resume` | Native agent; up to three work runs | Reuse the classic two-question story: answer a choice and a free-text question through the browser, verify each native continuation, then read the exact saved answers in an agent-authored task document after reload | | `task-properties` | Company, signed-in board user, task-write capability, prepared run-owned backlog task | Open its properties, assign it to the signed-in user, reload, check both UI and saved assignment; verify no agent run | | `task-search-filter` | Company, signed-in board user, task-write capability; zero model runs | Search three owned target/control tasks through the task list, require exact matching identities, filter Backlog to exclude the completed control, reload and return from the exact saved task with search/filter context intact | | `plan-revise-accept` | Native agent; three work runs | Reuse the classic Plan story: canonical draft, browser revision request, new revision-bound confirmation, approval and exact saved final response | @@ -35,6 +36,19 @@ never start a server or depend on a tenant filesystem. | `stop-redirect` | Native agent; three work runs; qualified Docker oracle | Download a verified checkpoint, Stop an active revision, require native cancellation acknowledgement, submit a new direction once and verify completion without original-run activity | | `cloud-onboarding` | Signed-in admitted Cloud account; fresh workspace; qualified bounded provider route | Create workspace in the portal, connect API key and native Codex agent in the wizard, complete a task, download and independently check its artifact | +The question-resumption journey uses the same natural-language scenario, browser +answer helper and behavioral grader as the classic `question-resume` suite. Native +provider questions can stay in one run; durable native questions create one exactly bound +continuation each. Creation must have an applied semantic receipt or a completed native +API mutation receipt with the original card, exact scope, form and idempotency key. +An API receipt must retain its creation identity unchanged after the answer. A form or +run count alone is never provenance. A target declares a three-run maximum and positive model allowance, +and a passing receipt must contain one to three attributed native runs. Other journeys +retain their exact run-count requirements. Both real board answers, no premature +output, the sole document's agent/company/task ownership and its reloaded revision +are independently checked. The private provider owns authentication, setup, accounting +and cleanup. Local calibration is not deployed-target qualification. + The shared Studio helper waits for the original saved file body before typing. An editor control can mount before the file query completes; editing that empty draft can otherwise lose the original content when the file arrives. diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index beb3ffe5c3b..df5a2cc39f4 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -192,8 +192,8 @@ The explicit-only `necessary-question` suite has four development cells: the recorded Codex and Claude models, on local and Daytona environments. It requests one essential booking reference that only the test user knows. The browser must show the actual free-text prompt and its explanation, then submit the real answer. -The oracle checks the unchanged question, its server-owned provider or semantic -receipt, the exact stored user answer, and one bound continuation. The saved +The oracle checks the unchanged question, its server-owned provider, semantic or +native API receipt, the exact stored user answer, and one bound continuation. The saved confirmation must contain that answer exactly. The final reply's link must open that same task document and display its confirmation. A title, success claim, replacement card, duplicate wake, or uncommitted result @@ -1705,13 +1705,18 @@ placement; it is not a statistical comparison with the former prompt arrangement The separate explicit-only `question-resume` suite uses the same neutral two-question request on native Codex and ACPX Claude. It qualifies the user journey through -either the provider's built-in question or Paperclip's semantic question tool. +the provider's built-in question, Paperclip's semantic question tool, or its native API fallback. It does not replace or regrade `question-tool-documentation`. The oracle classifies each wait from its exact task/agent/run identity. Provider questions require the server-generated request key and retain the paused run. -Semantic questions require an applied `request_human_input` receipt and one new -response wake bound to that interaction and source run. Only a verified provider +Durable questions require an applied `request_human_input` receipt or a completed +server-owned native `POST /api/issues/{id}/interactions` receipt. An API creation +receipt must report HTTP 201 and preserve the original pending card's exact task, +company, agent, source run, idempotency key and complete normalized form. It must +remain unchanged after the answer. Both durable paths require one new response +wake bound to that interaction and source run. Missing, pending, foreign or +mismatched receipts fail; form shape and run counts cannot replace them. Only a verified provider choice may carry the adapter's optional Other field. A second substantive question is still rejected; the subsequent reference question must be text-only. Both real board answers must remain bound to unchanged forms, and one saved document must diff --git a/tests/runner-e2e/catalog.test.ts b/tests/runner-e2e/catalog.test.ts index 5433370ea37..1e98adc848b 100644 --- a/tests/runner-e2e/catalog.test.ts +++ b/tests/runner-e2e/catalog.test.ts @@ -59,7 +59,7 @@ describe("runner E2E catalog", () => { }); it("exports deployment journeys separately from the model matrix and default selection", () => { - expect(productJourneys.map(journey => journey.id)).toEqual(["task-create", "project-task-context", "ask-question", "task-properties", "task-search-filter", "plan-revise-accept", "build-revise", "protected-actions", "create-skill-studio", "stop-redirect", "cloud-onboarding"]); + expect(productJourneys.map(journey => journey.id)).toEqual(["task-create", "project-task-context", "ask-question", "question-answer-resume", "task-properties", "task-search-filter", "plan-revise-accept", "build-revise", "protected-actions", "create-skill-studio", "stop-redirect", "cloud-onboarding"]); expect(runnerSuites.some(suite => suite.id === "product-journeys")).toBe(false); expect(runnerMatrix.some(execution => execution.suite.id === "product-journeys")).toBe(false); expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(execution => execution.suite.id === "product-journeys")).toBe(false); diff --git a/tests/runner-e2e/continuation-flow.ts b/tests/runner-e2e/continuation-flow.ts index e667575ea90..89072dc005e 100644 --- a/tests/runner-e2e/continuation-flow.ts +++ b/tests/runner-e2e/continuation-flow.ts @@ -1,6 +1,7 @@ import { gradeNecessaryQuestion } from "./necessary-question.js"; import { necessaryQuestionContract } from "./continuation-cases.js"; import { validResumeQuestionForm } from "./question-resume-scoring.js"; +import { answerResumeQuestionThroughUi } from "./question-resume-ui.js"; import { gradeLifecycleBaseline, type LifecycleCheckpoint } from "./lifecycle-baseline.js"; import { lifecycleLiveCase, lifecycleLiveContinuation, gradeLifecycleNarrative } from "./lifecycle-live-cases.js"; import { prepareLegacyContinuationSkill, prepareContinuationBudget } from "./continuation-fixtures.js"; @@ -224,29 +225,7 @@ export async function runContinuationFlow(input: { expect(isSingleClaudeQuestion(set.questions), "one choice question with only the optional provider Other field").toBe(true); } else expect(set.questions, "ask only the requested next question").toHaveLength(1); const before = new Set(runs.map((r) => r.id)); - if (choice) { - expect(set.questions[0].answerMode, "choices must use radio controls").toBe("single_select"); - const options = questions[0].payload.questionSet?.questions[0]?.options - ?? questions[0].payload.questions[0]?.options ?? []; - expect(new Set(options.map((o: Row) => String(o.label).trim().toLowerCase())).size).toBeGreaterThanOrEqual(2); - await page.getByRole("radio", { name: new RegExp(`^${choice}\\b`, "i") }).last().click(); - } else { - expect(set.questions[0].answerMode, "open answers must render a text field, not a choice question").toBe("text"); - await page.getByTestId("question-text-answer-composer").last() - .locator('[contenteditable="true"],textarea').first().fill(scenario.answer); - } - // Claude may add a separate optional Other field after its choice page. - // Navigate every rendered page before submitting; do not invent an answer. - for (let index = 1; index < set.questions.length; index += 1) { - await page.getByRole("button", { name: "Next", exact: true }).last().click(); - } - await page - .getByRole("button", { - name: set.submitLabel ?? "Submit answers", - exact: true, - }) - .last() - .click(); + await answerResumeQuestionThroughUi(page, questions[0], choice ? { choice } : { text: scenario.answer }); await settle(before, false, questions[0].id); } async function reply(body: string) { diff --git a/tests/runner-e2e/fixtures/native-api-question.json b/tests/runner-e2e/fixtures/native-api-question.json new file mode 100644 index 00000000000..48084858be0 --- /dev/null +++ b/tests/runner-e2e/fixtures/native-api-question.json @@ -0,0 +1,152 @@ +{ + "phase": "initial", + "issue": { + "id": "00000000-0000-4000-8000-000000000002", + "companyId": "00000000-0000-4000-8000-000000000001", + "assigneeAgentId": "00000000-0000-4000-8000-000000000003", + "status": "in_review" + }, + "runs": [ + { + "id": "00000000-0000-4000-8000-000000000004", + "companyId": "00000000-0000-4000-8000-000000000001", + "issueId": "00000000-0000-4000-8000-000000000002", + "agentId": "00000000-0000-4000-8000-000000000003", + "status": "succeeded", + "runtimeMode": "native", + "resultJson": { + "apiToolReceipts": { + "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb": { + "state": "completed", + "digest": "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc", + "operationId": "POST /api/issues/{id}/interactions", + "result": { + "ok": true, + "status": 201, + "contentType": "application/json; charset=utf-8", + "operationId": "POST /api/issues/{id}/interactions", + "data": { + "kind": "ask_user_questions", + "status": "pending", + "continuationPolicy": "wake_assignee", + "payload": { + "version": 1, + "supersedeOnUserComment": false, + "questions": [ + { + "id": "time_choice", + "prompt": "Should the club meetup welcome note reference Morning or Afternoon?", + "selectionMode": "single", + "required": true, + "allowOther": false, + "options": [ + { + "id": "morning", + "label": "Morning" + }, + { + "id": "afternoon", + "label": "Afternoon" + } + ] + } + ], + "questionSet": { + "schema": "paperclip.question_set.v1", + "questions": [ + { + "id": "time_choice", + "prompt": "Should the club meetup welcome note reference Morning or Afternoon?", + "required": true, + "answerMode": "single_select", + "options": [ + { + "id": "morning", + "label": "Morning" + }, + { + "id": "afternoon", + "label": "Afternoon" + } + ] + } + ] + } + }, + "id": "00000000-0000-4000-8000-000000000005", + "sourceRunId": "00000000-0000-4000-8000-000000000004", + "companyId": "00000000-0000-4000-8000-000000000001", + "issueId": "00000000-0000-4000-8000-000000000002", + "createdByAgentId": "00000000-0000-4000-8000-000000000003", + "createdByUserId": null, + "idempotencyKey": "meetup-time-choice-api-v1" + } + } + } + } + } + } + ], + "interactions": [ + { + "kind": "ask_user_questions", + "status": "pending", + "continuationPolicy": "wake_assignee", + "payload": { + "version": 1, + "supersedeOnUserComment": false, + "questions": [ + { + "id": "time_choice", + "prompt": "Should the club meetup welcome note reference Morning or Afternoon?", + "selectionMode": "single", + "required": true, + "allowOther": false, + "options": [ + { + "id": "morning", + "label": "Morning" + }, + { + "id": "afternoon", + "label": "Afternoon" + } + ] + } + ], + "questionSet": { + "schema": "paperclip.question_set.v1", + "questions": [ + { + "id": "time_choice", + "prompt": "Should the club meetup welcome note reference Morning or Afternoon?", + "required": true, + "answerMode": "single_select", + "options": [ + { + "id": "morning", + "label": "Morning" + }, + { + "id": "afternoon", + "label": "Afternoon" + } + ] + } + ] + } + }, + "id": "00000000-0000-4000-8000-000000000005", + "sourceRunId": "00000000-0000-4000-8000-000000000004", + "companyId": "00000000-0000-4000-8000-000000000001", + "issueId": "00000000-0000-4000-8000-000000000002", + "createdByAgentId": "00000000-0000-4000-8000-000000000003", + "createdByUserId": null, + "idempotencyKey": "meetup-time-choice-api-v1" + } + ], + "documents": [], + "attachments": [], + "children": [], + "comments": [] +} diff --git a/tests/runner-e2e/product-journeys/agent-flows.ts b/tests/runner-e2e/product-journeys/agent-flows.ts index 57a7c772565..37ccb8995bf 100644 --- a/tests/runner-e2e/product-journeys/agent-flows.ts +++ b/tests/runner-e2e/product-journeys/agent-flows.ts @@ -18,7 +18,7 @@ import { agentJourneyRuns } from "./catalog.js"; import { JourneyError, type JourneyContext, type JourneyEvidence, type TargetSession } from "./types.js"; type Row = Record; -type AgentJourney = Exclude; +type AgentJourney = Exclude; interface TaskState { currentIssue: Row & { id: string; status: string }; taskRuns: Array; comments: Row[]; interactions: Array; attachments: StoryDelivery[] } interface Options { deadlineAt: number; directory: string; capture(id: string, label: string, file: string): Promise } const active = (run: Row) => ["queued", "running"].includes(run.status); diff --git a/tests/runner-e2e/product-journeys/campaign.test.ts b/tests/runner-e2e/product-journeys/campaign.test.ts index e387eba67ab..444d2155169 100644 --- a/tests/runner-e2e/product-journeys/campaign.test.ts +++ b/tests/runner-e2e/product-journeys/campaign.test.ts @@ -46,7 +46,7 @@ async function run(options:{search?:boolean;authFailure?:boolean;onboardingFailu await writeFile(module, `import {readFile,writeFile} from 'node:fs/promises'; const identity={kind:'cloud',targetId:'qa',origin:'https://qa.example.test',commit:'${"a".repeat(40)}',cloudRelease:'v1',runtimeImage:'image',configurationHash:'config'}; let onboarded=false; -const context={identity,company:{id:'company-1',issuePrefix:'QA'},api:{origin:identity.origin,post:async()=>({id:'issue-fixture-1',identifier:'QA-1'})},startingState:'onboarding',capabilities:['company','board-user','task-write'],page:{url:()=>identity.origin+'/QA/issues/QA-1',locator:()=>({isVisible:async()=>true,screenshot:async()=>{}}) }}; +const context={identity,company:{id:'company-1',issuePrefix:'QA'},api:{origin:identity.origin,post:async()=>({id:'issue-fixture-1',identifier:'QA-1'})},startingState:'onboarding',capabilities:['company','board-user','task-write'],page:{url:()=>identity.origin+'/QA/issues/QA-1',locator:()=>({waitFor:async()=>{},isVisible:async()=>true,screenshot:async()=>{}}) }}; export const provider={version:1, async acquire(input){return { async authenticate(){${options.authFailure ? "throw {outcome:'failed',code:'login_code_expired'};" : ""}}, async onboard(evidence){${options.preAcceptanceFailure ? `evidence.onboardingPreAcceptance=${JSON.stringify(preAcceptanceEvidence)};throw {outcome:'failed',code:'onboarding_work_before_acceptance'};` : options.onboardingFailure ? "throw {outcome:'failed',code:'onboarding_failed'};" : `${options.missingArtifact ? "" : `evidence.artifact=${JSON.stringify(artifact)};`}evidence.checks.push(...['fresh-workspace','ui-connection-and-native-agent','first-task-complete','downloadable-artifact-verified'${options.missingCheck ? "" : ",'native-trace-verified'"}].map(id=>({id,passed:true})));onboarded=true;return context;`}}, diff --git a/tests/runner-e2e/product-journeys/catalog.ts b/tests/runner-e2e/product-journeys/catalog.ts index 048c39c49b1..3d40c2cdc66 100644 --- a/tests/runner-e2e/product-journeys/catalog.ts +++ b/tests/runner-e2e/product-journeys/catalog.ts @@ -5,6 +5,7 @@ export const productJourneys = [ { id: "task-create", label: "Create a task for yourself and reopen it", requires: ["company", "board-user", "task-write"], capabilities: ["tasks/create"], paid: false }, { id: "project-task-context", label: "Create a project and reopen its self-assigned task", requires: ["company", "board-user", "task-write"], capabilities: ["projects/lifecycle", "projects/task-context", "tasks/create"], paid: false }, { id: "ask-question", label: "Ask an agent a question and read its answer", requires: ["company", "board-user", "task-write", "agent"], capabilities: ["questions-and-approvals/questions"], paid: true }, + { id: "question-answer-resume", label: "Answer agent questions and verify the saved result", requires: ["company", "board-user", "task-write", "agent"], capabilities: ["questions-and-approvals/questions"], paid: true }, { id: "task-properties", label: "Assign a task to the signed-in user", requires: ["company", "board-user", "task-write", "backlog-task"], capabilities: ["tasks/properties"], paid: false }, { id: "task-search-filter", label: "Search and filter tasks, then reopen the exact saved result", requires: ["company", "board-user", "task-write"], capabilities: ["tasks/organize"], paid: false }, { id: "plan-revise-accept", label: "Revise and approve a saved Plan, then complete the task", requires: ["company", "board-user", "task-write", "agent"], capabilities: ["questions-and-approvals/plan-review"], paid: true }, diff --git a/tests/runner-e2e/product-journeys/clarification-campaign.test.ts b/tests/runner-e2e/product-journeys/clarification-campaign.test.ts new file mode 100644 index 00000000000..d4572b6323a --- /dev/null +++ b/tests/runner-e2e/product-journeys/clarification-campaign.test.ts @@ -0,0 +1,59 @@ +import { afterEach, expect, it, vi } from "vitest"; +import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import path from "node:path"; +import os from "node:os"; +import { journeyRequiredChecks } from "./success-contract.js"; + +vi.mock("@playwright/test", () => ({ chromium: { launch: async () => ({ + newContext: async () => ({ setDefaultTimeout() {}, pages: () => [] }), close: async () => {}, +}) }, expect: {} })); +vi.mock("./clarification-flow.js", () => ({ runClarificationJourney: async (_context: unknown, _session: unknown, + _campaignId: unknown, evidence: { checks: Array<{ id: string; passed: boolean }> }) => { + evidence.checks.push(...journeyRequiredChecks["question-answer-resume"]!.map(id => ({ id, passed: true }))); +} })); +vi.mock("../dashboard-regenerate.js", () => ({ regenerateRunnerDashboard: async ({ outputDirectory }: { outputDirectory: string }) => { + await Promise.all(["index.html", "dashboard.html"].map(file => writeFile(path.join(outputDirectory, file), "dashboard"))); +} })); +import { launchProductJourneys } from "./launch.js"; + +const roots: string[] = []; +afterEach(async () => { process.exitCode = 0; await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true }))); }); +async function campaign(count: number, omitReceipt = false) { + const root = await mkdtemp(path.join(os.tmpdir(), "clarification-collection-")); roots.push(root); + const module = path.join(root, "provider.mjs"), output = path.join(root, "report"); + const runIds = Array.from({ length: count }, (_, i) => `bound-run-${i}`); + const receipts = omitReceipt ? [] : [{ journeyId: "question-answer-resume", runIds, + usage: { runs: runIds.map(runId => ({ runId, usage: { costUsd: 0.01 } })) } }]; + await writeFile(module, ` +const identity = { kind: 'cloud', targetId: 'qa', origin: 'https://qa.example.test', commit: '${"a".repeat(40)}', cloudRelease: 'v1', runtimeImage: 'image', configurationHash: 'config' }; +const context = { identity, company: { id: 'company', issuePrefix: 'QA' }, user: { id: 'board', name: 'QA' }, + api: { origin: identity.origin }, startingState: 'recipe', capabilities: ['company', 'board-user', 'task-write', 'agent'], + agent: { id: 'agent', name: 'QA', provider: 'opencode', model: 'model', runtimeMode: 'native' }, environmentId: 'daytona', + page: { url: () => identity.origin + '/QA/issues/QA-1', locator: () => ({ waitFor: async () => {}, isVisible: async () => true, screenshot: async () => {} }) } }; +export const provider = { version: 1, async acquire() { return { + async authenticate() {}, async prepare() { return context; }, async identity() { return identity; }, + async collect() { return ${JSON.stringify(receipts)}; }, async own() {}, async release() { return 'released'; }, async stop() {} +}; } };`); + const file = path.join(root, "target.json"); + await writeFile(file, JSON.stringify({ version: 1, id: "qa", mode: "existing-cloud", adapterModule: module, + journeys: ["question-answer-resume"], executionEnvironment: "daytona", baseline: { features: {}, connections: [] }, + limits: { durationMs: 60_000, modelBudgetCents: 100, maxRuns: 3 } })); + await launchProductJourneys(["--target-config", file, "--output", output], path.resolve(import.meta.dirname, "../../..")); + return JSON.parse(await readFile(path.join(output, "normalized-results.json"), "utf8")).results[0]; +} + +it.each([1, 2, 3])("collects a successful clarification journey with %i native work runs", async count => { + const result = await campaign(count); + expect(result).toMatchObject({ status: "passed", cleanup: "passed", productJourney: { outcome: "passed" } }); + expect(result.runIds).toHaveLength(count); + expect(result.productJourney.phases).toContainEqual({ phase: "collection", outcome: "passed" }); + expect(result.usage.runs).toHaveLength(count); +}); +it.each([0, 4])("rejects a clarification collection with %i native work runs", async count => { + const result = await campaign(count); + expect(result).toMatchObject({ status: "failed", error: "paid_journey_receipt_missing" }); + expect(result.runIds).toHaveLength(count); +}); +it("rejects a passing story when its provider omits collection entirely", async () => { + expect(await campaign(1, true)).toMatchObject({ status: "failed", error: "paid_journey_receipt_missing" }); +}); diff --git a/tests/runner-e2e/product-journeys/clarification-evidence.test.ts b/tests/runner-e2e/product-journeys/clarification-evidence.test.ts new file mode 100644 index 00000000000..88c0ccb47cf --- /dev/null +++ b/tests/runner-e2e/product-journeys/clarification-evidence.test.ts @@ -0,0 +1,122 @@ +import { describe, expect, it } from "vitest"; +import { parseTargetConfig, productJourneys } from "./catalog.js"; +import { agentJourneyRunCountValid } from "./success-contract.js"; + +describe("portable clarification journey admission", () => { + it("declares the existing two-question story as a bounded native journey", () => { + expect(productJourneys.find(j => j.id === "question-answer-resume")).toMatchObject({ paid: true, requires: ["company", "board-user", "task-write", "agent"] }); + const config = { version: 1, id: "attached", mode: "existing-cloud", adapterModule: "/private/provider.mjs", + executionEnvironment: "daytona", journeys: ["question-answer-resume"], baseline: { features: {}, connections: [] }, + limits: { durationMs: 600_000, modelBudgetCents: 500, maxRuns: 3 } }; + expect(parseTargetConfig(config).journeys).toEqual(["question-answer-resume"]); + for (const maxRuns of [0, 1, 2, 4]) expect(() => parseTargetConfig({ ...config, limits: { ...config.limits, maxRuns } })).toThrow(); + }); + it("admits exactly the provider/semantic bounds without changing existing exact-count stories", () => { + for (const count of [1, 2, 3]) expect(agentJourneyRunCountValid("question-answer-resume", count)).toBe(true); + for (const count of [undefined, 0, 4, 1.5]) expect(agentJourneyRunCountValid("question-answer-resume", count)).toBe(false); + expect(agentJourneyRunCountValid("plan-revise-accept", 2)).toBe(false); + expect(agentJourneyRunCountValid("plan-revise-accept", 3)).toBe(true); + }); +}); + +import { clarificationQuestionFailure, gradeClarificationEvidence } from "./clarification-evidence.js"; +import { continuationScenario } from "../continuation-cases.js"; +import { validateRetainedRunnerResult } from "../result-validation.js"; +import { journeyRequiredChecks } from "./success-contract.js"; +import type { JourneyContext } from "./types.js"; + +type Row = Record; +const context = { company: { id: "company" }, user: { id: "board" }, environmentId: "daytona", + agent: { id: "agent", provider: "opencode", model: "model", runtimeMode: "native" } } as JourneyContext; +function recording(paths: Array<"provider" | "semantic" | "api">) { + const runs: Row[] = [], cards: Row[] = [], checkpoints: any[] = []; + const createRun = (previous?: Row) => ({ id: `run-${runs.length}`, status: "running", runtimeMode: "native", companyId: "company", issueId: "task", agentId: "agent", + contextSnapshot: { paperclipEnvironment: { id: "daytona" }, ...(previous ? { issueId: "task", sourceRunId: previous.sourceRunId, + interactionId: previous.id, interactionKind: "ask_user_questions", interactionStatus: "answered" } : {}) }, + resultJson: { semanticToolReceipts: {} }, runnerProfileJson: { nativeExecutionInput: { task: { prompt: continuationScenario("question-answer-resume", "nonce").prompt }, provider: { kind: "opencode", model: "model" } } } }); + const document = { issueId: "task", companyId: "company", key: "welcome-note", latestRevisionId: "revision", createdByAgentId: "agent", updatedByAgentId: "agent", body: "Welcome to the afternoon meetup. Use AMBERnonce." }; + const snapshot = (phase: string) => checkpoints.push(structuredClone({ phase, + issue: { id: "task", companyId: "company", assigneeAgentId: "agent", status: phase === "final" ? "done" : "in_review" }, + runs, interactions: cards, children: [], attachments: [], comments: [], documents: phase === "final" ? [document] : [] })); + runs.push(createRun()); + for (const [index, path] of paths.entries()) { + const run = runs.at(-1)!; + const field = index === 0 ? { id: "choice", answerMode: "single_select", options: [{ id: "am", label: "Morning" }, { id: "pm", label: "Afternoon" }] } : { id: "reference", answerMode: "text" }; + const request = `request-${index}`, semantic = path !== "provider"; + const card: Row = { id: `question-${index}`, kind: "ask_user_questions", status: "pending", companyId: "company", issueId: "task", createdByAgentId: "agent", sourceRunId: run.id, + idempotencyKey: semantic ? `semantic-${index}` : `paperclip-runner-question:${run.id}:${request}`, continuationPolicy: semantic ? "wake_assignee" : "none", createdByUserId: null, + payload: { ...(semantic ? {} : { runtimeRequestId: request }), questionSet: { questions: [field] } } }; + cards.push(card); + if (semantic) { + run.status = "succeeded"; + if (path === "api") run.resultJson.apiToolReceipts = { ["a".repeat(64)]: { state: "completed", digest: "b".repeat(64), + operationId: "POST /api/issues/{id}/interactions", result: { ok: true, status: 201, + operationId: "POST /api/issues/{id}/interactions", data: structuredClone(card) } } }; + else run.resultJson.semanticToolReceipts[card.idempotencyKey] = { operationId: "request_human_input", result: { disposition: "applied", interaction: structuredClone(card) } }; + } + snapshot(index === 0 ? "initial" : "answered"); + card.status = "answered"; card.resolvedByUserId = "board"; + card.result = { answers: [{ questionId: field.id, optionIds: index === 0 ? ["pm"] : [], ...(index === 1 ? { otherText: "Use AMBERnonce." } : {}) }] }; + if (semantic) runs.push(createRun(card)); + } + runs.at(-1)!.status = "succeeded"; snapshot("final"); + return { checkpoints, readback: { issueId: "task", key: document.key, revisionId: document.latestRevisionId, body: document.body, visibleBody: document.body } }; +} +const failures = (f: ReturnType) => gradeClarificationEvidence(context, "task", "AMBERnonce", f.checkpoints, f.readback).filter(c => !c.passed); + +describe("attached clarification evidence", () => { + it.each([true, false])("rejects an incomplete API receipt instead of inferring native question evidence (choice=%s)", choice => { + const f = recording(["semantic", "semantic"]), point = f.checkpoints[choice ? 0 : 1]; + const question = point.interactions.find((i: Row) => i.status === "pending"), source = point.runs.at(-1); + source.resultJson = { apiToolReceipts: { call: { state: "completed", operationId: "POST /api/issues/{id}/interactions", + result: { ok: true, data: { id: question.id } } } } }; + expect(clarificationQuestionFailure(point, question, choice)).toBe(`clarification_${choice ? "choice" : "text"}_api_fallback`); + expect(failures(f).some(check => check.id === `question-resume.${choice ? "choice" : "text"}.path`)).toBe(true); + delete source.resultJson.apiToolReceipts; + expect(clarificationQuestionFailure(point, question, choice)).toBe(`clarification_${choice ? "choice" : "text"}_native_path_invalid`); + }); + it("distinguishes an invalid form from a correctly bound semantic question", () => { + const point = recording(["semantic", "semantic"]).checkpoints[0], question = point.interactions[0]; + expect(clarificationQuestionFailure(point, question, true)).toBeUndefined(); + question.payload.questionSet.questions[0].answerMode = "text"; + point.runs[0].resultJson.semanticToolReceipts[question.idempotencyKey].result.interaction.payload = structuredClone(question.payload); + expect(clarificationQuestionFailure(point, question, true)).toBe("clarification_choice_form_invalid"); + }); + it.each([["provider", "provider"], ["provider", "semantic"], ["semantic", "provider"], ["semantic", "semantic"], + ["api", "api"], ["api", "semantic"], ["semantic", "api"], ["provider", "api"], ["api", "provider"]] as const)("accepts the real %s/%s paths with native binding and reload proof", (...paths) => { + expect(failures(recording([...paths]))).toEqual([]); + }); + it.each(["foreign-user", "foreign-agent-document", "foreign-task-document", "lost-revision", "wrong-visible-answer", "changed-readback-body", "premature-document", "extra-child", "foreign-model", "foreign-environment", "missing-native-input", "duplicate-question", "wrong-answer"])("rejects a plausible false pass: %s", kind => { + for (const path of ["semantic", "api"] as const) { + const f = recording([path, path]), first = f.checkpoints[0], final = f.checkpoints[2]; + if (kind === "foreign-user") final.interactions[1].resolvedByUserId = "other-board"; + if (kind === "foreign-agent-document") final.documents[0].updatedByAgentId = "other-agent"; + if (kind === "foreign-task-document") final.documents[0].issueId = "other-task"; + if (kind === "lost-revision") f.readback.revisionId = "old-revision"; + if (kind === "wrong-visible-answer") f.readback.visibleBody = "Welcome to the morning meetup."; + if (kind === "changed-readback-body") f.readback.body += " changed"; + if (kind === "premature-document") first.documents = [final.documents[0]]; + if (kind === "extra-child") final.children.push({ id: "extra-child" }); + if (kind === "foreign-model") final.runs[0].runnerProfileJson.nativeExecutionInput.provider.model = "other-model"; + if (kind === "foreign-environment") final.runs[0].contextSnapshot.paperclipEnvironment.id = "local"; + if (kind === "missing-native-input") delete final.runs[0].runnerProfileJson.nativeExecutionInput.task.prompt; + if (kind === "duplicate-question") final.interactions.push(structuredClone(final.interactions[1])); + if (kind === "wrong-answer") final.interactions[1].result.answers[0].otherText = "Use the old reference."; + expect(failures(f).length).toBeGreaterThan(0); + } + }); + it("requires reload evidence even for a correct API document", () => { + const f = recording(["provider", "provider"]); + expect(gradeClarificationEvidence(context, "task", "AMBERnonce", f.checkpoints).some(c => c.id === "question-document-reloaded" && !c.passed)).toBe(true); + }); + it("retains 1–3 bound native runs but rejects zero and excess work", () => { + const result = { schema: "paperclip.runner-e2e.result/v2", executionId: "product-journeys.attached.configured-agent.daytona.question-answer-resume", suiteId: "product-journeys", suiteDefinitionHash: "definition", + attempt: 1, status: "passed", profileId: "configured-agent", environmentId: "daytona", caseId: "question-answer-resume", provider: "opencode", model: "model", runtimeMode: "native", + startedAt: "2026-10-01T00:00:00Z", finishedAt: "2026-10-01T00:00:01Z", durationMs: 1000, cleanup: "passed", runIds: ["run-0"], + productJourney: { targetId: "attached", outcome: "passed", startingState: "recipe", privacy: "private", resourceDisposition: "retained", + phases: [{ phase: "journey", outcome: "passed" }], checks: journeyRequiredChecks["question-answer-resume"]!.map(id => ({ id, passed: true })), + deployment: { kind: "cloud", targetId: "attached", origin: "https://example.test", commit: "a".repeat(40), cloudRelease: "b".repeat(40), runtimeImage: "immutable-image", configurationHash: "config" } } }; + for (const count of [1, 2, 3]) expect(() => validateRetainedRunnerResult({ ...result, runIds: Array.from({ length: count }, (_, i) => `run-${i}`) })).not.toThrow(); + for (const count of [0, 4]) expect(() => validateRetainedRunnerResult({ ...result, runIds: Array.from({ length: count }, (_, i) => `run-${i}`) })).toThrow("result.agentJourneyRun"); + }); +}); diff --git a/tests/runner-e2e/product-journeys/clarification-evidence.ts b/tests/runner-e2e/product-journeys/clarification-evidence.ts new file mode 100644 index 00000000000..69e39b7fa62 --- /dev/null +++ b/tests/runner-e2e/product-journeys/clarification-evidence.ts @@ -0,0 +1,57 @@ +import { gradeQuestionResume, questionPath, validResumeQuestionForm } from "../question-resume-scoring.js"; +import type { ContinuationCheckpoint } from "../continuation-scoring.js"; +import { validateAgentTaskState } from "./agent-flows.js"; +import type { JourneyContext } from "./types.js"; + +type Row = Record; +export interface ClarificationReadback { + issueId: string; key: string; revisionId: string; body: string; visibleBody: string; +} + +export function clarificationQuestionFailure(point: ContinuationCheckpoint, question: Row, choice: boolean) { + const stage = choice ? "choice" : "text"; + if (!questionPath(point, question)) { + const source = point.runs.find(run => run.id === question.sourceRunId) as Row | undefined; + const apiCreated = Object.values(source?.resultJson?.apiToolReceipts ?? {}).some(value => { + const receipt = value as Row; + return receipt.state === "completed" && receipt.operationId === "POST /api/issues/{id}/interactions" + && receipt.result?.ok === true && receipt.result?.data?.id === question.id; + }); + return apiCreated ? `clarification_${stage}_api_fallback` : `clarification_${stage}_native_path_invalid`; + } + return validResumeQuestionForm(point, question, choice) ? undefined : `clarification_${stage}_form_invalid`; +} + +export function gradeClarificationEvidence(context: JourneyContext, issueId: string, marker: string, + checkpoints: ContinuationCheckpoint[], readback?: ClarificationReadback) { + const checks = gradeQuestionResume(checkpoints, marker); + const final = checkpoints.find(point => point.phase === "final"); + const initial = checkpoints.filter(point => point.phase !== "final"); + const document = final?.documents[0] as Row | undefined; + let bound = !!context.agent && context.agent.runtimeMode === "native" && !!context.environmentId; + try { + for (const point of checkpoints) validateAgentTaskState(context, issueId, { + currentIssue: point.issue as Row & { id: string; status: string }, taskRuns: point.runs, + comments: point.comments as Row[], interactions: point.interactions as Array, attachments: [], + }, 3); + } catch { bound = false; } + checks.push({ id: "question-no-premature-output", passed: initial.length === 2 && initial.every(point => + point.documents.length === 0 && point.attachments.length === 0 && point.children.length === 0 && point.issue.status !== "done") + && !!final && final.issue.status === "done" && final.attachments.length === 0 && final.children.length === 0, + detail: "Wait for both real answers before producing the sole task document, with no extra task or download." }, + { id: "question-answers-owned-by-board", passed: !!final && final.interactions.length === 2 && final.interactions.every(value => { + const question = value as Row; + return question.status === "answered" && question.resolvedByUserId === context.user.id && !question.resolvedByAgentId; + }), detail: "Both retained responses must come from the signed-in board user." }, + { id: "question-document-attributed", passed: !!document && document.issueId === issueId && document.companyId === context.company.id + && document.createdByAgentId === context.agent?.id && document.updatedByAgentId === context.agent?.id, + detail: "The owned native agent created and last wrote the document on this exact task and company." }, + { id: "question-document-reloaded", passed: !!document && !!readback && readback.issueId === issueId + && readback.key === document.key && readback.revisionId === document.latestRevisionId && readback.body === document.body + && readback.visibleBody.includes(`Use ${marker}.`) && /\bafternoon\b/i.test(readback.visibleBody), + detail: "The reloaded document must show both saved answers at the independently read revision." }, + { id: "configured-model-and-environment", passed: bound && checkpoints.length === 3 && !!final + && final.runs.length >= 1 && final.runs.length <= 3 && final.runs.every(run => run.status === "succeeded"), + detail: "Every observed run belongs to the configured native agent, task, company, model and environment." }); + return checks; +} diff --git a/tests/runner-e2e/product-journeys/clarification-flow.test.ts b/tests/runner-e2e/product-journeys/clarification-flow.test.ts new file mode 100644 index 00000000000..97711ca526a --- /dev/null +++ b/tests/runner-e2e/product-journeys/clarification-flow.test.ts @@ -0,0 +1,35 @@ +import { expect, it, vi } from "vitest"; +import { readClarificationState } from "./clarification-flow.js"; +import type { JourneyContext } from "./types.js"; + +const issueId = "11111111-1111-4111-8111-111111111111"; +const companyId = "22222222-2222-4222-8222-222222222222"; +const parent = { id: issueId, companyId, parentId: null, assigneeAgentId: "agent", status: "in_progress" }; +const child = { id: "child", companyId, parentId: issueId }; +function fixture(rows = [parent, child]) { + const get = vi.fn(async (route: string) => { + const url = new URL(route, "https://qa.example.test"); + if (url.pathname === `/api/companies/${companyId}/issues`) { + if (url.searchParams.get("view") !== "index") throw Error("unsupported child projection"); + return rows.filter(row => row.parentId === url.searchParams.get("parentIds")); + } + if (url.pathname === `/api/issues/${issueId}`) return parent; + return []; + }); + const context = { company: { id: companyId }, environmentId: "daytona", api: { get }, + agent: { id: "agent", provider: "opencode", model: "model", runtimeMode: "native" } } as unknown as JourneyContext; + return { context, get }; +} +it("reads the supported child projection while excluding the journey task and unrelated company work", async () => { + const { context, get } = fixture([parent, child, { id: "unrelated", companyId, parentId: "other" }]); + expect((await readClarificationState(context, issueId)).children).toEqual([child]); + expect(get).toHaveBeenCalledWith(`/api/companies/${companyId}/issues?view=index&parentIds=${issueId}&limit=2`); +}); +it("retains an empty child result for a task that did not delegate work", async () => { + expect((await readClarificationState(fixture([parent]).context, issueId)).children).toEqual([]); +}); +it.each([{ ...child, companyId: "other" }, { ...child, parentId: "other" }])("rejects a returned child outside the owned company or task", async foreign => { + const { context, get } = fixture(); + get.mockImplementation(async route => route.includes("/companies/") ? [foreign] : route === `/api/issues/${issueId}` ? parent : []); + await expect(readClarificationState(context, issueId)).rejects.toMatchObject({ code: "clarification_child_query_mismatch" }); +}); diff --git a/tests/runner-e2e/product-journeys/clarification-flow.ts b/tests/runner-e2e/product-journeys/clarification-flow.ts new file mode 100644 index 00000000000..932efbebf92 --- /dev/null +++ b/tests/runner-e2e/product-journeys/clarification-flow.ts @@ -0,0 +1,110 @@ +import { mkdir, writeFile } from "node:fs/promises"; +import path from "node:path"; +import { expect } from "@playwright/test"; +import { continuationScenario } from "../continuation-cases.js"; +import { continuationAnswerCommitted, continuationCheckpointReady } from "../continuation-readiness.js"; +import type { ContinuationCheckpoint } from "../continuation-scoring.js"; +import { answerResumeQuestionThroughUi } from "../question-resume-ui.js"; +import { createTaskThroughUi } from "../user-actions.js"; +import { validateAgentTaskState } from "./agent-flows.js"; +import { clarificationQuestionFailure, gradeClarificationEvidence, type ClarificationReadback } from "./clarification-evidence.js"; +import { JourneyError, type JourneyContext, type JourneyEvidence, type TargetSession } from "./types.js"; + +type Row = Record; +interface Options { deadlineAt: number; directory: string; capture(id: string, label: string, file: string): Promise } + +export async function readClarificationState(context: JourneyContext, issueId: string) { + const route = `/api/issues/${issueId}`; + const [currentIssue, summaries, interactions, documents, attachments, children] = await Promise.all([ + context.api.get(route), context.api.get(`${route}/runs`), context.api.get(`${route}/interactions`), + context.api.get(`${route}/documents`), context.api.get(`${route}/attachments`), + context.api.get(`/api/companies/${context.company.id}/issues?view=index&parentIds=${encodeURIComponent(issueId)}&limit=2`), + ]); + if (!Array.isArray(summaries) || summaries.length > 3) throw new JourneyError("failed", "clarification_run_limit_exceeded"); + const runs = await Promise.all(summaries.map(summary => context.api.get(`/api/heartbeat-runs/${summary.runId}`))); + validateAgentTaskState(context, issueId, { currentIssue: currentIssue as Row & { id: string; status: string }, + taskRuns: runs as Array, interactions: interactions as Array, comments: [], attachments: [] }, 3); + if (!Array.isArray(children) || children.some(child => child.parentId !== issueId || child.companyId !== context.company.id)) { + throw new JourneyError("failed", "clarification_child_query_mismatch"); + } + return { currentIssue, runs, interactions, documents, attachments, children }; +} + +export async function runClarificationJourney(context: JourneyContext, session: TargetSession, + campaignId: string, evidence: JourneyEvidence, options: Options) { + if (!context.agent || context.agent.runtimeMode !== "native" || !context.environmentId) { + throw new JourneyError("unsupported", "native_agent_and_environment_required"); + } + const agent = context.agent, scenario = continuationScenario("question-answer-resume", campaignId.replaceAll("-", "_")); + const checkpoints: ContinuationCheckpoint[] = []; + let issue: Row | undefined, readback: ClarificationReadback | undefined; + await mkdir(options.directory, { recursive: true, mode: 0o700 }); + const retain = (name: string, value: unknown) => writeFile(path.join(options.directory, name), JSON.stringify(value, null, 2), { mode: 0o600 }); + const read = () => readClarificationState(context, issue!.id); + const settle = async (answeredId?: string, final = false) => { + let stable = ""; + while (Date.now() < options.deadlineAt) { + const state = await read(); + const pending = state.interactions.filter(i => i.kind === "ask_user_questions" && i.status === "pending"); + const ready = continuationAnswerCommitted(state.interactions, answeredId) + && continuationCheckpointReady({ issue: state.currentIssue as { status: string; executionRunId?: string }, runs: state.runs as Array, interactions: state.interactions }) + && (final ? state.currentIssue.status === "done" && !pending.length : pending.length === 1) + && !state.currentIssue.scheduledRetry && !state.currentIssue.activeRecoveryAction; + const key = ready ? JSON.stringify([state.currentIssue.status, state.currentIssue.executionRunId, + state.runs.map(run => [run.id, run.status]), state.interactions.map(i => [i.id, i.status])]) : ""; + if (key && key === stable) return state; + stable = key; + await new Promise(resolve => setTimeout(resolve, 1000)); + } + throw new JourneyError("failed", final ? "clarification_not_completed" : "clarification_question_not_ready"); + }; + const open = () => context.page.goto(`${context.api.origin}/${encodeURIComponent(context.company.issuePrefix)}/issues/${encodeURIComponent(issue!.identifier ?? issue!.id)}`, { waitUntil: "domcontentloaded" }); + const snapshot = async (phase: "initial" | "answered" | "final", state: Awaited>) => { + const documents = await Promise.all(state.documents.map(document => context.api.get(`/api/issues/${issue!.id}/documents/${encodeURIComponent(document.key)}`))); + const point = { phase, issue: state.currentIssue, runs: structuredClone(state.runs), interactions: structuredClone(state.interactions), + documents, attachments: state.attachments, children: state.children, comments: [] } as unknown as ContinuationCheckpoint; + checkpoints.push(point); + await retain("clarification.json", { checkpoints }); + await open(); + await options.capture(phase, `Clarification ${phase}`, `clarification-${phase}.png`); + return point; + }; + try { + const title = `Journey QA question-answer-resume ${campaignId}`; + const created = await createTaskThroughUi({ page: context.page, origin: context.api.origin, + issuePrefix: context.company.issuePrefix, agentName: agent.name, title, prompt: scenario.prompt, + workMode: "standard", requireExplicitTitle: true, + beforeDispatch: () => session.own({ kind: "issue", name: title, journeyId: "question-answer-resume" }) }); + await session.own({ kind: "issue", name: title, id: created.issueId, journeyId: "question-answer-resume" }); + issue = await context.api.get(`/api/issues/${created.issueId}`); + let state = await settle(); + for (const [index, input] of [{ choice: "Afternoon" }, { text: scenario.answer }].entries()) { + const point = await snapshot(index === 0 ? "initial" : "answered", state); + const question = state.interactions.find(i => i.kind === "ask_user_questions" && i.status === "pending")!; + const questionFailure = clarificationQuestionFailure(point, question, index === 0); + if (questionFailure) throw new JourneyError("failed", questionFailure); + if (point.documents.length || point.attachments.length || point.children.length || point.issue.status === "done") { + throw new JourneyError("failed", "clarification_premature_output"); + } + await answerResumeQuestionThroughUi(context.page, question, input); + state = await settle(question.id, index === 1); + } + const final = await snapshot("final", state); + if (final.documents.length !== 1) throw new JourneyError("failed", "clarification_document_missing"); + const document = final.documents[0]!; + const href = `/${encodeURIComponent(context.company.issuePrefix)}/issues/${encodeURIComponent(issue.identifier ?? issue.id)}#document-${encodeURIComponent(document.key)}`; + await context.page.goto(new URL(href, context.api.origin).href, { waitUntil: "domcontentloaded" }); + await context.page.reload({ waitUntil: "domcontentloaded" }); + const visible = context.page.locator(`[id=${JSON.stringify(`document-${document.key}`)}]:visible, [id=${JSON.stringify(`side-panel-content-document:${document.key}`)}]:visible`); + await expect(visible).toHaveCount(1); await expect(visible).toBeVisible(); + await expect(visible).toContainText(scenario.answer); await expect(visible).toContainText(/afternoon/i); + const persisted = await context.api.get(`/api/issues/${issue.id}/documents/${encodeURIComponent(document.key)}`); + readback = { issueId: persisted.issueId, key: persisted.key, revisionId: persisted.latestRevisionId, body: persisted.body, visibleBody: await visible.innerText() }; + await options.capture("reloaded", "Saved answers after reload", "clarification-reloaded.png"); + } finally { + const checks = gradeClarificationEvidence(context, issue?.id ?? "", scenario.marker, checkpoints, readback); + evidence.checks.push(...checks.map(({ id, passed }) => ({ id, passed }))); + await retain("clarification.json", { checkpoints, readback, checks }); + } + if (evidence.checks.some(check => !check.passed)) throw new JourneyError("failed", "clarification_evidence_failed"); +} diff --git a/tests/runner-e2e/product-journeys/launch.ts b/tests/runner-e2e/product-journeys/launch.ts index aa65ce9b560..4be973f1e81 100644 --- a/tests/runner-e2e/product-journeys/launch.ts +++ b/tests/runner-e2e/product-journeys/launch.ts @@ -6,13 +6,15 @@ import path from "node:path"; import { pathToFileURL } from "node:url"; import { chromium, type Browser, type BrowserContext } from "@playwright/test"; import { runnerSuites } from "../catalog.js"; -import { productJourneys, parseTargetConfig, preparationPreview, isAgentJourney, agentJourneyRuns } from "./catalog.js"; +import { productJourneys, parseTargetConfig, preparationPreview, isAgentJourney } from "./catalog.js"; +import { agentJourneyRunCountValid } from "./success-contract.js"; import { journeyFixtures } from "./fixtures.js"; import { taskProperties } from "./task-properties.js"; import { taskSearchFilter } from "./task-search-filter.js"; import { askQuestion, taskCreate } from "./basic-flows.js"; import { projectTaskContext } from "./project-context-flow.js"; import { runAgentJourney } from "./agent-flows.js"; +import { runClarificationJourney } from "./clarification-flow.js"; import { JourneyError, type JourneyContext, type JourneyEvidence, type JourneyOutcome, type TargetLifecycleInput, type TargetProvider, type TargetSession } from "./types.js"; import type { RunnerE2EResult } from "../types.js"; import { regenerateRunnerDashboard } from "../dashboard-regenerate.js"; @@ -104,7 +106,7 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri try { sha = execFileSync("git", ["rev-parse", "HEAD"], { cwd: repositoryRoot, encoding: "utf8", stdio: ["ignore", "pipe", "ignore"] }).trim(); } catch { /* Packaged harnesses may not include Git metadata. */ } const source = resolveRunnerE2ESource({ sha, ref: null, workflowRunUrl: null }); const definitionHash = createHash("sha256").update(JSON.stringify(productJourneys)); - for (const name of ["catalog.ts", "task-properties.ts", "task-search-filter.ts", "task-search-evidence.ts", "basic-flows.ts", "basic-evidence.ts", "project-context-flow.ts", "project-context-readback.ts", "project-context-evidence.ts", "agent-flows.ts", "success-contract.ts", "product-screenshot.ts", "cloud-onboarding.ts", "onboarding-connect.ts", "onboarding-first-task.ts", "workspace-deferral.ts", "onboarding-pre-acceptance.ts", "artifact-evidence.ts", "fixtures.ts", "launch.ts", "types.ts", "collection.ts", "local-target.ts"]) definitionHash.update(await readFile(path.join(import.meta.dirname, name))); + for (const name of ["catalog.ts", "task-properties.ts", "task-search-filter.ts", "task-search-evidence.ts", "basic-flows.ts", "basic-evidence.ts", "project-context-flow.ts", "project-context-readback.ts", "project-context-evidence.ts", "agent-flows.ts", "clarification-flow.ts", "clarification-evidence.ts", "../question-resume-ui.ts", "../question-resume-scoring.ts", "../runtime-question-readiness.ts", "../chat-flow.ts", "../continuation-readiness.ts", "../continuation-cases.ts", "success-contract.ts", "product-screenshot.ts", "cloud-onboarding.ts", "onboarding-connect.ts", "onboarding-first-task.ts", "workspace-deferral.ts", "onboarding-pre-acceptance.ts", "artifact-evidence.ts", "fixtures.ts", "launch.ts", "types.ts", "collection.ts", "local-target.ts"]) definitionHash.update(await readFile(path.join(import.meta.dirname, name))); for (const file of ["catalog.ts", "api.ts", "run-observations.ts", "native-task-reply.ts", "native-event-envelope.ts", "user-actions.ts", "plan-revision-flow.ts", "skill-studio-flow.ts", "protected-action-flow.ts", "story-artifact.ts", "stop-redirect-flow.ts", "everyday-cases.ts", "everyday-delivery.ts", "everyday-decisions.ts", "connection-guidance-evidence.ts", "everyday-artifact.py"]) definitionHash.update(await readFile(path.join(import.meta.dirname, "..", file))); definitionHash.update(runnerSuites.flatMap(suite => suite.tasks).find(task => task.id === "ask-question")!.buildPrompt("definition")); const suiteDefinitionHash = definitionHash.digest("hex"); @@ -201,12 +203,19 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri result.model = ready.agent.model; result.runtimeMode = ready.agent.runtimeMode; if (id === "ask-question") await askQuestion(ready, session!, campaignId, evidence); + else if (id === "question-answer-resume") await runClarificationJourney(ready, session!, campaignId, evidence, { deadlineAt, + directory: path.join(outputDir, "private-stories", id), + capture: async (_id, _label, file) => { + const directory = path.join(outputDir, "private-stories", id); + await mkdir(directory, { recursive: true, mode: 0o700 }); + await captureProductScreenshot(ready!.page, ready!.api.origin, path.join(directory, file), deadlineAt); + } }); else await runAgentJourney(id, ready, session!, campaignId, evidence, { deadlineAt, directory: path.join(outputDir, "private-stories", id), capture: async (_id, _label, file) => { const directory = path.join(outputDir, "private-stories", id); await mkdir(directory, { recursive: true, mode: 0o700 }); - await captureProductScreenshot(ready!.page, ready!.api.origin, path.join(directory, file)); + await captureProductScreenshot(ready!.page, ready!.api.origin, path.join(directory, file), deadlineAt); } }); } else if (id === "task-create") await taskCreate(ready, session!, campaignId, evidence); else if (id === "project-task-context") await projectTaskContext(ready, session!, campaignId, evidence); @@ -221,7 +230,7 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri result.status = "passed"; const evidenceDir = path.join(outputDir, "evidence", result.executionId, "attempt-1"); await mkdir(evidenceDir, { recursive: true, mode: 0o700 }); - await captureProductScreenshot(ready.page, ready.api.origin, path.join(evidenceDir, `${id}.png`)); + await captureProductScreenshot(ready.page, ready.api.origin, path.join(evidenceDir, `${id}.png`), deadlineAt); result.screenshots = [{ id, label: productJourneys.find(j => j.id === id)!.label, file: `${id}.png` }]; } catch (error) { const failure = journeyFailure(error); @@ -237,7 +246,7 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri if (url.origin === ready.api.origin && /^\/[^/]+\/(issues|search)(\/|$)/.test(url.pathname)) { const evidenceDir = path.join(outputDir, "evidence", result.executionId, "attempt-1"); await mkdir(evidenceDir, { recursive: true, mode: 0o700 }); - await captureProductScreenshot(ready.page, ready.api.origin, path.join(evidenceDir, `${id}-failed.png`)); + await captureProductScreenshot(ready.page, ready.api.origin, path.join(evidenceDir, `${id}-failed.png`), deadlineAt); result.screenshots = [{ id: `${id}-failed`, label: "Failed journey screen", file: `${id}-failed.png` }]; } } catch { /* Preserve the original failure when its page is unavailable. */ } @@ -265,7 +274,7 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri } for (const result of results) { if (isAgentJourney(result.caseId as import("./types.js").JourneyId) && result.status === "passed" - && (!seen.has(result.caseId) || result.runIds?.length !== agentJourneyRuns[result.caseId as keyof typeof agentJourneyRuns])) throw new JourneyError("failed", "paid_journey_receipt_missing"); + && (!seen.has(result.caseId) || !agentJourneyRunCountValid(result.caseId, result.runIds?.length))) throw new JourneyError("failed", "paid_journey_receipt_missing"); if (result.caseId === "cloud-onboarding" && result.status === "passed" && (!seen.has(result.caseId) || !result.runIds?.length || !result.productJourney?.operations || !result.productJourney.operations.traceIds.length)) throw new JourneyError("failed", "paid_journey_receipt_missing"); // Success-only checks must run before a disposable target can be deleted. // Cleanup is provisional here; the observed cleanup result is retained below. diff --git a/tests/runner-e2e/product-journeys/product-screenshot.test.ts b/tests/runner-e2e/product-journeys/product-screenshot.test.ts index 9d1cfd911cb..37f079e6a4f 100644 --- a/tests/runner-e2e/product-journeys/product-screenshot.test.ts +++ b/tests/runner-e2e/product-journeys/product-screenshot.test.ts @@ -1,20 +1,63 @@ -import { describe, expect, it, vi } from "vitest"; +import { afterEach, describe, expect, it, vi } from "vitest"; import type { Page } from "@playwright/test"; import { captureProductScreenshot } from "./product-screenshot.js"; +const origin = "https://qa.example.test"; +afterEach(() => vi.restoreAllMocks()); +function fixture(options: { mounted?: boolean; redirect?: string; afterMount?(): void } = {}) { + let mounted = false, url = `${origin}/QA/issues/QA-1`; + const screenshot = vi.fn(async (_input: { path: string; timeout: number }) => {}); + const waitFor = vi.fn(async (_input: { state: string; timeout: number }) => { mounted = options.mounted !== false; if (options.redirect) url = options.redirect; options.afterMount?.(); }); + const locator = vi.fn(() => ({ waitFor, isVisible: async () => mounted, screenshot })); + const page = { url: () => url, locator } as unknown as Page; + return { page, locator, waitFor, screenshot }; +} describe("product screenshot boundary", () => { - it("excludes a sign-in screen rendered at an unchanged task URL", async () => { - const screenshot = vi.fn(); - const page = { url: () => "https://qa.example.test/QA/issues/QA-1", locator: () => ({ isVisible: async () => false, screenshot }) } as unknown as Page; - await expect(captureProductScreenshot(page, "https://qa.example.test", "shot.png")).rejects.toThrow("product_screen_unavailable"); - expect(screenshot).not.toHaveBeenCalled(); + it("waits for Layout to mount after the document navigation completes", async () => { + const f = fixture(); + await captureProductScreenshot(f.page, origin, "shot.png"); + expect(f.locator).toHaveBeenCalledWith("main#main-content"); + expect(f.waitFor).toHaveBeenCalledWith({ state: "visible", timeout: expect.any(Number) }); + expect(f.screenshot).toHaveBeenCalledTimes(1); + expect(f.waitFor.mock.invocationCallOrder[0]).toBeLessThan(f.screenshot.mock.invocationCallOrder[0]!); }); - it("captures only the signed-in product region and refuses other origins", async () => { - const screenshot = vi.fn(), locator = vi.fn(() => ({ isVisible: async () => true, screenshot })); - const page = { url: () => "https://qa.example.test/QA/issues/QA-1", locator } as unknown as Page; - await captureProductScreenshot(page, "https://qa.example.test", "shot.png"); - expect(locator).toHaveBeenCalledWith("main#main-content"); expect(screenshot).toHaveBeenCalledTimes(1); - await expect(captureProductScreenshot(page, "https://other.example.test", "shot.png")).rejects.toThrow("product_screen_unavailable"); - expect(screenshot).toHaveBeenCalledTimes(1); + it("excludes a sign-in screen rendered at an unchanged task URL", async () => { + const f = fixture({ mounted: false }); + await expect(captureProductScreenshot(f.page, origin, "shot.png")).rejects.toThrow("product_screen_unavailable"); + expect(f.screenshot).not.toHaveBeenCalled(); + }); + it("rejects a product region that never mounts", async () => { + const f = fixture(); f.waitFor.mockRejectedValueOnce(new Error("Timeout")); + await expect(captureProductScreenshot(f.page, origin, "shot.png")).rejects.toThrow("product_screen_unavailable"); + expect(f.screenshot).not.toHaveBeenCalled(); + }); + it.each([`${origin}/login`, "https://other.example.test/QA/issues/QA-1"])("excludes a redirect during readiness to %s", async redirect => { + const f = fixture({ redirect }); + await expect(captureProductScreenshot(f.page, origin, "shot.png")).rejects.toThrow("product_screen_unavailable"); + expect(f.screenshot).not.toHaveBeenCalled(); + }); + it("refuses another origin before waiting or taking a screenshot", async () => { + const f = fixture(); + await expect(captureProductScreenshot(f.page, "https://other.example.test", "shot.png")).rejects.toThrow("product_screen_unavailable"); + expect(f.waitFor).not.toHaveBeenCalled(); expect(f.screenshot).not.toHaveBeenCalled(); + }); + it("bounds readiness and capture by the campaign deadline", async () => { + vi.spyOn(Date, "now").mockReturnValue(1000); + const f = fixture(); + await captureProductScreenshot(f.page, origin, "shot.png", 1050); + for (const call of [f.waitFor.mock.calls[0], f.screenshot.mock.calls[0]]) { + expect((call?.[0] as { timeout: number }).timeout).toBeGreaterThan(0); + expect((call?.[0] as { timeout: number }).timeout).toBeLessThanOrEqual(50); + } + }); + it("does not capture after readiness consumes the remaining deadline", async () => { + const now = vi.spyOn(Date, "now").mockReturnValue(1000), f = fixture({ afterMount: () => { now.mockReturnValue(1050); } }); + await expect(captureProductScreenshot(f.page, origin, "shot.png", 1050)).rejects.toThrow("product_screen_unavailable"); + expect(f.screenshot).not.toHaveBeenCalled(); + }); + it("never disables a timeout when the campaign deadline has elapsed", async () => { + const f = fixture(); + await expect(captureProductScreenshot(f.page, origin, "shot.png", Date.now())).rejects.toThrow("product_screen_unavailable"); + expect(f.waitFor).not.toHaveBeenCalled(); expect(f.screenshot).not.toHaveBeenCalled(); }); }); diff --git a/tests/runner-e2e/product-journeys/product-screenshot.ts b/tests/runner-e2e/product-journeys/product-screenshot.ts index 14ae3d0a9c8..f8387e447e2 100644 --- a/tests/runner-e2e/product-journeys/product-screenshot.ts +++ b/tests/runner-e2e/product-journeys/product-screenshot.ts @@ -1,12 +1,22 @@ import type { Page } from "@playwright/test"; import { JourneyError } from "./types.js"; -export async function captureProductScreenshot(page: Page, origin: string, file: string) { - const url = new URL(page.url()); - if (url.origin !== origin || !/^\/[^/]+\/(issues|search)(\/|$)/.test(url.pathname)) throw new JourneyError("failed", "product_screen_unavailable"); +export async function captureProductScreenshot(page: Page, origin: string, file: string, deadlineAt = Date.now() + 10_000) { + const permitted = () => { + const url = new URL(page.url()); + return url.origin === origin && /^\/[^/]+\/(issues|search)(\/|$)/.test(url.pathname); + }; + const timeout = (maximum: number) => { + const remaining = deadlineAt - Date.now(); + if (remaining <= 0) throw new JourneyError("failed", "product_screen_unavailable"); + return Math.min(maximum, remaining); + }; + if (!permitted() || deadlineAt <= Date.now()) throw new JourneyError("failed", "product_screen_unavailable"); // CloudAccessGate replaces Layout on expiry without necessarily changing the // URL. Capture Layout's main element, so auth screens can never be included. const product = page.locator("main#main-content"); - if (!await product.isVisible()) throw new JourneyError("failed", "product_screen_unavailable"); - await product.screenshot({ path: file, timeout: 2000 }); + try { await product.waitFor({ state: "visible", timeout: timeout(10_000) }); } + catch { throw new JourneyError("failed", "product_screen_unavailable"); } + if (!permitted() || !await product.isVisible() || deadlineAt <= Date.now()) throw new JourneyError("failed", "product_screen_unavailable"); + await product.screenshot({ path: file, timeout: timeout(2000) }); } diff --git a/tests/runner-e2e/product-journeys/success-contract.ts b/tests/runner-e2e/product-journeys/success-contract.ts index 8e68da0a091..f49db32a938 100644 --- a/tests/runner-e2e/product-journeys/success-contract.ts +++ b/tests/runner-e2e/product-journeys/success-contract.ts @@ -1,12 +1,22 @@ /** Closed, target-independent success requirements used by dispatch and retained evidence. */ export const agentJourneyRuns = { "ask-question": 1, "plan-revise-accept": 3, "build-revise": 2, - "protected-actions": 4, "create-skill-studio": 1, "stop-redirect": 3 } as const; + "protected-actions": 4, "create-skill-studio": 1, "stop-redirect": 3, "question-answer-resume": 3 } as const; + +export function agentJourneyRunCountValid(id: string, count: unknown): boolean { + if (!Object.hasOwn(agentJourneyRuns, id) || !Number.isSafeInteger(count)) return false; + const maximum = agentJourneyRuns[id as keyof typeof agentJourneyRuns]; + const minimum = id === "question-answer-resume" ? 1 : maximum; + return (count as number) >= minimum && (count as number) <= maximum; +} export const journeyRequiredChecks: Record = { "task-properties": ["assignment-persisted", "no-agent-work"], "task-create": ["browser-created-task-persisted", "no-agent-work"], "project-task-context": ["browser-created-project-persisted", "browser-created-project-task-persisted", "project-task-association-reloaded", "project-task-context-visible", "no-agent-work"], "task-search-filter": ["task-search-results-exact", "task-status-filter-excludes-controls", "task-search-filter-reloaded", "task-search-result-reopened", "no-agent-work"], "ask-question": ["question-answer-persisted", "one-bound-successful-run", "configured-model-and-environment"], + "question-answer-resume": ["question-resume.forms", "question-resume.choice.answer", "question-resume.choice.path", + "question-resume.text.answer", "question-resume.text.path", "question-resume.complete-evidence", "question-no-premature-output", + "question-answers-owned-by-board", "question-document-attributed", "question-document-reloaded", "configured-model-and-environment"], "plan-revise-accept": ["plan-revision-and-approval", "plan-final-answer", "configured-model-and-environment"], "build-revise": ["initial-artifact-verified", "revised-artifact-verified", "artifact-versions-differ", "original-artifact-preserved", "user-reply-stored-once", "configured-model-and-environment"], "protected-actions": ["approve-waited-for-decision", "approve-decision-persisted", "approve-service-call-count", "approved-service-data-used", diff --git a/tests/runner-e2e/product-journeys/types.ts b/tests/runner-e2e/product-journeys/types.ts index ec7746cf303..0f899efca5e 100644 --- a/tests/runner-e2e/product-journeys/types.ts +++ b/tests/runner-e2e/product-journeys/types.ts @@ -3,8 +3,7 @@ import type { ConnectionApi } from "../connection-target.js"; export type JourneyOutcome = "passed" | "failed" | "blocked" | "unsupported"; export type JourneyId = "task-properties" | "task-create" | "project-task-context" | "ask-question" | "cloud-onboarding" - | "task-search-filter" - | "plan-revise-accept" | "build-revise" | "protected-actions" | "create-skill-studio" | "stop-redirect"; + | "question-answer-resume" | "task-search-filter" | "plan-revise-accept" | "build-revise" | "protected-actions" | "create-skill-studio" | "stop-redirect"; export type TargetMode = "managed-self-hosted" | "existing-cloud" | "new-cloud"; export interface DeploymentIdentity { kind: "self-hosted" | "cloud"; diff --git a/tests/runner-e2e/question-resume-scoring.ts b/tests/runner-e2e/question-resume-scoring.ts index 10e3f81e6bc..85f8902d95e 100644 --- a/tests/runner-e2e/question-resume-scoring.ts +++ b/tests/runner-e2e/question-resume-scoring.ts @@ -3,14 +3,33 @@ import type { ContinuationCheckpoint } from "./continuation-scoring.js"; import { isSingleClaudeQuestion } from "./runtime-question-readiness.js"; type Row = Record; -export type QuestionPath = "provider" | "semantic"; +export type QuestionPath = "provider" | "semantic" | "api"; const nonempty = (v: unknown): v is string => typeof v === "string" && v.trim().length > 0; const questions = (c?: ContinuationCheckpoint): Row[] => (c?.interactions ?? []).filter((i: any) => i.kind === "ask_user_questions") as Row[]; export const resumeQuestionFields = (i?: Row): Row[] => Array.isArray(i?.payload?.questionSet?.questions) ? i.payload.questionSet.questions : []; -/** Classify by server-owned identities and the applied semantic receipt. A +function nativeApiQuestionReceipt(run: Row | undefined, question: Row) { + const hash = (value: unknown) => typeof value === "string" && /^[a-f0-9]{64}$/.test(value); + return Object.entries(run?.resultJson?.apiToolReceipts ?? {}).find(([key, value]) => { + const receipt = value as Row, original = receipt?.result?.data; + return hash(key) && hash(receipt?.digest) && receipt?.state === "completed" + && receipt.operationId === "POST /api/issues/{id}/interactions" && receipt.result?.operationId === receipt.operationId + && receipt.result.ok === true && receipt.result.status === 201 && original?.status === "pending" + && original.kind === "ask_user_questions" && original.createdByUserId === null && question.createdByUserId === null + && sameQuestion(original, question); + }); +} + +function sameNativeApiQuestionReceipt(before: ContinuationCheckpoint | undefined, after: ContinuationCheckpoint | undefined, question?: Row) { + if (!question) return false; + const receipt = nativeApiQuestionReceipt(before?.runs.find(run => run.id === question.sourceRunId), question); + return !!receipt && isDeepStrictEqual(receipt, + nativeApiQuestionReceipt(after?.runs.find(run => run.id === question.sourceRunId), question)); +} + +/** Classify by server-owned identities and an applied native mutation receipt. A * missing receipt is unknown, never inferred from form shape or run count. */ export function questionPath(checkpoint: ContinuationCheckpoint | undefined, i?: Row): QuestionPath | null { if (!checkpoint || !i || !nonempty(i.id) || !nonempty(i.sourceRunId) || !nonempty(i.idempotencyKey)) return null; @@ -29,11 +48,13 @@ export function questionPath(checkpoint: ContinuationCheckpoint | undefined, i?: } const receipt = run.resultJson?.semanticToolReceipts?.[i.idempotencyKey]; const created = receipt?.result?.interaction; - return run.status === "succeeded" && i.continuationPolicy === "wake_assignee" && + if (run.status !== "succeeded" || i.continuationPolicy !== "wake_assignee") return null; + if ( receipt?.operationId === "request_human_input" && receipt.result?.disposition === "applied" && created?.id === i.id && created?.sourceRunId === run.id && created?.issueId === i.issueId && created?.companyId === i.companyId && created?.kind === "ask_user_questions" && - isDeepStrictEqual(created?.payload?.questionSet, i.payload?.questionSet) ? "semantic" : null; + isDeepStrictEqual(created?.payload?.questionSet, i.payload?.questionSet)) return "semantic"; + return nativeApiQuestionReceipt(run, i) ? "api" : null; } export function validResumeQuestionForm(checkpoint: ContinuationCheckpoint, i: Row, choice: boolean): boolean { @@ -62,14 +83,15 @@ export function questionResumedOnce(before: ContinuationCheckpoint | undefined, before.runs.some(r => !after.runs.some(n => n.id === r.id))) return false; if (path === "provider") return next.length === 0 && after.runs.some(r => r.id === i.sourceRunId && ["running", "succeeded"].includes(r.status)); - if (path !== "semantic" || next.length !== 1) return false; + if ((path !== "semantic" && path !== "api") || next.length !== 1) return false; + if (path === "api" && !sameNativeApiQuestionReceipt(before, after, i)) return false; const context = next[0].contextSnapshot; return context?.issueId === i.issueId && context?.sourceRunId === i.sourceRunId && context?.interactionId === i.id && context?.interactionKind === "ask_user_questions" && context?.interactionStatus === "answered"; } -/** Behavioral qualification supports either valid native path. It does not +/** Behavioral qualification supports each verified native path. It does not * replace the separate documentation-placement/semantic-tool qualification. */ export function gradeQuestionResume(checkpoints: ContinuationCheckpoint[], marker: string) { const initial = checkpoints.find(c => c.phase === "initial"); @@ -112,8 +134,9 @@ export function gradeQuestionResume(checkpoints: ContinuationCheckpoint[], marke detail: "The real board answer must remain bound to the unchanged question, with no fabricated or substituted response.", }, { id: `question-resume.${answer}.path`, - passed: questionResumedOnce(before, after, i), - detail: `Observed path: ${questionPath(before, i) ?? "unknown"}; provider answers resume the same run; semantic answers require one exactly bound response wake.`, + passed: questionResumedOnce(before, after, i) && + (questionPath(before, i) !== "api" || sameNativeApiQuestionReceipt(before, final, i)), + detail: `Observed path: ${questionPath(before, i) ?? "unknown"}; provider answers resume the same run; durable native questions require one exactly bound response wake.`, }); } checks.push({ diff --git a/tests/runner-e2e/question-resume-ui.ts b/tests/runner-e2e/question-resume-ui.ts new file mode 100644 index 00000000000..ac6fa3f9de8 --- /dev/null +++ b/tests/runner-e2e/question-resume-ui.ts @@ -0,0 +1,23 @@ +import { expect, type Page } from "@playwright/test"; +import { chatQuestionPresentation } from "./chat-flow.js"; + +type Row = Record; + +export async function answerResumeQuestionThroughUi(page: Page, interaction: Row, input: { choice: string } | { text: string }) { + const set = chatQuestionPresentation(interaction.payload); + if ("choice" in input) { + expect(set.questions[0]?.answerMode, "choices must use radio controls").toBe("single_select"); + const options = interaction.payload.questionSet?.questions[0]?.options ?? interaction.payload.questions[0]?.options ?? []; + expect(new Set(options.map((o: Row) => String(o.label).trim().toLowerCase())).size).toBeGreaterThanOrEqual(2); + const choice = input.choice.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); + await page.getByRole("radio", { name: new RegExp(`^${choice}\\b`, "i") }).last().click(); + } else { + expect(set.questions[0]?.answerMode, "open answers must render a text field, not a choice question").toBe("text"); + await page.getByTestId("question-text-answer-composer").last() + .locator('[contenteditable="true"],textarea').first().fill(input.text); + } + for (let index = 1; index < set.questions.length; index += 1) { + await page.getByRole("button", { name: "Next", exact: true }).last().click(); + } + await page.getByRole("button", { name: set.submitLabel ?? "Submit answers", exact: true }).last().click(); +} diff --git a/tests/runner-e2e/question-resume.test.ts b/tests/runner-e2e/question-resume.test.ts index 525bce0c199..bded4d1c8d2 100644 --- a/tests/runner-e2e/question-resume.test.ts +++ b/tests/runner-e2e/question-resume.test.ts @@ -1,4 +1,5 @@ import { describe, expect, it } from "vitest"; +import { readFileSync } from "node:fs"; import { gradeQuestionResume, questionPath, validResumeQuestionForm, type QuestionPath } from "./question-resume-scoring.js"; import { gradeContinuation } from "./continuation-scoring.js"; import { gradeLifecycleBaseline } from "./lifecycle-baseline.js"; @@ -26,7 +27,7 @@ function recording(paths: QuestionPath[] = ["provider", "semantic"]) { }; for (let n = 0; n < 2; n++) { const run = runs.at(-1)!; - const provider = paths[n] === "provider"; + const provider = paths[n] === "provider", api = paths[n] === "api"; const request = `request-${n}`; const idempotencyKey = provider ? `paperclip-runner-question:${run.id}:${request}` : `question-${n}`; const field = n === 0 ? { id: "field-1-question_0-hash", answerMode: "single_select", options: [ @@ -36,12 +37,15 @@ function recording(paths: QuestionPath[] = ["provider", "semantic"]) { if (provider && n === 0) questions.push({ id: "field-2-question_0_custom-hash", answerMode: "text", header: "Other", required: false }); const card: Row = { id: `card-${n}`, kind: "ask_user_questions", status: "pending", companyId: "company", issueId: "task", createdByAgentId: "agent", sourceRunId: run.id, idempotencyKey, - continuationPolicy: provider ? "none" : "wake_assignee", + continuationPolicy: provider ? "none" : "wake_assignee", createdByUserId: null, payload: { ...(provider ? { runtimeRequestId: request } : {}), questionSet: { questions } } }; cards.push(card); if (!provider) { run.status = "succeeded"; - run.resultJson.semanticToolReceipts[idempotencyKey] = { operationId: "request_human_input", + if (api) run.resultJson.apiToolReceipts = { ["a".repeat(64)]: { state: "completed", digest: "b".repeat(64), + operationId: "POST /api/issues/{id}/interactions", result: { ok: true, status: 201, + operationId: "POST /api/issues/{id}/interactions", data: structuredClone(card) } } }; + else run.resultJson.semanticToolReceipts[idempotencyKey] = { operationId: "request_human_input", result: { disposition: "applied", interaction: structuredClone(card) } }; } snap(n === 0 ? "initial" : "answered"); @@ -58,6 +62,58 @@ const failed = (r: any[]) => gradeQuestionResume(r, "AMBERnonce").filter(c => !c const question = (r: any[], n = 0) => r[n].interactions[n]; describe("native question/resume path qualification", () => { + it("accepts a native API creation receipt with the retained server response shape", () => { + const point = JSON.parse(readFileSync(new URL("./fixtures/native-api-question.json", import.meta.url), "utf8")); + expect(questionPath(point, point.interactions[0])).toBe("api"); + expect(validResumeQuestionForm(point, point.interactions[0], true)).toBe(true); + }); + it.each(["missing", "pending", "wrong-operation", "wrong-result-operation", "unknown-outcome", "wrong-status", "missing-digest", "wrong-key", + "foreign-card", "foreign-company", "foreign-issue", "foreign-agent", "foreign-source", "wrong-idempotency", "changed-form", "changed-compatibility-form", + "board-created", "resolved-creation", "wrong-policy", "runtime-request", "legacy-run", "active-source", "foreign-run", "duplicate-source", "spoofed-store"])("rejects unverified native API provenance: %s", kind => { + const point = JSON.parse(readFileSync(new URL("./fixtures/native-api-question.json", import.meta.url), "utf8")); + const card = point.interactions[0], run = point.runs[0], receipts = run.resultJson.apiToolReceipts; + const receipt = Object.values(receipts)[0] as Row, original = receipt.result.data; + if (kind === "missing") delete run.resultJson.apiToolReceipts; + if (kind === "pending") receipt.state = "pending"; + if (kind === "wrong-operation") receipt.operationId = "POST /api/issues/{id}/comments"; + if (kind === "wrong-result-operation") receipt.result.operationId = "POST /api/issues/{id}/comments"; + if (kind === "unknown-outcome") receipt.result.ok = false; + if (kind === "wrong-status") receipt.result.status = 200; + if (kind === "missing-digest") delete receipt.digest; + if (kind === "wrong-key") run.resultJson.apiToolReceipts = { "model-authored-claim": receipt }; + if (kind === "foreign-card") original.id = "other"; + if (kind === "foreign-company") original.companyId = "other"; + if (kind === "foreign-issue") original.issueId = "other"; + if (kind === "foreign-agent") original.createdByAgentId = "other"; + if (kind === "foreign-source") original.sourceRunId = "other"; + if (kind === "wrong-idempotency") original.idempotencyKey = "other"; + if (kind === "changed-form") original.payload.questionSet.questions[0].options[1].label = "Other"; + if (kind === "changed-compatibility-form") original.payload.questions[0].prompt = "Other"; + if (kind === "board-created") original.createdByUserId = "other"; + if (kind === "resolved-creation") original.status = "answered"; + if (kind === "wrong-policy") card.continuationPolicy = "none"; + if (kind === "runtime-request") card.payload.runtimeRequestId = "fabricated-provider-request"; + if (kind === "legacy-run") run.runtimeMode = "legacy"; + if (kind === "active-source") run.status = "running"; + if (kind === "foreign-run") run.agentId = "other"; + if (kind === "duplicate-source") point.runs.push(structuredClone(run)); + if (kind === "spoofed-store") { run.resultJson.modelClaim = receipts; delete run.resultJson.apiToolReceipts; } + expect(questionPath(point, card)).toBeNull(); + }); + it.each(["lost-receipt", "changed-receipt", "interactionId", "sourceRunId", "issueId", "interactionStatus", "duplicate-continuation"])("retains one immutable API receipt and one bound answer wake: %s", kind => { + const r = recording(["api", "api"]), source = r[1].runs[0], next = r[1].runs[1]; + if (kind === "lost-receipt") delete source.resultJson.apiToolReceipts; + else if (kind === "changed-receipt") source.resultJson.apiToolReceipts["a".repeat(64)].digest = "c".repeat(64); + else if (kind === "duplicate-continuation") r[1].runs.push({ ...structuredClone(next), id: "extra" }); + else next.contextSnapshot[kind] = "other"; + expect(failed(r)).toContain("question-resume.choice.path"); + }); + it.each(["lost", "changed"])("rejects the first API receipt %s after the second answer", kind => { + const r = recording(["api", "api"]), source = r[2].runs[0]; + if (kind === "lost") delete source.resultJson.apiToolReceipts; + else source.resultJson.apiToolReceipts["a".repeat(64)].digest = "c".repeat(64); + expect(failed(r)).toContain("question-resume.choice.path"); + }); it("keeps the new journey explicit-only and the original documentation cells intact", () => { const cells = runnerMatrix.filter(c => c.suite.id === "question-resume"); expect(cells.map(c => c.profile.id).sort()).toEqual(["runner-acpx-claude", "runner-codex"]); @@ -70,6 +126,7 @@ describe("native question/resume path qualification", () => { }); it.each([ ["provider", "provider"], ["provider", "semantic"], ["semantic", "provider"], ["semantic", "semantic"], + ["api", "api"], ["api", "semantic"], ["semantic", "api"], ["provider", "api"], ["api", "provider"], ] as QuestionPath[][])("accepts exactly bound %s then %s journeys", (a, b) => { const r = recording([a,b]); expect(failed(r)).toEqual([]); diff --git a/tests/runner-e2e/result-validation.ts b/tests/runner-e2e/result-validation.ts index 8399905bf7e..d85b195394e 100644 --- a/tests/runner-e2e/result-validation.ts +++ b/tests/runner-e2e/result-validation.ts @@ -1,4 +1,4 @@ -import { agentJourneyRuns, journeyRequiredChecks } from "./product-journeys/success-contract.js"; +import { agentJourneyRuns, agentJourneyRunCountValid, journeyRequiredChecks } from "./product-journeys/success-contract.js"; import { validateWorkerTiming } from "./worker-timing.js"; import { connectionCheckpoints, connectionEvidencePasses, connectionRunSignals } from "./connection-evidence.js"; import { githubJourneyCheckpoints, gradeGitHubReviewJourney } from "./github-review-evidence.js"; @@ -435,7 +435,7 @@ export function validateRetainedRunnerResult( const required = journeyRequiredChecks[checked.caseId]; if (evidence.outcome !== "passed" || !evidence.deployment || !required || !required.every(id => evidence.checks.some(c => c.id === id && c.passed)) || evidence.checks.some(check => !check.passed) || evidence.phases.some(p => p.outcome !== "passed") || checked.cleanup !== "passed") invalid("result.productJourney"); if (["task-properties", "task-create", "project-task-context", "task-search-filter"].includes(checked.caseId) && (checked.runtimeMode !== "none" || checked.provider !== null || checked.model !== null || checked.runIds?.length)) invalid("result.browserOnly"); - if (Object.hasOwn(agentJourneyRuns, checked.caseId) && (checked.runIds?.length !== agentJourneyRuns[checked.caseId as keyof typeof agentJourneyRuns] + if (Object.hasOwn(agentJourneyRuns, checked.caseId) && (!agentJourneyRunCountValid(checked.caseId, checked.runIds?.length) || !checked.model || !checked.provider || checked.runtimeMode !== "native")) invalid("result.agentJourneyRun"); if (checked.caseId === "cloud-onboarding" && (!checked.runIds?.length || !evidence.operations || !evidence.operations.requestCount || !evidence.operations.traceIds.length || evidence.operations.campaignId !== evidence.campaignId || evidence.operations.attemptId !== evidence.attemptId)) invalid("result.paidJourneyReceipt"); }