Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-10-31 22:24:16 +01:00
Code Issues Packages Projects Releases Wiki Activity

Add a portable agent question and answer resumption journey (#18346)

Browse Source 
## Thinking Path

> - Paperclip helps people manage agent work and keep human decisions
attached to tasks.
> - Product E2E journeys verify the browser, native agent, saved state,
and cleanup.
> - The classic suite already checks a choice question followed by a
text question.
> - Deployment-targeted journeys need the same behavior on an attached
instance.
> - Production questions can use provider input, the dedicated semantic
tool, or the authenticated native API fallback.
> - The basic journey must verify the original card and native
continuation for each supported path.
> - This pull request shares the scenario and browser actions, adds
strict API receipt evidence, and verifies the saved document after
reload.
> - Separate tool-documentation qualification and the original failed
measurements keep their existing scope and outcomes.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

It adds portable coverage for answering agent questions and resuming the
same task on an attached deployment.

**Current behavior**

The classic suite has the two-question scenario. The portable catalog
does not expose it. Its shared native-path grader recognizes provider
and dedicated semantic receipts, but rejects a valid completed native
API creation receipt.

**Proposed behavior**

Expose `question-answer-resume` as a deployment-targeted journey. Answer
one choice question and one text question in the browser. Require
server-owned creation evidence, unchanged forms, exact board answers,
bound continuation, an agent-authored task document, and readback after
reload.

**Reason and benefit**

The same story can qualify local and hosted targets through their normal
provider setup and cleanup. It catches lost answers, duplicate or
foreign work, premature output, and stale documents.

## What Changed

- Register the journey with a three-run maximum and positive model
allowance. Accept one to three attributed work runs at both collection
and retained-result boundaries.
- Share the existing two-question scenario, browser answer actions, and
behavioral grader. Keep separate tool-documentation grading unchanged.
- Recognize a completed native API creation receipt only when its HTTP
201 result retains the original pending card, exact
task/company/agent/source run, idempotency key, and full normalized
form. Require the receipt to remain unchanged after the answer and
through the final checkpoint. Missing, pending, foreign, or altered
proof still fails.
- Check signed-in board ownership, each native continuation, no
premature or extra output, document authorship, and the saved revision
after reload.
- Wait for the authenticated product region before screenshots. Recheck
origin and path after readiness. Bound readiness and capture by the
campaign deadline.
- Add closed stage-specific errors, catalog coverage, sanitized receipt
calibration, and documentation. Include the provider-form and
browser-question presentation helpers in the definition fingerprint.

## Verification

- Fresh QA staging qualification passed at
`7f7de8be286a436010a48af89aba1727b4451493`: all 11 required checks
passed, cleanup passed, and all three native runs used the configured QA
target and model. The [sanitized run
report](https://pages.paperclip.ing/evals-qa-smoke-41121966520-1/)
records the measured pass. This is one successful qualification, not a
reliability claim.
- Final integration head `ae0d9d289831c770a413658d3520fb1dc8ad60e3`
contains the one main merge through
`fd008979f5794fda3d5f54151d36acd31d0de140` and an exact cherry-pick of
the four-file qualified search fix
`4edd10804e2b42941aa2038e90c362c13a0b98dc` from #18370. Patch identity
matches. It preserves the project and search registrations and atomic
search readback. The question live qualification ran at the earlier SHA
above; the final integration head has separate local and CI
verification.
- Eighteen question journey, helper, and test file blobs are identical
between the qualified and integration SHAs. This includes clarification
flow and evidence, question grader and browser helper, the sanitized
receipt fixture, screenshot helper, provider-form helper, browser
question presentation, continuation scenario/readiness/flow, shared
agent validation, and user actions. The question run-count contract and
dispatch branch are also unchanged. Reviewers can compare those paths
directly between the two commits.
- All 622 selected Product E2E, catalog, question-resumption,
necessary-question, report, and readiness tests pass after the main
merge. After the exact search delta, all 145 narrow
question/search/project/contract tests and all nine real Chromium search
regressions pass. Map checks, combined credential-free preview, source
fingerprints, and the earlier workspace-link preflight pass. The local
TypeScript attempt reports stale Runner declaration errors for
`onAccountingEvent` in unchanged native-session code; fresh official CI
provides the clean-build type gate. The three search source/test blobs
match the qualified search merge, and the three project source blobs are
unchanged.
- An in-memory probe executes the actual launcher fingerprint
expression. Each project, search, and question implementation input
changes the digest. Separate red probes first reproduced the omitted
provider-form and browser presentation helpers; both pass after their
source inputs are added.
- The sanitized retained native API response-shape regression fails
before the grading correction and passes afterward. Negative controls
reject missing, pending, foreign, changed, spoofed, or wrongly bound
receipts, answers, continuation, model, environment, and document
evidence. All nine provider/semantic/API path combinations pass
calibration.
- The real local Chromium screenshot probe fails before the readiness
repair and passes afterward. Login redirects and sign-in pages at task
URLs remain excluded with no screenshot files.
- The two original attached failures remain retained: the first stopped
at early screenshot capture; the second stopped before answering at the
former API provenance gate. An earlier sandbox run also retained one
report subprocess failure; the full selected tests then passed with
local subprocess IPC access.
- The prior integration CI had an unchanged canary mock `EADDRINUSE`
failure. That history is retained; its documented retry is owned by the
coordinator. No application API, schema, production UI, model, runtime
policy, credential, or deployment configuration change is included.
Final integration head ae0d9d2898 passed
all 58 CI checks, the external Snyk status, and fresh Greptile 5/5 with
no unresolved findings. This clean CI includes the official TypeScript
and build gates. Recurring selection follows merge and a separate normal
TestOps job verification.

## Risks

An API result must prove native creation; form shape and run count
cannot establish it. The grader checks the server receipt, exact scope,
unchanged card and receipt, real board answers, and bound native
continuation. The shared answer helper still serves classic journeys.
The screenshot wait affects attached journeys and remains limited to the
authenticated product region and campaign deadline. One live QA pass
establishes the measured journey outcome at the qualified SHA. It does
not establish reliability across models or deployments. The integration
SHA preserves its journey and helper blobs, while shared registry
changes receive fresh CI and review.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, source inspection, and code
execution. The host does not expose the exact underlying model version
or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-10-30 16:18:21 -05:00
1 parent 56957f2bd9
commit d9354bbfc3
23 files changed
+789 -72
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

doc/evals.md
+10 -1
View File
@@ -61,6 +61,15 @@ self-hosted/Cloud targets and separately selected local/Daytona agents. Account
and credential bindings stay in the private eval repo; registration and oracle
tests do not establish a completed live pass or unattended CI qualification.
The deployment-targeted `question-answer-resume` journey reuses the classic
two-question scenario and native continuation oracle on an attached target.
It checks the actual board answers, one to three bound work runs, an agent-authored
task document and browser readback after reload. It accepts provider questions and
durable questions proven by an applied semantic receipt or an immutable completed
native API mutation receipt. It does not qualify named-tool selection. Its provider owns target setup,
usage and cleanup. Registration and local calibration do not establish a fresh
Cloud pass or inclusion in a recurring selection.
The explicit-only [necessary free-text question journey](../tests/runner-e2e/README.md#necessary-free-text-question)
checks one essential input, the rendered prompt and context, the real answer,
one bound continuation, and an exact saved confirmation. Its four development
@@ -547,7 +556,7 @@ any qualification claim; catalog presence alone is not a pass.
The explicit-only [native question/resume qualification](../tests/runner-e2e/README.md#task-continuation)
separates a completed two-answer user journey from semantic-tool documentation
qualification. It verifies the exact provider-pause or semantic-response-wake
qualification. It verifies the exact provider-pause or native response-wake
binding for each answer and preserves prior grades when the definition changes.
## Lifecycle behavior baseline
tests/runner-e2e/PRODUCT-JOURNEYS.md
+14
View File
@@ -26,6 +26,7 @@ never start a server or depend on a tenant filesystem.
| `task-create` | Company and signed-in board user | Create a task through the shared production composer, assign it to yourself, reopen and reload it, verify the saved prompt and no agent work |
| `project-task-context` | Company, signed-in board user, task-write capability; no model runs | Create a project and a self-assigned task with the project selected through the production dialogs. Reopen the project, open its task, reload both, and verify exact saved company, project, prompt, assignment, standard work mode and zero runs |
| `ask-question` | Company, configured agent capability, positive model allowance and exactly one permitted work run | Reuse the existing cross-model Ask question prompt; require one successful run with matching company/task/agent/model/environment, one saved exact answer, and visible readback after reload |
| `question-answer-resume` | Native agent; up to three work runs | Reuse the classic two-question story: answer a choice and a free-text question through the browser, verify each native continuation, then read the exact saved answers in an agent-authored task document after reload |
| `task-properties` | Company, signed-in board user, task-write capability, prepared run-owned backlog task | Open its properties, assign it to the signed-in user, reload, check both UI and saved assignment; verify no agent run |
| `task-search-filter` | Company, signed-in board user, task-write capability; zero model runs | Search three owned target/control tasks through the task list, require exact matching identities, filter Backlog to exclude the completed control, reload and return from the exact saved task with search/filter context intact |
| `plan-revise-accept` | Native agent; three work runs | Reuse the classic Plan story: canonical draft, browser revision request, new revision-bound confirmation, approval and exact saved final response |
@@ -35,6 +36,19 @@ never start a server or depend on a tenant filesystem.
| `stop-redirect` | Native agent; three work runs; qualified Docker oracle | Download a verified checkpoint, Stop an active revision, require native cancellation acknowledgement, submit a new direction once and verify completion without original-run activity |
| `cloud-onboarding` | Signed-in admitted Cloud account; fresh workspace; qualified bounded provider route | Create workspace in the portal, connect API key and native Codex agent in the wizard, complete a task, download and independently check its artifact |
The question-resumption journey uses the same natural-language scenario, browser
answer helper and behavioral grader as the classic `question-resume` suite. Native
provider questions can stay in one run; durable native questions create one exactly bound
continuation each. Creation must have an applied semantic receipt or a completed native
API mutation receipt with the original card, exact scope, form and idempotency key.
An API receipt must retain its creation identity unchanged after the answer. A form or
run count alone is never provenance. A target declares a three-run maximum and positive model allowance,
and a passing receipt must contain one to three attributed native runs. Other journeys
retain their exact run-count requirements. Both real board answers, no premature
output, the sole document's agent/company/task ownership and its reloaded revision
are independently checked. The private provider owns authentication, setup, accounting
and cleanup. Local calibration is not deployed-target qualification.
The shared Studio helper waits for the original saved file body before typing.
An editor control can mount before the file query completes; editing that empty
draft can otherwise lose the original content when the file arrives.
tests/runner-e2e/README.md
+10 -5
View File
@@ -192,8 +192,8 @@ The explicit-only `necessary-question` suite has four development cells: the
recorded Codex and Claude models, on local and Daytona environments. It requests
one essential booking reference that only the test user knows. The browser must
show the actual free-text prompt and its explanation, then submit the real answer.
The oracle checks the unchanged question, its server-owned provider or semantic
receipt, the exact stored user answer, and one bound continuation. The saved
The oracle checks the unchanged question, its server-owned provider, semantic or
native API receipt, the exact stored user answer, and one bound continuation. The saved
confirmation must contain that answer exactly. The final reply's link must open
that same task document and display its confirmation.
A title, success claim, replacement card, duplicate wake, or uncommitted result
@@ -1705,13 +1705,18 @@ placement; it is not a statistical comparison with the former prompt arrangement
The separate explicit-only `question-resume` suite uses the same neutral two-question
request on native Codex and ACPX Claude. It qualifies the user journey through
either the provider's built-in question or Paperclip's semantic question tool.
the provider's built-in question, Paperclip's semantic question tool, or its native API fallback.
It does not replace or regrade `question-tool-documentation`.
The oracle classifies each wait from its exact task/agent/run identity. Provider
questions require the server-generated request key and retain the paused run.
Semantic questions require an applied `request_human_input` receipt and one new
response wake bound to that interaction and source run. Only a verified provider
Durable questions require an applied `request_human_input` receipt or a completed
server-owned native `POST /api/issues/{id}/interactions` receipt. An API creation
receipt must report HTTP 201 and preserve the original pending card's exact task,
company, agent, source run, idempotency key and complete normalized form. It must
remain unchanged after the answer. Both durable paths require one new response
wake bound to that interaction and source run. Missing, pending, foreign or
mismatched receipts fail; form shape and run counts cannot replace them. Only a verified provider
choice may carry the adapter's optional Other field. A second substantive question
is still rejected; the subsequent reference question must be text-only. Both real
board answers must remain bound to unchanged forms, and one saved document must
tests/runner-e2e/catalog.test.ts
+1 -1
View File
@@ -59,7 +59,7 @@ describe("runner E2E catalog", () => {
});
it("exports deployment journeys separately from the model matrix and default selection", () => {
expect(productJourneys.map(journey => journey.id)).toEqual(["task-create", "project-task-context", "ask-question", "task-properties", "task-search-filter", "plan-revise-accept", "build-revise", "protected-actions", "create-skill-studio", "stop-redirect", "cloud-onboarding"]);
expect(productJourneys.map(journey => journey.id)).toEqual(["task-create", "project-task-context", "ask-question", "question-answer-resume", "task-properties", "task-search-filter", "plan-revise-accept", "build-revise", "protected-actions", "create-skill-studio", "stop-redirect", "cloud-onboarding"]);
expect(runnerSuites.some(suite => suite.id === "product-journeys")).toBe(false);
expect(runnerMatrix.some(execution => execution.suite.id === "product-journeys")).toBe(false);
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(execution => execution.suite.id === "product-journeys")).toBe(false);
tests/runner-e2e/continuation-flow.ts
+2 -23
View File
@@ -1,6 +1,7 @@
import { gradeNecessaryQuestion } from "./necessary-question.js";
import { necessaryQuestionContract } from "./continuation-cases.js";
import { validResumeQuestionForm } from "./question-resume-scoring.js";
import { answerResumeQuestionThroughUi } from "./question-resume-ui.js";
import { gradeLifecycleBaseline, type LifecycleCheckpoint } from "./lifecycle-baseline.js";
import { lifecycleLiveCase, lifecycleLiveContinuation, gradeLifecycleNarrative } from "./lifecycle-live-cases.js";
import { prepareLegacyContinuationSkill, prepareContinuationBudget } from "./continuation-fixtures.js";
@@ -224,29 +225,7 @@ export async function runContinuationFlow(input: {
expect(isSingleClaudeQuestion(set.questions), "one choice question with only the optional provider Other field").toBe(true);
} else expect(set.questions, "ask only the requested next question").toHaveLength(1);
const before = new Set(runs.map((r) => r.id));
if (choice) {
expect(set.questions[0].answerMode, "choices must use radio controls").toBe("single_select");
const options = questions[0].payload.questionSet?.questions[0]?.options
?? questions[0].payload.questions[0]?.options ?? [];
expect(new Set(options.map((o: Row) => String(o.label).trim().toLowerCase())).size).toBeGreaterThanOrEqual(2);
await page.getByRole("radio", { name: new RegExp(`^${choice}\\b`, "i") }).last().click();
} else {
expect(set.questions[0].answerMode, "open answers must render a text field, not a choice question").toBe("text");
await page.getByTestId("question-text-answer-composer").last()
.locator('[contenteditable="true"],textarea').first().fill(scenario.answer);
}
// Claude may add a separate optional Other field after its choice page.
// Navigate every rendered page before submitting; do not invent an answer.
for (let index = 1; index < set.questions.length; index += 1) {
await page.getByRole("button", { name: "Next", exact: true }).last().click();
}
await page
.getByRole("button", {
name: set.submitLabel ?? "Submit answers",
exact: true,
})
.last()
.click();
await answerResumeQuestionThroughUi(page, questions[0], choice ? { choice } : { text: scenario.answer });
await settle(before, false, questions[0].id);
}
async function reply(body: string) {
tests/runner-e2e/fixtures/native-api-question.json
+152
View File
@@ -0,0 +1,152 @@
{
"phase": "initial",
"issue": {
"id": "00000000-0000-4000-8000-000000000002",
"companyId": "00000000-0000-4000-8000-000000000001",
"assigneeAgentId": "00000000-0000-4000-8000-000000000003",
"status": "in_review"
},
"runs": [
{
"id": "00000000-0000-4000-8000-000000000004",
"companyId": "00000000-0000-4000-8000-000000000001",
"issueId": "00000000-0000-4000-8000-000000000002",
"agentId": "00000000-0000-4000-8000-000000000003",
"status": "succeeded",
"runtimeMode": "native",
"resultJson": {
"apiToolReceipts": {
"bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb": {
"state": "completed",
"digest": "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc",
"operationId": "POST /api/issues/{id}/interactions",
"result": {
"ok": true,
"status": 201,
"contentType": "application/json; charset=utf-8",
"operationId": "POST /api/issues/{id}/interactions",
"data": {
"kind": "ask_user_questions",
"status": "pending",
"continuationPolicy": "wake_assignee",
"payload": {
"version": 1,
"supersedeOnUserComment": false,
"questions": [
{
"id": "time_choice",
"prompt": "Should the club meetup welcome note reference Morning or Afternoon?",
"selectionMode": "single",
"required": true,
"allowOther": false,
"options": [
{
"id": "morning",
"label": "Morning"
},
{
"id": "afternoon",
"label": "Afternoon"
}
]
}
],
"questionSet": {
"schema": "paperclip.question_set.v1",
"questions": [
{
"id": "time_choice",
"prompt": "Should the club meetup welcome note reference Morning or Afternoon?",
"required": true,
"answerMode": "single_select",
"options": [
{
"id": "morning",
"label": "Morning"
},
{
"id": "afternoon",
"label": "Afternoon"
}
]
}
]
}
},
"id": "00000000-0000-4000-8000-000000000005",
"sourceRunId": "00000000-0000-4000-8000-000000000004",
"companyId": "00000000-0000-4000-8000-000000000001",
"issueId": "00000000-0000-4000-8000-000000000002",
"createdByAgentId": "00000000-0000-4000-8000-000000000003",
"createdByUserId": null,
"idempotencyKey": "meetup-time-choice-api-v1"
}
}
}
}
}
}
],
"interactions": [
{
"kind": "ask_user_questions",
"status": "pending",
"continuationPolicy": "wake_assignee",
"payload": {
"version": 1,
"supersedeOnUserComment": false,
"questions": [
{
"id": "time_choice",
"prompt": "Should the club meetup welcome note reference Morning or Afternoon?",
"selectionMode": "single",
"required": true,
"allowOther": false,
"options": [
{
"id": "morning",
"label": "Morning"
},
{
"id": "afternoon",
"label": "Afternoon"
}
]
}
],
"questionSet": {
"schema": "paperclip.question_set.v1",
"questions": [
{
"id": "time_choice",
"prompt": "Should the club meetup welcome note reference Morning or Afternoon?",
"required": true,
"answerMode": "single_select",
"options": [
{
"id": "morning",
"label": "Morning"
},
{
"id": "afternoon",
"label": "Afternoon"
}
]
}
]
}
},
"id": "00000000-0000-4000-8000-000000000005",
"sourceRunId": "00000000-0000-4000-8000-000000000004",
"companyId": "00000000-0000-4000-8000-000000000001",
"issueId": "00000000-0000-4000-8000-000000000002",
"createdByAgentId": "00000000-0000-4000-8000-000000000003",
"createdByUserId": null,
"idempotencyKey": "meetup-time-choice-api-v1"
}
],
"documents": [],
"attachments": [],
"children": [],
"comments": []
}
tests/runner-e2e/product-journeys/agent-flows.ts
+1 -1
View File
@@ -18,7 +18,7 @@ import { agentJourneyRuns } from "./catalog.js";
import { JourneyError, type JourneyContext, type JourneyEvidence, type TargetSession } from "./types.js";
type Row = Record<string, any>;
type AgentJourney = Exclude<keyof typeof agentJourneyRuns, "ask-question">;
type AgentJourney = Exclude<keyof typeof agentJourneyRuns, "ask-question" | "question-answer-resume">;
interface TaskState { currentIssue: Row & { id: string; status: string }; taskRuns: Array<Row & { status: string }>; comments: Row[]; interactions: Array<Row & { id: string; status: string }>; attachments: StoryDelivery[] }
interface Options { deadlineAt: number; directory: string; capture(id: string, label: string, file: string): Promise<void> }
const active = (run: Row) => ["queued", "running"].includes(run.status);
tests/runner-e2e/product-journeys/campaign.test.ts
+1 -1
View File
@@ -46,7 +46,7 @@ async function run(options:{search?:boolean;authFailure?:boolean;onboardingFailu
await writeFile(module, `import {readFile,writeFile} from 'node:fs/promises';
const identity={kind:'cloud',targetId:'qa',origin:'https://qa.example.test',commit:'${"a".repeat(40)}',cloudRelease:'v1',runtimeImage:'image',configurationHash:'config'};
let onboarded=false;
const context={identity,company:{id:'company-1',issuePrefix:'QA'},api:{origin:identity.origin,post:async()=>({id:'issue-fixture-1',identifier:'QA-1'})},startingState:'onboarding',capabilities:['company','board-user','task-write'],page:{url:()=>identity.origin+'/QA/issues/QA-1',locator:()=>({isVisible:async()=>true,screenshot:async()=>{}}) }};
const context={identity,company:{id:'company-1',issuePrefix:'QA'},api:{origin:identity.origin,post:async()=>({id:'issue-fixture-1',identifier:'QA-1'})},startingState:'onboarding',capabilities:['company','board-user','task-write'],page:{url:()=>identity.origin+'/QA/issues/QA-1',locator:()=>({waitFor:async()=>{},isVisible:async()=>true,screenshot:async()=>{}}) }};
export const provider={version:1, async acquire(input){return {
async authenticate(){${options.authFailure ? "throw {outcome:'failed',code:'login_code_expired'};" : ""}},
async onboard(evidence){${options.preAcceptanceFailure ? `evidence.onboardingPreAcceptance=${JSON.stringify(preAcceptanceEvidence)};throw {outcome:'failed',code:'onboarding_work_before_acceptance'};` : options.onboardingFailure ? "throw {outcome:'failed',code:'onboarding_failed'};" : `${options.missingArtifact ? "" : `evidence.artifact=${JSON.stringify(artifact)};`}evidence.checks.push(...['fresh-workspace','ui-connection-and-native-agent','first-task-complete','downloadable-artifact-verified'${options.missingCheck ? "" : ",'native-trace-verified'"}].map(id=>({id,passed:true})));onboarded=true;return context;`}},
tests/runner-e2e/product-journeys/catalog.ts
+1
View File
@@ -5,6 +5,7 @@ export const productJourneys = [
{ id: "task-create", label: "Create a task for yourself and reopen it", requires: ["company", "board-user", "task-write"], capabilities: ["tasks/create"], paid: false },
{ id: "project-task-context", label: "Create a project and reopen its self-assigned task", requires: ["company", "board-user", "task-write"], capabilities: ["projects/lifecycle", "projects/task-context", "tasks/create"], paid: false },
{ id: "ask-question", label: "Ask an agent a question and read its answer", requires: ["company", "board-user", "task-write", "agent"], capabilities: ["questions-and-approvals/questions"], paid: true },
{ id: "question-answer-resume", label: "Answer agent questions and verify the saved result", requires: ["company", "board-user", "task-write", "agent"], capabilities: ["questions-and-approvals/questions"], paid: true },
{ id: "task-properties", label: "Assign a task to the signed-in user", requires: ["company", "board-user", "task-write", "backlog-task"], capabilities: ["tasks/properties"], paid: false },
{ id: "task-search-filter", label: "Search and filter tasks, then reopen the exact saved result", requires: ["company", "board-user", "task-write"], capabilities: ["tasks/organize"], paid: false },
{ id: "plan-revise-accept", label: "Revise and approve a saved Plan, then complete the task", requires: ["company", "board-user", "task-write", "agent"], capabilities: ["questions-and-approvals/plan-review"], paid: true },
tests/runner-e2e/product-journeys/clarification-campaign.test.ts
+59
View File
@@ -0,0 +1,59 @@
import { afterEach, expect, it, vi } from "vitest";
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
import path from "node:path";
import os from "node:os";
import { journeyRequiredChecks } from "./success-contract.js";
vi.mock("@playwright/test", () => ({ chromium: { launch: async () => ({
newContext: async () => ({ setDefaultTimeout() {}, pages: () => [] }), close: async () => {},
}) }, expect: {} }));
vi.mock("./clarification-flow.js", () => ({ runClarificationJourney: async (_context: unknown, _session: unknown,
_campaignId: unknown, evidence: { checks: Array<{ id: string; passed: boolean }> }) => {
evidence.checks.push(...journeyRequiredChecks["question-answer-resume"]!.map(id => ({ id, passed: true })));
} }));
vi.mock("../dashboard-regenerate.js", () => ({ regenerateRunnerDashboard: async ({ outputDirectory }: { outputDirectory: string }) => {
await Promise.all(["index.html", "dashboard.html"].map(file => writeFile(path.join(outputDirectory, file), "dashboard")));
} }));
import { launchProductJourneys } from "./launch.js";
const roots: string[] = [];
afterEach(async () => { process.exitCode = 0; await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true }))); });
async function campaign(count: number, omitReceipt = false) {
const root = await mkdtemp(path.join(os.tmpdir(), "clarification-collection-")); roots.push(root);
const module = path.join(root, "provider.mjs"), output = path.join(root, "report");
const runIds = Array.from({ length: count }, (_, i) => `bound-run-${i}`);
const receipts = omitReceipt ? [] : [{ journeyId: "question-answer-resume", runIds,
usage: { runs: runIds.map(runId => ({ runId, usage: { costUsd: 0.01 } })) } }];
await writeFile(module, `
const identity = { kind: 'cloud', targetId: 'qa', origin: 'https://qa.example.test', commit: '${"a".repeat(40)}', cloudRelease: 'v1', runtimeImage: 'image', configurationHash: 'config' };
const context = { identity, company: { id: 'company', issuePrefix: 'QA' }, user: { id: 'board', name: 'QA' },
api: { origin: identity.origin }, startingState: 'recipe', capabilities: ['company', 'board-user', 'task-write', 'agent'],
agent: { id: 'agent', name: 'QA', provider: 'opencode', model: 'model', runtimeMode: 'native' }, environmentId: 'daytona',
page: { url: () => identity.origin + '/QA/issues/QA-1', locator: () => ({ waitFor: async () => {}, isVisible: async () => true, screenshot: async () => {} }) } };
export const provider = { version: 1, async acquire() { return {
async authenticate() {}, async prepare() { return context; }, async identity() { return identity; },
async collect() { return ${JSON.stringify(receipts)}; }, async own() {}, async release() { return 'released'; }, async stop() {}
}; } };`);
const file = path.join(root, "target.json");
await writeFile(file, JSON.stringify({ version: 1, id: "qa", mode: "existing-cloud", adapterModule: module,
journeys: ["question-answer-resume"], executionEnvironment: "daytona", baseline: { features: {}, connections: [] },
limits: { durationMs: 60_000, modelBudgetCents: 100, maxRuns: 3 } }));
await launchProductJourneys(["--target-config", file, "--output", output], path.resolve(import.meta.dirname, "../../.."));
return JSON.parse(await readFile(path.join(output, "normalized-results.json"), "utf8")).results[0];
}
it.each([1, 2, 3])("collects a successful clarification journey with %i native work runs", async count => {
const result = await campaign(count);
expect(result).toMatchObject({ status: "passed", cleanup: "passed", productJourney: { outcome: "passed" } });
expect(result.runIds).toHaveLength(count);
expect(result.productJourney.phases).toContainEqual({ phase: "collection", outcome: "passed" });
expect(result.usage.runs).toHaveLength(count);
});
it.each([0, 4])("rejects a clarification collection with %i native work runs", async count => {
const result = await campaign(count);
expect(result).toMatchObject({ status: "failed", error: "paid_journey_receipt_missing" });
expect(result.runIds).toHaveLength(count);
});
it("rejects a passing story when its provider omits collection entirely", async () => {
expect(await campaign(1, true)).toMatchObject({ status: "failed", error: "paid_journey_receipt_missing" });
});
tests/runner-e2e/product-journeys/clarification-evidence.test.ts
+122
View File
@@ -0,0 +1,122 @@
import { describe, expect, it } from "vitest";
import { parseTargetConfig, productJourneys } from "./catalog.js";
import { agentJourneyRunCountValid } from "./success-contract.js";
describe("portable clarification journey admission", () => {
it("declares the existing two-question story as a bounded native journey", () => {
expect(productJourneys.find(j => j.id === "question-answer-resume")).toMatchObject({ paid: true, requires: ["company", "board-user", "task-write", "agent"] });
const config = { version: 1, id: "attached", mode: "existing-cloud", adapterModule: "/private/provider.mjs",
executionEnvironment: "daytona", journeys: ["question-answer-resume"], baseline: { features: {}, connections: [] },
limits: { durationMs: 600_000, modelBudgetCents: 500, maxRuns: 3 } };
expect(parseTargetConfig(config).journeys).toEqual(["question-answer-resume"]);
for (const maxRuns of [0, 1, 2, 4]) expect(() => parseTargetConfig({ ...config, limits: { ...config.limits, maxRuns } })).toThrow();
});
it("admits exactly the provider/semantic bounds without changing existing exact-count stories", () => {
for (const count of [1, 2, 3]) expect(agentJourneyRunCountValid("question-answer-resume", count)).toBe(true);
for (const count of [undefined, 0, 4, 1.5]) expect(agentJourneyRunCountValid("question-answer-resume", count)).toBe(false);
expect(agentJourneyRunCountValid("plan-revise-accept", 2)).toBe(false);
expect(agentJourneyRunCountValid("plan-revise-accept", 3)).toBe(true);
});
});
import { clarificationQuestionFailure, gradeClarificationEvidence } from "./clarification-evidence.js";
import { continuationScenario } from "../continuation-cases.js";
import { validateRetainedRunnerResult } from "../result-validation.js";
import { journeyRequiredChecks } from "./success-contract.js";
import type { JourneyContext } from "./types.js";
type Row = Record<string, any>;
const context = { company: { id: "company" }, user: { id: "board" }, environmentId: "daytona",
agent: { id: "agent", provider: "opencode", model: "model", runtimeMode: "native" } } as JourneyContext;
function recording(paths: Array<"provider" | "semantic" | "api">) {
const runs: Row[] = [], cards: Row[] = [], checkpoints: any[] = [];
const createRun = (previous?: Row) => ({ id: `run-${runs.length}`, status: "running", runtimeMode: "native", companyId: "company", issueId: "task", agentId: "agent",
contextSnapshot: { paperclipEnvironment: { id: "daytona" }, ...(previous ? { issueId: "task", sourceRunId: previous.sourceRunId,
interactionId: previous.id, interactionKind: "ask_user_questions", interactionStatus: "answered" } : {}) },
resultJson: { semanticToolReceipts: {} }, runnerProfileJson: { nativeExecutionInput: { task: { prompt: continuationScenario("question-answer-resume", "nonce").prompt }, provider: { kind: "opencode", model: "model" } } } });
const document = { issueId: "task", companyId: "company", key: "welcome-note", latestRevisionId: "revision", createdByAgentId: "agent", updatedByAgentId: "agent", body: "Welcome to the afternoon meetup. Use AMBERnonce." };
const snapshot = (phase: string) => checkpoints.push(structuredClone({ phase,
issue: { id: "task", companyId: "company", assigneeAgentId: "agent", status: phase === "final" ? "done" : "in_review" },
runs, interactions: cards, children: [], attachments: [], comments: [], documents: phase === "final" ? [document] : [] }));
runs.push(createRun());
for (const [index, path] of paths.entries()) {
const run = runs.at(-1)!;
const field = index === 0 ? { id: "choice", answerMode: "single_select", options: [{ id: "am", label: "Morning" }, { id: "pm", label: "Afternoon" }] } : { id: "reference", answerMode: "text" };
const request = `request-${index}`, semantic = path !== "provider";
const card: Row = { id: `question-${index}`, kind: "ask_user_questions", status: "pending", companyId: "company", issueId: "task", createdByAgentId: "agent", sourceRunId: run.id,
idempotencyKey: semantic ? `semantic-${index}` : `paperclip-runner-question:${run.id}:${request}`, continuationPolicy: semantic ? "wake_assignee" : "none", createdByUserId: null,
payload: { ...(semantic ? {} : { runtimeRequestId: request }), questionSet: { questions: [field] } } };
cards.push(card);
if (semantic) {
run.status = "succeeded";
if (path === "api") run.resultJson.apiToolReceipts = { ["a".repeat(64)]: { state: "completed", digest: "b".repeat(64),
operationId: "POST /api/issues/{id}/interactions", result: { ok: true, status: 201,
operationId: "POST /api/issues/{id}/interactions", data: structuredClone(card) } } };
else run.resultJson.semanticToolReceipts[card.idempotencyKey] = { operationId: "request_human_input", result: { disposition: "applied", interaction: structuredClone(card) } };
}
snapshot(index === 0 ? "initial" : "answered");
card.status = "answered"; card.resolvedByUserId = "board";
card.result = { answers: [{ questionId: field.id, optionIds: index === 0 ? ["pm"] : [], ...(index === 1 ? { otherText: "Use AMBERnonce." } : {}) }] };
if (semantic) runs.push(createRun(card));
}
runs.at(-1)!.status = "succeeded"; snapshot("final");
return { checkpoints, readback: { issueId: "task", key: document.key, revisionId: document.latestRevisionId, body: document.body, visibleBody: document.body } };
}
const failures = (f: ReturnType<typeof recording>) => gradeClarificationEvidence(context, "task", "AMBERnonce", f.checkpoints, f.readback).filter(c => !c.passed);
describe("attached clarification evidence", () => {
it.each([true, false])("rejects an incomplete API receipt instead of inferring native question evidence (choice=%s)", choice => {
const f = recording(["semantic", "semantic"]), point = f.checkpoints[choice ? 0 : 1];
const question = point.interactions.find((i: Row) => i.status === "pending"), source = point.runs.at(-1);
source.resultJson = { apiToolReceipts: { call: { state: "completed", operationId: "POST /api/issues/{id}/interactions",
result: { ok: true, data: { id: question.id } } } } };
expect(clarificationQuestionFailure(point, question, choice)).toBe(`clarification_${choice ? "choice" : "text"}_api_fallback`);
expect(failures(f).some(check => check.id === `question-resume.${choice ? "choice" : "text"}.path`)).toBe(true);
delete source.resultJson.apiToolReceipts;
expect(clarificationQuestionFailure(point, question, choice)).toBe(`clarification_${choice ? "choice" : "text"}_native_path_invalid`);
});
it("distinguishes an invalid form from a correctly bound semantic question", () => {
const point = recording(["semantic", "semantic"]).checkpoints[0], question = point.interactions[0];
expect(clarificationQuestionFailure(point, question, true)).toBeUndefined();
question.payload.questionSet.questions[0].answerMode = "text";
point.runs[0].resultJson.semanticToolReceipts[question.idempotencyKey].result.interaction.payload = structuredClone(question.payload);
expect(clarificationQuestionFailure(point, question, true)).toBe("clarification_choice_form_invalid");
});
it.each([["provider", "provider"], ["provider", "semantic"], ["semantic", "provider"], ["semantic", "semantic"],
["api", "api"], ["api", "semantic"], ["semantic", "api"], ["provider", "api"], ["api", "provider"]] as const)("accepts the real %s/%s paths with native binding and reload proof", (...paths) => {
expect(failures(recording([...paths]))).toEqual([]);
});
it.each(["foreign-user", "foreign-agent-document", "foreign-task-document", "lost-revision", "wrong-visible-answer", "changed-readback-body", "premature-document", "extra-child", "foreign-model", "foreign-environment", "missing-native-input", "duplicate-question", "wrong-answer"])("rejects a plausible false pass: %s", kind => {
for (const path of ["semantic", "api"] as const) {
const f = recording([path, path]), first = f.checkpoints[0], final = f.checkpoints[2];
if (kind === "foreign-user") final.interactions[1].resolvedByUserId = "other-board";
if (kind === "foreign-agent-document") final.documents[0].updatedByAgentId = "other-agent";
if (kind === "foreign-task-document") final.documents[0].issueId = "other-task";
if (kind === "lost-revision") f.readback.revisionId = "old-revision";
if (kind === "wrong-visible-answer") f.readback.visibleBody = "Welcome to the morning meetup.";
if (kind === "changed-readback-body") f.readback.body += " changed";
if (kind === "premature-document") first.documents = [final.documents[0]];
if (kind === "extra-child") final.children.push({ id: "extra-child" });
if (kind === "foreign-model") final.runs[0].runnerProfileJson.nativeExecutionInput.provider.model = "other-model";
if (kind === "foreign-environment") final.runs[0].contextSnapshot.paperclipEnvironment.id = "local";
if (kind === "missing-native-input") delete final.runs[0].runnerProfileJson.nativeExecutionInput.task.prompt;
if (kind === "duplicate-question") final.interactions.push(structuredClone(final.interactions[1]));
if (kind === "wrong-answer") final.interactions[1].result.answers[0].otherText = "Use the old reference.";
expect(failures(f).length).toBeGreaterThan(0);
}
});
it("requires reload evidence even for a correct API document", () => {
const f = recording(["provider", "provider"]);
expect(gradeClarificationEvidence(context, "task", "AMBERnonce", f.checkpoints).some(c => c.id === "question-document-reloaded" && !c.passed)).toBe(true);
});
it("retains 1–3 bound native runs but rejects zero and excess work", () => {
const result = { schema: "paperclip.runner-e2e.result/v2", executionId: "product-journeys.attached.configured-agent.daytona.question-answer-resume", suiteId: "product-journeys", suiteDefinitionHash: "definition",
attempt: 1, status: "passed", profileId: "configured-agent", environmentId: "daytona", caseId: "question-answer-resume", provider: "opencode", model: "model", runtimeMode: "native",
startedAt: "2026-10-01T00:00:00Z", finishedAt: "2026-10-01T00:00:01Z", durationMs: 1000, cleanup: "passed", runIds: ["run-0"],
productJourney: { targetId: "attached", outcome: "passed", startingState: "recipe", privacy: "private", resourceDisposition: "retained",
phases: [{ phase: "journey", outcome: "passed" }], checks: journeyRequiredChecks["question-answer-resume"]!.map(id => ({ id, passed: true })),
deployment: { kind: "cloud", targetId: "attached", origin: "https://example.test", commit: "a".repeat(40), cloudRelease: "b".repeat(40), runtimeImage: "immutable-image", configurationHash: "config" } } };
for (const count of [1, 2, 3]) expect(() => validateRetainedRunnerResult({ ...result, runIds: Array.from({ length: count }, (_, i) => `run-${i}`) })).not.toThrow();
for (const count of [0, 4]) expect(() => validateRetainedRunnerResult({ ...result, runIds: Array.from({ length: count }, (_, i) => `run-${i}`) })).toThrow("result.agentJourneyRun");
});
});
tests/runner-e2e/product-journeys/clarification-evidence.ts
+57
View File
@@ -0,0 +1,57 @@
import { gradeQuestionResume, questionPath, validResumeQuestionForm } from "../question-resume-scoring.js";
import type { ContinuationCheckpoint } from "../continuation-scoring.js";
import { validateAgentTaskState } from "./agent-flows.js";
import type { JourneyContext } from "./types.js";
type Row = Record<string, any>;
export interface ClarificationReadback {
issueId: string; key: string; revisionId: string; body: string; visibleBody: string;
}
export function clarificationQuestionFailure(point: ContinuationCheckpoint, question: Row, choice: boolean) {
const stage = choice ? "choice" : "text";
if (!questionPath(point, question)) {
const source = point.runs.find(run => run.id === question.sourceRunId) as Row | undefined;
const apiCreated = Object.values(source?.resultJson?.apiToolReceipts ?? {}).some(value => {
const receipt = value as Row;
return receipt.state === "completed" && receipt.operationId === "POST /api/issues/{id}/interactions"
&& receipt.result?.ok === true && receipt.result?.data?.id === question.id;
});
return apiCreated ? `clarification_${stage}_api_fallback` : `clarification_${stage}_native_path_invalid`;
}
return validResumeQuestionForm(point, question, choice) ? undefined : `clarification_${stage}_form_invalid`;
}
export function gradeClarificationEvidence(context: JourneyContext, issueId: string, marker: string,
checkpoints: ContinuationCheckpoint[], readback?: ClarificationReadback) {
const checks = gradeQuestionResume(checkpoints, marker);
const final = checkpoints.find(point => point.phase === "final");
const initial = checkpoints.filter(point => point.phase !== "final");
const document = final?.documents[0] as Row | undefined;
let bound = !!context.agent && context.agent.runtimeMode === "native" && !!context.environmentId;
try {
for (const point of checkpoints) validateAgentTaskState(context, issueId, {
currentIssue: point.issue as Row & { id: string; status: string }, taskRuns: point.runs,
comments: point.comments as Row[], interactions: point.interactions as Array<Row & { id: string; status: string }>, attachments: [],
}, 3);
} catch { bound = false; }
checks.push({ id: "question-no-premature-output", passed: initial.length === 2 && initial.every(point =>
point.documents.length === 0 && point.attachments.length === 0 && point.children.length === 0 && point.issue.status !== "done")
&& !!final && final.issue.status === "done" && final.attachments.length === 0 && final.children.length === 0,
detail: "Wait for both real answers before producing the sole task document, with no extra task or download." },
{ id: "question-answers-owned-by-board", passed: !!final && final.interactions.length === 2 && final.interactions.every(value => {
const question = value as Row;
return question.status === "answered" && question.resolvedByUserId === context.user.id && !question.resolvedByAgentId;
}), detail: "Both retained responses must come from the signed-in board user." },
{ id: "question-document-attributed", passed: !!document && document.issueId === issueId && document.companyId === context.company.id
&& document.createdByAgentId === context.agent?.id && document.updatedByAgentId === context.agent?.id,
detail: "The owned native agent created and last wrote the document on this exact task and company." },
{ id: "question-document-reloaded", passed: !!document && !!readback && readback.issueId === issueId
&& readback.key === document.key && readback.revisionId === document.latestRevisionId && readback.body === document.body
&& readback.visibleBody.includes(`Use ${marker}.`) && /\bafternoon\b/i.test(readback.visibleBody),
detail: "The reloaded document must show both saved answers at the independently read revision." },
{ id: "configured-model-and-environment", passed: bound && checkpoints.length === 3 && !!final
&& final.runs.length >= 1 && final.runs.length <= 3 && final.runs.every(run => run.status === "succeeded"),
detail: "Every observed run belongs to the configured native agent, task, company, model and environment." });
return checks;
}
tests/runner-e2e/product-journeys/clarification-flow.test.ts
+35
View File
@@ -0,0 +1,35 @@
import { expect, it, vi } from "vitest";
import { readClarificationState } from "./clarification-flow.js";
import type { JourneyContext } from "./types.js";
const issueId = "11111111-1111-4111-8111-111111111111";
const companyId = "22222222-2222-4222-8222-222222222222";
const parent = { id: issueId, companyId, parentId: null, assigneeAgentId: "agent", status: "in_progress" };
const child = { id: "child", companyId, parentId: issueId };
function fixture(rows = [parent, child]) {
const get = vi.fn(async (route: string) => {
const url = new URL(route, "https://qa.example.test");
if (url.pathname === `/api/companies/${companyId}/issues`) {
if (url.searchParams.get("view") !== "index") throw Error("unsupported child projection");
return rows.filter(row => row.parentId === url.searchParams.get("parentIds"));
}
if (url.pathname === `/api/issues/${issueId}`) return parent;
return [];
});
const context = { company: { id: companyId }, environmentId: "daytona", api: { get },
agent: { id: "agent", provider: "opencode", model: "model", runtimeMode: "native" } } as unknown as JourneyContext;
return { context, get };
}
it("reads the supported child projection while excluding the journey task and unrelated company work", async () => {
const { context, get } = fixture([parent, child, { id: "unrelated", companyId, parentId: "other" }]);
expect((await readClarificationState(context, issueId)).children).toEqual([child]);
expect(get).toHaveBeenCalledWith(`/api/companies/${companyId}/issues?view=index&parentIds=${issueId}&limit=2`);
});
it("retains an empty child result for a task that did not delegate work", async () => {
expect((await readClarificationState(fixture([parent]).context, issueId)).children).toEqual([]);
});
it.each([{ ...child, companyId: "other" }, { ...child, parentId: "other" }])("rejects a returned child outside the owned company or task", async foreign => {
const { context, get } = fixture();
get.mockImplementation(async route => route.includes("/companies/") ? [foreign] : route === `/api/issues/${issueId}` ? parent : []);
await expect(readClarificationState(context, issueId)).rejects.toMatchObject({ code: "clarification_child_query_mismatch" });
});
tests/runner-e2e/product-journeys/clarification-flow.ts
+110
View File
@@ -0,0 +1,110 @@
import { mkdir, writeFile } from "node:fs/promises";
import path from "node:path";
import { expect } from "@playwright/test";
import { continuationScenario } from "../continuation-cases.js";
import { continuationAnswerCommitted, continuationCheckpointReady } from "../continuation-readiness.js";
import type { ContinuationCheckpoint } from "../continuation-scoring.js";
import { answerResumeQuestionThroughUi } from "../question-resume-ui.js";
import { createTaskThroughUi } from "../user-actions.js";
import { validateAgentTaskState } from "./agent-flows.js";
import { clarificationQuestionFailure, gradeClarificationEvidence, type ClarificationReadback } from "./clarification-evidence.js";
import { JourneyError, type JourneyContext, type JourneyEvidence, type TargetSession } from "./types.js";
type Row = Record<string, any>;
interface Options { deadlineAt: number; directory: string; capture(id: string, label: string, file: string): Promise<void> }
export async function readClarificationState(context: JourneyContext, issueId: string) {
const route = `/api/issues/${issueId}`;
const [currentIssue, summaries, interactions, documents, attachments, children] = await Promise.all([
context.api.get<Row>(route), context.api.get<Row[]>(`${route}/runs`), context.api.get<Row[]>(`${route}/interactions`),
context.api.get<Row[]>(`${route}/documents`), context.api.get<Row[]>(`${route}/attachments`),
context.api.get<Row[]>(`/api/companies/${context.company.id}/issues?view=index&parentIds=${encodeURIComponent(issueId)}&limit=2`),
]);
if (!Array.isArray(summaries) || summaries.length > 3) throw new JourneyError("failed", "clarification_run_limit_exceeded");
const runs = await Promise.all(summaries.map(summary => context.api.get<Row>(`/api/heartbeat-runs/${summary.runId}`)));
validateAgentTaskState(context, issueId, { currentIssue: currentIssue as Row & { id: string; status: string },
taskRuns: runs as Array<Row & { status: string }>, interactions: interactions as Array<Row & { id: string; status: string }>, comments: [], attachments: [] }, 3);
if (!Array.isArray(children) || children.some(child => child.parentId !== issueId || child.companyId !== context.company.id)) {
throw new JourneyError("failed", "clarification_child_query_mismatch");
}
return { currentIssue, runs, interactions, documents, attachments, children };
}
export async function runClarificationJourney(context: JourneyContext, session: TargetSession,
campaignId: string, evidence: JourneyEvidence, options: Options) {
if (!context.agent || context.agent.runtimeMode !== "native" || !context.environmentId) {
throw new JourneyError("unsupported", "native_agent_and_environment_required");
}
const agent = context.agent, scenario = continuationScenario("question-answer-resume", campaignId.replaceAll("-", "_"));
const checkpoints: ContinuationCheckpoint[] = [];
let issue: Row | undefined, readback: ClarificationReadback | undefined;
await mkdir(options.directory, { recursive: true, mode: 0o700 });
const retain = (name: string, value: unknown) => writeFile(path.join(options.directory, name), JSON.stringify(value, null, 2), { mode: 0o600 });
const read = () => readClarificationState(context, issue!.id);
const settle = async (answeredId?: string, final = false) => {
let stable = "";
while (Date.now() < options.deadlineAt) {
const state = await read();
const pending = state.interactions.filter(i => i.kind === "ask_user_questions" && i.status === "pending");
const ready = continuationAnswerCommitted(state.interactions, answeredId)
&& continuationCheckpointReady({ issue: state.currentIssue as { status: string; executionRunId?: string }, runs: state.runs as Array<Row & { id: string; status: string }>, interactions: state.interactions })
&& (final ? state.currentIssue.status === "done" && !pending.length : pending.length === 1)
&& !state.currentIssue.scheduledRetry && !state.currentIssue.activeRecoveryAction;
const key = ready ? JSON.stringify([state.currentIssue.status, state.currentIssue.executionRunId,
state.runs.map(run => [run.id, run.status]), state.interactions.map(i => [i.id, i.status])]) : "";
if (key && key === stable) return state;
stable = key;
await new Promise(resolve => setTimeout(resolve, 1000));
}
throw new JourneyError("failed", final ? "clarification_not_completed" : "clarification_question_not_ready");
};
const open = () => context.page.goto(`${context.api.origin}/${encodeURIComponent(context.company.issuePrefix)}/issues/${encodeURIComponent(issue!.identifier ?? issue!.id)}`, { waitUntil: "domcontentloaded" });
const snapshot = async (phase: "initial" | "answered" | "final", state: Awaited<ReturnType<typeof read>>) => {
const documents = await Promise.all(state.documents.map(document => context.api.get<Row>(`/api/issues/${issue!.id}/documents/${encodeURIComponent(document.key)}`)));
const point = { phase, issue: state.currentIssue, runs: structuredClone(state.runs), interactions: structuredClone(state.interactions),
documents, attachments: state.attachments, children: state.children, comments: [] } as unknown as ContinuationCheckpoint;
checkpoints.push(point);
await retain("clarification.json", { checkpoints });
await open();
await options.capture(phase, `Clarification ${phase}`, `clarification-${phase}.png`);
return point;
};
try {
const title = `Journey QA question-answer-resume ${campaignId}`;
const created = await createTaskThroughUi({ page: context.page, origin: context.api.origin,
issuePrefix: context.company.issuePrefix, agentName: agent.name, title, prompt: scenario.prompt,
workMode: "standard", requireExplicitTitle: true,
beforeDispatch: () => session.own({ kind: "issue", name: title, journeyId: "question-answer-resume" }) });
await session.own({ kind: "issue", name: title, id: created.issueId, journeyId: "question-answer-resume" });
issue = await context.api.get<Row>(`/api/issues/${created.issueId}`);
let state = await settle();
for (const [index, input] of [{ choice: "Afternoon" }, { text: scenario.answer }].entries()) {
const point = await snapshot(index === 0 ? "initial" : "answered", state);
const question = state.interactions.find(i => i.kind === "ask_user_questions" && i.status === "pending")!;
const questionFailure = clarificationQuestionFailure(point, question, index === 0);
if (questionFailure) throw new JourneyError("failed", questionFailure);
if (point.documents.length || point.attachments.length || point.children.length || point.issue.status === "done") {
throw new JourneyError("failed", "clarification_premature_output");
}
await answerResumeQuestionThroughUi(context.page, question, input);
state = await settle(question.id, index === 1);
}
const final = await snapshot("final", state);
if (final.documents.length !== 1) throw new JourneyError("failed", "clarification_document_missing");
const document = final.documents[0]!;
const href = `/${encodeURIComponent(context.company.issuePrefix)}/issues/${encodeURIComponent(issue.identifier ?? issue.id)}#document-${encodeURIComponent(document.key)}`;
await context.page.goto(new URL(href, context.api.origin).href, { waitUntil: "domcontentloaded" });
await context.page.reload({ waitUntil: "domcontentloaded" });
const visible = context.page.locator(`[id=${JSON.stringify(`document-${document.key}`)}]:visible, [id=${JSON.stringify(`side-panel-content-document:${document.key}`)}]:visible`);
await expect(visible).toHaveCount(1); await expect(visible).toBeVisible();
await expect(visible).toContainText(scenario.answer); await expect(visible).toContainText(/afternoon/i);
const persisted = await context.api.get<Row>(`/api/issues/${issue.id}/documents/${encodeURIComponent(document.key)}`);
readback = { issueId: persisted.issueId, key: persisted.key, revisionId: persisted.latestRevisionId, body: persisted.body, visibleBody: await visible.innerText() };
await options.capture("reloaded", "Saved answers after reload", "clarification-reloaded.png");
} finally {
const checks = gradeClarificationEvidence(context, issue?.id ?? "", scenario.marker, checkpoints, readback);
evidence.checks.push(...checks.map(({ id, passed }) => ({ id, passed })));
await retain("clarification.json", { checkpoints, readback, checks });
}
if (evidence.checks.some(check => !check.passed)) throw new JourneyError("failed", "clarification_evidence_failed");
}
tests/runner-e2e/product-journeys/launch.ts
+15 -6
View File
@@ -6,13 +6,15 @@ import path from "node:path";
import { pathToFileURL } from "node:url";
import { chromium, type Browser, type BrowserContext } from "@playwright/test";
import { runnerSuites } from "../catalog.js";
import { productJourneys, parseTargetConfig, preparationPreview, isAgentJourney, agentJourneyRuns } from "./catalog.js";
import { productJourneys, parseTargetConfig, preparationPreview, isAgentJourney } from "./catalog.js";
import { agentJourneyRunCountValid } from "./success-contract.js";
import { journeyFixtures } from "./fixtures.js";
import { taskProperties } from "./task-properties.js";
import { taskSearchFilter } from "./task-search-filter.js";
import { askQuestion, taskCreate } from "./basic-flows.js";
import { projectTaskContext } from "./project-context-flow.js";
import { runAgentJourney } from "./agent-flows.js";
import { runClarificationJourney } from "./clarification-flow.js";
import { JourneyError, type JourneyContext, type JourneyEvidence, type JourneyOutcome, type TargetLifecycleInput, type TargetProvider, type TargetSession } from "./types.js";
import type { RunnerE2EResult } from "../types.js";
import { regenerateRunnerDashboard } from "../dashboard-regenerate.js";
@@ -104,7 +106,7 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri
try { sha = execFileSync("git", ["rev-parse", "HEAD"], { cwd: repositoryRoot, encoding: "utf8", stdio: ["ignore", "pipe", "ignore"] }).trim(); } catch { /* Packaged harnesses may not include Git metadata. */ }
const source = resolveRunnerE2ESource({ sha, ref: null, workflowRunUrl: null });
const definitionHash = createHash("sha256").update(JSON.stringify(productJourneys));
for (const name of ["catalog.ts", "task-properties.ts", "task-search-filter.ts", "task-search-evidence.ts", "basic-flows.ts", "basic-evidence.ts", "project-context-flow.ts", "project-context-readback.ts", "project-context-evidence.ts", "agent-flows.ts", "success-contract.ts", "product-screenshot.ts", "cloud-onboarding.ts", "onboarding-connect.ts", "onboarding-first-task.ts", "workspace-deferral.ts", "onboarding-pre-acceptance.ts", "artifact-evidence.ts", "fixtures.ts", "launch.ts", "types.ts", "collection.ts", "local-target.ts"]) definitionHash.update(await readFile(path.join(import.meta.dirname, name)));
for (const name of ["catalog.ts", "task-properties.ts", "task-search-filter.ts", "task-search-evidence.ts", "basic-flows.ts", "basic-evidence.ts", "project-context-flow.ts", "project-context-readback.ts", "project-context-evidence.ts", "agent-flows.ts", "clarification-flow.ts", "clarification-evidence.ts", "../question-resume-ui.ts", "../question-resume-scoring.ts", "../runtime-question-readiness.ts", "../chat-flow.ts", "../continuation-readiness.ts", "../continuation-cases.ts", "success-contract.ts", "product-screenshot.ts", "cloud-onboarding.ts", "onboarding-connect.ts", "onboarding-first-task.ts", "workspace-deferral.ts", "onboarding-pre-acceptance.ts", "artifact-evidence.ts", "fixtures.ts", "launch.ts", "types.ts", "collection.ts", "local-target.ts"]) definitionHash.update(await readFile(path.join(import.meta.dirname, name)));
for (const file of ["catalog.ts", "api.ts", "run-observations.ts", "native-task-reply.ts", "native-event-envelope.ts", "user-actions.ts", "plan-revision-flow.ts", "skill-studio-flow.ts", "protected-action-flow.ts", "story-artifact.ts", "stop-redirect-flow.ts", "everyday-cases.ts", "everyday-delivery.ts", "everyday-decisions.ts", "connection-guidance-evidence.ts", "everyday-artifact.py"]) definitionHash.update(await readFile(path.join(import.meta.dirname, "..", file)));
definitionHash.update(runnerSuites.flatMap(suite => suite.tasks).find(task => task.id === "ask-question")!.buildPrompt("definition"));
const suiteDefinitionHash = definitionHash.digest("hex");
@@ -201,12 +203,19 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri
result.model = ready.agent.model;
result.runtimeMode = ready.agent.runtimeMode;
if (id === "ask-question") await askQuestion(ready, session!, campaignId, evidence);
else if (id === "question-answer-resume") await runClarificationJourney(ready, session!, campaignId, evidence, { deadlineAt,
directory: path.join(outputDir, "private-stories", id),
capture: async (_id, _label, file) => {
const directory = path.join(outputDir, "private-stories", id);
await mkdir(directory, { recursive: true, mode: 0o700 });
await captureProductScreenshot(ready!.page, ready!.api.origin, path.join(directory, file), deadlineAt);
} });
else await runAgentJourney(id, ready, session!, campaignId, evidence, { deadlineAt,
directory: path.join(outputDir, "private-stories", id),
capture: async (_id, _label, file) => {
const directory = path.join(outputDir, "private-stories", id);
await mkdir(directory, { recursive: true, mode: 0o700 });
await captureProductScreenshot(ready!.page, ready!.api.origin, path.join(directory, file));
await captureProductScreenshot(ready!.page, ready!.api.origin, path.join(directory, file), deadlineAt);
} });
} else if (id === "task-create") await taskCreate(ready, session!, campaignId, evidence);
else if (id === "project-task-context") await projectTaskContext(ready, session!, campaignId, evidence);
@@ -221,7 +230,7 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri
result.status = "passed";
const evidenceDir = path.join(outputDir, "evidence", result.executionId, "attempt-1");
await mkdir(evidenceDir, { recursive: true, mode: 0o700 });
await captureProductScreenshot(ready.page, ready.api.origin, path.join(evidenceDir, `${id}.png`));
await captureProductScreenshot(ready.page, ready.api.origin, path.join(evidenceDir, `${id}.png`), deadlineAt);
result.screenshots = [{ id, label: productJourneys.find(j => j.id === id)!.label, file: `${id}.png` }];
} catch (error) {
const failure = journeyFailure(error);
@@ -237,7 +246,7 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri
if (url.origin === ready.api.origin && /^\/[^/]+\/(issues|search)(\/|$)/.test(url.pathname)) {
const evidenceDir = path.join(outputDir, "evidence", result.executionId, "attempt-1");
await mkdir(evidenceDir, { recursive: true, mode: 0o700 });
await captureProductScreenshot(ready.page, ready.api.origin, path.join(evidenceDir, `${id}-failed.png`));
await captureProductScreenshot(ready.page, ready.api.origin, path.join(evidenceDir, `${id}-failed.png`), deadlineAt);
result.screenshots = [{ id: `${id}-failed`, label: "Failed journey screen", file: `${id}-failed.png` }];
}
} catch { /* Preserve the original failure when its page is unavailable. */ }
@@ -265,7 +274,7 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri
}
for (const result of results) {
if (isAgentJourney(result.caseId as import("./types.js").JourneyId) && result.status === "passed"
&& (!seen.has(result.caseId) || result.runIds?.length !== agentJourneyRuns[result.caseId as keyof typeof agentJourneyRuns])) throw new JourneyError("failed", "paid_journey_receipt_missing");
&& (!seen.has(result.caseId) || !agentJourneyRunCountValid(result.caseId, result.runIds?.length))) throw new JourneyError("failed", "paid_journey_receipt_missing");
if (result.caseId === "cloud-onboarding" && result.status === "passed" && (!seen.has(result.caseId) || !result.runIds?.length || !result.productJourney?.operations || !result.productJourney.operations.traceIds.length)) throw new JourneyError("failed", "paid_journey_receipt_missing");
// Success-only checks must run before a disposable target can be deleted.
// Cleanup is provisional here; the observed cleanup result is retained below.
tests/runner-e2e/product-journeys/product-screenshot.test.ts
+56 -13
View File
@@ -1,20 +1,63 @@
import { describe, expect, it, vi } from "vitest";
import { afterEach, describe, expect, it, vi } from "vitest";
import type { Page } from "@playwright/test";
import { captureProductScreenshot } from "./product-screenshot.js";
const origin = "https://qa.example.test";
afterEach(() => vi.restoreAllMocks());
function fixture(options: { mounted?: boolean; redirect?: string; afterMount?(): void } = {}) {
let mounted = false, url = `${origin}/QA/issues/QA-1`;
const screenshot = vi.fn(async (_input: { path: string; timeout: number }) => {});
const waitFor = vi.fn(async (_input: { state: string; timeout: number }) => { mounted = options.mounted !== false; if (options.redirect) url = options.redirect; options.afterMount?.(); });
const locator = vi.fn(() => ({ waitFor, isVisible: async () => mounted, screenshot }));
const page = { url: () => url, locator } as unknown as Page;
return { page, locator, waitFor, screenshot };
}
describe("product screenshot boundary", () => {
it("excludes a sign-in screen rendered at an unchanged task URL", async () => {
const screenshot = vi.fn();
const page = { url: () => "https://qa.example.test/QA/issues/QA-1", locator: () => ({ isVisible: async () => false, screenshot }) } as unknown as Page;
await expect(captureProductScreenshot(page, "https://qa.example.test", "shot.png")).rejects.toThrow("product_screen_unavailable");
expect(screenshot).not.toHaveBeenCalled();
it("waits for Layout to mount after the document navigation completes", async () => {
const f = fixture();
await captureProductScreenshot(f.page, origin, "shot.png");
expect(f.locator).toHaveBeenCalledWith("main#main-content");
expect(f.waitFor).toHaveBeenCalledWith({ state: "visible", timeout: expect.any(Number) });
expect(f.screenshot).toHaveBeenCalledTimes(1);
expect(f.waitFor.mock.invocationCallOrder[0]).toBeLessThan(f.screenshot.mock.invocationCallOrder[0]!);
});
it("captures only the signed-in product region and refuses other origins", async () => {
const screenshot = vi.fn(), locator = vi.fn(() => ({ isVisible: async () => true, screenshot }));
const page = { url: () => "https://qa.example.test/QA/issues/QA-1", locator } as unknown as Page;
await captureProductScreenshot(page, "https://qa.example.test", "shot.png");
expect(locator).toHaveBeenCalledWith("main#main-content"); expect(screenshot).toHaveBeenCalledTimes(1);
await expect(captureProductScreenshot(page, "https://other.example.test", "shot.png")).rejects.toThrow("product_screen_unavailable");
expect(screenshot).toHaveBeenCalledTimes(1);
it("excludes a sign-in screen rendered at an unchanged task URL", async () => {
const f = fixture({ mounted: false });
await expect(captureProductScreenshot(f.page, origin, "shot.png")).rejects.toThrow("product_screen_unavailable");
expect(f.screenshot).not.toHaveBeenCalled();
});
it("rejects a product region that never mounts", async () => {
const f = fixture(); f.waitFor.mockRejectedValueOnce(new Error("Timeout"));
await expect(captureProductScreenshot(f.page, origin, "shot.png")).rejects.toThrow("product_screen_unavailable");
expect(f.screenshot).not.toHaveBeenCalled();
});
it.each([`${origin}/login`, "https://other.example.test/QA/issues/QA-1"])("excludes a redirect during readiness to %s", async redirect => {
const f = fixture({ redirect });
await expect(captureProductScreenshot(f.page, origin, "shot.png")).rejects.toThrow("product_screen_unavailable");
expect(f.screenshot).not.toHaveBeenCalled();
});
it("refuses another origin before waiting or taking a screenshot", async () => {
const f = fixture();
await expect(captureProductScreenshot(f.page, "https://other.example.test", "shot.png")).rejects.toThrow("product_screen_unavailable");
expect(f.waitFor).not.toHaveBeenCalled(); expect(f.screenshot).not.toHaveBeenCalled();
});
it("bounds readiness and capture by the campaign deadline", async () => {
vi.spyOn(Date, "now").mockReturnValue(1000);
const f = fixture();
await captureProductScreenshot(f.page, origin, "shot.png", 1050);
for (const call of [f.waitFor.mock.calls[0], f.screenshot.mock.calls[0]]) {
expect((call?.[0] as { timeout: number }).timeout).toBeGreaterThan(0);
expect((call?.[0] as { timeout: number }).timeout).toBeLessThanOrEqual(50);
}
});
it("does not capture after readiness consumes the remaining deadline", async () => {
const now = vi.spyOn(Date, "now").mockReturnValue(1000), f = fixture({ afterMount: () => { now.mockReturnValue(1050); } });
await expect(captureProductScreenshot(f.page, origin, "shot.png", 1050)).rejects.toThrow("product_screen_unavailable");
expect(f.screenshot).not.toHaveBeenCalled();
});
it("never disables a timeout when the campaign deadline has elapsed", async () => {
const f = fixture();
await expect(captureProductScreenshot(f.page, origin, "shot.png", Date.now())).rejects.toThrow("product_screen_unavailable");
expect(f.waitFor).not.toHaveBeenCalled(); expect(f.screenshot).not.toHaveBeenCalled();
});
});
tests/runner-e2e/product-journeys/product-screenshot.ts
+15 -5
View File
@@ -1,12 +1,22 @@
import type { Page } from "@playwright/test";
import { JourneyError } from "./types.js";
export async function captureProductScreenshot(page: Page, origin: string, file: string) {
const url = new URL(page.url());
if (url.origin !== origin || !/^\/[^/]+\/(issues|search)(\/|$)/.test(url.pathname)) throw new JourneyError("failed", "product_screen_unavailable");
export async function captureProductScreenshot(page: Page, origin: string, file: string, deadlineAt = Date.now() + 10_000) {
const permitted = () => {
const url = new URL(page.url());
return url.origin === origin && /^\/[^/]+\/(issues|search)(\/|$)/.test(url.pathname);
};
const timeout = (maximum: number) => {
const remaining = deadlineAt - Date.now();
if (remaining <= 0) throw new JourneyError("failed", "product_screen_unavailable");
return Math.min(maximum, remaining);
};
if (!permitted() || deadlineAt <= Date.now()) throw new JourneyError("failed", "product_screen_unavailable");
// CloudAccessGate replaces Layout on expiry without necessarily changing the
// URL. Capture Layout's main element, so auth screens can never be included.
const product = page.locator("main#main-content");
if (!await product.isVisible()) throw new JourneyError("failed", "product_screen_unavailable");
await product.screenshot({ path: file, timeout: 2000 });
try { await product.waitFor({ state: "visible", timeout: timeout(10_000) }); }
catch { throw new JourneyError("failed", "product_screen_unavailable"); }
if (!permitted() || !await product.isVisible() || deadlineAt <= Date.now()) throw new JourneyError("failed", "product_screen_unavailable");
await product.screenshot({ path: file, timeout: timeout(2000) });
}
tests/runner-e2e/product-journeys/success-contract.ts
+11 -1
View File
@@ -1,12 +1,22 @@
/** Closed, target-independent success requirements used by dispatch and retained evidence. */
export const agentJourneyRuns = { "ask-question": 1, "plan-revise-accept": 3, "build-revise": 2,
"protected-actions": 4, "create-skill-studio": 1, "stop-redirect": 3 } as const;
"protected-actions": 4, "create-skill-studio": 1, "stop-redirect": 3, "question-answer-resume": 3 } as const;
export function agentJourneyRunCountValid(id: string, count: unknown): boolean {
if (!Object.hasOwn(agentJourneyRuns, id) || !Number.isSafeInteger(count)) return false;
const maximum = agentJourneyRuns[id as keyof typeof agentJourneyRuns];
const minimum = id === "question-answer-resume" ? 1 : maximum;
return (count as number) >= minimum && (count as number) <= maximum;
}
export const journeyRequiredChecks: Record<string, readonly string[]> = {
"task-properties": ["assignment-persisted", "no-agent-work"],
"task-create": ["browser-created-task-persisted", "no-agent-work"],
"project-task-context": ["browser-created-project-persisted", "browser-created-project-task-persisted", "project-task-association-reloaded", "project-task-context-visible", "no-agent-work"],
"task-search-filter": ["task-search-results-exact", "task-status-filter-excludes-controls", "task-search-filter-reloaded", "task-search-result-reopened", "no-agent-work"],
"ask-question": ["question-answer-persisted", "one-bound-successful-run", "configured-model-and-environment"],
"question-answer-resume": ["question-resume.forms", "question-resume.choice.answer", "question-resume.choice.path",
"question-resume.text.answer", "question-resume.text.path", "question-resume.complete-evidence", "question-no-premature-output",
"question-answers-owned-by-board", "question-document-attributed", "question-document-reloaded", "configured-model-and-environment"],
"plan-revise-accept": ["plan-revision-and-approval", "plan-final-answer", "configured-model-and-environment"],
"build-revise": ["initial-artifact-verified", "revised-artifact-verified", "artifact-versions-differ", "original-artifact-preserved", "user-reply-stored-once", "configured-model-and-environment"],
"protected-actions": ["approve-waited-for-decision", "approve-decision-persisted", "approve-service-call-count", "approved-service-data-used",
tests/runner-e2e/product-journeys/types.ts
+1 -2
View File
@@ -3,8 +3,7 @@ import type { ConnectionApi } from "../connection-target.js";
export type JourneyOutcome = "passed" | "failed" | "blocked" | "unsupported";
export type JourneyId = "task-properties" | "task-create" | "project-task-context" | "ask-question" | "cloud-onboarding"
| "task-search-filter"
| "plan-revise-accept" | "build-revise" | "protected-actions" | "create-skill-studio" | "stop-redirect";
| "question-answer-resume" | "task-search-filter" | "plan-revise-accept" | "build-revise" | "protected-actions" | "create-skill-studio" | "stop-redirect";
export type TargetMode = "managed-self-hosted" | "existing-cloud" | "new-cloud";
export interface DeploymentIdentity {
kind: "self-hosted" | "cloud";
tests/runner-e2e/question-resume-scoring.ts
+31 -8
View File
@@ -3,14 +3,33 @@ import type { ContinuationCheckpoint } from "./continuation-scoring.js";
import { isSingleClaudeQuestion } from "./runtime-question-readiness.js";
type Row = Record<string, any>;
export type QuestionPath = "provider" | "semantic";
export type QuestionPath = "provider" | "semantic" | "api";
const nonempty = (v: unknown): v is string => typeof v === "string" && v.trim().length > 0;
const questions = (c?: ContinuationCheckpoint): Row[] =>
(c?.interactions ?? []).filter((i: any) => i.kind === "ask_user_questions") as Row[];
export const resumeQuestionFields = (i?: Row): Row[] =>
Array.isArray(i?.payload?.questionSet?.questions) ? i.payload.questionSet.questions : [];
/** Classify by server-owned identities and the applied semantic receipt. A
function nativeApiQuestionReceipt(run: Row | undefined, question: Row) {
const hash = (value: unknown) => typeof value === "string" && /^[a-f0-9]{64}$/.test(value);
return Object.entries(run?.resultJson?.apiToolReceipts ?? {}).find(([key, value]) => {
const receipt = value as Row, original = receipt?.result?.data;
return hash(key) && hash(receipt?.digest) && receipt?.state === "completed"
&& receipt.operationId === "POST /api/issues/{id}/interactions" && receipt.result?.operationId === receipt.operationId
&& receipt.result.ok === true && receipt.result.status === 201 && original?.status === "pending"
&& original.kind === "ask_user_questions" && original.createdByUserId === null && question.createdByUserId === null
&& sameQuestion(original, question);
});
}
function sameNativeApiQuestionReceipt(before: ContinuationCheckpoint | undefined, after: ContinuationCheckpoint | undefined, question?: Row) {
if (!question) return false;
const receipt = nativeApiQuestionReceipt(before?.runs.find(run => run.id === question.sourceRunId), question);
return !!receipt && isDeepStrictEqual(receipt,
nativeApiQuestionReceipt(after?.runs.find(run => run.id === question.sourceRunId), question));
}
/** Classify by server-owned identities and an applied native mutation receipt. A
* missing receipt is unknown, never inferred from form shape or run count. */
export function questionPath(checkpoint: ContinuationCheckpoint | undefined, i?: Row): QuestionPath | null {
if (!checkpoint || !i || !nonempty(i.id) || !nonempty(i.sourceRunId) || !nonempty(i.idempotencyKey)) return null;
@@ -29,11 +48,13 @@ export function questionPath(checkpoint: ContinuationCheckpoint | undefined, i?:
}
const receipt = run.resultJson?.semanticToolReceipts?.[i.idempotencyKey];
const created = receipt?.result?.interaction;
return run.status === "succeeded" && i.continuationPolicy === "wake_assignee" &&
if (run.status !== "succeeded" || i.continuationPolicy !== "wake_assignee") return null;
if (
receipt?.operationId === "request_human_input" && receipt.result?.disposition === "applied" &&
created?.id === i.id && created?.sourceRunId === run.id && created?.issueId === i.issueId &&
created?.companyId === i.companyId && created?.kind === "ask_user_questions" &&
isDeepStrictEqual(created?.payload?.questionSet, i.payload?.questionSet) ? "semantic" : null;
isDeepStrictEqual(created?.payload?.questionSet, i.payload?.questionSet)) return "semantic";
return nativeApiQuestionReceipt(run, i) ? "api" : null;
}
export function validResumeQuestionForm(checkpoint: ContinuationCheckpoint, i: Row, choice: boolean): boolean {
@@ -62,14 +83,15 @@ export function questionResumedOnce(before: ContinuationCheckpoint | undefined,
before.runs.some(r => !after.runs.some(n => n.id === r.id))) return false;
if (path === "provider") return next.length === 0 &&
after.runs.some(r => r.id === i.sourceRunId && ["running", "succeeded"].includes(r.status));
if (path !== "semantic" || next.length !== 1) return false;
if ((path !== "semantic" && path !== "api") || next.length !== 1) return false;
if (path === "api" && !sameNativeApiQuestionReceipt(before, after, i)) return false;
const context = next[0].contextSnapshot;
return context?.issueId === i.issueId && context?.sourceRunId === i.sourceRunId &&
context?.interactionId === i.id && context?.interactionKind === "ask_user_questions" &&
context?.interactionStatus === "answered";
}
/** Behavioral qualification supports either valid native path. It does not
/** Behavioral qualification supports each verified native path. It does not
* replace the separate documentation-placement/semantic-tool qualification. */
export function gradeQuestionResume(checkpoints: ContinuationCheckpoint[], marker: string) {
const initial = checkpoints.find(c => c.phase === "initial");
@@ -112,8 +134,9 @@ export function gradeQuestionResume(checkpoints: ContinuationCheckpoint[], marke
detail: "The real board answer must remain bound to the unchanged question, with no fabricated or substituted response.",
}, {
id: `question-resume.${answer}.path`,
passed: questionResumedOnce(before, after, i),
detail: `Observed path: ${questionPath(before, i) ?? "unknown"}; provider answers resume the same run; semantic answers require one exactly bound response wake.`,
passed: questionResumedOnce(before, after, i) &&
(questionPath(before, i) !== "api" || sameNativeApiQuestionReceipt(before, final, i)),
detail: `Observed path: ${questionPath(before, i) ?? "unknown"}; provider answers resume the same run; durable native questions require one exactly bound response wake.`,
});
}
checks.push({
tests/runner-e2e/question-resume-ui.ts
+23
View File
@@ -0,0 +1,23 @@
import { expect, type Page } from "@playwright/test";
import { chatQuestionPresentation } from "./chat-flow.js";
type Row = Record<string, any>;
export async function answerResumeQuestionThroughUi(page: Page, interaction: Row, input: { choice: string } | { text: string }) {
const set = chatQuestionPresentation(interaction.payload);
if ("choice" in input) {
expect(set.questions[0]?.answerMode, "choices must use radio controls").toBe("single_select");
const options = interaction.payload.questionSet?.questions[0]?.options ?? interaction.payload.questions[0]?.options ?? [];
expect(new Set(options.map((o: Row) => String(o.label).trim().toLowerCase())).size).toBeGreaterThanOrEqual(2);
const choice = input.choice.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
await page.getByRole("radio", { name: new RegExp(`^${choice}\\b`, "i") }).last().click();
} else {
expect(set.questions[0]?.answerMode, "open answers must render a text field, not a choice question").toBe("text");
await page.getByTestId("question-text-answer-composer").last()
.locator('[contenteditable="true"],textarea').first().fill(input.text);
}
for (let index = 1; index < set.questions.length; index += 1) {
await page.getByRole("button", { name: "Next", exact: true }).last().click();
}
await page.getByRole("button", { name: set.submitLabel ?? "Submit answers", exact: true }).last().click();
}
tests/runner-e2e/question-resume.test.ts
+60 -3
View File
@@ -1,4 +1,5 @@
import { describe, expect, it } from "vitest";
import { readFileSync } from "node:fs";
import { gradeQuestionResume, questionPath, validResumeQuestionForm, type QuestionPath } from "./question-resume-scoring.js";
import { gradeContinuation } from "./continuation-scoring.js";
import { gradeLifecycleBaseline } from "./lifecycle-baseline.js";
@@ -26,7 +27,7 @@ function recording(paths: QuestionPath[] = ["provider", "semantic"]) {
};
for (let n = 0; n < 2; n++) {
const run = runs.at(-1)!;
const provider = paths[n] === "provider";
const provider = paths[n] === "provider", api = paths[n] === "api";
const request = `request-${n}`;
const idempotencyKey = provider ? `paperclip-runner-question:${run.id}:${request}` : `question-${n}`;
const field = n === 0 ? { id: "field-1-question_0-hash", answerMode: "single_select", options: [
@@ -36,12 +37,15 @@ function recording(paths: QuestionPath[] = ["provider", "semantic"]) {
if (provider && n === 0) questions.push({ id: "field-2-question_0_custom-hash", answerMode: "text", header: "Other", required: false });
const card: Row = { id: `card-${n}`, kind: "ask_user_questions", status: "pending",
companyId: "company", issueId: "task", createdByAgentId: "agent", sourceRunId: run.id, idempotencyKey,
continuationPolicy: provider ? "none" : "wake_assignee",
continuationPolicy: provider ? "none" : "wake_assignee", createdByUserId: null,
payload: { ...(provider ? { runtimeRequestId: request } : {}), questionSet: { questions } } };
cards.push(card);
if (!provider) {
run.status = "succeeded";
run.resultJson.semanticToolReceipts[idempotencyKey] = { operationId: "request_human_input",
if (api) run.resultJson.apiToolReceipts = { ["a".repeat(64)]: { state: "completed", digest: "b".repeat(64),
operationId: "POST /api/issues/{id}/interactions", result: { ok: true, status: 201,
operationId: "POST /api/issues/{id}/interactions", data: structuredClone(card) } } };
else run.resultJson.semanticToolReceipts[idempotencyKey] = { operationId: "request_human_input",
result: { disposition: "applied", interaction: structuredClone(card) } };
}
snap(n === 0 ? "initial" : "answered");
@@ -58,6 +62,58 @@ const failed = (r: any[]) => gradeQuestionResume(r, "AMBERnonce").filter(c => !c
const question = (r: any[], n = 0) => r[n].interactions[n];
describe("native question/resume path qualification", () => {
it("accepts a native API creation receipt with the retained server response shape", () => {
const point = JSON.parse(readFileSync(new URL("./fixtures/native-api-question.json", import.meta.url), "utf8"));
expect(questionPath(point, point.interactions[0])).toBe("api");
expect(validResumeQuestionForm(point, point.interactions[0], true)).toBe(true);
});
it.each(["missing", "pending", "wrong-operation", "wrong-result-operation", "unknown-outcome", "wrong-status", "missing-digest", "wrong-key",
"foreign-card", "foreign-company", "foreign-issue", "foreign-agent", "foreign-source", "wrong-idempotency", "changed-form", "changed-compatibility-form",
"board-created", "resolved-creation", "wrong-policy", "runtime-request", "legacy-run", "active-source", "foreign-run", "duplicate-source", "spoofed-store"])("rejects unverified native API provenance: %s", kind => {
const point = JSON.parse(readFileSync(new URL("./fixtures/native-api-question.json", import.meta.url), "utf8"));
const card = point.interactions[0], run = point.runs[0], receipts = run.resultJson.apiToolReceipts;
const receipt = Object.values(receipts)[0] as Row, original = receipt.result.data;
if (kind === "missing") delete run.resultJson.apiToolReceipts;
if (kind === "pending") receipt.state = "pending";
if (kind === "wrong-operation") receipt.operationId = "POST /api/issues/{id}/comments";
if (kind === "wrong-result-operation") receipt.result.operationId = "POST /api/issues/{id}/comments";
if (kind === "unknown-outcome") receipt.result.ok = false;
if (kind === "wrong-status") receipt.result.status = 200;
if (kind === "missing-digest") delete receipt.digest;
if (kind === "wrong-key") run.resultJson.apiToolReceipts = { "model-authored-claim": receipt };
if (kind === "foreign-card") original.id = "other";
if (kind === "foreign-company") original.companyId = "other";
if (kind === "foreign-issue") original.issueId = "other";
if (kind === "foreign-agent") original.createdByAgentId = "other";
if (kind === "foreign-source") original.sourceRunId = "other";
if (kind === "wrong-idempotency") original.idempotencyKey = "other";
if (kind === "changed-form") original.payload.questionSet.questions[0].options[1].label = "Other";
if (kind === "changed-compatibility-form") original.payload.questions[0].prompt = "Other";
if (kind === "board-created") original.createdByUserId = "other";
if (kind === "resolved-creation") original.status = "answered";
if (kind === "wrong-policy") card.continuationPolicy = "none";
if (kind === "runtime-request") card.payload.runtimeRequestId = "fabricated-provider-request";
if (kind === "legacy-run") run.runtimeMode = "legacy";
if (kind === "active-source") run.status = "running";
if (kind === "foreign-run") run.agentId = "other";
if (kind === "duplicate-source") point.runs.push(structuredClone(run));
if (kind === "spoofed-store") { run.resultJson.modelClaim = receipts; delete run.resultJson.apiToolReceipts; }
expect(questionPath(point, card)).toBeNull();
});
it.each(["lost-receipt", "changed-receipt", "interactionId", "sourceRunId", "issueId", "interactionStatus", "duplicate-continuation"])("retains one immutable API receipt and one bound answer wake: %s", kind => {
const r = recording(["api", "api"]), source = r[1].runs[0], next = r[1].runs[1];
if (kind === "lost-receipt") delete source.resultJson.apiToolReceipts;
else if (kind === "changed-receipt") source.resultJson.apiToolReceipts["a".repeat(64)].digest = "c".repeat(64);
else if (kind === "duplicate-continuation") r[1].runs.push({ ...structuredClone(next), id: "extra" });
else next.contextSnapshot[kind] = "other";
expect(failed(r)).toContain("question-resume.choice.path");
});
it.each(["lost", "changed"])("rejects the first API receipt %s after the second answer", kind => {
const r = recording(["api", "api"]), source = r[2].runs[0];
if (kind === "lost") delete source.resultJson.apiToolReceipts;
else source.resultJson.apiToolReceipts["a".repeat(64)].digest = "c".repeat(64);
expect(failed(r)).toContain("question-resume.choice.path");
});
it("keeps the new journey explicit-only and the original documentation cells intact", () => {
const cells = runnerMatrix.filter(c => c.suite.id === "question-resume");
expect(cells.map(c => c.profile.id).sort()).toEqual(["runner-acpx-claude", "runner-codex"]);
@@ -70,6 +126,7 @@ describe("native question/resume path qualification", () => {
});
it.each([
["provider", "provider"], ["provider", "semantic"], ["semantic", "provider"], ["semantic", "semantic"],
["api", "api"], ["api", "semantic"], ["semantic", "api"], ["provider", "api"], ["api", "provider"],
] as QuestionPath[][])("accepts exactly bound %s then %s journeys", (a, b) => {
const r = recording([a,b]);
expect(failed(r)).toEqual([]);
tests/runner-e2e/result-validation.ts
+2 -2
View File
@@ -1,4 +1,4 @@
import { agentJourneyRuns, journeyRequiredChecks } from "./product-journeys/success-contract.js";
import { agentJourneyRuns, agentJourneyRunCountValid, journeyRequiredChecks } from "./product-journeys/success-contract.js";
import { validateWorkerTiming } from "./worker-timing.js";
import { connectionCheckpoints, connectionEvidencePasses, connectionRunSignals } from "./connection-evidence.js";
import { githubJourneyCheckpoints, gradeGitHubReviewJourney } from "./github-review-evidence.js";
@@ -435,7 +435,7 @@ export function validateRetainedRunnerResult(
const required = journeyRequiredChecks[checked.caseId];
if (evidence.outcome !== "passed" || !evidence.deployment || !required || !required.every(id => evidence.checks.some(c => c.id === id && c.passed)) || evidence.checks.some(check => !check.passed) || evidence.phases.some(p => p.outcome !== "passed") || checked.cleanup !== "passed") invalid("result.productJourney");
if (["task-properties", "task-create", "project-task-context", "task-search-filter"].includes(checked.caseId) && (checked.runtimeMode !== "none" || checked.provider !== null || checked.model !== null || checked.runIds?.length)) invalid("result.browserOnly");
if (Object.hasOwn(agentJourneyRuns, checked.caseId) && (checked.runIds?.length !== agentJourneyRuns[checked.caseId as keyof typeof agentJourneyRuns]
if (Object.hasOwn(agentJourneyRuns, checked.caseId) && (!agentJourneyRunCountValid(checked.caseId, checked.runIds?.length)
|| !checked.model || !checked.provider || checked.runtimeMode !== "native")) invalid("result.agentJourneyRun");
if (checked.caseId === "cloud-onboarding" && (!checked.runIds?.length || !evidence.operations || !evidence.operations.requestCount || !evidence.operations.traceIds.length || evidence.operations.campaignId !== evidence.campaignId || evidence.operations.attemptId !== evidence.attemptId)) invalid("result.paidJourneyReceipt");
}