diff --git a/doc/plans/2026-10-29-cloud-qa-basic-journeys.md b/doc/plans/2026-10-29-cloud-qa-basic-journeys.md new file mode 100644 index 00000000000..7b5f7f554d0 --- /dev/null +++ b/doc/plans/2026-10-29-cloud-qa-basic-journeys.md @@ -0,0 +1,41 @@ +# Five additional daily Cloud QA journeys + +The existing daily QA plan runs three deployment-targeted Product E2E journeys. +Extend it with the five workflows requested in this chat, qualifying every one +against the registered testing Cloud QA instance before promotion. GitHub hosts +the browser driver; the application and native agents remain on Cloud/Daytona. + +## Acceptance + +| Journey | Required live evidence | +| --- | --- | +| Plan, revise, approve, complete | Persisted canonical Plan, a distinct revision-bound replacement confirmation, browser revision and approval, no premature completion, final native completion. | +| Deliver and revise an artifact | Browser downloads of both versions, independent content validation, different hashes, original download still available. | +| Approve and decline protected actions | Authenticated controlled service, zero accepted invocations before approval, exactly one after approval, zero after decline, saved UI decisions and truthful responses. | +| Create and edit a company skill | Native skill-creation receipt, persisted owned skill, activity card, browser Skill Studio edit, reload and task readback. | +| Stop and redirect | Observed active native work, browser Stop, authoritative retirement of the original work, one saved new-direction message, correct new response, no old work continuing. | + +## Ownership and delivery + +- Shared Product journey code owns user actions, prompts, assertions and evidence. +- Target adapters own authentication, registered target identity, independent + fixtures, native agent/connection setup, run accounting and owned cleanup. +- The private Cloud QA workflow owns TestOps selection, upload, reports and Slack. +- Reuse the existing classic story logic and graders where applicable; keep + deployment-specific setup outside shared stories. Keep local and Cloud target + selection explicit and fail closed on missing capabilities or evidence. +- Retain unsuccessful attempts. Product grades, infrastructure failures, + cleanup, usage coverage and trace delivery remain distinct observations. +- Promote each new case to the saved daily TestOps plan only after its real + Cloud pass, independent evidence inspection and successful cleanup. +- Completion requires all five new journeys passing and present in the daily + plan, the existing three controls still passing, passing applicable repository + checks/reviews, and a real combined CI/TestOps/report/Slack verification. + +## Feedback path + +Calibrate assertions and selectors without paid work first. Qualify a narrow +Cloud case, inspect its retained evidence, then add the other cases. Prefer the +existing protected QA credential and workflow paths; do not weaken source review +or secret boundaries to run candidate harness code. Record configuration, +source revisions, target deployment identity, duration and measured usage. diff --git a/maps/feature-map/documents-artifacts.md b/maps/feature-map/documents-artifacts.md index dc4b185897b..efc9ef87bf6 100644 --- a/maps/feature-map/documents-artifacts.md +++ b/maps/feature-map/documents-artifacts.md @@ -47,6 +47,8 @@ Open `/artifacts` and its grouping, search, and media controls. ## Driving it with Playwright and the Paperclip CLI +The explicit `build-revise` [Product journey](../../tests/runner-e2e/PRODUCT-JOURNEYS.md) covers `documents-artifacts/attachments`: browser downloads of two ZIP versions, an independent content oracle, and preservation of the original. Its shared source is not proof of a pass on any deployed target. + Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use a task with a text document, a small attachment, and a generated work product. Use nonsensitive fixture files. ### `task-document` diff --git a/maps/feature-map/questions-and-approvals.md b/maps/feature-map/questions-and-approvals.md index 59924ed889a..8c400bfd5c1 100644 --- a/maps/feature-map/questions-and-approvals.md +++ b/maps/feature-map/questions-and-approvals.md @@ -79,6 +79,8 @@ Channel-specific forms and identity linking require separate provider evidence. ## Driving it +The explicit `plan-revise-accept` and `protected-actions` [Product journeys](../../tests/runner-e2e/PRODUCT-JOURNEYS.md) cover `questions-and-approvals/plan-review` and `questions-and-approvals/tool-review`. They share the classic browser actions and saved-state checks. Cloud qualification remains separate; source registration is not a passing result. + Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use separate disposable requests for accept, decline, and revision paths. Record the actor and request revision. Prepare a waiting question through a test agent; diff --git a/maps/feature-map/skills.md b/maps/feature-map/skills.md index 279b7d95344..c2cea1247ac 100644 --- a/maps/feature-map/skills.md +++ b/maps/feature-map/skills.md @@ -39,6 +39,8 @@ Use skill policy/agent selection in the library or an agent’s Skills tab. ## Driving it with Playwright and the Paperclip CLI +The explicit `create-skill-studio` [Product journey](../../tests/runner-e2e/PRODUCT-JOURNEYS.md) covers `skills/authoring`: native creation of an owned company skill, its task activity card, a Studio edit, reload, and task-card readback. It does not establish policy, source-update, or Studio-test coverage. A deployed target requires its own live result. + Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use a harmless skill fixture and test agent with skill support. Record whether the source is local, bundled, project, or remote. ### `library-sources` diff --git a/maps/feature-map/steering.md b/maps/feature-map/steering.md index 3adb8f1f748..840fcd1ba74 100644 --- a/maps/feature-map/steering.md +++ b/maps/feature-map/steering.md @@ -58,6 +58,8 @@ inspect the resulting task and run in Paperclip. ## Driving it +The explicit `stop-redirect` [Product journey](../../tests/runner-e2e/PRODUCT-JOURNEYS.md) covers `steering/stop-and-pause` only for Stop and one new direction. It requires active work, native cancellation acknowledgement, resource cleanup, and completion without later original-turn activity. It does not qualify pause, interrupt, or all queued-message behavior. Source registration is not a live Cloud pass. + Preconditions: follow the [baseline](./README.md#before-driving-a-journey). Use an agent that stays busy long enough to inspect the queue. Record the adapter, native/legacy mode, and advertised steering capability. Test supported and diff --git a/tests/fixtures/connection-review-provider.ts b/tests/fixtures/connection-review-provider.ts index fc9435380a2..5fd80040fc1 100644 --- a/tests/fixtures/connection-review-provider.ts +++ b/tests/fixtures/connection-review-provider.ts @@ -8,26 +8,34 @@ export async function startReviewProvider( ) { const captures: Array<{ method: string; toolName: string | null; authorized?: boolean }> = []; const server: Server = createServer(async (req, res) => { - const chunks: Buffer[] = []; - for await (const chunk of req) chunks.push(chunk as Buffer); - const payload = JSON.parse( - Buffer.concat(chunks).toString("utf8") || "{}", - ) as { - id?: string | number; - method?: string; - params?: { name?: string; arguments?: { query?: string } }; - }; const authorized = !bearerToken || req.headers.authorization === `Bearer ${bearerToken}`; - captures.push({ - method: String(payload.method ?? ""), - toolName: payload.params?.name ?? null, - ...(bearerToken ? { authorized } : {}), - }); if (!authorized) { + if (captures.length < 10_000) captures.push({ method: "", toolName: null, authorized: false }); + req.resume(); res.writeHead(401, { "Content-Type": "application/json" }); res.end(JSON.stringify({ error: "Service credential required" })); return; } + const chunks: Buffer[] = []; + let size = 0; + try { + for await (const chunk of req) { + size += Buffer.byteLength(chunk); + if (size > 65_536) { res.writeHead(413); res.end(); return; } + chunks.push(chunk as Buffer); + } + } catch { res.destroy(); return; } + let payload: { id?: string | number; method?: string; params?: { name?: string; arguments?: { query?: string } } }; + try { + payload = JSON.parse(Buffer.concat(chunks).toString("utf8") || "{}"); + if (!payload || typeof payload !== "object" || Array.isArray(payload) + || payload.method !== undefined && typeof payload.method !== "string") throw new Error("invalid_request"); + } catch { res.writeHead(400); res.end(); return; } + if (captures.length >= 10_000) { res.writeHead(429); res.end(); return; } + captures.push({ method: String(payload.method ?? "").slice(0, 120), + toolName: typeof payload.params?.name === "string" ? payload.params.name.slice(0, 128) : null, + ...(bearerToken ? { authorized: true } : {}), + }); res.writeHead(200, { "Content-Type": "application/json" }); if (payload.method === "tools/list") { res.end( @@ -54,17 +62,18 @@ export async function startReviewProvider( return; } if (payload.method === "tools/call") { + const unavailable = payload.params?.arguments?.query === "fail" || payload.params?.name !== (tool?.name ?? "notion:list_pages"); res.end( JSON.stringify({ jsonrpc: "2.0", id: payload.id ?? null, result: { - isError: payload.params?.arguments?.query === "fail", + isError: unavailable, content: [ { type: "text", text: - payload.params?.arguments?.query === "fail" + unavailable ? "Fixture provider unavailable" : successText, }, @@ -78,10 +87,12 @@ export async function startReviewProvider( JSON.stringify({ jsonrpc: "2.0", id: payload.id ?? null, result: {} }), ); }); + server.requestTimeout = 15_000; + server.headersTimeout = 10_000; const port = await listenOnFetchAllowedPort(server); return { url: `http://127.0.0.1:${port}/`, captures, - close: () => new Promise((resolve) => server.close(() => resolve())), + close: () => new Promise((resolve) => { server.close(() => resolve()); server.closeAllConnections(); }), }; } diff --git a/tests/runner-e2e/PRODUCT-JOURNEYS.md b/tests/runner-e2e/PRODUCT-JOURNEYS.md index 24393e099ee..bcf42b014dc 100644 --- a/tests/runner-e2e/PRODUCT-JOURNEYS.md +++ b/tests/runner-e2e/PRODUCT-JOURNEYS.md @@ -26,6 +26,11 @@ never start a server or depend on a tenant filesystem. | `task-create` | Company and signed-in board user | Create a task through the shared production composer, assign it to yourself, reopen and reload it, verify the saved prompt and no agent work | | `ask-question` | Company, configured agent capability, positive model allowance and exactly one permitted work run | Reuse the existing cross-model Ask question prompt; require one successful run with matching company/task/agent/model/environment, one saved exact answer, and visible readback after reload | | `task-properties` | Company, signed-in board user, task-write capability, prepared run-owned backlog task | Open its properties, assign it to the signed-in user, reload, check both UI and saved assignment; verify no agent run | +| `plan-revise-accept` | Native agent; three work runs | Reuse the classic Plan story: canonical draft, browser revision request, new revision-bound confirmation, approval and exact saved final response | +| `build-revise` | Native agent; two work runs; qualified Docker artifact oracle | Browser-download both ZIP versions, independently exercise source/CLI behavior, verify different hashes and preserve the original download | +| `protected-actions` | Native agent; four work runs; authenticated reachable review-service fixture | Approve one action and decline another on separate owned tasks; verify zero calls before either decision, one after approval, zero after decline and a truthful saved explanation | +| `create-skill-studio` | Native agent with skill-write permission; one work run | Create a uniquely named company skill, open its activity card, edit in Studio, reload and verify task-card readback | +| `stop-redirect` | Native agent; three work runs; qualified Docker oracle | Download a verified checkpoint, Stop an active revision, require native cancellation acknowledgement, submit a new direction once and verify completion without original-run activity | | `cloud-onboarding` | Signed-in admitted Cloud account; fresh workspace; qualified bounded provider route | Create workspace in the portal, connect API key and native Codex agent in the wizard, complete a task, download and independently check its artifact | Supported acquisitions are managed self-hosted (`local`), registered existing @@ -135,7 +140,7 @@ contract before successful cleanup. Missing assertions or paid accounting proof turn the campaign into a failure while preserving valid cost receipts, so an invalid success cannot trigger disposable-stack deletion. -Only post-journey screenshots are captured. Failed task/search screens are +The shared Plan story additionally retains draft and revised task screenshots privately. Other screenshots capture the post-journey task. Failed task/search screens are retained only after tenant setup; authentication pages are excluded. Browser state, login codes, raw errors, credentials, request headers, and auth screenshots are excluded. All Cloud evidence is private; the public history publisher rejects these bundles. @@ -180,3 +185,25 @@ separate onboarding gateway product change, trusted CI selection, and real runtime gateway admission remain live qualification prerequisites. Testing and staging use separate registries and credentials; only the declared testing control plane is admitted. Defining a story or passing structural checks is not evidence that its Cloud journey passed. + +## Reuse and daily QA promotion + +The classic matrix and deployment-targeted stories share Plan revision actions, +artifact download/oracle code, Skill Studio editing and protected-action decisions. +Their target adapters own login, agent/connection provisioning, environment +identity, and cleanup. The managed local recipe currently provides browser-only +capabilities; paid stories fail as unsupported until a native agent fixture is +supplied. This does not turn a classic local pass into a QA Cloud pass. + +Paid deployment campaigns declare the exact sum of expected work runs (Ask 1, +Plan 3, artifact revision 2, protected actions 4, Skill Studio 1, Stop/redirect 3). +Each observer reads only its owned tasks and checks native company, agent, model +and execution-environment identity. Setup is separately unpriced. Skills and +service connections are uniquely owned; customer credentials and company-wide +configuration are never copied into a journey. + +The private QA adapter keeps candidate cases available for manual qualification. +A candidate's presence in this source catalog is not qualification. It enters the +saved daily TestOps plan only after an inspected real QA Cloud pass including +usage, artifact/decision evidence and cleanup. The daily driver runs that saved +plan against the registered QA tenant; it does not substitute a local instance. diff --git a/tests/runner-e2e/api.ts b/tests/runner-e2e/api.ts index 5032c3e56f9..317401d7736 100644 --- a/tests/runner-e2e/api.ts +++ b/tests/runner-e2e/api.ts @@ -46,6 +46,12 @@ export class RunnerApi { return response.json() as Promise; } + async bytes(path: string): Promise { + const response = await this.request.get(path, this.sessionOptions()); + if (!response.ok()) throw new RunnerApiHttpError(response.status(), await failureMessage(response, "GET")); + return response.body(); + } + async post(path: string, data?: unknown, options?: { timeout: number }): Promise { const response = await this.request.post(path, { data, ...this.sessionOptions(), ...options }); if (!response.ok()) throw new Error(await failureMessage(response, "POST")); diff --git a/tests/runner-e2e/catalog.test.ts b/tests/runner-e2e/catalog.test.ts index d50c16df3c4..60b9f4a25ae 100644 --- a/tests/runner-e2e/catalog.test.ts +++ b/tests/runner-e2e/catalog.test.ts @@ -59,7 +59,7 @@ describe("runner E2E catalog", () => { }); it("exports deployment journeys separately from the model matrix and default selection", () => { - expect(productJourneys.map(journey => journey.id)).toEqual(["task-create", "ask-question", "task-properties", "cloud-onboarding"]); + expect(productJourneys.map(journey => journey.id)).toEqual(["task-create", "ask-question", "task-properties", "plan-revise-accept", "build-revise", "protected-actions", "create-skill-studio", "stop-redirect", "cloud-onboarding"]); expect(runnerSuites.some(suite => suite.id === "product-journeys")).toBe(false); expect(runnerMatrix.some(execution => execution.suite.id === "product-journeys")).toBe(false); expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(execution => execution.suite.id === "product-journeys")).toBe(false); diff --git a/tests/runner-e2e/connection-sensitive.test.ts b/tests/runner-e2e/connection-sensitive.test.ts new file mode 100644 index 00000000000..f67091692e4 --- /dev/null +++ b/tests/runner-e2e/connection-sensitive.test.ts @@ -0,0 +1,41 @@ +import { afterEach, expect, it, vi } from "vitest"; +import type { APIRequestContext } from "@playwright/test"; +import { ConnectionApi } from "./connection-target.js"; + +afterEach(() => vi.unstubAllGlobals()); +it("uses same-origin session cookies without sending fixture credentials through the traced request transport", async () => { + const cookie = (domain: string, path = "/") => ({ name: domain, value: "session", domain, path, expires: -1, secure: true }); + const request = { storageState: vi.fn(async () => ({ cookies: [cookie("qa.example.test"), cookie("other.example.test"), cookie("qa.example.test", "/other"), cookie("qa.example.test", "/api/company")] })), + fetch: vi.fn(() => { throw new Error("Sensitive body reached the traced transport"); }) }; + const send = vi.fn(async (_url: URL, options: RequestInit) => { + expect(options.redirect).toBe("error"); + expect(options.headers).toMatchObject({ Origin: "https://qa.example.test", Cookie: "qa.example.test=session" }); + expect(options.body).toBe(JSON.stringify({ credential: "fixture-only" })); + return Response.json({ id: "owned-connection" }); + }); + vi.stubGlobal("fetch", send); + const api = new ConnectionApi(request as unknown as APIRequestContext, "https://qa.example.test"); + await expect(api.postSensitive("/api/companies/owned/tools/apps/connect", { credential: "fixture-only" })).resolves.toEqual({ id: "owned-connection" }); + expect(request.fetch).not.toHaveBeenCalled(); + await expect(api.postSensitive("https://other.example.test/api/key", {})).rejects.toThrow("invalid_api_route"); + await expect(api.postSensitive("/login", {})).rejects.toThrow("invalid_api_route"); + expect(send).toHaveBeenCalledTimes(1); +}); +it.each(["/api/companies", "/api/companies/", "/api/companies/owned/tools/apps/connect"])("sends a cookie scoped to the exact path or a containing directory: %s", async (path) => { + const request = { storageState: async () => ({ cookies: [{ name: "scoped", value: "session", domain: "qa.example.test", path, expires: -1, secure: true }] }) } as unknown as APIRequestContext; + const send = vi.fn(async (_url: URL, options: RequestInit) => { + expect(options.headers).toMatchObject({ Cookie: "scoped=session" }); + return Response.json({ id: "owned" }); + }); + vi.stubGlobal("fetch", send); + await new ConnectionApi(request, "https://qa.example.test").postSensitive("/api/companies/owned/tools/apps/connect", {}); + expect(send).toHaveBeenCalledOnce(); +}); +it("withholds provider bodies and thrown transport details on sensitive failures", async () => { + const request = { storageState: async () => ({ cookies: [] }) } as unknown as APIRequestContext; + const api = new ConnectionApi(request, "https://qa.example.test"); + vi.stubGlobal("fetch", async () => new Response("SECRET response", { status: 400 })); + await expect(api.postSensitive("/api/connection", {})).rejects.toThrow("api_status_400"); + vi.stubGlobal("fetch", async () => { throw new Error("SECRET transport"); }); + await expect(api.postSensitive("/api/connection", {})).rejects.toThrow("sensitive_api_request_failed"); +}); diff --git a/tests/runner-e2e/connection-target.ts b/tests/runner-e2e/connection-target.ts index fafecc2c0e4..72234cdda30 100644 --- a/tests/runner-e2e/connection-target.ts +++ b/tests/runner-e2e/connection-target.ts @@ -34,6 +34,25 @@ export class ConnectionApi { async patch(route: string, data: unknown): Promise { return (await this.response(route, "PATCH", data)).json(); } async delete(route: string) { await this.response(route, "DELETE"); } async bytes(route: string) { return (await this.response(route)).body(); } + /** Keep fixture credentials out of Playwright request traces. */ + async postSensitive(route: string, data: unknown): Promise { + const target = new URL(route, this.origin); + if (!route.startsWith("/api/") || target.origin !== this.origin || !target.pathname.startsWith("/api/")) throw new ConnectionFailure("invalid_api_route"); + const { cookies } = await this.request.storageState(); + const cookie = cookies.filter(c => (target.hostname === c.domain || c.domain.startsWith(".") && target.hostname.endsWith(c.domain)) + && (target.pathname === c.path || target.pathname.startsWith(c.path.endsWith("/") ? c.path : `${c.path}/`)) + && (!c.secure || target.protocol === "https:") + && (c.expires === -1 || c.expires * 1000 > Date.now())).map(c => `${c.name}=${c.value}`).join("; "); + try { + const response = await fetch(target, { method: "POST", redirect: "error", signal: AbortSignal.timeout(30_000), + headers: { "content-type": "application/json", Origin: this.origin, ...(cookie ? { Cookie: cookie } : {}) }, body: JSON.stringify(data) }); + if (!response.ok) { await response.body?.cancel(); throw new ConnectionFailure(`api_status_${response.status}`); } + return await response.json() as T; + } catch (error) { + if (error instanceof ConnectionFailure) throw error; + throw new ConnectionFailure("sensitive_api_request_failed"); + } + } } async function stopOwnedServer(child: ChildProcess) { diff --git a/tests/runner-e2e/everyday-flow.ts b/tests/runner-e2e/everyday-flow.ts index 9cffbbe678c..9f24264026f 100644 --- a/tests/runner-e2e/everyday-flow.ts +++ b/tests/runner-e2e/everyday-flow.ts @@ -1,3 +1,7 @@ +import { stopAndRedirectThroughUi } from "./stop-redirect-flow.js"; +import { downloadStoryArtifact } from "./story-artifact.js"; +import { editCreatedSkillThroughStudio } from "./skill-studio-flow.js"; +import { decideProtectedActionThroughUi, readServiceBriefing, serviceBriefingChecks } from "./protected-action-flow.js"; import { gradeAgentmailSetup } from "./agentmail-setup-evidence.js"; import { CONNECTION_GUIDANCE_SUITE, CONNECTION_GUIDANCE_BUDGET_CENTS } from "./connection-guidance-cases.js"; import { gradeConnectionGuidanceDecline, hasConnectionGuidanceDeclineReply, explainsConnectionUnavailable, type ConnectionGuidanceDeclineInput } from "./connection-guidance-evidence.js"; @@ -477,41 +481,15 @@ export async function runEverydayFlow(input: Input) { if (!attachment) throw new Error("Selected delivery is no longer available"); const issue = ev.issues.find((i) => i.id === issueId)!; await openTask(issue); - const links = page.locator( - `a[href*="/api/attachments/${attachment.id}/content"]`, - ); - const link = links.filter({ hasText: /Download/i }).first(); - await expect(link).toBeVisible({ timeout: 15_000 }); - const pending = page.waitForEvent("download", { timeout: 30_000 }); - await link.click(); - const file = await pending; - const target = path.join(input.privateDir, "snapshots", `${phase}.zip`); - await file.saveAs(target); - const bytes = await readFile(target); - const digest = createHash("sha256").update(bytes).digest("hex"); - const result = await runCommand(process.env.PYTHON ?? "python3", [ - path.join(import.meta.dirname, "everyday-artifact.py"), - target, - "--mode", - mode, - ], Math.max(1, Math.min(60_000, input.deadlineAt - Date.now()))); - const oracle = JSON.parse(result.stdout) as { checks: StoryCheck[] }; - ev.checks.push( - ...oracle.checks.map((c) => ({ ...c, id: `${phase}.${c.id}` })), - ); - ev.downloads.push({ - phase, - attachmentId: attachment.id, - issueId, - sha256: digest, - bytes: bytes.length, - filename: file.suggestedFilename(), - mode, - }); + const delivered = await downloadStoryArtifact({ page, attachmentId: attachment.id, + directory: path.join(input.privateDir, "snapshots"), phase, mode, deadlineAt: input.deadlineAt }); + ev.checks.push(...delivered.checks.map(c => ({ ...c, id: `${phase}.${c.id}` }))); + const { checks: _checks, ...receipt } = delivered; + ev.downloads.push({ phase, issueId, mode, ...receipt }); note("download-verified", { phase, attachmentId: attachment.id, - sha256: digest, + sha256: delivered.sha256, }); await openParent(); } @@ -921,24 +899,20 @@ export async function runEverydayFlow(input: Input) { "Fault boundary was not exercised: no active execution", ); if (caseId === "stop-redirect") { - await page.getByTestId("task-chat-composer-stop").last().click(); ev.allowedInterruptedRuns.push(active.id); - note("stop-clicked", { runId: active.id }); - await pollUntil({ - label: "owned runner stopped", - deadlineAt: input.deadlineAt, - load: () => runnerStopped(active), - accept: Boolean, - intervalMs: 250, + await stopAndRedirectThroughUi({ page, nonce, + awaitRetirement: async () => { + note("stop-clicked", { runId: active.id }); + await pollUntil({ label: "owned runner stopped", deadlineAt: input.deadlineAt, + load: () => runnerStopped(active), accept: Boolean, intervalMs: 250 }); + }, + beforeRedirect: async () => { + await recordSource("source-saved-after-interruption", "source-after-interruption.json"); + stoppedWorkspace = await workspaceFiles(); + note("stopped-workspace-snapshot", stoppedWorkspace); + }, + reply: async message => { await reply(message); note("new-direction-submitted"); }, }); - await recordSource("source-saved-after-interruption", "source-after-interruption.json"); - stoppedWorkspace = await workspaceFiles(); - note("stopped-workspace-snapshot", stoppedWorkspace); - await reply( - `Change direction. Leave the project as it is. Reply with just this short note: "The studio is ready. Reference ${nonce}."`, - ); - note("new-direction-submitted"); - await page.reload(); } else { await reply(LATE_REQUIREMENT); note("followup-submitted-before-interruption"); @@ -1122,19 +1096,24 @@ export async function runEverydayFlow(input: Input) { .getByRole("button", { name: "Not now", exact: true }) .click(); } else { - const dismiss = page.getByRole("button", { - name: "Dismiss Approve tool action", - }); - if (await dismiss.isVisible()) await dismiss.click(); - await page - .getByRole("button", { name: "Review request", exact: true }) - .click(); - await page - .getByRole("button", { - name: declining ? "Decline" : "Approve & run", - exact: true, - }) - .click(); + if (review) await decideProtectedActionThroughUi({ page, + rows: [pendingInteraction], connectionId: review.connectionId, decline: declining, + invocationCount: review.invocationCount }); + else { + const dismiss = page.getByRole("button", { + name: "Dismiss Approve tool action", + }); + if (await dismiss.isVisible()) await dismiss.click(); + await page + .getByRole("button", { name: "Review request", exact: true }) + .click(); + await page + .getByRole("button", { + name: declining ? "Decline" : "Approve & run", + exact: true, + }) + .click(); + } } const decisionStatus = declining ? "rejected" : "accepted"; const decided = await pollUntil({ @@ -1178,56 +1157,13 @@ export async function runEverydayFlow(input: Input) { ); if (created) { await openParent(); - const card = page.getByRole("article", { - name: "Skill created: release-readiness-checklist", - }); - await expect(card).toHaveCount(1); - await expect(card).toBeVisible(); - await card.getByRole("button").click(); - await expect(page.getByRole("heading", { name: "release-readiness-checklist" })).toBeVisible(); - await expect(page.getByText("Verify checks.", { exact: true })).toBeVisible(); - check("feed-card-opened", true, "The task thread card opened the created skill sidebar."); - const openStudio = page.getByRole("button", { name: "Open in Skill Studio", exact: true }); - await expect(openStudio).toBeVisible(); - await openStudio.click(); - await expect(page).toHaveURL(new RegExp(`/skills/studio/${created.id}$`)); - // Studio's skill selector identifies the resource. Headings inside the - // authored document can differ from its canonical skill name. - await expect(page.getByRole("combobox").filter({ hasText: "release-readiness-checklist" })).toBeVisible(); - const editor = page.getByRole("textbox", { name: "editable markdown", exact: true }); - await editor.click(); - await editor.press("ControlOrMeta+End"); - await editor.press("Enter"); - await editor.press("Enter"); - await editor.pressSequentially("Studio edit marker: verified"); - await page.getByRole("button", { name: /^Save$/ }).click(); - await pollUntil({ - label: "Skill Studio edit persisted", - deadlineAt: Math.min(input.deadlineAt, Date.now() + 30_000), - load: () => api.get( - `/api/companies/${fixtures.company.id}/skills/${encodeURIComponent(String(created.id))}`, - ), - accept: (detail) => JSON.stringify(detail).includes("Studio edit marker: verified"), - }); - const detail = await api.get( - `/api/companies/${fixtures.company.id}/skills/${encodeURIComponent(String(created.id))}`, - ); - check( - "studio-edit-persisted", - JSON.stringify(detail).includes("Studio edit marker: verified"), - "The Skill Studio edit remains in the persisted skill after returning to the page.", - ); - check("studio-opened", true, "The skill detail opened in Skill Studio."); - await page.goBack(); - await expect(page).toHaveURL(new RegExp(`/issues/`)); - const returnedCard = page.getByRole("article", { - name: "Skill created: release-readiness-checklist", - }); - await expect(returnedCard).toBeVisible(); - await returnedCard.getByRole("button").click(); - await expect(page.getByRole("heading", { name: "release-readiness-checklist" })).toBeVisible(); - await expect(page.getByText("Studio edit marker: verified", { exact: true })).toBeVisible(); - check("return-content-persisted", true, "Returning to the task shows the saved Skill Studio edit."); + await editCreatedSkillThroughStudio({ page, skillId: created.id, skillName: created.name, + marker: "Studio edit marker: verified", deadlineAt: input.deadlineAt, + read: () => api.get(`/api/companies/${fixtures.company.id}/skills/${created.id}`) }); + check("feed-card-opened", true, "The task activity card opens the created skill."); + check("studio-opened", true, "The skill opens in Skill Studio."); + check("studio-edit-persisted", true, "The Studio edit persists across reload."); + check("return-content-persisted", true, "The saved edit is readable from the task card."); } } if (providerChoice) { @@ -1538,48 +1474,8 @@ export async function runEverydayFlow(input: Input) { "Exactly one approved service call; none after decline.", ); if (caseId === "service-approve") { - const docs = await api.get( - `/api/issues/${parent!.id}/documents`, - ); - const bodies = await Promise.all( - docs.map((d) => - api.get( - `/api/issues/${parent!.id}/documents/${encodeURIComponent(d.key)}`, - ), - ), - ); - const attachments = await api.get( - `/api/issues/${parent!.id}/attachments`, - ); - for (const attachment of attachments.filter( - (a) => - /\.(?:md|txt)$/i.test(a.originalFilename ?? a.filename ?? "") || - ["text/markdown", "text/plain"].includes(a.contentType), - )) { - const url = `/api/attachments/${attachment.id}/content`; - const response = await api.request.get(url); - if (!response.ok()) continue; - await expect(page.locator(`a[href*="${url}"]`).first()).toBeVisible(); - bodies.push({ - id: attachment.id, - title: attachment.originalFilename ?? attachment.filename, - body: await response.text(), - source: "delivered-attachment", - }); - } - const text = bodies - .map((d) => String(d.body ?? d.revision?.body ?? "")) - .join("\n"); - check( - "briefing-uses-real-result", - text.includes(`SERVICE_${nonce}`), - "A delivered issue document or Markdown attachment contains the actual service verification code.", - ); - check( - "briefing-includes-page-titles", - text.includes("Roadmap") && text.includes("Meeting notes"), - "The briefing includes both page titles returned by the service.", - ); + const bodies = await readServiceBriefing({ page, api, issueId: parent!.id }); + for (const result of serviceBriefingChecks(bodies, `SERVICE_${nonce}`)) check(result.id, result.passed, result.detail); ev.documents = bodies; ev.issues.find((i) => i.id === parent!.id)!.documents = bodies; } diff --git a/tests/runner-e2e/manual-private-policy.d.mts b/tests/runner-e2e/manual-private-policy.d.mts new file mode 100644 index 00000000000..d9dadc7bb8c --- /dev/null +++ b/tests/runner-e2e/manual-private-policy.d.mts @@ -0,0 +1,5 @@ +/** Public contract for the credential-free manual evidence authorization script. */ +export function manualEvidenceRouting(env: Record): { + retention: "recurring" | "actions" | "private"; executionIds: string[]; +}; +export function authorizeManualCellRoute(retention: string | undefined, executionId: string, privateIds: string): "actions" | "private"; diff --git a/tests/runner-e2e/plan-revision-flow.test.ts b/tests/runner-e2e/plan-revision-flow.test.ts new file mode 100644 index 00000000000..57d55a59897 --- /dev/null +++ b/tests/runner-e2e/plan-revision-flow.test.ts @@ -0,0 +1,38 @@ +import { expect, it, vi } from "vitest"; +import type { Page } from "@playwright/test"; +import { runPlanRevisionAcceptance } from "./plan-revision-flow.js"; + +const state = { + currentIssue: { status: "in_progress" }, + taskRuns: [{ status: "succeeded" }], + interactions: [{ id: "confirmation", kind: "request_confirmation", status: "pending", + payload: { target: { type: "issue_document", key: "plan", revisionId: "draft" } } }], +}; + +it("keeps the classic Plan story alive through a transient observation failure", async () => { + const reachedDocument = new Error("reached the canonical Plan read"); + const load = vi.fn().mockRejectedValueOnce(new Error("temporary read failure")).mockResolvedValue(state); + const get = vi.fn(async () => { throw reachedDocument; }); + await expect(runPlanRevisionAcceptance({ + page: {} as Page, api: { get }, origin: "https://qa.example.test", issue: { id: "owned" }, + issuePrefix: "QA", native: true, planMarkers: { draft: "draft", revised: "revised" }, + revisionRequest: "revise", deadlineAt: Date.now() + 5000, + load, reject: () => undefined, capture: async () => {}, + })).rejects.toBe(reachedDocument); + expect(load).toHaveBeenCalledTimes(2); + expect(get).toHaveBeenCalledWith("/api/issues/owned/documents/plan"); +}); + +it("lets deployment callers reject a binding failure without another observation", async () => { + const bindingFailure = new Error("owned task binding changed"); + const load = vi.fn().mockRejectedValueOnce(bindingFailure).mockResolvedValue(state); + const get = vi.fn(async () => { throw new Error("must not read an unbound document"); }); + await expect(runPlanRevisionAcceptance({ + page: {} as Page, api: { get }, origin: "https://qa.example.test", issue: { id: "owned" }, + issuePrefix: "QA", native: true, planMarkers: { draft: "draft", revised: "revised" }, + revisionRequest: "revise", deadlineAt: Date.now() + 5000, + load, retryOnError: error => error !== bindingFailure, reject: () => undefined, capture: async () => {}, + })).rejects.toBe(bindingFailure); + expect(load).toHaveBeenCalledOnce(); + expect(get).not.toHaveBeenCalled(); +}); diff --git a/tests/runner-e2e/plan-revision-flow.ts b/tests/runner-e2e/plan-revision-flow.ts new file mode 100644 index 00000000000..e8d7bd84a7b --- /dev/null +++ b/tests/runner-e2e/plan-revision-flow.ts @@ -0,0 +1,198 @@ +import { expect, type Page } from "@playwright/test"; +import { pollUntil } from "./api.js"; +import { numberedPlanStepCount } from "./run-observations.js"; + +export interface PlanInteraction { + id: string; kind?: string; status: string; + payload?: { acceptLabel?: string; rejectLabel?: string; target?: { type?: string; key?: string; revisionId?: string } }; +} +export interface PlanDocument { id: string; key: string; body?: string | null; latestRevisionId?: string | null; latestRevisionNumber?: number } +interface PlanState { + currentIssue: { status: string }; + taskRuns: Array<{ status: string }>; + interactions: PlanInteraction[]; +} +const TERMINAL_RUN_STATUSES = new Set(["succeeded", "interrupted", "failed", "cancelled", "timed_out"]); +export function isPendingPlanConfirmation(interaction: PlanInteraction) { + return interaction.kind === "request_confirmation" && interaction.status === "pending" + && interaction.payload?.target?.type === "issue_document" && interaction.payload.target.key === "plan" + && typeof interaction.payload.target.revisionId === "string"; +} +export function normalizePlanMarkdown(body: string | null | undefined) { return (body ?? "").replaceAll("\\_", "_"); } +export async function expectPlanStageVisible(page: Page, options: { native: boolean; revision: number | null }) { + const root = options.native ? page.locator('[data-testid="task-chat-turn"][data-settled="true"]') : page.locator("body"); + const preview = root.getByTestId("task-chat-plan-preview").last(); + await expect(preview).toBeVisible({ timeout: 30_000 }); + if (options.revision !== null) await expect(preview).toHaveAttribute("aria-label", `Open Plan revision ${options.revision}`); +} + +/** The classic matrix and deployment-targeted journeys share this exact user story. */ +export async function runPlanRevisionAcceptance(input: { + page: Page; api: { get(route: string): Promise }; origin: string; + issue: { id: string; identifier?: string | null }; issuePrefix: string; native: boolean; + planMarkers: { draft: string; revised: string }; revisionRequest: string; deadlineAt: number; + load(): Promise; + retryOnError?: (error: unknown) => boolean; + reject(state: { taskRuns: S["taskRuns"]; interactions: S["interactions"]; minimumRunCount: number }): string | undefined; + capture(id: string, label: string, file: string): Promise; +}) { + const { page, api, origin, issue, issuePrefix, native, planMarkers, revisionRequest, deadlineAt, load, reject, capture } = input; + const draftState = await pollUntil({ + label: `initial plan confirmation for issue ${issue.id}`, + deadlineAt, + load, + retryOnError: input.retryOnError, + accept: ({ currentIssue, taskRuns, interactions }) => + currentIssue.status !== "done" && + taskRuns.length >= 1 && + taskRuns.every((run) => TERMINAL_RUN_STATUSES.has(run.status)) && + interactions.some(isPendingPlanConfirmation), + reject: ({ currentIssue, taskRuns, interactions }) => + currentIssue.status === "done" ? "Plan completed before approval" : reject({ + taskRuns, + interactions, + minimumRunCount: 1, + }), + }); + const draftInteraction = draftState.interactions.find( + isPendingPlanConfirmation, + )!; + const draftPlan = await api.get( + `/api/issues/${issue.id}/documents/plan`, + ); + if ( + !normalizePlanMarkdown(draftPlan.body).includes(planMarkers.draft) + ) { + throw new Error( + `Initial Plan document did not contain ${planMarkers.draft}`, + ); + } + if (numberedPlanStepCount(draftPlan.body) !== 2) { + throw new Error( + "Initial Plan must contain exactly two numbered steps", + ); + } + if ( + draftInteraction.payload?.target?.revisionId !== + draftPlan.latestRevisionId + ) { + throw new Error( + "Initial plan confirmation did not target the latest Plan revision", + ); + } + await page.goto( + `${origin}/${encodeURIComponent(issuePrefix)}/issues/${encodeURIComponent(issue.identifier ?? issue.id)}`, + { waitUntil: "domcontentloaded" }, + ); + await expectPlanStageVisible(page, { + native, + revision: draftPlan.latestRevisionNumber ?? null, + }); + await capture( + "plan-draft", + "Initial plan awaiting revision", + "plan-draft.png", + ); + await page + .getByRole("button", { + name: draftInteraction.payload?.rejectLabel ?? "Reject", + exact: true, + }) + .last() + .click(); + const revisionComposer = page + .getByTestId("plan-revision-composer") + .last(); + await expect(revisionComposer).toBeVisible({ timeout: 10_000 }); + await revisionComposer + .locator('[contenteditable="true"], textarea') + .first() + .fill(revisionRequest); + await page + .getByRole("button", { + name: draftInteraction.payload?.rejectLabel ?? "Reject", + exact: true, + }) + .last() + .click(); + + const revisedState = await pollUntil({ + label: `revised plan confirmation for issue ${issue.id}`, + deadlineAt, + load, + retryOnError: input.retryOnError, + accept: ({ currentIssue, taskRuns, interactions }) => + currentIssue.status !== "done" && + taskRuns.length >= 2 && + taskRuns.every((run) => TERMINAL_RUN_STATUSES.has(run.status)) && + interactions.some( + (interaction) => + isPendingPlanConfirmation(interaction) && + interaction.id !== draftInteraction.id, + ), + reject: ({ currentIssue, taskRuns, interactions }) => + currentIssue.status === "done" ? "Plan completed before approval" : reject({ + taskRuns, + interactions, + minimumRunCount: 2, + }), + }); + const revisedInteraction = revisedState.interactions.find( + (interaction) => + isPendingPlanConfirmation(interaction) && + interaction.id !== draftInteraction.id, + )!; + const revisedPlan = await api.get( + `/api/issues/${issue.id}/documents/plan`, + ); + const normalizedRevisedPlan = normalizePlanMarkdown(revisedPlan.body); + if ( + !normalizedRevisedPlan.includes(planMarkers.revised) || + normalizedRevisedPlan.includes(planMarkers.draft) + ) { + throw new Error( + `Revised Plan must replace ${planMarkers.draft} with ${planMarkers.revised}`, + ); + } + if (numberedPlanStepCount(revisedPlan.body) !== 3) { + throw new Error( + "Revised Plan must contain exactly three numbered steps", + ); + } + if ( + revisedInteraction.payload?.target?.revisionId !== + revisedPlan.latestRevisionId || + revisedPlan.latestRevisionId === draftPlan.latestRevisionId + ) { + throw new Error( + "Revised confirmation did not target a new latest Plan revision", + ); + } + await page.goto( + `${origin}/${encodeURIComponent(issuePrefix)}/issues/${encodeURIComponent(issue.identifier ?? issue.id)}`, + { waitUntil: "domcontentloaded" }, + ); + await expectPlanStageVisible(page, { + native, + revision: revisedPlan.latestRevisionNumber ?? null, + }); + await capture( + "plan-revised", + "Revised plan awaiting acceptance", + "plan-revised.png", + ); + await page + .getByRole("button", { + name: revisedInteraction.payload?.acceptLabel ?? "Approve", + exact: true, + }) + .last() + .click(); + return { + draftInteraction, + draftPlan, + revisedInteraction, + revisedPlan, + revisionRequest, + }; +} diff --git a/tests/runner-e2e/product-journeys/agent-evidence.test.ts b/tests/runner-e2e/product-journeys/agent-evidence.test.ts new file mode 100644 index 00000000000..0157a707209 --- /dev/null +++ b/tests/runner-e2e/product-journeys/agent-evidence.test.ts @@ -0,0 +1,109 @@ +import { describe, expect, it } from "vitest"; +import { validateAgentTaskState } from "./agent-flows.js"; +import { activeNativeTurnStarted, nativeCancellationHasNoLaterWork, nativeStopAcknowledged, nativeWorkEvents } from "../stop-redirect-flow.js"; +import { parseTargetConfig } from "./catalog.js"; +import type { JourneyContext } from "./types.js"; +import { validateRetainedRunnerResult } from "../result-validation.js"; +import { agentJourneyRuns, journeyRequiredChecks } from "./success-contract.js"; + +const context = { company: { id: "qa-company" }, environmentId: "qa-daytona", + agent: { id: "owned-agent", provider: "opencode", model: "qa-model", runtimeMode: "native" } } as JourneyContext; +function state() { + return { currentIssue: { id: "owned-task", companyId: "qa-company", assigneeAgentId: "owned-agent", status: "done" }, + taskRuns: [{ id: "run", issueId: "owned-task", companyId: "qa-company", agentId: "owned-agent", status: "succeeded", runtimeMode: "native", + contextSnapshot: { paperclipEnvironment: { id: "qa-daytona" } }, runnerProfileJson: { nativeExecutionInput: { provider: { kind: "opencode", model: "qa-model" } } } }], + comments: [], interactions: [], attachments: [] }; +} +describe("deployed agent journey evidence", () => { + it.each(["tool.execution.started", "turn.started", "message.delta", "item.delta", "item.completed"])("rejects %s after cancellation even when cleanup's first snapshot already contains it", eventType => { + const event = (type: string) => ({ id: type, payload: { prpEvent: { eventType: type } } }); + const stopped = [event("turn.started"), event("item.delta"), event("turn.cancelled")]; + expect(nativeCancellationHasNoLaterWork(stopped)).toBe(true); + expect(nativeCancellationHasNoLaterWork([...stopped, event("usage.reported")])).toBe(true); + expect(nativeCancellationHasNoLaterWork([...stopped, event(eventType)])).toBe(false); + expect(nativeCancellationHasNoLaterWork(stopped.slice(0, -1))).toBe(false); + }); + it("requires an active native turn before the Stop boundary, including the native completion race", () => { + const run = { runtimeMode: "native", status: "running" }; + const event = (eventType: string) => ({ payload: { prpEvent: { eventType } } }); + expect(activeNativeTurnStarted(run, [])).toBe(false); + expect(activeNativeTurnStarted(run, [event("session.opened")])).toBe(false); + expect(activeNativeTurnStarted(run, [event("turn.started")])).toBe(true); + for (const terminal of ["turn.completed", "turn.failed", "turn.cancelled", "turn.interrupted"]) { + expect(activeNativeTurnStarted(run, [event("turn.started"), event(terminal)])).toBe(false); + } + expect(activeNativeTurnStarted({ ...run, status: "succeeded" }, [event("turn.started")])).toBe(false); + expect(activeNativeTurnStarted({ ...run, runtimeMode: "legacy" }, [event("turn.started")])).toBe(false); + }); + it.each(["tool.execution.started", "turn.started", "message.delta", "item.delta", "item.completed"])("rejects a quiet-Stop claim when the retired turn emits %s", eventType => { + const event = (type: string) => ({ id: type, payload: { prpEvent: { eventType: type, payload: { text: "old turn kept writing" } } } }); + const before = [event("turn.cancelled")]; + expect(nativeWorkEvents([...before, event(eventType)])).not.toEqual(nativeWorkEvents(before)); + expect(nativeWorkEvents([...before, event("usage.reported")])).toEqual(nativeWorkEvents(before)); + }); + it("accepts bound native work and fails plausible passes from a different target, agent, model or task", () => { + expect(() => validateAgentTaskState(context, "owned-task", state(), 3)).not.toThrow(); + const substitutions = [ + (s: ReturnType) => { s.currentIssue.companyId = "other"; }, + (s: ReturnType) => { s.currentIssue.assigneeAgentId = "other"; }, + (s: ReturnType) => { s.taskRuns[0]!.issueId = "other"; }, + (s: ReturnType) => { s.taskRuns[0]!.agentId = "other"; }, + (s: ReturnType) => { s.taskRuns[0]!.contextSnapshot.paperclipEnvironment.id = "local"; }, + (s: ReturnType) => { s.taskRuns[0]!.runnerProfileJson.nativeExecutionInput.provider.model = "other"; }, + (s: ReturnType) => { s.taskRuns[0]!.runtimeMode = "legacy"; }, + (s: ReturnType) => { s.taskRuns[0]!.status = "failed"; }, + (s: ReturnType) => { s.taskRuns[0]!.status = "cancelled"; }, + ]; + for (const substitute of substitutions) { const s = state(); substitute(s); expect(() => validateAgentTaskState(context, "owned-task", s, 3)).toThrow(); } + const s = state(); s.taskRuns.push(s.taskRuns[0]!); + expect(() => validateAgentTaskState(context, "owned-task", s, 1)).toThrow(); + }); + it("permits only the explicitly stopped run, while still checking its deployment binding", () => { + const s = state(); s.taskRuns[0]!.status = "cancelled"; + expect(() => validateAgentTaskState(context, "owned-task", s, 3, ["run"])).not.toThrow(); + s.taskRuns[0]!.companyId = "other"; + expect(() => validateAgentTaskState(context, "owned-task", s, 3, ["run"])).toThrow(); + }); + it("requires explicit paid run limits for a mixed deployment campaign", () => { + const config = { version: 1, id: "qa", mode: "existing-cloud", adapterModule: "/private/provider.mjs", + executionEnvironment: "daytona", journeys: ["task-create", "ask-question", "plan-revise-accept", "build-revise", "protected-actions", "create-skill-studio", "stop-redirect"], + baseline: { features: {}, connections: [] }, limits: { durationMs: 1_800_000, modelBudgetCents: 500, maxRuns: 14 } }; + expect(parseTargetConfig(config).journeys).toEqual(config.journeys); + expect(() => parseTargetConfig({ ...config, limits: { ...config.limits, maxRuns: 1 } })).toThrow("agent_journeys_require_bounded_runs"); + }); + it("does not mistake a cancelled status or pending native intent for an acknowledged Stop", () => { + const run = { id: "run", companyId: "qa-company", issueId: "owned-task", status: "cancelled", runtimeMode: "native", resultJson: { cancelledByActorType: "user", cancelledByUserId: "qa-user", + cancellation: { source: "operator", expected: true, initiator: { type: "user", id: "qa-user" } }, + startupCancellation: { requestedAt: "2026-10-29T00:00:00Z", beforeNativeSelection: false }, nativeCancellation: { + schema: "paperclip.native-cancellation.v1", intentId: "native-cancellation:00000000-0000-4000-8000-000000000001", companyId: "qa-company", runId: "run", issueId: "owned-task", scope: "run", + dispatched: true, dispatchState: "acknowledged", reasonCode: "cancellation_run_only", effects: ["release_run_resources"], intentAuditId: "intent", acknowledgementAuditId: "ack" } } }; + expect(nativeStopAcknowledged(run, "qa-company", "owned-task", "qa-user")).toBe(true); + for (const change of [{ dispatchState: "pending" }, { runId: "other" }, { scope: "task" }, { effects: [] }, { acknowledgementAuditId: "intent" }]) { + const changed = structuredClone(run); Object.assign(changed.resultJson.nativeCancellation, change); + expect(nativeStopAcknowledged(changed, "qa-company", "owned-task", "qa-user")).toBe(false); + } + expect(nativeStopAcknowledged(run, "qa-company", "owned-task", "other-user")).toBe(false); + expect(nativeStopAcknowledged({ ...run, resultJson: {} }, "qa-company", "owned-task", "qa-user")).toBe(false); + }); + it.each(Object.keys(agentJourneyRuns))("retained %s successes require every story check and the exact run receipts", id => { + const count = agentJourneyRuns[id as keyof typeof agentJourneyRuns]; + const result = { schema: "paperclip.runner-e2e.result/v2", executionId: `product-journeys.qa.configured-agent.daytona.${id}`, + suiteId: "product-journeys", suiteDefinitionHash: "test", profileId: "configured-agent", environmentId: "daytona", caseId: id, + provider: "opencode", model: "qa-model", runtimeMode: "native", status: "passed", cleanup: "passed", attempt: 1, + startedAt: "2026-10-29T00:00:00Z", finishedAt: "2026-10-29T00:00:01Z", durationMs: 1000, + runIds: Array.from({ length: count }, (_, index) => `owned-run-${index}`), + productJourney: { outcome: "passed", startingState: "recipe", privacy: "private", resourceDisposition: "retained", + phases: [{ phase: "journey", outcome: "passed" }], checks: journeyRequiredChecks[id]!.map(id => ({ id, passed: true })), + deployment: { kind: "cloud", targetId: "qa", origin: "https://qa.test.paperclip.app", commit: "a".repeat(40), + cloudRelease: "b".repeat(40), runtimeImage: "pinned-image", configurationHash: "test" } } }; + expect(() => validateRetainedRunnerResult(result)).not.toThrow(); + for (let index = 0; index < result.productJourney.checks.length; index++) { + const missing = structuredClone(result); missing.productJourney.checks.splice(index, 1); + expect(() => validateRetainedRunnerResult(missing)).toThrow("result.productJourney"); + } + const extraRun = structuredClone(result); extraRun.runIds.push("foreign-run"); + expect(() => validateRetainedRunnerResult(extraRun)).toThrow("result.agentJourneyRun"); + const falseCheck = structuredClone(result); falseCheck.productJourney.checks.push({ id: "independent-extra-check", passed: false }); + expect(() => validateRetainedRunnerResult(falseCheck)).toThrow("result.productJourney"); + }); +}); diff --git a/tests/runner-e2e/product-journeys/agent-flows.ts b/tests/runner-e2e/product-journeys/agent-flows.ts new file mode 100644 index 00000000000..0ec4f5b98da --- /dev/null +++ b/tests/runner-e2e/product-journeys/agent-flows.ts @@ -0,0 +1,248 @@ +import { createHash } from "node:crypto"; +import { mkdir, writeFile } from "node:fs/promises"; +import path from "node:path"; +import { expect } from "@playwright/test"; +import { activeNativeTurnStarted, nativeCancellationHasNoLaterWork, nativeStopAcknowledged, nativeWorkEvents, stopAndRedirectThroughUi } from "../stop-redirect-flow.js"; +import { runnerTasks } from "../catalog.js"; +import { everydayTasks, SLUGIFY_REQUIREMENTS, SLUGIFY_REVISION } from "../everyday-cases.js"; +import { latestStoryDelivery, type StoryDelivery } from "../everyday-delivery.js"; +import { gradeConnectionGuidanceDecline } from "../connection-guidance-evidence.js"; +import { runPlanRevisionAcceptance } from "../plan-revision-flow.js"; +import { decideProtectedActionThroughUi, readServiceBriefing, serviceBriefingChecks } from "../protected-action-flow.js"; +import { editCreatedSkillThroughStudio } from "../skill-studio-flow.js"; +import { downloadStoryArtifact, runArtifactOracle } from "../story-artifact.js"; +import { collectRunEvents } from "../run-observations.js"; +import { createTaskThroughUi, submitTaskReply } from "../user-actions.js"; +import { agentJourneyRuns } from "./catalog.js"; +import { JourneyError, type JourneyContext, type JourneyEvidence, type TargetSession } from "./types.js"; + +type Row = Record; +type AgentJourney = Exclude; +interface TaskState { currentIssue: Row & { id: string; status: string }; taskRuns: Array; comments: Row[]; interactions: Array; attachments: StoryDelivery[] } +interface Options { deadlineAt: number; directory: string; capture(id: string, label: string, file: string): Promise } +const active = (run: Row) => ["queued", "running"].includes(run.status); + +/** Owned-task observations only: other work in a persistent QA company cannot affect this grade. */ +export function validateAgentTaskState(context: JourneyContext, issueId: string, state: TaskState, maxRuns: number, allowedCancelled: string[] = []) { + const agent = context.agent!; + if (state.currentIssue.id !== issueId || state.currentIssue.companyId !== context.company.id + || state.currentIssue.assigneeAgentId !== agent.id || state.taskRuns.length > maxRuns) throw new JourneyError("failed", "agent_task_binding_mismatch"); + for (const run of state.taskRuns) { + const provider = run.runnerProfileJson?.nativeExecutionInput?.provider; + const boundIssueId = run.issueId ?? run.nativeIssueId ?? run.contextSnapshot?.issueId ?? run.contextSnapshot?.taskId; + if (run.companyId !== context.company.id || run.agentId !== agent.id || boundIssueId !== issueId) throw new JourneyError("failed", "agent_run_binding_mismatch"); + if (!active(run) && (run.runtimeMode !== "native" || run.contextSnapshot?.paperclipEnvironment?.id !== context.environmentId + || provider?.kind !== agent.provider || provider?.model !== agent.model)) throw new JourneyError("failed", "agent_run_binding_mismatch"); + if (!["queued", "running", "succeeded"].includes(run.status) && !(run.status === "cancelled" && allowedCancelled.includes(run.id))) throw new JourneyError("failed", "agent_run_failed"); + } +} + +async function waitFor(deadlineAt: number, load: () => Promise, accept: (value: T) => boolean, code: string): Promise { + while (Date.now() < deadlineAt) { + const value = await load(); + if (accept(value)) return value; + await new Promise(resolve => setTimeout(resolve, 1000)); + } + throw new JourneyError("failed", code); +} + +export async function runAgentJourney(id: AgentJourney, context: JourneyContext, session: TargetSession, + campaignId: string, evidence: JourneyEvidence, options: Options) { + const agent = context.agent; + if (!agent || agent.runtimeMode !== "native" || !context.environmentId) throw new JourneyError("unsupported", "native_agent_and_environment_required"); + const nonce = campaignId.replaceAll("-", "_"); + const maxRuns = agentJourneyRuns[id]; + const cancelled: string[] = []; + await mkdir(options.directory, { recursive: true, mode: 0o700 }); + const check = (id: string, passed = true) => { + evidence.checks.push({ id, passed }); + if (!passed) throw new JourneyError("failed", id.replaceAll("-", "_")); + }; + const retain = async (name: string, value: unknown) => writeFile(path.join(options.directory, name), JSON.stringify(value, null, 2), { mode: 0o600 }); + const open = async (issue: Row) => context.page.goto(`${context.api.origin}/${context.company.issuePrefix}/issues/${issue.identifier ?? issue.id}`, { waitUntil: "domcontentloaded" }); + const create = async (suffix: string, prompt: string, workMode: "standard" | "planning" = "standard") => { + const title = `Journey QA ${id}${suffix} ${campaignId}`; + const created = await createTaskThroughUi({ page: context.page, origin: context.api.origin, issuePrefix: context.company.issuePrefix, + agentName: agent.name, title, prompt, workMode, requireExplicitTitle: true, + beforeDispatch: () => session.own({ kind: "issue", name: title, journeyId: id }) }); + await session.own({ kind: "issue", name: title, id: created.issueId, journeyId: id }); + return context.api.get(`/api/issues/${created.issueId}`); + }; + const read = async (issue: Row): Promise => { + const route = `/api/issues/${issue.id}`; + const [currentIssue, summaries, comments, interactions, attachments] = await Promise.all([ + context.api.get(route), context.api.get(`${route}/runs`), context.api.get(`${route}/comments?order=asc`), + context.api.get(`${route}/interactions`), context.api.get(`${route}/attachments`), + ]); + if (!Array.isArray(summaries) || summaries.length > maxRuns) throw new JourneyError("failed", "agent_run_limit_exceeded"); + const taskRuns = await Promise.all(summaries.map(summary => context.api.get(`/api/heartbeat-runs/${summary.runId}`))); + const state = { currentIssue, taskRuns, comments, interactions, attachments }; + await retain(`${issue.id}-latest.json`, state); + validateAgentTaskState(context, issue.id, state, maxRuns, cancelled); + return state; + }; + const settled = (issue: Row, count: number) => waitFor(options.deadlineAt, () => read(issue), state => + state.currentIssue.status === "done" && state.taskRuns.length === count && state.taskRuns.every(run => !active(run)) + && state.interactions.every(row => row.status !== "pending"), "agent_task_not_completed"); + const reply = async (issue: Row, message: string) => { + const before = (await read(issue)).comments; + await open(issue); + await submitTaskReply(context.page, message); + const after = await waitFor(Math.min(options.deadlineAt, Date.now() + 30_000), () => read(issue), state => + state.comments.some(row => row.authorUserId === context.user.id && !before.some(old => old.id === row.id)), "reply_not_persisted"); + const added = after.comments.filter(row => row.authorUserId === context.user.id && !before.some(old => old.id === row.id)); + check("user-reply-stored-once", added.length === 1 && added[0].body === message); + return added[0]; + }; + const download = async (issue: Row, state: TaskState, phase: string, mode: "base" | "separator", preserved: string[] = []) => { + const delivery = latestStoryDelivery(state.attachments, [issue.id], undefined, preserved); + if (!delivery) throw new JourneyError("failed", "artifact_delivery_missing"); + await open(issue); + const receipt = await downloadStoryArtifact({ page: context.page, attachmentId: delivery.id, directory: options.directory, + phase, mode, deadlineAt: options.deadlineAt }); + await retain(`${phase}-download.json`, receipt); + check(`${phase}-artifact-verified`, receipt.checks.length > 0 && receipt.checks.every(row => row.passed)); + return receipt; + }; + + if (id === "plan-revise-accept") { + const fixture = runnerTasks.find(task => task.id === id)!; + const issue = await create("", fixture.buildPrompt(nonce), "planning"); + const receipt = await runPlanRevisionAcceptance({ page: context.page, api: context.api, origin: context.api.origin, + issue: { id: issue.id, identifier: issue.identifier }, issuePrefix: context.company.issuePrefix, native: true, + planMarkers: fixture.buildPlanMarkers!(nonce), revisionRequest: fixture.buildRevisionRequest!(nonce), deadlineAt: options.deadlineAt, + load: () => read(issue), retryOnError: error => !(error instanceof JourneyError), reject: () => undefined, capture: options.capture }); + await retain("plan-revision.json", receipt); + const final = await settled(issue, 3); + const marker = fixture.buildVisibleMarker(nonce); + check("plan-revision-and-approval", receipt.draftInteraction.id !== receipt.revisedInteraction.id); + check("plan-final-answer", final.comments.some(row => row.authorAgentId === agent.id && row.body?.trim() === marker)); + await open(issue); await context.page.reload(); + await expect(context.page.getByText(marker, { exact: true }).last()).toBeVisible(); + } else if (id === "build-revise") { + await runArtifactOracle(null, "base", options.deadlineAt); + const issue = await create("", SLUGIFY_REQUIREMENTS); + const initial = await download(issue, await settled(issue, 1), "initial", "base"); + await reply(issue, SLUGIFY_REVISION); + const revised = await download(issue, await settled(issue, 2), "revised", "separator", [initial.sha256]); + check("artifact-versions-differ", revised.attachmentId !== initial.attachmentId && revised.sha256 !== initial.sha256); + check("original-artifact-preserved", createHash("sha256").update(await context.api.bytes(`/api/attachments/${initial.attachmentId}/content`)).digest("hex") === initial.sha256); + } else if (id === "create-skill-studio") { + const skillName = `qa-skill-${campaignId}`; + const fixture = everydayTasks.find(task => task.id === id)!; + const prompt = fixture.buildPrompt(nonce).replaceAll("release-readiness-checklist", skillName).replaceAll("create-skill-e2e-001", campaignId); + await session.own({ kind: "skill", name: skillName }); + const issue = await create("", prompt); + const completed = await settled(issue, 1); + const skills = await context.api.get(`/api/companies/${context.company.id}/skills`); + const owned = skills.filter(skill => skill.slug === skillName && skill.name === skillName); + check("created-skill-persisted", owned.length === 1); + const skill = owned[0]; + await session.own({ kind: "skill", name: skillName, id: skill.id }); + const activity = await context.api.get(`/api/issues/${issue.id}/activity`); + check("skill-creation-attributed", activity.filter(row => row.action === "company.skill_created" && row.entityId === skill.id + && row.runId === completed.taskRuns[0]!.id && row.agentId === agent.id && row.details?.sourceIssueId === issue.id).length === 1); + await open(issue); + await editCreatedSkillThroughStudio({ page: context.page, skillId: skill.id, skillName, + marker: `Studio edit marker: ${campaignId}`, deadlineAt: options.deadlineAt, + read: () => context.api.get(`/api/companies/${context.company.id}/skills/${skill.id}`) }); + check("skill-studio-edit-persisted"); + } else if (id === "protected-actions") { + const service = context.fixtures?.reviewService; + if (!service) throw new JourneyError("unsupported", "review_service_required"); + for (const decline of [false, true]) { + const phase = decline ? "decline" : "approve"; + const callsBefore = service.invocationCount(); + const connectionBefore = await context.api.get(`/api/tool-connections/${service.connectionId}`); + const fixture = everydayTasks.find(task => task.id === `service-${phase}`)!; + const issue = await create(`-${phase}`, `${fixture.buildPrompt(nonce)}\nUse only the connected service named "${service.name}". Do not use any other service.`); + const pending = await waitFor(options.deadlineAt, () => read(issue), state => + state.taskRuns.length === 1 && !state.taskRuns.some(active) && state.interactions.some(row => row.status === "pending"), "protected_action_not_requested"); + check(`${phase}-waited-for-decision`, pending.currentIssue.status !== "done" && service.invocationCount() === callsBefore); + await open(issue); await context.page.reload(); + const decision = await decideProtectedActionThroughUi({ page: context.page, rows: pending.interactions as any, + connectionId: service.connectionId, decline, invocationCount: service.invocationCount, expectedCallsBefore: callsBefore }); + const final = await settled(issue, 2); + check(`${phase}-decision-persisted`, final.interactions.some(row => row.id === decision.id && row.status === (decline ? "rejected" : "accepted"))); + check(`${phase}-service-call-count`, service.invocationCount() === callsBefore + (decline ? 0 : 1)); + if (decline) { + const connectionAfter = await context.api.get(`/api/tool-connections/${service.connectionId}`); + const unchanged = (row: Row) => JSON.stringify({ id: row.id, companyId: row.companyId, status: row.status, enabled: row.enabled, + installs: row.installs?.filter((install: Row) => install.targetType === "agent" && install.targetId === agent.id) }); + const checks = gradeConnectionGuidanceDecline({ caseId: "service-decline", decisionId: decision.id, + decisions: final.interactions as any, leadAgentId: agent.id, issueId: issue.id, replies: final.comments, + runs: final.taskRuns as any, marker: service.marker, calls: service.invocationCount() - callsBefore, sameConnections: unchanged(connectionBefore) === unchanged(connectionAfter) }); + await retain("service-decline-grade.json", checks); + check("decline-truthful-explanation", checks.every(row => row.passed)); + } else { + await open(issue); + const bodies = await readServiceBriefing({ page: context.page, api: context.api, issueId: issue.id }); + await retain("service-briefing.json", bodies); + check("approved-service-data-used", serviceBriefingChecks(bodies, service.marker).every(row => row.passed)); + } + await open(issue); + } + } else { + await runArtifactOracle(null, "base", options.deadlineAt); + const issue = await create("", SLUGIFY_REQUIREMENTS); + const initial = await download(issue, await settled(issue, 1), "checkpoint", "base"); + await reply(issue, SLUGIFY_REVISION); + const state = await waitFor(options.deadlineAt, () => read(issue), value => value.taskRuns.some(run => run.status === "running"), "stop_boundary_not_active"); + const original = state.taskRuns.find(run => run.status === "running")!; + await open(issue); + const originalEvents = () => collectRunEvents((afterSeq, limit) => + context.api.get(`/api/heartbeat-runs/${original.id}/events?afterSeq=${afterSeq}&limit=${limit}`)); + const boundary = await waitFor(options.deadlineAt, async () => { + const events = await originalEvents(); + await retain("stop-startup-events.json", events); + const current = await read(issue); + const run = current.taskRuns.find(row => row.id === original.id); + if (!run || !active(run)) throw new JourneyError("failed", "stop_boundary_already_settled"); + return { state: current, run, events }; + }, value => activeNativeTurnStarted(value.run, value.events), "stop_native_turn_not_active"); + await retain("stop-before.json", boundary.state); + cancelled.push(original.id); + const runActivity = async () => nativeWorkEvents(await originalEvents()); + let beforeEvents: Row[]; + const redirected = await stopAndRedirectThroughUi({ page: context.page, nonce, + awaitRetirement: async () => { + const stopped = await waitFor(options.deadlineAt, () => read(issue), value => { + const run = value.taskRuns.find(row => row.id === original.id); + return run?.status === "cancelled" && nativeStopAcknowledged(run, context.company.id, issue.id, context.user.id); + }, "stop_not_acknowledged"); + await retain("stop-after.json", stopped); + beforeEvents = await waitFor(options.deadlineAt, async () => { + const events = await runActivity(); + await retain("stop-cancellation-events.json", events); + return events; + }, events => events.some(row => row.payload?.prpEvent?.eventType === "turn.cancelled"), "stop_native_terminal_not_retained"); + check("stop-native-terminal-retained", beforeEvents.some(row => row.payload?.prpEvent?.eventType === "turn.cancelled")); + if (!nativeCancellationHasNoLaterWork(beforeEvents)) check("stopped-run-no-later-events", false); + }, + beforeRedirect: async () => { + const leases = await waitFor(options.deadlineAt, () => context.api.get(`/api/environments/${context.environmentId}/leases`), rows => { + const owned = rows.filter(row => row.companyId === context.company.id && row.issueId === issue.id && row.heartbeatRunId === original.id); + return owned.length > 0 && owned.every(row => ["released", "retained", "expired"].includes(row.status) && row.cleanupStatus === "success"); + }, "stopped_run_resources_not_released"); + await retain("stop-leases.json", leases.filter(row => row.issueId === issue.id && row.heartbeatRunId === original.id)); + check("active-work-stopped"); + }, + reply: message => reply(issue, message), + }); + const marker = redirected.marker; + const userReply = redirected.reply as Row; + const final = await settled(issue, 3); + check("new-direction-completed", final.comments.some(row => row.authorAgentId === agent.id && row.body?.trim() === marker + && row.createdByRunId !== original.id)); + check("stopped-run-remained-terminal", final.taskRuns.find(run => run.id === original.id)?.status === "cancelled"); + const afterEvents = await runActivity(); + check("stopped-run-no-later-events", nativeCancellationHasNoLaterWork(afterEvents) + && JSON.stringify(beforeEvents!) === JSON.stringify(afterEvents)); + check("stop-preserved-download", createHash("sha256").update(await context.api.bytes(`/api/attachments/${initial.attachmentId}/content`)).digest("hex") === initial.sha256); + const queue = await context.api.get(`/api/issues/${issue.id}/queued-comments`); + check("redirect-consumed-once", queue.entries.length === 0 && final.comments.filter(row => row.id === userReply.id).length === 1); + await open(issue); await context.page.reload(); + await expect(context.page.getByText(marker, { exact: true }).last()).toBeVisible(); + } + check("configured-model-and-environment"); +} diff --git a/tests/runner-e2e/product-journeys/catalog.ts b/tests/runner-e2e/product-journeys/catalog.ts index 30f1d516faa..9ac222f0079 100644 --- a/tests/runner-e2e/product-journeys/catalog.ts +++ b/tests/runner-e2e/product-journeys/catalog.ts @@ -1,11 +1,20 @@ +import { agentJourneyRuns } from "./success-contract.js"; +export { agentJourneyRuns } from "./success-contract.js"; import type { JourneyId, TargetConfig } from "./types.js"; export const productJourneys = [ { id: "task-create", label: "Create a task for yourself and reopen it", requires: ["company", "board-user", "task-write"], capabilities: ["tasks/create"], paid: false }, { id: "ask-question", label: "Ask an agent a question and read its answer", requires: ["company", "board-user", "task-write", "agent"], capabilities: ["questions-and-approvals/questions"], paid: true }, { id: "task-properties", label: "Assign a task to the signed-in user", requires: ["company", "board-user", "task-write", "backlog-task"], capabilities: ["tasks/properties"], paid: false }, + { id: "plan-revise-accept", label: "Revise and approve a saved Plan, then complete the task", requires: ["company", "board-user", "task-write", "agent"], capabilities: ["questions-and-approvals/plan-review"], paid: true }, + { id: "build-revise", label: "Download and independently verify both artifact versions", requires: ["company", "board-user", "task-write", "agent"], capabilities: ["documents-artifacts/attachments"], paid: true }, + { id: "protected-actions", label: "Approve and decline protected service actions", requires: ["company", "board-user", "task-write", "agent", "review-service"], capabilities: ["questions-and-approvals/tool-review"], paid: true }, + { id: "create-skill-studio", label: "Create a company skill and persist a Skill Studio edit", requires: ["company", "board-user", "task-write", "agent", "skill-write"], capabilities: ["skills/authoring"], paid: true }, + { id: "stop-redirect", label: "Stop active work and send a new direction once", requires: ["company", "board-user", "task-write", "agent"], capabilities: ["steering/stop-and-pause"], paid: true }, { id: "cloud-onboarding", label: "Create a Cloud workspace and complete a first task", requires: ["cloud-account", "fresh-workspace", "provider-key"], capabilities: ["onboarding/first-task"], paid: true }, ] as const; +export function isAgentJourney(id: JourneyId): id is keyof typeof agentJourneyRuns { return Object.hasOwn(agentJourneyRuns, id); } + export function preparationPreview(config: TargetConfig) { return { target: { id: config.id, mode: config.mode, executionEnvironment: config.executionEnvironment }, @@ -40,7 +49,8 @@ export function parseTargetConfig(value: unknown): TargetConfig { if (c.mode === "new-cloud" && c.baseline.connections.length) throw new Error("onboarding_requires_ui_connection_setup"); if (c.mode !== "new-cloud" && c.journeys.includes("cloud-onboarding")) throw new Error("onboarding_requires_fresh_cloud_workspace"); if (c.journeys.includes("cloud-onboarding") && (!c.limits.modelBudgetCents || !c.limits.maxRuns)) throw new Error("paid_journey_requires_limits"); - if (c.journeys.includes("ask-question") && (!c.limits.modelBudgetCents || c.limits.maxRuns !== 1)) throw new Error("question_requires_one_bounded_run"); + const requiredRuns = c.journeys.reduce((total, id) => total + (isAgentJourney(id) ? agentJourneyRuns[id] : 0), 0); + if (requiredRuns && (!c.limits.modelBudgetCents || c.limits.maxRuns !== requiredRuns)) throw new Error(c.journeys.includes("ask-question") && requiredRuns === 1 ? "question_requires_one_bounded_run" : "agent_journeys_require_bounded_runs"); if (c.mode !== "managed-self-hosted" && !c.adapterModule) throw new Error("cloud_target_requires_private_adapter"); if (c.mode === "managed-self-hosted" && c.executionEnvironment !== "local") throw new Error("unsupported_self_hosted_environment"); return { ...c, ...(c.journeys.includes("cloud-onboarding") ? { modelProfileId: c.modelProfileId ?? "runner-codex" as const } : {}), headed: c.headed ?? false, journeys: c.journeys as JourneyId[] }; diff --git a/tests/runner-e2e/product-journeys/fixtures.ts b/tests/runner-e2e/product-journeys/fixtures.ts index efd632b761e..2d057b3cf69 100644 --- a/tests/runner-e2e/product-journeys/fixtures.ts +++ b/tests/runner-e2e/product-journeys/fixtures.ts @@ -11,7 +11,7 @@ export function journeyFixtures(session: TargetSession, campaignId?: string) { id: "ready-context", async setup() { return ready ??= await (preparation ??= session.prepare()); }, }); - for (const id of ["company", "board-user", "task-write", "agent"]) registry.register({ + for (const id of ["company", "board-user", "task-write", "agent", "skill-write", "review-service"]) registry.register({ id, dependencies: ["ready-context"], async setup(resolved) { const context = resolved.get("ready-context") as JourneyContext; diff --git a/tests/runner-e2e/product-journeys/launch.ts b/tests/runner-e2e/product-journeys/launch.ts index f98430df099..bb18c868ea4 100644 --- a/tests/runner-e2e/product-journeys/launch.ts +++ b/tests/runner-e2e/product-journeys/launch.ts @@ -6,10 +6,11 @@ import path from "node:path"; import { pathToFileURL } from "node:url"; import { chromium, type Browser, type BrowserContext } from "@playwright/test"; import { runnerSuites } from "../catalog.js"; -import { productJourneys, parseTargetConfig, preparationPreview } from "./catalog.js"; +import { productJourneys, parseTargetConfig, preparationPreview, isAgentJourney, agentJourneyRuns } from "./catalog.js"; import { journeyFixtures } from "./fixtures.js"; import { taskProperties } from "./task-properties.js"; import { askQuestion, taskCreate } from "./basic-flows.js"; +import { runAgentJourney } from "./agent-flows.js"; import { JourneyError, type JourneyContext, type JourneyEvidence, type JourneyOutcome, type TargetLifecycleInput, type TargetProvider, type TargetSession } from "./types.js"; import type { RunnerE2EResult } from "../types.js"; import { regenerateRunnerDashboard } from "../dashboard-regenerate.js"; @@ -101,11 +102,11 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri try { sha = execFileSync("git", ["rev-parse", "HEAD"], { cwd: repositoryRoot, encoding: "utf8", stdio: ["ignore", "pipe", "ignore"] }).trim(); } catch { /* Packaged harnesses may not include Git metadata. */ } const source = resolveRunnerE2ESource({ sha, ref: null, workflowRunUrl: null }); const definitionHash = createHash("sha256").update(JSON.stringify(productJourneys)); - for (const name of ["catalog.ts", "task-properties.ts", "basic-flows.ts", "basic-evidence.ts", "product-screenshot.ts", "cloud-onboarding.ts", "onboarding-connect.ts", "onboarding-first-task.ts", "workspace-deferral.ts", "onboarding-pre-acceptance.ts", "artifact-evidence.ts", "fixtures.ts", "launch.ts", "types.ts", "collection.ts", "local-target.ts"]) definitionHash.update(await readFile(path.join(import.meta.dirname, name))); - definitionHash.update(await readFile(path.join(import.meta.dirname, "../user-actions.ts"))); + for (const name of ["catalog.ts", "task-properties.ts", "basic-flows.ts", "basic-evidence.ts", "agent-flows.ts", "success-contract.ts", "product-screenshot.ts", "cloud-onboarding.ts", "onboarding-connect.ts", "onboarding-first-task.ts", "workspace-deferral.ts", "onboarding-pre-acceptance.ts", "artifact-evidence.ts", "fixtures.ts", "launch.ts", "types.ts", "collection.ts", "local-target.ts"]) definitionHash.update(await readFile(path.join(import.meta.dirname, name))); + for (const file of ["catalog.ts", "api.ts", "run-observations.ts", "user-actions.ts", "plan-revision-flow.ts", "skill-studio-flow.ts", "protected-action-flow.ts", "story-artifact.ts", "stop-redirect-flow.ts", "everyday-cases.ts", "everyday-delivery.ts", "everyday-decisions.ts", "connection-guidance-evidence.ts", "everyday-artifact.py"]) definitionHash.update(await readFile(path.join(import.meta.dirname, "..", file))); definitionHash.update(runnerSuites.flatMap(suite => suite.tasks).find(task => task.id === "ask-question")!.buildPrompt("definition")); const suiteDefinitionHash = definitionHash.digest("hex"); - const expected = config.journeys.map(id => `product-journeys.${config.id}.${id === "cloud-onboarding" ? config.modelProfileId : id === "ask-question" ? "configured-agent" : "browser"}.${config.executionEnvironment}.${id}`); + const expected = config.journeys.map(id => `product-journeys.${config.id}.${id === "cloud-onboarding" ? config.modelProfileId : isAgentJourney(id) ? "configured-agent" : "browser"}.${config.executionEnvironment}.${id}`); let reportPersisted = false; const failFinalization = (code: string) => { for (const result of results) { @@ -164,7 +165,7 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri const profile = id === "cloud-onboarding" ? runnerSuites.flatMap(suite => suite.profiles).find(p => p.id === config.modelProfileId) : undefined; const result: RunnerE2EResult = { schema: "paperclip.runner-e2e.result/v2", executionId: expected[index]!, suiteId: "product-journeys", suiteDefinitionHash, - source, attempt: 1, status: "failed", profileId: id === "ask-question" ? "configured-agent" : profile?.id ?? "browser", environmentId: config.executionEnvironment, caseId: id, + source, attempt: 1, status: "failed", profileId: isAgentJourney(id) ? "configured-agent" : profile?.id ?? "browser", environmentId: config.executionEnvironment, caseId: id, provider: profile?.provider ?? null, model: profile?.model ?? null, runtimeMode: profile?.generation ?? "none", startedAt, finishedAt: startedAt, durationMs: 0, cleanup: "not_started", productJourney: evidence, }; results.push(result); @@ -191,13 +192,20 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri evidence.startingState = ready.startingState; await persist(); phase = "journey"; - if (id === "ask-question") { + if (isAgentJourney(id)) { if (!ready.agent) throw new JourneyError("blocked", "configured_agent_required"); result.profileId = "configured-agent"; result.provider = ready.agent.provider; result.model = ready.agent.model; result.runtimeMode = ready.agent.runtimeMode; - await askQuestion(ready, session!, campaignId, evidence); + if (id === "ask-question") await askQuestion(ready, session!, campaignId, evidence); + else await runAgentJourney(id, ready, session!, campaignId, evidence, { deadlineAt, + directory: path.join(outputDir, "private-stories", id), + capture: async (_id, _label, file) => { + const directory = path.join(outputDir, "private-stories", id); + await mkdir(directory, { recursive: true, mode: 0o700 }); + await captureProductScreenshot(ready!.page, ready!.api.origin, path.join(directory, file)); + } }); } else if (id === "task-create") await taskCreate(ready, session!, campaignId, evidence); else await taskProperties(ready, session!, campaignId, evidence); } @@ -252,6 +260,8 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri if (receipt.operations) result.productJourney!.operations = receipt.operations; } for (const result of results) { + if (isAgentJourney(result.caseId as import("./types.js").JourneyId) && result.status === "passed" + && (!seen.has(result.caseId) || result.runIds?.length !== agentJourneyRuns[result.caseId as keyof typeof agentJourneyRuns])) throw new JourneyError("failed", "paid_journey_receipt_missing"); if (result.caseId === "cloud-onboarding" && result.status === "passed" && (!seen.has(result.caseId) || !result.runIds?.length || !result.productJourney?.operations || !result.productJourney.operations.traceIds.length)) throw new JourneyError("failed", "paid_journey_receipt_missing"); // Success-only checks must run before a disposable target can be deleted. // Cleanup is provisional here; the observed cleanup result is retained below. diff --git a/tests/runner-e2e/product-journeys/success-contract.ts b/tests/runner-e2e/product-journeys/success-contract.ts new file mode 100644 index 00000000000..2c03d3de0ea --- /dev/null +++ b/tests/runner-e2e/product-journeys/success-contract.ts @@ -0,0 +1,16 @@ +/** Closed, target-independent success requirements used by dispatch and retained evidence. */ +export const agentJourneyRuns = { "ask-question": 1, "plan-revise-accept": 3, "build-revise": 2, + "protected-actions": 4, "create-skill-studio": 1, "stop-redirect": 3 } as const; +export const journeyRequiredChecks: Record = { + "task-properties": ["assignment-persisted", "no-agent-work"], + "task-create": ["browser-created-task-persisted", "no-agent-work"], + "ask-question": ["question-answer-persisted", "one-bound-successful-run", "configured-model-and-environment"], + "plan-revise-accept": ["plan-revision-and-approval", "plan-final-answer", "configured-model-and-environment"], + "build-revise": ["initial-artifact-verified", "revised-artifact-verified", "artifact-versions-differ", "original-artifact-preserved", "user-reply-stored-once", "configured-model-and-environment"], + "protected-actions": ["approve-waited-for-decision", "approve-decision-persisted", "approve-service-call-count", "approved-service-data-used", + "decline-waited-for-decision", "decline-decision-persisted", "decline-service-call-count", "decline-truthful-explanation", "configured-model-and-environment"], + "create-skill-studio": ["created-skill-persisted", "skill-creation-attributed", "skill-studio-edit-persisted", "configured-model-and-environment"], + "stop-redirect": ["checkpoint-artifact-verified", "active-work-stopped", "stop-native-terminal-retained", "new-direction-completed", "stopped-run-remained-terminal", + "stopped-run-no-later-events", "stop-preserved-download", "redirect-consumed-once", "user-reply-stored-once", "configured-model-and-environment"], + "cloud-onboarding": ["fresh-workspace", "ui-connection-and-native-agent", "first-task-complete", "downloadable-artifact-verified", "native-trace-verified"], +}; diff --git a/tests/runner-e2e/product-journeys/types.ts b/tests/runner-e2e/product-journeys/types.ts index 54b2f53eb0c..10e50287e0a 100644 --- a/tests/runner-e2e/product-journeys/types.ts +++ b/tests/runner-e2e/product-journeys/types.ts @@ -2,7 +2,8 @@ import type { BrowserContext, Page } from "@playwright/test"; import type { ConnectionApi } from "../connection-target.js"; export type JourneyOutcome = "passed" | "failed" | "blocked" | "unsupported"; -export type JourneyId = "task-properties" | "task-create" | "ask-question" | "cloud-onboarding"; +export type JourneyId = "task-properties" | "task-create" | "ask-question" | "cloud-onboarding" + | "plan-revise-accept" | "build-revise" | "protected-actions" | "create-skill-studio" | "stop-redirect"; export type TargetMode = "managed-self-hosted" | "existing-cloud" | "new-cloud"; export interface DeploymentIdentity { kind: "self-hosted" | "cloud"; @@ -81,7 +82,10 @@ export interface JourneyContext { capabilities: string[]; startingState: "recipe" | "onboarding"; agent?: { id: string; name: string; provider: string; model: string; runtimeMode: "native" | "legacy" }; - fixtures?: { backlogTask?: PreparedBacklogTask }; + fixtures?: { + backlogTask?: PreparedBacklogTask; + reviewService?: { connectionId: string; name: string; marker: string; invocationCount(): number }; + }; } export interface PreparedBacklogTask { id: string; @@ -107,7 +111,7 @@ export interface TargetSession { /** Publish the observed finalization failure without releasing resources or leases. */ checkpointEvidence?(input?: TargetLifecycleInput): Promise; /** Durable intent before creation; receipt immediately after creation. */ - own(resource: { kind: "issue" | "agent" | "connection"; name: string; id?: string }): Promise; + own(resource: { kind: "issue" | "agent" | "connection" | "tool-connection" | "skill"; name: string; id?: string; journeyId?: JourneyId }): Promise; } export interface TargetProvider { version: 1; diff --git a/tests/runner-e2e/protected-action-flow.ts b/tests/runner-e2e/protected-action-flow.ts new file mode 100644 index 00000000000..1aa6278c8fd --- /dev/null +++ b/tests/runner-e2e/protected-action-flow.ts @@ -0,0 +1,43 @@ +import { expect, type Page } from "@playwright/test"; +import { pendingStoryDecision, type StoryInteraction } from "./everyday-decisions.js"; + +/** Both delivery forms are supported by the product; a visible link must back an attachment. */ +export async function readServiceBriefing(input: { + page: Page; issueId: string; api: { get(route: string): Promise; bytes(route: string): Promise }; +}) { + type Row = Record; + const docs = await input.api.get(`/api/issues/${input.issueId}/documents`); + const bodies = await Promise.all(docs.map(doc => input.api.get(`/api/issues/${input.issueId}/documents/${encodeURIComponent(doc.key)}`))); + const attachments = await input.api.get(`/api/issues/${input.issueId}/attachments`); + for (const attachment of attachments.filter(row => /\.(?:md|txt)$/i.test(row.originalFilename ?? row.filename ?? "") + || ["text/markdown", "text/plain"].includes(row.contentType))) { + const route = `/api/attachments/${attachment.id}/content`; + await expect(input.page.locator(`a[href*="${route}"]`).first()).toBeVisible(); + const bytes = await input.api.bytes(route); + if (bytes.length > 65_536) throw new Error("service_briefing_exceeds_limit"); + bodies.push({ id: attachment.id, title: attachment.originalFilename ?? attachment.filename, body: bytes.toString("utf8"), source: "delivered-attachment" }); + } + return bodies; +} + +export function serviceBriefingChecks(bodies: Array>, marker: string) { + const text = bodies.map(doc => String(doc.body ?? doc.revision?.body ?? "")).join("\n"); + return [ + { id: "briefing-uses-real-result", passed: text.includes(marker), detail: "The saved document or attachment includes the actual service verification code." }, + { id: "briefing-includes-page-titles", passed: text.includes("Roadmap") && text.includes("Meeting notes"), detail: "The saved briefing includes both returned page titles." }, + ]; +} + +/** A saved decision for this exact connection must exist before any service call. */ +export async function decideProtectedActionThroughUi(input: { + page: Page; rows: StoryInteraction[]; connectionId: string; decline: boolean; + invocationCount(): number; expectedCallsBefore?: number; +}) { + const interaction = pendingStoryDecision(input.rows, { kind: "tool", connectionId: input.connectionId }); + expect(input.invocationCount()).toBe(input.expectedCallsBefore ?? 0); + const dismiss = input.page.getByRole("button", { name: "Dismiss Approve tool action" }); + if (await dismiss.isVisible()) await dismiss.click(); + await input.page.getByRole("button", { name: "Review request", exact: true }).click(); + await input.page.getByRole("button", { name: input.decline ? "Decline" : "Approve & run", exact: true }).click(); + return interaction; +} diff --git a/tests/runner-e2e/result-validation.ts b/tests/runner-e2e/result-validation.ts index 1ca685369a7..9b3afe0d9c4 100644 --- a/tests/runner-e2e/result-validation.ts +++ b/tests/runner-e2e/result-validation.ts @@ -1,3 +1,4 @@ +import { agentJourneyRuns, journeyRequiredChecks } from "./product-journeys/success-contract.js"; import { validateWorkerTiming } from "./worker-timing.js"; import { connectionCheckpoints, connectionEvidencePasses, connectionRunSignals } from "./connection-evidence.js"; import { githubJourneyCheckpoints, gradeGitHubReviewJourney } from "./github-review-evidence.js"; @@ -429,13 +430,11 @@ export function validateRetainedRunnerResult( if (checked.suiteId === "product-journeys" && checked.status === "passed") { const evidence = checked.productJourney; if (!evidence) return invalid("result.productJourney"); - const required = checked.caseId === "task-properties" ? ["assignment-persisted", "no-agent-work"] - : checked.caseId === "task-create" ? ["browser-created-task-persisted", "no-agent-work"] - : checked.caseId === "ask-question" ? ["question-answer-persisted", "one-bound-successful-run", "configured-model-and-environment"] - : ["fresh-workspace", "ui-connection-and-native-agent", "first-task-complete", "downloadable-artifact-verified", "native-trace-verified"]; - if (evidence.outcome !== "passed" || !evidence.deployment || !required.every(id => evidence.checks.some(c => c.id === id && c.passed)) || evidence.phases.some(p => p.outcome !== "passed") || checked.cleanup !== "passed") invalid("result.productJourney"); + const required = journeyRequiredChecks[checked.caseId]; + if (evidence.outcome !== "passed" || !evidence.deployment || !required || !required.every(id => evidence.checks.some(c => c.id === id && c.passed)) || evidence.checks.some(check => !check.passed) || evidence.phases.some(p => p.outcome !== "passed") || checked.cleanup !== "passed") invalid("result.productJourney"); if (["task-properties", "task-create"].includes(checked.caseId) && (checked.runtimeMode !== "none" || checked.provider !== null || checked.model !== null || checked.runIds?.length)) invalid("result.browserOnly"); - if (checked.caseId === "ask-question" && (checked.runIds?.length !== 1 || !checked.model || !checked.provider || checked.runtimeMode === "none")) invalid("result.questionRun"); + if (Object.hasOwn(agentJourneyRuns, checked.caseId) && (checked.runIds?.length !== agentJourneyRuns[checked.caseId as keyof typeof agentJourneyRuns] + || !checked.model || !checked.provider || checked.runtimeMode !== "native")) invalid("result.agentJourneyRun"); if (checked.caseId === "cloud-onboarding" && (!checked.runIds?.length || !evidence.operations || !evidence.operations.requestCount || !evidence.operations.traceIds.length || evidence.operations.campaignId !== evidence.campaignId || evidence.operations.attemptId !== evidence.attemptId)) invalid("result.paidJourneyReceipt"); } if (checked.suiteId === "provider-connections" && checked.status === "passed" && (!checked.providerConnection || !connectionEvidencePasses(checked.providerConnection))) invalid("result.providerConnection"); diff --git a/tests/runner-e2e/review-service.test.ts b/tests/runner-e2e/review-service.test.ts new file mode 100644 index 00000000000..5c9714ea925 --- /dev/null +++ b/tests/runner-e2e/review-service.test.ts @@ -0,0 +1,18 @@ +import { expect, it } from "vitest"; +import { startReviewProvider } from "../fixtures/connection-review-provider.js"; + +it("authenticates the reachable review fixture before parsing and bounds accepted requests", async () => { + const service = await startReviewProvider("synthetic-service-proof", "fixture-only-token"); + const request = (body: string, token?: string) => fetch(service.url, { method: "POST", headers: token ? { authorization: `Bearer ${token}` } : {}, body }); + try { + expect((await request("malformed unauthenticated input")).status).toBe(401); + expect((await request("not json", "fixture-only-token")).status).toBe(400); + expect((await request("null", "fixture-only-token")).status).toBe(400); + expect((await request(JSON.stringify({ method: "tools/call", params: { name: "notion:list_pages", arguments: { query: "x".repeat(70_000) } } }), "fixture-only-token")).status).toBe(413); + const wrong = await request(JSON.stringify({ id: 1, method: "tools/call", params: { name: "uninstalled-tool" } }), "fixture-only-token"); + expect((await wrong.json()).result.isError).toBe(true); + const valid = await request(JSON.stringify({ id: 2, method: "tools/call", params: { name: "notion:list_pages" } }), "fixture-only-token"); + expect((await valid.json()).result).toEqual({ isError: false, content: [{ type: "text", text: "synthetic-service-proof" }] }); + expect(service.captures.filter(call => call.authorized && call.method === "tools/call")).toHaveLength(2); + } finally { await service.close(); } +}); diff --git a/tests/runner-e2e/runner.spec.ts b/tests/runner-e2e/runner.spec.ts index 97a429fecb1..080188c2cf7 100644 --- a/tests/runner-e2e/runner.spec.ts +++ b/tests/runner-e2e/runner.spec.ts @@ -1,3 +1,4 @@ +import { runPlanRevisionAcceptance, isPendingPlanConfirmation, normalizePlanMarkdown, expectPlanStageVisible } from "./plan-revision-flow.js"; import { createWorkerTiming } from "./worker-timing.js"; import { captureTaskChat, type TaskChatEvidence } from "./task-chat.js"; import { FEEDBACK_PERSISTENCE_SUITE, gradeFeedbackPersistence } from "./feedback-persistence.js"; @@ -425,16 +426,6 @@ function matchingRuns(runs: RunRecord[], issue: IssueRecord) { }); } -function isPendingPlanConfirmation(interaction: InteractionRecord) { - return ( - interaction.kind === "request_confirmation" && - interaction.status === "pending" && - interaction.payload?.target?.type === "issue_document" && - interaction.payload.target.key === "plan" && - typeof interaction.payload.target.revisionId === "string" - ); -} - function isPendingQuestion(interaction: InteractionRecord) { return ( interaction.kind === "ask_user_questions" && @@ -452,12 +443,6 @@ function isPendingWarmConfirmation(interaction: InteractionRecord) { ); } -function normalizePlanMarkdown(body: string | null | undefined) { - // Provider plan renderers may defensively escape underscores in plain-text - // markers. The rendered document and semantic marker are identical. - return (body ?? "").replaceAll("\\_", "_"); -} - function nativeRunEventIntegrityFailures( run: RunRecord, events: readonly RunEventRecord[], @@ -568,28 +553,6 @@ function nativeRunEventIntegrityFailures( return failures; } -async function expectPlanStageVisible( - page: Page, - options: { native: boolean; revision: number | null }, -) { - // A Plan flow is only qualified when the canonical saved document is - // projected as a card. Native profiles additionally require that card to be - // inside the runner turn, which proves write-boundary embedding rather than - // a standalone fallback. The API assertions own exact body-marker identity - // because the compact card intentionally shows only its first three lines. - const root = options.native - ? page.locator('[data-testid="task-chat-turn"][data-settled="true"]') - : page.locator("body"); - const preview = root.getByTestId("task-chat-plan-preview").last(); - await expect(preview).toBeVisible({ timeout: 30_000 }); - if (options.revision !== null) { - await expect(preview).toHaveAttribute( - "aria-label", - `Open Plan revision ${options.revision}`, - ); - } -} - for (const execution of executions) { const controllerOutage = controllerOutageForExecution(execution, process.env); const privateDir = path.join(privateRoot, "cases", execution.task.id); @@ -1478,165 +1441,10 @@ for (const execution of executions) { } else if (execution.task.flow === "plan_revision_acceptance") { const planMarkers = execution.task.buildPlanMarkers?.(nonce); const revisionRequest = execution.task.buildRevisionRequest?.(nonce); - if (!planMarkers || !revisionRequest) { - throw new Error( - `Plan fixture ${execution.task.id} is missing lifecycle factories`, - ); - } - const draftState = await pollUntil({ - label: `initial plan confirmation for issue ${issue.id}`, - deadlineAt, - load: loadTaskState, - accept: ({ taskRuns, interactions }) => - taskRuns.length >= 1 && - taskRuns.every((run) => TERMINAL_RUN_STATUSES.has(run.status)) && - interactions.some(isPendingPlanConfirmation), - reject: ({ taskRuns, interactions }) => - rejectPlanConfirmationPoll({ - taskRuns, - interactions, - minimumRunCount: 1, - }), - }); - const draftInteraction = draftState.interactions.find( - isPendingPlanConfirmation, - )!; - const draftPlan = await api.get( - `/api/issues/${issue.id}/documents/plan`, - ); - if ( - !normalizePlanMarkdown(draftPlan.body).includes(planMarkers.draft) - ) { - throw new Error( - `Initial Plan document did not contain ${planMarkers.draft}`, - ); - } - if (numberedPlanStepCount(draftPlan.body) !== 2) { - throw new Error( - "Initial Plan must contain exactly two numbered steps", - ); - } - if ( - draftInteraction.payload?.target?.revisionId !== - draftPlan.latestRevisionId - ) { - throw new Error( - "Initial plan confirmation did not target the latest Plan revision", - ); - } - await page.goto( - `/${encodeURIComponent(issuePrefix)}/issues/${encodeURIComponent(issue.identifier ?? issue.id)}`, - { waitUntil: "domcontentloaded" }, - ); - await expectPlanStageVisible(page, { - native: execution.profile.expectedRuntimeMode === "native", - revision: draftPlan.latestRevisionNumber ?? null, - }); - await captureScreenshot( - "plan-draft", - "Initial plan awaiting revision", - "plan-draft.png", - ); - await page - .getByRole("button", { - name: draftInteraction.payload?.rejectLabel ?? "Reject", - exact: true, - }) - .last() - .click(); - const revisionComposer = page - .getByTestId("plan-revision-composer") - .last(); - await expect(revisionComposer).toBeVisible({ timeout: 10_000 }); - await revisionComposer - .locator('[contenteditable="true"], textarea') - .first() - .fill(revisionRequest); - await page - .getByRole("button", { - name: draftInteraction.payload?.rejectLabel ?? "Reject", - exact: true, - }) - .last() - .click(); - - const revisedState = await pollUntil({ - label: `revised plan confirmation for issue ${issue.id}`, - deadlineAt, - load: loadTaskState, - accept: ({ taskRuns, interactions }) => - taskRuns.length >= 2 && - taskRuns.every((run) => TERMINAL_RUN_STATUSES.has(run.status)) && - interactions.some( - (interaction) => - isPendingPlanConfirmation(interaction) && - interaction.id !== draftInteraction.id, - ), - reject: ({ taskRuns, interactions }) => - rejectPlanConfirmationPoll({ - taskRuns, - interactions, - minimumRunCount: 2, - }), - }); - const revisedInteraction = revisedState.interactions.find( - (interaction) => - isPendingPlanConfirmation(interaction) && - interaction.id !== draftInteraction.id, - )!; - const revisedPlan = await api.get( - `/api/issues/${issue.id}/documents/plan`, - ); - const normalizedRevisedPlan = normalizePlanMarkdown(revisedPlan.body); - if ( - !normalizedRevisedPlan.includes(planMarkers.revised) || - normalizedRevisedPlan.includes(planMarkers.draft) - ) { - throw new Error( - `Revised Plan must replace ${planMarkers.draft} with ${planMarkers.revised}`, - ); - } - if (numberedPlanStepCount(revisedPlan.body) !== 3) { - throw new Error( - "Revised Plan must contain exactly three numbered steps", - ); - } - if ( - revisedInteraction.payload?.target?.revisionId !== - revisedPlan.latestRevisionId || - revisedPlan.latestRevisionId === draftPlan.latestRevisionId - ) { - throw new Error( - "Revised confirmation did not target a new latest Plan revision", - ); - } - await page.goto( - `/${encodeURIComponent(issuePrefix)}/issues/${encodeURIComponent(issue.identifier ?? issue.id)}`, - { waitUntil: "domcontentloaded" }, - ); - await expectPlanStageVisible(page, { - native: execution.profile.expectedRuntimeMode === "native", - revision: revisedPlan.latestRevisionNumber ?? null, - }); - await captureScreenshot( - "plan-revised", - "Revised plan awaiting acceptance", - "plan-revised.png", - ); - await page - .getByRole("button", { - name: revisedInteraction.payload?.acceptLabel ?? "Approve", - exact: true, - }) - .last() - .click(); - planLifecycleEvidence = { - draftInteraction, - draftPlan, - revisedInteraction, - revisedPlan, - revisionRequest, - }; + if (!planMarkers || !revisionRequest) throw new Error(`Plan fixture ${execution.task.id} is missing lifecycle factories`); + planLifecycleEvidence = await runPlanRevisionAcceptance({ page, api, origin: api.baseURL, + issue, issuePrefix, native: execution.profile.expectedRuntimeMode === "native", planMarkers, revisionRequest, + deadlineAt, load: loadTaskState, reject: rejectPlanConfirmationPoll, capture: captureScreenshot }); } else if (execution.task.flow === "question_resume_completion") { const expectedAnswer = execution.task.buildQuestionAnswer?.(nonce); if (!expectedAnswer) { diff --git a/tests/runner-e2e/skill-studio-flow.ts b/tests/runner-e2e/skill-studio-flow.ts new file mode 100644 index 00000000000..7b12ba3e82c --- /dev/null +++ b/tests/runner-e2e/skill-studio-flow.ts @@ -0,0 +1,35 @@ +import { expect, type Page } from "@playwright/test"; + +/** Shared browser story for classic and Cloud: activity card -> Studio -> saved task readback. */ +export async function editCreatedSkillThroughStudio(input: { + page: Page; skillId: string; skillName: string; marker: string; deadlineAt: number; + read(): Promise; +}) { + const { page, skillName } = input; + const card = page.getByRole("article", { name: `Skill created: ${skillName}`, exact: true }); + await expect(card).toHaveCount(1); + await expect(card).toBeVisible(); + await card.getByRole("button").click(); + await expect(page.getByRole("heading", { name: skillName, exact: true })).toBeVisible(); + await expect(page.getByText("Verify checks.", { exact: true })).toBeVisible(); + await page.getByRole("button", { name: "Open in Skill Studio", exact: true }).click(); + await expect(page).toHaveURL(new RegExp(`/skills/studio/${input.skillId}$`)); + await expect(page.getByRole("combobox").filter({ hasText: skillName })).toBeVisible(); + const editor = page.getByRole("textbox", { name: "editable markdown", exact: true }); + await editor.click(); + await editor.press("ControlOrMeta+End"); + await editor.press("Enter"); + await editor.press("Enter"); + await editor.pressSequentially(input.marker); + await page.getByRole("button", { name: /^Save$/ }).click(); + await expect.poll(async () => JSON.stringify(await input.read()).includes(input.marker), + { timeout: Math.max(1, Math.min(30_000, input.deadlineAt - Date.now())) }).toBe(true); + await page.reload(); + await expect(editor).toContainText(input.marker); + await page.goBack(); + await expect(page).toHaveURL(/\/issues\//); + await expect(card).toBeVisible(); + await card.getByRole("button").click(); + await expect(page.getByRole("heading", { name: skillName, exact: true })).toBeVisible(); + await expect(page.getByText(input.marker, { exact: true })).toBeVisible(); +} diff --git a/tests/runner-e2e/stop-redirect-flow.ts b/tests/runner-e2e/stop-redirect-flow.ts new file mode 100644 index 00000000000..4580bd605a4 --- /dev/null +++ b/tests/runner-e2e/stop-redirect-flow.ts @@ -0,0 +1,61 @@ +import type { Page } from "@playwright/test"; +type Row = Record; + +/** Work emitted by a retired turn, excluding accounting and control-plane bookkeeping. */ +export function nativeWorkEvents(events: T[]): T[] { + return events.filter(row => { + const kind = row.payload?.prpEvent?.eventType; + return typeof kind === "string" && /^(tool\.execution\.|turn\.|message\.|item\.)/.test(kind); + }); +} + +/** Detect post-cancellation output even if it arrived before the first snapshot. */ +export function nativeCancellationHasNoLaterWork(events: Row[]): boolean { + const work = nativeWorkEvents(events); + const cancellation = work.findIndex(row => row.payload.prpEvent.eventType === "turn.cancelled"); + return cancellation >= 0 && cancellation === work.length - 1; +} + +/** A claimed run is not proof that a cancellable native turn has started. */ +export function activeNativeTurnStarted(run: Row | undefined, events: Row[]): boolean { + const latestTurn = events.map(row => row.payload?.prpEvent?.eventType) + .filter(kind => ["turn.started", "turn.completed", "turn.failed", "turn.cancelled", "turn.interrupted"].includes(kind)).at(-1); + return run?.runtimeMode === "native" && run.status === "running" && latestTurn === "turn.started"; +} + +/** Shared story order; target adapters supply authoritative retirement observations. */ +export async function stopAndRedirectThroughUi(input: { + page: Page; nonce: string; + awaitRetirement(): Promise; + beforeRedirect(): Promise; + reply(message: string): Promise; +}) { + await input.page.getByTestId("task-chat-composer-stop").last().click(); + await input.awaitRetirement(); + await input.beforeRedirect(); + const marker = `The studio is ready. Reference ${input.nonce}.`; + const reply = await input.reply(`Change direction. Leave the project as it is. Reply with just this short note: "${marker}"`); + await input.page.reload(); + return { marker, reply }; +} + +/** The native control plane must acknowledge this run's Stop, including resource release. */ +export function nativeStopAcknowledged(run: Row, companyId: string, issueId: string, userId: string) { + const stop = run.resultJson?.nativeCancellation; + const result = run.resultJson ?? {}; + const startup = result.startupCancellation; + const cancellation = result.cancellation; + return run.runtimeMode === "native" && run.status === "cancelled" && run.companyId === companyId && (run.nativeIssueId ?? run.issueId) === issueId + && stop?.schema === "paperclip.native-cancellation.v1" && stop.companyId === companyId + && stop.runId === run.id && stop.issueId === issueId && stop.scope === "run" && stop.dispatched === true + && stop.dispatchState === "acknowledged" && stop.reasonCode === "cancellation_run_only" + && stop.effects?.length === 1 && stop.effects[0] === "release_run_resources" + && typeof stop.intentAuditId === "string" && typeof stop.acknowledgementAuditId === "string" + && stop.intentAuditId !== stop.acknowledgementAuditId + && /^native-cancellation:[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/.test(stop.intentId ?? "") + && result.cancelledByActorType === "user" && result.cancelledByUserId === userId + && cancellation?.source === "operator" && cancellation.expected === true + && cancellation.initiator?.type === "user" && cancellation.initiator.id === userId + && typeof startup?.requestedAt === "string" && Number.isFinite(Date.parse(startup.requestedAt)) + && startup.beforeNativeSelection === false; +} diff --git a/tests/runner-e2e/story-artifact.test.ts b/tests/runner-e2e/story-artifact.test.ts new file mode 100644 index 00000000000..2962e5626ee --- /dev/null +++ b/tests/runner-e2e/story-artifact.test.ts @@ -0,0 +1,17 @@ +import { expect, it } from "vitest"; +import { parseArtifactOracleChecks } from "./story-artifact.js"; + +it("preserves the independent oracle's named failing checks on its documented exit 1", () => { + const checks = [{ id: "separator-underscore", passed: false, detail: "Expected queue_ready; received queue-ready" }]; + expect(parseArtifactOracleChecks(JSON.stringify({ passed: false, checks }), 1)).toEqual(checks); + expect(() => parseArtifactOracleChecks(JSON.stringify({ passed: false, checks }), 1, true)).toThrow("artifact_oracle_preflight_failed"); +}); +it("accepts valid success but rejects malformed, contradictory and process-failure receipts", () => { + const checks = [{ id: "cli-output", passed: true, detail: "Independent input matched" }]; + const output = JSON.stringify({ passed: true, checks }); + expect(parseArtifactOracleChecks(output, 0)).toEqual(checks); + for (const [body, code] of [[output, 1], [output, 2], ["not json", 0], [JSON.stringify({ passed: true, checks: [] }), 0], + [JSON.stringify({ passed: true, checks: [{ ...checks[0], passed: false }] }), 0]] as const) { + expect(() => parseArtifactOracleChecks(body, code)).toThrow(); + } +}); diff --git a/tests/runner-e2e/story-artifact.ts b/tests/runner-e2e/story-artifact.ts new file mode 100644 index 00000000000..de2e411903e --- /dev/null +++ b/tests/runner-e2e/story-artifact.ts @@ -0,0 +1,51 @@ +import { spawn } from "node:child_process"; +import { createHash } from "node:crypto"; +import { mkdir, readFile } from "node:fs/promises"; +import path from "node:path"; +import { expect, type Page } from "@playwright/test"; +import type { StoryCheck } from "./everyday-observations.js"; + +/** The oracle uses exit 1 for a valid failing receipt, not just a process failure. */ +export function parseArtifactOracleChecks(stdout: string, code: number, preflight = false): StoryCheck[] { + if (![0, 1].includes(code)) throw new Error("artifact_oracle_failed"); + const value = JSON.parse(stdout) as { passed: boolean; checks: StoryCheck[] }; + if (!Array.isArray(value.checks) || !value.checks.length || value.checks.some(check => typeof check.id !== "string" + || typeof check.passed !== "boolean" || typeof check.detail !== "string")) throw new Error("artifact_oracle_receipt_invalid"); + const passed = value.checks.every(check => check.passed); + if (value.passed !== passed || (code === 0) !== passed) throw new Error("artifact_oracle_receipt_invalid"); + if (preflight && !passed) throw new Error("artifact_oracle_preflight_failed"); + return value.checks; +} + +/** Runs only the maintained independent oracle, with no provider credentials. */ +export async function runArtifactOracle(file: string | null, mode: "base" | "separator" | "max-length", deadlineAt: number) { + const args = [path.join(import.meta.dirname, "everyday-artifact.py"), ...(file ? [file, "--mode", mode] : ["--preflight"])]; + const output = await new Promise<{ stdout: string; code: number }>((resolve, reject) => { + const env = Object.fromEntries(Object.entries(process.env).filter(([key]) => ["PATH", "SYSTEMROOT", "TMPDIR", "DOCKER_HOST", "DOCKER_CONTEXT", "HOME"].includes(key))); + const child = spawn(process.env.PYTHON ?? "python3", args, { env, stdio: ["ignore", "pipe", "pipe"] }); + let stdout = "", exceeded = false; + const timer = setTimeout(() => child.kill("SIGKILL"), Math.max(1, Math.min(90_000, deadlineAt - Date.now()))); + child.stdout.on("data", chunk => { stdout += String(chunk); if (stdout.length > 1_000_000) { exceeded = true; child.kill("SIGKILL"); } }); + child.stderr.resume(); + child.once("error", () => { clearTimeout(timer); reject(new Error("artifact_oracle_unavailable")); }); + child.once("exit", code => { clearTimeout(timer); code !== null && !exceeded ? resolve({ stdout, code }) : reject(new Error("artifact_oracle_failed")); }); + }); + return parseArtifactOracleChecks(output.stdout, output.code, file === null); +} + +/** Download through the visible task control; attachment metadata is not a content oracle. */ +export async function downloadStoryArtifact(input: { + page: Page; attachmentId: string; directory: string; phase: string; + mode: "base" | "separator" | "max-length"; deadlineAt: number; +}) { + const link = input.page.locator(`a[href*="/api/attachments/${input.attachmentId}/content"]`).filter({ hasText: /Download/i }).first(); + await expect(link).toBeVisible({ timeout: 30_000 }); + const [download] = await Promise.all([input.page.waitForEvent("download", { timeout: 30_000 }), link.click()]); + await mkdir(input.directory, { recursive: true, mode: 0o700 }); + const file = path.join(input.directory, `${input.phase}.zip`); + await download.saveAs(file); + const bytes = await readFile(file); + const checks = await runArtifactOracle(file, input.mode, input.deadlineAt); + return { attachmentId: input.attachmentId, sha256: createHash("sha256").update(bytes).digest("hex"), bytes: bytes.length, + filename: download.suggestedFilename(), checks }; +}