mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-31 06:04:11 +01:00
test(evals): add reusable task search and filter journey (#18355)
## Thinking Path > - Paperclip lets people manage AI agents and their work. > - The task list lets people find saved work and open its context. > - Product journeys can exercise the same task UI on local and Cloud targets. > - The shared suite lacks a search and status-filter journey with control tasks. > - This pull request adds that journey with exact task identity checks. > - The benefit is reusable evidence that search, filters, and task navigation work together. ## Linked Issues or Issue Description **What existing behavior does this improve?** Product E2E coverage for task search, status filters, and saved task navigation. **Current behavior** The shared journey suite covers task creation and assignment. It does not prove that a search excludes an unrelated task, that a status filter excludes a matching completed task, or that the selected task retains its saved content. **Proposed behavior** An explicit `task-search-filter` selection creates three owned tasks. It searches through the production task list, applies Backlog, and opens the exact target. It verifies reload and Back navigation, saved task fields, and zero agent runs for all three tasks. **Reason and benefit** The same journey can run against a managed local stack or an attached Cloud target. Independent assertions reject missing evidence and plausible wrong results. Deployment qualification and daily-plan admission remain separate gates. **Breaking changes** None. Existing journey selections keep their behavior. Related work: #16979 fixes status filtering before pagination. #15403 covers saved task views. This journey tests the existing search/filter interaction and does not add a new UI or saved-view feature. ## What Changed - Add a zero-run browser journey with one target and two owned controls. - Compare exact visible task identities after search, filtering, reload, and Back navigation. - Verify saved company, owner, title, description, status, and zero agent runs through the public API. - Register dispatch, source hashing, closed success checks, and browser reporting validation. - Add adversarial oracle tests and a campaign dispatch test. Document the selection and its feature-map scope. - Exercise the real browser flow with a stateful UI/API port across Coordinator, combined Tasks, and legacy shells. Inject missing ownership, wrong destinations, lost filters, changed saved context, and unexpected native work. - Use the combined Tasks shell's named Backlog view for reload persistence, as required by its existing view-preset contract. Preserve the successful polled result snapshot and link the case to feature coverage. ## Verification - First attached Cloud attempt failed during Backlog selection. Its cleanup passed and it started no model work. The original failed result is preserved; the corrected candidate passed a fresh attached campaign below. - Local Chromium reproduces the cause: the legacy StatusIcon and text label produce the accessible checkbox name `BacklogBacklog`, so the prior exact `Backlog` lookup finds no status control. - The selector now stays inside the Status group and accepts the explicit and duplicated names. Three real Chromium regressions pass, including a separate assignee called Backlog. - Latest focused journey checks: 130 passed. Runner typecheck and map checks passed. Unexpected browser/API failures retain only closed source-owned stage codes. Screenshot privacy guards and all exact identity, content, filter, and zero-run checks are unchanged. - Fresh attached Cloud qualification passed at `e3dfc0ae37e8ac74e9a07ca24421ea6c96f9bcfc`: [sanitized report](https://pages.paperclip.ing/evals-qa-smoke-41114233380-1/). All five independent search/filter/reload/reopen/no-agent checks and all five phases, including cleanup, passed. The browser journey started zero agent runs. - Runner typecheck passed: `node node_modules/typescript/bin/tsc -p tests/runner-e2e/tsconfig.json`. - Focused oracle, contract, and campaign tests passed: 101 unique tests. The campaign suite was rerun after its new dispatch test: 30 passed. - Direct browser-flow tests passed: 23 tests. These call the real `taskSearchFilter` function through a stateful UI/API port. They do not establish live browser or Cloud qualification. - Latest focused search, oracle, catalog, and report checks passed: 127 tests across five files. - Full runner Vitest completed with 5,204 passes, two skips, and three failures outside this change. Two local embedded PostgreSQL fixtures failed at bootstrap. The Pi controls test expects profile version 35 while the unchanged base declares 37. These failures remain recorded; the PR does not change those fixtures or pins. - The remaining Node support suite passed separately: 137 tests. The standard unit launcher stops before these tests after a Vitest failure. - Credential-free discovery passed: `node --import ./cli/node_modules/tsx/dist/loader.mjs tests/runner-e2e/launch.ts --suite product-journeys --list`. - `node scripts/check-maps.mjs` passed. `git diff --check` passed. - All 58 current-head check runs are green, the additional security status passed, Greptile reviewed this exact head at 5/5, and no review threads remain unresolved. Typecheck, test, build, browser CI, and fresh attached Cloud acceptance passed. Daily-plan admission follows merge. ## Risks - Task-list control labels and link structure can change. Such drift fails the journey and needs an update. - The journey creates three board-assigned tasks. It records ownership before creation and records each returned ID immediately. The existing target provider owns cleanup. - Local assertion tests do not prove that a deployed Cloud target passes. The target owner must record a fresh live pass before admission to a daily plan. ## Model Used OpenAI Codex, GPT-6 family. The host does not expose an exact model version or context limit. Used repository search, TypeScript tools, and tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
8f822bfcf7
commit
a0e3ac62a7
17 files changed
+688
-7
No files matched your search
@@ -27,6 +27,12 @@ checks only assignment to the signed-in user and persistence after reload. Its
|
||||
source is reusable across self-hosted and Cloud; Cloud live qualification is
|
||||
separate. It does not cover other property operations or agent authorization.
|
||||
|
||||
The [Product task-search-filter journey](../../tests/runner-e2e/PRODUCT-JOURNEYS.md)
|
||||
checks `tasks/organize` with owned matching and nonmatching controls, exact task
|
||||
identities, saved task content, and search/status persistence after reload and
|
||||
detail/back navigation. It is model-free and reusable across deployment targets.
|
||||
Live Cloud qualification is separate from the journey's assertion unit tests.
|
||||
|
||||
## How to get to it (user POV)
|
||||
|
||||
Cold collection loads use a lightweight index and batched hierarchy context,
|
||||
|
||||
@@ -26,6 +26,7 @@ never start a server or depend on a tenant filesystem.
|
||||
| `task-create` | Company and signed-in board user | Create a task through the shared production composer, assign it to yourself, reopen and reload it, verify the saved prompt and no agent work |
|
||||
| `ask-question` | Company, configured agent capability, positive model allowance and exactly one permitted work run | Reuse the existing cross-model Ask question prompt; require one successful run with matching company/task/agent/model/environment, one saved exact answer, and visible readback after reload |
|
||||
| `task-properties` | Company, signed-in board user, task-write capability, prepared run-owned backlog task | Open its properties, assign it to the signed-in user, reload, check both UI and saved assignment; verify no agent run |
|
||||
| `task-search-filter` | Company, signed-in board user, task-write capability; zero model runs | Search three owned target/control tasks through the task list, require exact matching identities, filter Backlog to exclude the completed control, reload and return from the exact saved task with search/filter context intact |
|
||||
| `plan-revise-accept` | Native agent; three work runs | Reuse the classic Plan story: canonical draft, browser revision request, new revision-bound confirmation, approval and exact saved final response |
|
||||
| `build-revise` | Native agent; two work runs; qualified Docker artifact oracle | Browser-download both ZIP versions, independently exercise source/CLI behavior, verify different hashes and preserve the original download |
|
||||
| `protected-actions` | Native agent; four work runs; authenticated reachable review-service fixture | Approve one action and decline another on separate owned tasks; verify zero calls before either decision, one after approval, zero after decline and a truthful saved explanation |
|
||||
@@ -37,6 +38,29 @@ The shared Studio helper waits for the original saved file body before typing.
|
||||
An editor control can mount before the file query completes; editing that empty
|
||||
draft can otherwise lose the original content when the file arrives.
|
||||
|
||||
The search/filter journey creates only campaign-owned, board-assigned tasks through
|
||||
the public API. The target and completed control share a unique search marker;
|
||||
a second Backlog control has a different marker. Search and status selection use
|
||||
the production controls. The oracle compares the complete visible task identity
|
||||
set at each step, verifies the exact saved company, owner, title, description and
|
||||
status, and checks all three tasks have no native runs. It reopens the target after
|
||||
reload and browser Back, and leaves the target visible for the harness screenshot.
|
||||
The combined Tasks shell reapplies its named view's status preset on mount, so
|
||||
the journey first proves the checkbox filter, then selects the normal Backlog
|
||||
view and enters the same search again before checking reload/Back persistence.
|
||||
Coordinator and legacy task lists keep their supported saved-filter behavior.
|
||||
The status selector stays inside the Status group. Legacy status glyphs contribute
|
||||
their label to the checkbox's accessible name, producing `BacklogBacklog`; the
|
||||
refreshed panel supplies the explicit `Backlog` name. A credential-free browser
|
||||
regression covers both forms and a separate assignee called Backlog.
|
||||
Unexpected browser/API errors retain only a closed `task_search_<stage>_failed`
|
||||
code. Stage names come from the journey source; raw exception text, URLs, form
|
||||
values, and response bodies are not added to reporting. Existing owned evidence
|
||||
failures keep their specific codes and the screenshot privacy guard stays in place.
|
||||
It does not depend on row order or an otherwise empty company. This selection is
|
||||
manual until a separate target owner records a fresh pass and admits it to a daily
|
||||
plan; adding the shared journey does not qualify any deployed target.
|
||||
|
||||
Supported acquisitions are managed self-hosted (`local`), registered existing
|
||||
Cloud (`local` or `daytona` when present), and new Cloud. New Cloud must start
|
||||
with onboarding. Its context can then satisfy the identical task-properties
|
||||
|
||||
@@ -59,7 +59,7 @@ describe("runner E2E catalog", () => {
|
||||
});
|
||||
|
||||
it("exports deployment journeys separately from the model matrix and default selection", () => {
|
||||
expect(productJourneys.map(journey => journey.id)).toEqual(["task-create", "ask-question", "task-properties", "plan-revise-accept", "build-revise", "protected-actions", "create-skill-studio", "stop-redirect", "cloud-onboarding"]);
|
||||
expect(productJourneys.map(journey => journey.id)).toEqual(["task-create", "ask-question", "task-properties", "task-search-filter", "plan-revise-accept", "build-revise", "protected-actions", "create-skill-studio", "stop-redirect", "cloud-onboarding"]);
|
||||
expect(runnerSuites.some(suite => suite.id === "product-journeys")).toBe(false);
|
||||
expect(runnerMatrix.some(execution => execution.suite.id === "product-journeys")).toBe(false);
|
||||
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(execution => execution.suite.id === "product-journeys")).toBe(false);
|
||||
|
||||
@@ -19,7 +19,7 @@ function associate(suites: readonly string[], cases: Record<string, readonly str
|
||||
}
|
||||
|
||||
associate(["product-journeys"], { "task-create": ["tasks/create"], "ask-question": ["questions-and-approvals/questions"],
|
||||
"task-properties": ["tasks/properties"], "cloud-onboarding": ["onboarding/first-task"] });
|
||||
"task-properties": ["tasks/properties"], "task-search-filter": ["tasks/organize"], "cloud-onboarding": ["onboarding/first-task"] });
|
||||
associate(["antigravity-agent-journey"], {
|
||||
"saved-subscription-two-turn": ["agents/hire", "agents/instructions", "runs-adapters/validation", "runs-adapters/session",
|
||||
"tasks/create", "tasks/completion", "documents-artifacts/attachments", "documents-artifacts/work-products"],
|
||||
|
||||
@@ -3,7 +3,7 @@ import { defineConfig } from "@playwright/test";
|
||||
/** Browser-only harness regressions: no Paperclip instance or provider credentials. */
|
||||
export default defineConfig({
|
||||
testDir: ".",
|
||||
testMatch: ["skill-studio-flow.spec.ts", "native-task-reply.spec.ts", "screenshot-readiness.spec.ts", "service-worker-reload.spec.ts", "lost-send.spec.ts", "chat-restart.spec.ts", "settings-toggle.spec.ts", "browser-bootstrap-diagnostics.spec.ts", "service-worker-module-loading.spec.ts", "browser-bootstrap-recovery.spec.ts", "blocker-input.spec.ts", "native-blocker-visible.spec.ts", "copilot-permission-card.spec.ts", "github-review-check.spec.ts", "antigravity-journey-browser.spec.ts"],
|
||||
testMatch: ["skill-studio-flow.spec.ts", "native-task-reply.spec.ts", "screenshot-readiness.spec.ts", "service-worker-reload.spec.ts", "lost-send.spec.ts", "chat-restart.spec.ts", "settings-toggle.spec.ts", "browser-bootstrap-diagnostics.spec.ts", "service-worker-module-loading.spec.ts", "browser-bootstrap-recovery.spec.ts", "blocker-input.spec.ts", "native-blocker-visible.spec.ts", "copilot-permission-card.spec.ts", "github-review-check.spec.ts", "antigravity-journey-browser.spec.ts", "task-search-filter.spec.ts"],
|
||||
workers: 1,
|
||||
retries: 0,
|
||||
timeout: 10_000,
|
||||
|
||||
@@ -31,13 +31,14 @@ vi.mock("../dashboard-regenerate.js", () => ({ regenerateRunnerDashboard: vi.fn(
|
||||
if(input.outputDirectory){const {writeFile}=await import("node:fs/promises");await Promise.all(["index.html","dashboard.html"].map(name=>writeFile(path.join(input.outputDirectory!,name),"dashboard")));}
|
||||
}) }));
|
||||
vi.mock("./task-properties.js", () => ({taskProperties: async (_context:unknown,_session:unknown,_id:unknown,evidence:any) => {evidence.checks.push({id:"assignment-persisted",passed:true},{id:"no-agent-work",passed:true});}}));
|
||||
vi.mock("./task-search-filter.js", () => ({taskSearchFilter: async (_context:unknown,_session:unknown,_id:unknown,evidence:any) => {evidence.checks.push(...["task-search-results-exact","task-status-filter-excludes-controls","task-search-filter-reloaded","task-search-result-reopened","no-agent-work"].map(id=>({id,passed:true})));}}));
|
||||
import { chromium } from "@playwright/test";
|
||||
import { launchProductJourneys } from "./launch.js";
|
||||
import { regenerateRunnerDashboard } from "../dashboard-regenerate.js";
|
||||
const roots: string[]=[];
|
||||
afterEach(async()=>{vi.useRealTimers(); vi.restoreAllMocks(); persistenceFault.kind=null; persistenceFault.active=false; persistenceFault.latestReport=null; process.exitCode=0; await Promise.all(roots.splice(0).map(root=>rm(root,{recursive:true,force:true})));});
|
||||
type Hook = "collect" | "release" | "stop" | "finalizeEvidence" | "checkpointEvidence";
|
||||
async function run(options:{authFailure?:boolean;onboardingFailure?:boolean;preAcceptanceFailure?:boolean;missingArtifact?:boolean;drift?:boolean;cleanupFailure?:boolean;collectionFailure?:boolean;mini?:boolean;missingReceipt?:boolean;zeroRequests?:boolean;missingCheck?:boolean;finalizationFailure?:boolean;persistenceFailure?:"write"|"rename";persistenceAfter?:"collection"|"release";stallHook?:Hook;onHook?:(hook:Hook,signal?:AbortSignal)=>void}={}) {
|
||||
async function run(options:{search?:boolean;authFailure?:boolean;onboardingFailure?:boolean;preAcceptanceFailure?:boolean;missingArtifact?:boolean;drift?:boolean;cleanupFailure?:boolean;collectionFailure?:boolean;mini?:boolean;missingReceipt?:boolean;zeroRequests?:boolean;missingCheck?:boolean;finalizationFailure?:boolean;persistenceFailure?:"write"|"rename";persistenceAfter?:"collection"|"release";stallHook?:Hook;onHook?:(hook:Hook,signal?:AbortSignal)=>void}={}) {
|
||||
const root=await mkdtemp(path.join(os.tmpdir(),"journey-campaign-test-")); roots.push(root);
|
||||
const module=path.join(root,"provider.mjs");
|
||||
const proof = createConnectionProof();
|
||||
@@ -61,7 +62,7 @@ const report=JSON.parse(await readFile(input.outputDir+'/normalized-results.json
|
||||
if(report.results.some(r=>r.cleanup!=='failed'))throw Error('failure checkpoint preceded failure persistence');
|
||||
await writeFile(input.outputDir+'/failure-checkpoint.json',JSON.stringify(report));
|
||||
}}}};`);
|
||||
const config={version:1,id:"qa",mode:"new-cloud",adapterModule:module,journeys:["cloud-onboarding","task-properties"],executionEnvironment:"daytona",baseline:{features:{},connections:[]},limits:{durationMs:60000,modelBudgetCents:1000,maxRuns:1},...(options.mini?{modelProfileId:"runner-codex-mini"}:{})};
|
||||
const config={version:1,id:"qa",mode:"new-cloud",adapterModule:module,journeys:["cloud-onboarding",options.search?"task-search-filter":"task-properties"],executionEnvironment:"daytona",baseline:{features:{},connections:[]},limits:{durationMs:60000,modelBudgetCents:1000,maxRuns:1},...(options.mini?{modelProfileId:"runner-codex-mini"}:{})};
|
||||
const file=path.join(root,"target.json"),output=path.join(root,"report"); await writeFile(file,JSON.stringify(config));
|
||||
const lifecycle: string[] = [];
|
||||
let artifactBeforeRelease: unknown;
|
||||
@@ -86,6 +87,13 @@ await writeFile(input.outputDir+'/failure-checkpoint.json',JSON.stringify(report
|
||||
failureCheckpoint: await readFile(path.join(output,"failure-checkpoint.json"),"utf8").then(JSON.parse).catch(()=>null) };
|
||||
}
|
||||
describe("product journey campaign boundary outcomes",()=>{
|
||||
it("dispatches task search separately and reports it without model work", async () => {
|
||||
const campaign = await run({search:true});
|
||||
const search = campaign.results[1];
|
||||
expect(search).toMatchObject({caseId:"task-search-filter",profileId:"browser",model:null,provider:null,runtimeMode:"none",status:"passed"});
|
||||
expect(search.runIds ?? []).toEqual([]);
|
||||
expect(search.productJourney.checks).toHaveLength(5);
|
||||
});
|
||||
it("accepts the conventional pnpm separator in a credential-free preview", async () => {
|
||||
const root=await mkdtemp(path.join(os.tmpdir(),"journey-preview-test-")); roots.push(root);
|
||||
const file=path.join(root,"target.json");
|
||||
|
||||
@@ -5,6 +5,7 @@ export const productJourneys = [
|
||||
{ id: "task-create", label: "Create a task for yourself and reopen it", requires: ["company", "board-user", "task-write"], capabilities: ["tasks/create"], paid: false },
|
||||
{ id: "ask-question", label: "Ask an agent a question and read its answer", requires: ["company", "board-user", "task-write", "agent"], capabilities: ["questions-and-approvals/questions"], paid: true },
|
||||
{ id: "task-properties", label: "Assign a task to the signed-in user", requires: ["company", "board-user", "task-write", "backlog-task"], capabilities: ["tasks/properties"], paid: false },
|
||||
{ id: "task-search-filter", label: "Search and filter tasks, then reopen the exact saved result", requires: ["company", "board-user", "task-write"], capabilities: ["tasks/organize"], paid: false },
|
||||
{ id: "plan-revise-accept", label: "Revise and approve a saved Plan, then complete the task", requires: ["company", "board-user", "task-write", "agent"], capabilities: ["questions-and-approvals/plan-review"], paid: true },
|
||||
{ id: "build-revise", label: "Download and independently verify both artifact versions", requires: ["company", "board-user", "task-write", "agent"], capabilities: ["documents-artifacts/attachments"], paid: true },
|
||||
{ id: "protected-actions", label: "Approve and decline protected service actions", requires: ["company", "board-user", "task-write", "agent", "review-service"], capabilities: ["questions-and-approvals/tool-review"], paid: true },
|
||||
|
||||
@@ -9,6 +9,7 @@ import { runnerSuites } from "../catalog.js";
|
||||
import { productJourneys, parseTargetConfig, preparationPreview, isAgentJourney, agentJourneyRuns } from "./catalog.js";
|
||||
import { journeyFixtures } from "./fixtures.js";
|
||||
import { taskProperties } from "./task-properties.js";
|
||||
import { taskSearchFilter } from "./task-search-filter.js";
|
||||
import { askQuestion, taskCreate } from "./basic-flows.js";
|
||||
import { runAgentJourney } from "./agent-flows.js";
|
||||
import { JourneyError, type JourneyContext, type JourneyEvidence, type JourneyOutcome, type TargetLifecycleInput, type TargetProvider, type TargetSession } from "./types.js";
|
||||
@@ -102,7 +103,7 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri
|
||||
try { sha = execFileSync("git", ["rev-parse", "HEAD"], { cwd: repositoryRoot, encoding: "utf8", stdio: ["ignore", "pipe", "ignore"] }).trim(); } catch { /* Packaged harnesses may not include Git metadata. */ }
|
||||
const source = resolveRunnerE2ESource({ sha, ref: null, workflowRunUrl: null });
|
||||
const definitionHash = createHash("sha256").update(JSON.stringify(productJourneys));
|
||||
for (const name of ["catalog.ts", "task-properties.ts", "basic-flows.ts", "basic-evidence.ts", "agent-flows.ts", "success-contract.ts", "product-screenshot.ts", "cloud-onboarding.ts", "onboarding-connect.ts", "onboarding-first-task.ts", "workspace-deferral.ts", "onboarding-pre-acceptance.ts", "artifact-evidence.ts", "fixtures.ts", "launch.ts", "types.ts", "collection.ts", "local-target.ts"]) definitionHash.update(await readFile(path.join(import.meta.dirname, name)));
|
||||
for (const name of ["catalog.ts", "task-properties.ts", "task-search-filter.ts", "task-search-evidence.ts", "basic-flows.ts", "basic-evidence.ts", "agent-flows.ts", "success-contract.ts", "product-screenshot.ts", "cloud-onboarding.ts", "onboarding-connect.ts", "onboarding-first-task.ts", "workspace-deferral.ts", "onboarding-pre-acceptance.ts", "artifact-evidence.ts", "fixtures.ts", "launch.ts", "types.ts", "collection.ts", "local-target.ts"]) definitionHash.update(await readFile(path.join(import.meta.dirname, name)));
|
||||
for (const file of ["catalog.ts", "api.ts", "run-observations.ts", "native-task-reply.ts", "native-event-envelope.ts", "user-actions.ts", "plan-revision-flow.ts", "skill-studio-flow.ts", "protected-action-flow.ts", "story-artifact.ts", "stop-redirect-flow.ts", "everyday-cases.ts", "everyday-delivery.ts", "everyday-decisions.ts", "connection-guidance-evidence.ts", "everyday-artifact.py"]) definitionHash.update(await readFile(path.join(import.meta.dirname, "..", file)));
|
||||
definitionHash.update(runnerSuites.flatMap(suite => suite.tasks).find(task => task.id === "ask-question")!.buildPrompt("definition"));
|
||||
const suiteDefinitionHash = definitionHash.digest("hex");
|
||||
@@ -207,6 +208,7 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri
|
||||
await captureProductScreenshot(ready!.page, ready!.api.origin, path.join(directory, file));
|
||||
} });
|
||||
} else if (id === "task-create") await taskCreate(ready, session!, campaignId, evidence);
|
||||
else if (id === "task-search-filter") await taskSearchFilter(ready, session!, campaignId, evidence);
|
||||
else await taskProperties(ready, session!, campaignId, evidence);
|
||||
}
|
||||
evidence.deployment = ready.identity;
|
||||
|
||||
@@ -4,6 +4,7 @@ export const agentJourneyRuns = { "ask-question": 1, "plan-revise-accept": 3, "b
|
||||
export const journeyRequiredChecks: Record<string, readonly string[]> = {
|
||||
"task-properties": ["assignment-persisted", "no-agent-work"],
|
||||
"task-create": ["browser-created-task-persisted", "no-agent-work"],
|
||||
"task-search-filter": ["task-search-results-exact", "task-status-filter-excludes-controls", "task-search-filter-reloaded", "task-search-result-reopened", "no-agent-work"],
|
||||
"ask-question": ["question-answer-persisted", "one-bound-successful-run", "configured-model-and-environment"],
|
||||
"plan-revise-accept": ["plan-revision-and-approval", "plan-final-answer", "configured-model-and-environment"],
|
||||
"build-revise": ["initial-artifact-verified", "revised-artifact-verified", "artifact-versions-differ", "original-artifact-preserved", "user-reply-stored-once", "configured-model-and-environment"],
|
||||
|
||||
@@ -0,0 +1,101 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { taskSearchEvidencePasses, type SearchTask } from "./task-search-evidence.js";
|
||||
import { parseTargetConfig, preparationPreview } from "./catalog.js";
|
||||
import { journeyRequiredChecks } from "./success-contract.js";
|
||||
import { validateRetainedRunnerResult } from "../result-validation.js";
|
||||
import type { RunnerE2EResult } from "../types.js";
|
||||
|
||||
function evidence(): Parameters<typeof taskSearchEvidencePasses>[0] {
|
||||
const common = { companyId: "company-1", assigneeUserId: "user-1", assigneeAgentId: null } as const;
|
||||
const expected: SearchTask[] = [
|
||||
{ ...common, id: "target", identifier: "QA-1", title: "needle-campaign target", description: "Exact target context.", status: "backlog" },
|
||||
{ ...common, id: "status-control", identifier: "QA-2", title: "needle-campaign control", description: "Completed control context.", status: "done" },
|
||||
{ ...common, id: "search-control", identifier: "QA-3", title: "unrelated control", description: "Other search context.", status: "backlog" },
|
||||
];
|
||||
return { expected, saved: structuredClone(expected), runs: [[], [], []], marker: "needle-campaign",
|
||||
searchIds: ["status-control", "target"], filteredIds: ["target"], reloadedIds: ["target"], returnedIds: ["target"],
|
||||
reloadedQuery: "needle-campaign", returnedQuery: "needle-campaign", reloadedStatusSelected: true, returnedStatusSelected: true,
|
||||
openedIdentifier: "QA-1", reopenedIdentifier: "QA-1", titleVisible: true, descriptionVisible: true };
|
||||
}
|
||||
|
||||
describe("task search and filter independent evidence", () => {
|
||||
it("accepts the exact target and controls independent of result order", () => {
|
||||
expect(taskSearchEvidencePasses(evidence())).toEqual({ search: true, filter: true, persistence: true, reopen: true, noAgentWork: true });
|
||||
});
|
||||
it.each([
|
||||
["missing target", ["status-control"]], ["missing status control", ["target"]],
|
||||
["unrelated control", ["target", "status-control", "search-control"]],
|
||||
["duplicate target", ["target", "target"]], ["unknown result", ["target", "unknown:QA-9"]],
|
||||
])("rejects search evidence with %s", (_name, searchIds) => {
|
||||
expect(taskSearchEvidencePasses({ ...evidence(), searchIds }).search).toBe(false);
|
||||
});
|
||||
it.each([[], ["status-control"], ["target", "status-control"], ["target", "search-control"]])("rejects wrong or unfiltered results %j", (...filteredIds) => {
|
||||
expect(taskSearchEvidencePasses({ ...evidence(), filteredIds }).filter).toBe(false);
|
||||
});
|
||||
it.each(["reloadedQuery", "returnedQuery"] as const)("rejects lost %s", key => {
|
||||
expect(taskSearchEvidencePasses({ ...evidence(), [key]: "" }).persistence).toBe(false);
|
||||
});
|
||||
it.each(["reloadedStatusSelected", "returnedStatusSelected"] as const)("rejects a lost status filter at %s", key => {
|
||||
expect(taskSearchEvidencePasses({ ...evidence(), [key]: false }).persistence).toBe(false);
|
||||
});
|
||||
it.each(["reloadedIds", "returnedIds"] as const)("rejects a control after %s", key => {
|
||||
expect(taskSearchEvidencePasses({ ...evidence(), [key]: ["target", "status-control"] }).persistence).toBe(false);
|
||||
});
|
||||
it.each(["openedIdentifier", "reopenedIdentifier"] as const)("rejects the wrong task at %s", key => {
|
||||
expect(taskSearchEvidencePasses({ ...evidence(), [key]: "QA-2" }).reopen).toBe(false);
|
||||
});
|
||||
it.each(["titleVisible", "descriptionVisible"] as const)("requires visible saved %s", key => {
|
||||
expect(taskSearchEvidencePasses({ ...evidence(), [key]: false }).reopen).toBe(false);
|
||||
});
|
||||
it.each(["id", "identifier", "companyId", "title", "description", "status", "assigneeUserId", "assigneeAgentId"] as const)("rejects changed saved %s", key => {
|
||||
const input = evidence();
|
||||
Object.assign(input.saved[0]!, { [key]: "wrong" });
|
||||
expect(taskSearchEvidencePasses(input).reopen).toBe(false);
|
||||
});
|
||||
it("rejects missing saved records and incomplete control ownership", () => {
|
||||
const input = evidence(); input.saved = [null];
|
||||
expect(taskSearchEvidencePasses(input).reopen).toBe(false);
|
||||
const missingControl = evidence(); missingControl.expected = missingControl.expected.slice(0, 2);
|
||||
expect(Object.values(taskSearchEvidencePasses(missingControl))).toEqual([false, false, false, false, false]);
|
||||
});
|
||||
it("requires the controls to differ in the property each one tests", () => {
|
||||
const input = evidence(); input.expected[1]!.status = "backlog";
|
||||
expect(taskSearchEvidencePasses(input).filter).toBe(false);
|
||||
const sameMarker = evidence(); sameMarker.expected[2]!.description = sameMarker.marker;
|
||||
expect(taskSearchEvidencePasses(sameMarker).search).toBe(false);
|
||||
});
|
||||
it.each([[[], []], [[], null, []], [[{ runId: "unexpected" }], [], []], [[], [], [{ runId: "control-run" }]]])("rejects missing or nonzero run evidence %j", (...runs) => {
|
||||
expect(taskSearchEvidencePasses({ ...evidence(), runs }).noAgentWork).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("task search journey reporting contract", () => {
|
||||
it("previews a reusable browser journey without private access or model allowance", () => {
|
||||
const preview = preparationPreview(parseTargetConfig({ version: 1, id: "qa", mode: "existing-cloud",
|
||||
adapterModule: "/not-installed-private-adapter.mjs", journeys: ["task-search-filter"], executionEnvironment: "daytona",
|
||||
baseline: { features: {}, connections: [] }, limits: { durationMs: 60000, modelBudgetCents: 0, maxRuns: 0 } }));
|
||||
expect(preview.journeys).toMatchObject([{ id: "task-search-filter", capabilities: ["tasks/organize"], paid: false }]);
|
||||
expect(preview.limits.maxRuns).toBe(0);
|
||||
expect(preview.modelProfileId).toBeNull();
|
||||
});
|
||||
function result(): RunnerE2EResult {
|
||||
return { schema: "paperclip.runner-e2e.result/v2", suiteId: "product-journeys", suiteDefinitionHash: "test",
|
||||
executionId: "product-journeys.qa.browser.daytona.task-search-filter", profileId: "browser", environmentId: "daytona",
|
||||
caseId: "task-search-filter", provider: null, model: null, runtimeMode: "none", status: "passed", cleanup: "passed",
|
||||
attempt: 1, startedAt: "2026-01-01T00:00:00Z", finishedAt: "2026-01-01T00:00:01Z", durationMs: 1000,
|
||||
productJourney: { outcome: "passed", startingState: "recipe", phases: [{ phase: "journey", outcome: "passed" }],
|
||||
checks: journeyRequiredChecks["task-search-filter"]!.map(id => ({ id, passed: true })),
|
||||
privacy: "private", resourceDisposition: "released", deployment: { kind: "cloud", targetId: "qa",
|
||||
origin: "https://qa.example.test", commit: "a".repeat(40), cloudRelease: "test", runtimeImage: null, configurationHash: "test" } } };
|
||||
}
|
||||
it("requires every closed check and zero model/native work", () => {
|
||||
expect(() => validateRetainedRunnerResult(result())).not.toThrow();
|
||||
for (const id of journeyRequiredChecks["task-search-filter"]!) {
|
||||
const missing = result(); missing.productJourney!.checks = missing.productJourney!.checks.filter(check => check.id !== id);
|
||||
expect(() => validateRetainedRunnerResult(missing)).toThrow();
|
||||
}
|
||||
for (const extra of [{ provider: "openai" }, { model: "unexpected" }, { runtimeMode: "native" }, { runIds: ["unexpected"] }]) {
|
||||
expect(() => validateRetainedRunnerResult({ ...result(), ...extra })).toThrow();
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,60 @@
|
||||
export interface SearchTask {
|
||||
id: string;
|
||||
identifier: string;
|
||||
companyId: string;
|
||||
title: string;
|
||||
description: string;
|
||||
status: string;
|
||||
assigneeUserId: string;
|
||||
assigneeAgentId: null;
|
||||
}
|
||||
|
||||
function exactIds(actual: readonly string[], expected: readonly string[]) {
|
||||
return actual.length === expected.length && new Set(actual).size === actual.length
|
||||
&& expected.every(id => actual.includes(id));
|
||||
}
|
||||
|
||||
/** Saved records and observed UI identities must agree; missing evidence fails. */
|
||||
export function taskSearchEvidencePasses(input: {
|
||||
expected: readonly SearchTask[];
|
||||
saved: readonly (SearchTask | null)[];
|
||||
runs: readonly (readonly unknown[] | null)[];
|
||||
searchIds: readonly string[];
|
||||
filteredIds: readonly string[];
|
||||
reloadedIds: readonly string[];
|
||||
returnedIds: readonly string[];
|
||||
reloadedQuery: string;
|
||||
returnedQuery: string;
|
||||
marker: string;
|
||||
reloadedStatusSelected: boolean;
|
||||
returnedStatusSelected: boolean;
|
||||
openedIdentifier: string | null;
|
||||
reopenedIdentifier: string | null;
|
||||
titleVisible: boolean;
|
||||
descriptionVisible: boolean;
|
||||
}) {
|
||||
const [target, statusControl, searchControl] = input.expected;
|
||||
const validFixture = input.expected.length === 3 && Boolean(target?.id && target.identifier && target.companyId
|
||||
&& target.assigneeUserId && input.marker) && new Set(input.expected.map(task => task.id)).size === 3
|
||||
&& new Set(input.expected.map(task => task.identifier)).size === 3
|
||||
&& input.expected.every(task => task.companyId === target!.companyId && task.assigneeUserId === target!.assigneeUserId
|
||||
&& task.assigneeAgentId === null && task.description.length > 0)
|
||||
&& target!.status === "backlog" && statusControl?.status === "done" && searchControl?.status === "backlog"
|
||||
&& target!.title.includes(input.marker) && statusControl!.title.includes(input.marker)
|
||||
&& !`${searchControl!.title} ${searchControl!.description}`.includes(input.marker);
|
||||
const persisted = validFixture && input.saved.length === 3 && input.expected.every((expected, index) => {
|
||||
const saved = input.saved[index];
|
||||
return saved !== null && saved !== undefined && Object.keys(expected).every(key =>
|
||||
saved[key as keyof SearchTask] === expected[key as keyof SearchTask]);
|
||||
});
|
||||
return {
|
||||
search: validFixture && exactIds(input.searchIds, [target!.id, statusControl!.id]),
|
||||
filter: validFixture && exactIds(input.filteredIds, [target!.id]),
|
||||
persistence: validFixture && input.reloadedQuery === input.marker && input.returnedQuery === input.marker
|
||||
&& input.reloadedStatusSelected && input.returnedStatusSelected
|
||||
&& exactIds(input.reloadedIds, [target!.id]) && exactIds(input.returnedIds, [target!.id]),
|
||||
reopen: persisted && input.openedIdentifier === target!.identifier && input.reopenedIdentifier === target!.identifier
|
||||
&& input.titleVisible && input.descriptionVisible,
|
||||
noAgentWork: persisted && input.runs.length === 3 && input.runs.every(runs => Array.isArray(runs) && runs.length === 0),
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,266 @@
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { taskSearchFilter } from "./task-search-filter.js";
|
||||
import type { JourneyContext, JourneyEvidence, TargetSession } from "./types.js";
|
||||
import type { SearchTask } from "./task-search-evidence.js";
|
||||
|
||||
// Use immediate assertions for this deterministic UI port. Live Playwright still
|
||||
// owns browser rendering, locator compatibility, and asynchronous convergence.
|
||||
vi.mock("@playwright/test", () => ({ expect: Object.assign(
|
||||
(locator: { isVisible(): Promise<boolean>; inputValue(): Promise<string> }) => ({
|
||||
async toBeVisible() { expect(await locator.isVisible()).toBe(true); },
|
||||
async toHaveValue(value: string) { expect(await locator.inputValue()).toBe(value); },
|
||||
}),
|
||||
{ poll: (read: () => Promise<unknown>) => ({ async toEqual(value: unknown) { expect(await read()).toEqual(value); } }) },
|
||||
) }));
|
||||
|
||||
type Fault = "ignore-search" | "ignore-status" | "duplicate-result" | "unknown-result" | "reload-query"
|
||||
| "reload-status" | "back-query" | "back-status" | "wrong-route" | "reload-content"
|
||||
| "saved-company" | "saved-description" | "target-run" | "control-run" | "missing-run-evidence"
|
||||
| "intent" | "receipt" | "missing-identifier" | "ignore-named-view";
|
||||
type UiNode = { key: string; role?: string; name?: string; title?: string; text?: string; href?: string;
|
||||
visible?: boolean; children?: UiNode[]; click?(): void; fill?(value: string): void; check?(): void;
|
||||
value?(): string; checked?(): boolean };
|
||||
type CollectionState = { query: string; backlog: boolean };
|
||||
|
||||
/** A stateful task-list port, not an always-successful locator stub. Search and
|
||||
* filters compute their results from the same owned records that the API stores.
|
||||
* Navigation can lose UI state independently of saved state and model activity. */
|
||||
function fixture(fault?: Fault, shell: "coordinator" | "streamlined" | "legacy" = "coordinator") {
|
||||
const origin = "https://qa.example.test";
|
||||
const tasks: SearchTask[] = [];
|
||||
const ownership: Array<{ kind: string; name: string; id?: string; journeyId?: string }> = [];
|
||||
const reads: string[] = [];
|
||||
const observed: Array<{ event: "reload" | "back"; route: "collection" | "detail"; query: string;
|
||||
backlog: boolean; taskIds: string[]; displayedId: string | null; view: string }> = [];
|
||||
let pathname = "/QA/issues";
|
||||
let collection: CollectionState = { query: "old search", backlog: true };
|
||||
let previous: CollectionState | undefined;
|
||||
let allTasks = shell !== "streamlined";
|
||||
let selectedView = allTasks ? "all" : "mine";
|
||||
let viewMenu = false;
|
||||
let filterMenu = false;
|
||||
let displayedId: string | null = null;
|
||||
let lostContent = false;
|
||||
let navigated = false;
|
||||
const visibleTasks = () => {
|
||||
let rows = tasks.filter(task => (fault === "ignore-search" || task.title.includes(collection.query))
|
||||
&& (fault === "ignore-status" || !collection.backlog || task.status === "backlog"));
|
||||
if (fault === "duplicate-result") rows = [...rows, ...rows.slice(0, 1)];
|
||||
if (fault === "unknown-result") rows = [...rows, { ...tasks[0]!, id: "foreign", identifier: "QA-999", title: "Other task" }];
|
||||
return rows.slice().reverse(); // The journey must not depend on row order.
|
||||
};
|
||||
const observe = (event: "reload" | "back") => observed.push({ event, route: displayedId ? "detail" : "collection",
|
||||
...collection, taskIds: displayedId ? [] : visibleTasks().map(task => task.id), displayedId, view: selectedView });
|
||||
const openTask = (task: SearchTask) => {
|
||||
previous = { ...collection };
|
||||
// Model a stale prefetched header on an incorrectly bound link destination.
|
||||
pathname = `/QA/issues/${fault === "wrong-route" ? tasks[1]!.identifier : task.identifier}`;
|
||||
displayedId = task.id;
|
||||
};
|
||||
const links = (): UiNode[] => displayedId || !allTasks ? [] : visibleTasks().map((task, index) => ({
|
||||
key: `task-${task.id}-${index}`, role: "link", name: task.title, text: task.title,
|
||||
href: `/QA/issues/${task.identifier}`, click: () => openTask(task),
|
||||
}));
|
||||
const nodes = (): UiNode[] => {
|
||||
const shown = tasks.find(task => task.id === displayedId);
|
||||
const status: UiNode = { key: "backlog", role: "checkbox", name: shell === "coordinator" ? "Backlog" : "BacklogBacklog", checked: () => collection.backlog,
|
||||
check: () => { collection.backlog = true; } };
|
||||
const clear: UiNode = { key: "clear", role: "button", name: "Clear", visible: collection.backlog,
|
||||
click: () => { collection.backlog = false; } };
|
||||
return [
|
||||
{ key: "main" },
|
||||
{ key: "views", role: "button", name: shell === "streamlined" ? `Change view — currently ${selectedView}` : "Task view: All tasks",
|
||||
visible: !displayedId && shell !== "legacy", click: () => { viewMenu = true; } },
|
||||
{ key: "all-tasks", role: shell === "streamlined" ? "menuitem" : "menuitemradio", name: "All tasks", visible: viewMenu,
|
||||
click: () => { allTasks = true; selectedView = "all"; viewMenu = false; } },
|
||||
{ key: "backlog-view", role: "menuitem", name: "Backlog Not started yet", visible: viewMenu && shell === "streamlined",
|
||||
click: () => {
|
||||
if (fault !== "ignore-named-view") selectedView = "backlog";
|
||||
collection = { query: "", backlog: true }; viewMenu = false;
|
||||
} },
|
||||
{ key: "search", role: "textbox", name: "Search tasks", visible: allTasks && !displayedId,
|
||||
fill: value => { collection.query = value; }, value: () => collection.query },
|
||||
{ key: "layout", role: "button", name: "List view", visible: allTasks && !displayedId, click: () => {} },
|
||||
{ key: "filter", role: "button", name: collection.backlog ? "1" : "Filter",
|
||||
title: collection.backlog ? "Filters: 1" : "Filter", visible: allTasks && !displayedId,
|
||||
click: () => { filterMenu = true; } },
|
||||
{ key: "filters", role: "dialog", name: "Filters", visible: filterMenu, children: [clear,
|
||||
{ key: "status-group", children: [{ key: "status-heading", text: "Status" }, status] },
|
||||
{ key: "responsible-group", children: [{ key: "responsible-heading", text: "Responsible" },
|
||||
{ key: "responsible-backlog", role: "checkbox", name: "Backlog", checked: () => false,
|
||||
check: () => { throw Error("Selected an assignee instead of a status"); } }] },
|
||||
] },
|
||||
{ key: "heading", role: "heading", name: shown?.title, text: shown?.title, visible: Boolean(shown && !lostContent) },
|
||||
{ key: "description", text: shown?.description, visible: Boolean(shown && !lostContent) },
|
||||
...links(),
|
||||
];
|
||||
};
|
||||
const flatten = (input: UiNode[]): UiNode[] => input.flatMap(node => node.visible === false ? [] : [node, ...flatten(node.children ?? [])]);
|
||||
const matches = (text: string | undefined, wanted: string | RegExp) => wanted instanceof RegExp ? wanted.test(text ?? "") : text === wanted;
|
||||
const locator = (resolve: () => UiNode[]) => {
|
||||
const unique = () => {
|
||||
const found = resolve();
|
||||
if (found.length !== 1) throw Error(`Expected one visible UI control, found ${found.length}`);
|
||||
return found[0]!;
|
||||
};
|
||||
return {
|
||||
resolve,
|
||||
or(other: { resolve(): UiNode[] }) { return locator(() => [...new Map([...resolve(), ...other.resolve()].map(node => [node.key, node])).values()]); },
|
||||
first() { return locator(() => resolve().slice(0, 1)); },
|
||||
filter({ hasText }: { hasText: string }) { return locator(() => resolve().filter(node => node.text?.includes(hasText))); },
|
||||
locator(selector: string) {
|
||||
if (selector !== "..") throw Error(`Unsupported control-relative selector: ${selector}`);
|
||||
return locator(() => {
|
||||
const keys = new Set(resolve().map(node => node.key));
|
||||
return flatten(nodes()).filter(node => node.children?.some(child => keys.has(child.key)));
|
||||
});
|
||||
},
|
||||
getByText(text: string) {
|
||||
return locator(() => flatten(resolve().flatMap(node => node.children ?? [])).filter(node => node.text === text));
|
||||
},
|
||||
getByRole(role: string, { name }: { name: string | RegExp }) {
|
||||
return locator(() => flatten(resolve().flatMap(node => node.children ?? [])).filter(node => node.role === role && matches(node.name, name)));
|
||||
},
|
||||
async all() { return resolve().map(node => locator(() => [node])); },
|
||||
async isVisible() { return resolve().length > 0; },
|
||||
async getAttribute(name: string) { expect(name).toBe("href"); return unique().href ?? null; },
|
||||
async inputValue() { const read = unique().value; if (!read) throw Error("Not an input"); return read(); },
|
||||
async isChecked() { const read = unique().checked; if (!read) throw Error("Not a checkbox"); return read(); },
|
||||
async click() { const action = unique().click; if (!action) throw Error("Not clickable"); action(); },
|
||||
async fill(value: string) { const action = unique().fill; if (!action) throw Error("Not fillable"); action(value); },
|
||||
async check() { const action = unique().check; if (!action) throw Error("Not checkable"); action(); },
|
||||
};
|
||||
};
|
||||
const page = {
|
||||
url: () => `${origin}${pathname}`,
|
||||
locator(selector: string) {
|
||||
if (selector === "main#main-content") return locator(() => [{ key: "main" }]);
|
||||
if (selector === "main#main-content a[data-inbox-issue-link]") return locator(links);
|
||||
throw Error(`Unsupported product selector: ${selector}`);
|
||||
},
|
||||
getByRole(role: string, { name }: { name: string | RegExp }) {
|
||||
return locator(() => flatten(nodes()).filter(node => node.role === role && matches(node.name, name)));
|
||||
},
|
||||
getByTitle(title: RegExp) { return locator(() => flatten(nodes()).filter(node => matches(node.title, title))); },
|
||||
getByText(text: string) { return locator(() => flatten(nodes()).filter(node => node.text === text)); },
|
||||
async goto(url: string) { expect(url).toBe(`${origin}/QA/issues`); navigated = true; },
|
||||
keyboard: { async press(key: string) { expect(key).toBe("Escape"); filterMenu = false; } },
|
||||
async reload() {
|
||||
// Combined Tasks explicitly overrides saved checkbox state with the
|
||||
// named view's preset. The journey must choose a supported named view.
|
||||
if (!displayedId && shell === "streamlined") collection.backlog = selectedView === "backlog";
|
||||
if (!displayedId && fault === "reload-query") collection.query = "";
|
||||
if (!displayedId && fault === "reload-status") collection.backlog = false;
|
||||
if (displayedId && fault === "reload-content") lostContent = true;
|
||||
observe("reload");
|
||||
},
|
||||
async goBack() {
|
||||
if (!displayedId || !previous) throw Error("No previous collection");
|
||||
displayedId = null; pathname = "/QA/issues"; collection = { ...previous };
|
||||
if (shell === "streamlined") collection.backlog = selectedView === "backlog";
|
||||
if (fault === "back-query") collection.query = "";
|
||||
if (fault === "back-status") collection.backlog = false;
|
||||
observe("back");
|
||||
},
|
||||
};
|
||||
const api = {
|
||||
origin,
|
||||
async post(route: string, body: Omit<SearchTask, "id" | "identifier" | "companyId">) {
|
||||
expect(route).toBe("/api/companies/company-1/issues");
|
||||
// A mutation without durable ownership intent is unsafe, even if the
|
||||
// browser journey would otherwise produce all its success checks.
|
||||
expect(ownership.some(resource => !resource.id && resource.name === body.title)).toBe(true);
|
||||
const task = { ...body, companyId: "company-1", id: `task-${tasks.length + 1}`, identifier: `QA-${tasks.length + 1}` };
|
||||
tasks.push(task);
|
||||
return fault === "missing-identifier" ? { id: task.id } : { id: task.id, identifier: task.identifier };
|
||||
},
|
||||
async get(route: string) {
|
||||
reads.push(route);
|
||||
const match = route.match(/^\/api\/issues\/(task-\d)(\/runs)?$/);
|
||||
if (!match) throw Error(`Unexpected public read: ${route}`);
|
||||
const task = tasks.find(task => task.id === match[1]);
|
||||
if (!task) throw Error("Read outside owned task set");
|
||||
if (match[2]) {
|
||||
if (fault === "missing-run-evidence") return null;
|
||||
return (fault === "target-run" && task.id === "task-1") || (fault === "control-run" && task.id === "task-3") ? [{ runId: "unexpected" }] : [];
|
||||
}
|
||||
return { ...task, ...(fault === "saved-company" && task.id === "task-1" ? { companyId: "other-company" } : {}),
|
||||
...(fault === "saved-description" && task.id === "task-1" ? { description: "Different saved context" } : {}) };
|
||||
},
|
||||
};
|
||||
const session = { async own(resource: (typeof ownership)[number]) {
|
||||
if ((fault === "intent" && !resource.id) || (fault === "receipt" && resource.id)) throw Error("Ownership storage unavailable");
|
||||
ownership.push(resource);
|
||||
} } as unknown as TargetSession;
|
||||
const evidence = { checks: [] } as unknown as JourneyEvidence;
|
||||
const context = { page, api, company: { id: "company-1", issuePrefix: "QA" }, user: { id: "user-1", name: "Board" },
|
||||
capabilities: ["company", "board-user", "task-write"] } as unknown as JourneyContext;
|
||||
return { run: () => taskSearchFilter(context, session, "campaign-1", evidence), tasks, ownership, reads, observed, evidence,
|
||||
navigated: () => navigated, context };
|
||||
}
|
||||
|
||||
describe("taskSearchFilter browser journey", () => {
|
||||
it.each(["coordinator", "streamlined", "legacy"] as const)("checks owned UI results and persisted content in the %s task shell", async shell => {
|
||||
const test = fixture(undefined, shell);
|
||||
await test.run();
|
||||
expect(test.evidence.checks).toEqual([
|
||||
{ id: "task-search-results-exact", passed: true }, { id: "task-status-filter-excludes-controls", passed: true },
|
||||
{ id: "task-search-filter-reloaded", passed: true }, { id: "task-search-result-reopened", passed: true },
|
||||
{ id: "no-agent-work", passed: true },
|
||||
]);
|
||||
for (const task of test.tasks) {
|
||||
expect(test.ownership).toContainEqual({ kind: "issue", name: task.title, journeyId: "task-search-filter" });
|
||||
expect(test.ownership).toContainEqual({ kind: "issue", name: task.title, id: task.id, journeyId: "task-search-filter" });
|
||||
expect(test.reads).toContain(`/api/issues/${task.id}`);
|
||||
expect(test.reads).toContain(`/api/issues/${task.id}/runs`);
|
||||
}
|
||||
const filtered = { query: "needle-campaign-1", backlog: true, taskIds: ["task-1"], displayedId: null,
|
||||
view: shell === "streamlined" ? "backlog" : "all" };
|
||||
expect(test.observed).toContainEqual({ event: "reload", route: "collection", ...filtered });
|
||||
expect(test.observed).toContainEqual({ event: "back", route: "collection", ...filtered });
|
||||
expect(test.observed.filter(view => view.event === "reload" && view.route === "detail").map(view => view.displayedId))
|
||||
.toEqual(["task-1", "task-1"]);
|
||||
});
|
||||
it("requires the streamlined named status view when All tasks clears temporary filters on mount", async () => {
|
||||
const test = fixture("ignore-named-view", "streamlined");
|
||||
await expect(test.run()).rejects.toThrow();
|
||||
expect(test.observed).toContainEqual(expect.objectContaining({ event: "reload", view: "all", backlog: false,
|
||||
taskIds: ["task-2", "task-1"] }));
|
||||
expect(test.evidence.checks).toEqual([]);
|
||||
});
|
||||
it.each<Fault>(["ignore-search", "ignore-status", "duplicate-result", "unknown-result", "reload-query", "reload-status",
|
||||
"back-query", "back-status", "reload-content"])("does not report success when the UI has %s", async fault => {
|
||||
const test = fixture(fault);
|
||||
await expect(test.run()).rejects.toThrow();
|
||||
expect(test.evidence.checks.some(check => check.passed)).toBe(false);
|
||||
});
|
||||
it.each<Fault>(["wrong-route", "saved-company", "saved-description", "target-run", "control-run", "missing-run-evidence"])
|
||||
("fails independent verification for %s after otherwise valid UI actions", async fault => {
|
||||
const test = fixture(fault);
|
||||
await expect(test.run()).rejects.toMatchObject({ outcome: "failed", code: "task_search_evidence_mismatch" });
|
||||
expect(test.evidence.checks.some(check => !check.passed)).toBe(true);
|
||||
});
|
||||
it.each<Fault>(["intent", "receipt", "missing-identifier"])("stops setup safely on %s failure", async fault => {
|
||||
const test = fixture(fault);
|
||||
await expect(test.run()).rejects.toThrow();
|
||||
expect(test.tasks).toHaveLength(fault === "intent" ? 0 : 1);
|
||||
expect(test.navigated()).toBe(false);
|
||||
expect(test.evidence.checks).toEqual([]);
|
||||
if (fault === "missing-identifier") expect(test.ownership.at(-1)).toMatchObject({ id: "task-1" });
|
||||
});
|
||||
it("rejects unsupported capability before creating resources", async () => {
|
||||
const test = fixture(); test.context.capabilities = ["company", "board-user"];
|
||||
await expect(test.run()).rejects.toMatchObject({ outcome: "unsupported", code: "missing_task-write" });
|
||||
expect(test.tasks).toEqual([]);
|
||||
expect(test.ownership).toEqual([]);
|
||||
});
|
||||
it.each([
|
||||
["intent", "prepare"], ["ignore-search", "search_results"], ["ignore-status", "apply_status"],
|
||||
["reload-query", "reload_collection"], ["reload-content", "reload_result"], ["back-query", "return_collection"],
|
||||
] as const)("retains only a closed stage code for %s failure", async (fault, stage) => {
|
||||
const test = fixture(fault);
|
||||
await expect(test.run()).rejects.toMatchObject({ outcome: "failed", code: `task_search_${stage}_failed`,
|
||||
message: `task_search_${stage}_failed` });
|
||||
expect(test.evidence.checks).toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,177 @@
|
||||
import { expect, type Locator, type Page } from "@playwright/test";
|
||||
import { taskSearchEvidencePasses, type SearchTask } from "./task-search-evidence.js";
|
||||
import { JourneyError, type JourneyContext, type JourneyEvidence, type TargetSession } from "./types.js";
|
||||
|
||||
type TaskSearchStage = "prepare" | "open_collection" | "clear_filters" | "search_results" | "apply_status"
|
||||
| "persist_view" | "reload_collection" | "open_result" | "reload_result" | "return_collection"
|
||||
| "reopen_result" | "verify_saved";
|
||||
|
||||
/** The legacy glyph contributes a second Backlog to the checkbox's name. */
|
||||
export function taskSearchBacklogCheckbox(filters: Locator) {
|
||||
return filters.getByText("Status", { exact: true }).locator("..")
|
||||
.getByRole("checkbox", { name: /^Backlog(?:\s*Backlog)?$/ });
|
||||
}
|
||||
|
||||
async function resultIds(page: Page, tasks: readonly SearchTask[]) {
|
||||
const ids: string[] = [];
|
||||
for (const link of await page.locator("main#main-content a[data-inbox-issue-link]").all()) {
|
||||
if (!await link.isVisible()) continue;
|
||||
const href = await link.getAttribute("href");
|
||||
const identifier = href ? decodeURIComponent(new URL(href, page.url()).pathname.split("/").at(-1)!) : "";
|
||||
ids.push(tasks.find(task => task.identifier === identifier || task.id === identifier)?.id ?? `unknown:${identifier}`);
|
||||
}
|
||||
return ids.sort();
|
||||
}
|
||||
|
||||
async function expectResults(page: Page, tasks: readonly SearchTask[], expectedIds: readonly string[]) {
|
||||
let observed: string[] = [];
|
||||
await expect.poll(async () => { observed = await resultIds(page, tasks); return observed; }, { timeout: 30_000 })
|
||||
.toEqual([...expectedIds].sort());
|
||||
return observed;
|
||||
}
|
||||
|
||||
async function openFilters(page: Page) {
|
||||
await page.getByRole("button", { name: /^(Filter|Filters: \d+)$/ })
|
||||
.or(page.getByTitle(/^(Filter|Filters: \d+)$/)).click();
|
||||
const filters = page.getByRole("dialog", { name: "Filters", exact: true });
|
||||
await expect(filters).toBeVisible();
|
||||
return filters;
|
||||
}
|
||||
|
||||
async function readStatusSelection(page: Page) {
|
||||
const filters = await openFilters(page);
|
||||
const selected = await taskSearchBacklogCheckbox(filters).isChecked();
|
||||
await page.keyboard.press("Escape");
|
||||
return selected;
|
||||
}
|
||||
|
||||
function openedIdentifier(page: Page, context: JourneyContext) {
|
||||
const url = new URL(page.url());
|
||||
const prefix = `/${context.company.issuePrefix}/issues/`;
|
||||
return url.origin === context.api.origin && url.pathname.startsWith(prefix)
|
||||
? decodeURIComponent(url.pathname.slice(prefix.length)) : null;
|
||||
}
|
||||
|
||||
/** API setup owns three tasks; search, filtering and opening use the shipped UI. */
|
||||
export async function taskSearchFilter(context: JourneyContext, session: TargetSession, campaignId: string, evidence: JourneyEvidence) {
|
||||
for (const capability of ["company", "board-user", "task-write"]) {
|
||||
if (!context.capabilities.includes(capability)) throw new JourneyError("unsupported", `missing_${capability}`);
|
||||
}
|
||||
if (!campaignId) throw new JourneyError("blocked", "task_search_campaign_required");
|
||||
const { api, page, company, user } = context;
|
||||
let stage: TaskSearchStage = "prepare";
|
||||
try {
|
||||
const marker = `needle-${campaignId}`;
|
||||
const tasks: SearchTask[] = [];
|
||||
for (const fixture of [
|
||||
{ title: `Journey QA ${marker} target`, status: "backlog", description: `Search target context ${campaignId}. Keep this for board review.` },
|
||||
{ title: `Journey QA ${marker} status control`, status: "done", description: `Completed control context ${campaignId}.` },
|
||||
{ title: `Journey QA unrelated ${campaignId}`, status: "backlog", description: `Different search control context ${campaignId}.` },
|
||||
]) {
|
||||
const ownership = { kind: "issue" as const, name: fixture.title, journeyId: "task-search-filter" as const };
|
||||
await session.own(ownership);
|
||||
const created = await api.post<{ id: string; identifier: string }>(`/api/companies/${company.id}/issues`, {
|
||||
...fixture, assigneeUserId: user.id, assigneeAgentId: null,
|
||||
});
|
||||
if (created.id) await session.own({ ...ownership, id: created.id });
|
||||
if (!created.id || !created.identifier) throw new JourneyError("failed", "task_search_fixture_receipt_missing");
|
||||
tasks.push({ ...fixture, id: created.id, identifier: created.identifier, companyId: company.id,
|
||||
assigneeUserId: user.id, assigneeAgentId: null });
|
||||
}
|
||||
const [target, statusControl] = tasks as [SearchTask, SearchTask, SearchTask];
|
||||
stage = "open_collection";
|
||||
await page.goto(`${api.origin}/${company.issuePrefix}/issues`);
|
||||
await expect(page.locator("main#main-content")).toBeVisible();
|
||||
const search = page.getByRole("textbox", { name: "Search tasks", exact: true });
|
||||
// Select the task collection through its normal view control when present.
|
||||
const views = page.getByRole("button", { name: /^(Change view — currently |Task view: )/ }).first();
|
||||
await expect(views.or(search).first()).toBeVisible();
|
||||
if (await views.isVisible()) {
|
||||
await views.click();
|
||||
await page.getByRole("menuitem", { name: /^All tasks/ }).or(page.getByRole("menuitemradio", { name: "All tasks", exact: true })).click();
|
||||
}
|
||||
await expect(search).toBeVisible();
|
||||
const listView = page.getByRole("button", { name: "List view", exact: true });
|
||||
if (await listView.isVisible()) await listView.click();
|
||||
stage = "clear_filters";
|
||||
const filters = await openFilters(page);
|
||||
const clear = filters.getByRole("button", { name: "Clear", exact: true });
|
||||
if (await clear.isVisible()) await clear.click();
|
||||
await page.keyboard.press("Escape");
|
||||
stage = "search_results";
|
||||
await search.fill(marker);
|
||||
const searchIds = await expectResults(page, tasks, [target.id, statusControl.id]);
|
||||
stage = "apply_status";
|
||||
const statusFilters = await openFilters(page);
|
||||
await taskSearchBacklogCheckbox(statusFilters).check();
|
||||
await page.keyboard.press("Escape");
|
||||
const filteredIds = await expectResults(page, tasks, [target.id]);
|
||||
|
||||
// The combined Tasks shell defines status through its named view on mount.
|
||||
// Its Backlog view preserves the chosen status across reload/Back navigation.
|
||||
stage = "persist_view";
|
||||
const namedView = page.getByRole("button", { name: /^Change view — currently / }).first();
|
||||
if (await namedView.isVisible()) {
|
||||
await namedView.click();
|
||||
await page.getByRole("menuitem", { name: /^Backlog/ }).click();
|
||||
await expect(search).toHaveValue("");
|
||||
await search.fill(marker);
|
||||
await expectResults(page, tasks, [target.id]);
|
||||
}
|
||||
|
||||
stage = "reload_collection";
|
||||
await page.reload();
|
||||
await expect(search).toHaveValue(marker);
|
||||
const reloadedStatusSelected = await readStatusSelection(page);
|
||||
const reloadedIds = await expectResults(page, tasks, [target.id]);
|
||||
const reloadedQuery = await search.inputValue();
|
||||
const targetLink = () => page.locator("main#main-content a[data-inbox-issue-link]")
|
||||
.filter({ hasText: target.title });
|
||||
stage = "open_result";
|
||||
await targetLink().click();
|
||||
await expect(page.getByRole("heading", { name: target.title, exact: true })).toBeVisible();
|
||||
const firstOpened = openedIdentifier(page, context);
|
||||
stage = "reload_result";
|
||||
await page.reload();
|
||||
await expect(page.getByRole("heading", { name: target.title, exact: true })).toBeVisible();
|
||||
await expect(page.getByText(target.description, { exact: true })).toBeVisible();
|
||||
|
||||
stage = "return_collection";
|
||||
await page.goBack();
|
||||
await expect(search).toHaveValue(marker);
|
||||
const returnedStatusSelected = await readStatusSelection(page);
|
||||
const returnedIds = await expectResults(page, tasks, [target.id]);
|
||||
const returnedQuery = await search.inputValue();
|
||||
stage = "reopen_result";
|
||||
await targetLink().click();
|
||||
await page.reload();
|
||||
await expect(page.getByRole("heading", { name: target.title, exact: true })).toBeVisible();
|
||||
await expect(page.getByText(target.description, { exact: true })).toBeVisible();
|
||||
stage = "verify_saved";
|
||||
const saved: SearchTask[] = [];
|
||||
const runs: unknown[][] = [];
|
||||
for (const task of tasks) {
|
||||
saved.push(await api.get<SearchTask>(`/api/issues/${task.id}`));
|
||||
runs.push(await api.get<unknown[]>(`/api/issues/${task.id}/runs`));
|
||||
}
|
||||
const checks = taskSearchEvidencePasses({ expected: tasks, saved, runs, marker, searchIds, filteredIds,
|
||||
reloadedIds, returnedIds, reloadedQuery, returnedQuery, reloadedStatusSelected, returnedStatusSelected,
|
||||
openedIdentifier: firstOpened, reopenedIdentifier: openedIdentifier(page, context),
|
||||
titleVisible: await page.getByRole("heading", { name: target.title, exact: true }).isVisible(),
|
||||
descriptionVisible: await page.getByText(target.description, { exact: true }).isVisible() });
|
||||
const entries = [
|
||||
{ id: "task-search-results-exact", passed: checks.search },
|
||||
{ id: "task-status-filter-excludes-controls", passed: checks.filter },
|
||||
{ id: "task-search-filter-reloaded", passed: checks.persistence },
|
||||
{ id: "task-search-result-reopened", passed: checks.reopen },
|
||||
{ id: "no-agent-work", passed: checks.noAgentWork },
|
||||
];
|
||||
evidence.checks.push(...entries);
|
||||
if (entries.some(check => !check.passed)) throw new JourneyError("failed", "task_search_evidence_mismatch");
|
||||
} catch (error) {
|
||||
if (error instanceof JourneyError) throw error;
|
||||
// Only a closed stage identifier survives; browser errors can contain
|
||||
// private URLs, form values, or response data.
|
||||
throw new JourneyError("failed", `task_search_${stage}_failed`);
|
||||
}
|
||||
}
|
||||
@@ -3,6 +3,7 @@ import type { ConnectionApi } from "../connection-target.js";
|
||||
|
||||
export type JourneyOutcome = "passed" | "failed" | "blocked" | "unsupported";
|
||||
export type JourneyId = "task-properties" | "task-create" | "ask-question" | "cloud-onboarding"
|
||||
| "task-search-filter"
|
||||
| "plan-revise-accept" | "build-revise" | "protected-actions" | "create-skill-studio" | "stop-redirect";
|
||||
export type TargetMode = "managed-self-hosted" | "existing-cloud" | "new-cloud";
|
||||
export interface DeploymentIdentity {
|
||||
|
||||
@@ -88,6 +88,7 @@ describe("trusted report catalog discovery", () => {
|
||||
const product = { ...known, suite: { ...known.suite, id: "product-journeys" } };
|
||||
expect(journeyCapabilities(product).map(capability => capability.id)).toEqual(["questions-and-approvals/questions"]);
|
||||
expect(journeyCapabilities({ ...product, task: { ...known.task, id: "task-create" } }).map(capability => capability.id)).toEqual(["tasks/create"]);
|
||||
expect(journeyCapabilities({ ...product, task: { ...known.task, id: "task-search-filter" } }).map(capability => capability.id)).toEqual(["tasks/organize"]);
|
||||
});
|
||||
|
||||
it("aggregates unknown suites and renders their cases, missing results, and screenshot gallery", () => {
|
||||
|
||||
@@ -434,7 +434,7 @@ export function validateRetainedRunnerResult(
|
||||
if (!evidence) return invalid("result.productJourney");
|
||||
const required = journeyRequiredChecks[checked.caseId];
|
||||
if (evidence.outcome !== "passed" || !evidence.deployment || !required || !required.every(id => evidence.checks.some(c => c.id === id && c.passed)) || evidence.checks.some(check => !check.passed) || evidence.phases.some(p => p.outcome !== "passed") || checked.cleanup !== "passed") invalid("result.productJourney");
|
||||
if (["task-properties", "task-create"].includes(checked.caseId) && (checked.runtimeMode !== "none" || checked.provider !== null || checked.model !== null || checked.runIds?.length)) invalid("result.browserOnly");
|
||||
if (["task-properties", "task-create", "task-search-filter"].includes(checked.caseId) && (checked.runtimeMode !== "none" || checked.provider !== null || checked.model !== null || checked.runIds?.length)) invalid("result.browserOnly");
|
||||
if (Object.hasOwn(agentJourneyRuns, checked.caseId) && (checked.runIds?.length !== agentJourneyRuns[checked.caseId as keyof typeof agentJourneyRuns]
|
||||
|| !checked.model || !checked.provider || checked.runtimeMode !== "native")) invalid("result.agentJourneyRun");
|
||||
if (checked.caseId === "cloud-onboarding" && (!checked.runIds?.length || !evidence.operations || !evidence.operations.requestCount || !evidence.operations.traceIds.length || evidence.operations.campaignId !== evidence.campaignId || evidence.operations.attemptId !== evidence.attemptId)) invalid("result.paidJourneyReceipt");
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
import { expect, test } from "@playwright/test";
|
||||
import { taskSearchBacklogCheckbox } from "./product-journeys/task-search-filter.js";
|
||||
|
||||
// Reproduce the observed Radix checkbox + StatusIcon + visible text label.
|
||||
// No app server, provider, browser profile, or target credentials are involved.
|
||||
for (const presentation of ["legacy", "spaced-legacy", "refreshed"] as const) {
|
||||
test(`task search selects Status/Backlog with ${presentation} labels`, async ({ page }) => {
|
||||
const explicit = presentation === "refreshed" ? 'aria-label="Backlog"' : "";
|
||||
const gap = presentation === "spaced-legacy" ? " " : "";
|
||||
await page.setContent(`<div role="dialog" aria-label="Filters">
|
||||
<button type="button">Backlog</button>
|
||||
<div><span>Status</span><div><label>
|
||||
<button id="status-backlog" type="button" role="checkbox" aria-checked="false" ${explicit}></button><svg role="img" aria-label="Backlog"><title>Backlog</title></svg>${gap}<span>Backlog</span>
|
||||
</label></div></div>
|
||||
<div><span>Responsible</span><label><input id="assignee-backlog" type="checkbox">Backlog</label></div>
|
||||
</div><script>
|
||||
document.getElementById('status-backlog').addEventListener('click', event => {
|
||||
const button = event.currentTarget;
|
||||
button.setAttribute('aria-checked', button.getAttribute('aria-checked') !== 'true' ? 'true' : 'false');
|
||||
});
|
||||
</script>`);
|
||||
const filters = page.getByRole("dialog", { name: "Filters", exact: true });
|
||||
const statusGroup = filters.getByText("Status", { exact: true }).locator("..");
|
||||
// The old exact-name lookup fails for the duplicated legacy label.
|
||||
await expect(statusGroup.getByRole("checkbox", { name: "Backlog", exact: true }))
|
||||
.toHaveCount(presentation === "refreshed" ? 1 : 0);
|
||||
const status = taskSearchBacklogCheckbox(filters);
|
||||
await expect(status).toHaveCount(1);
|
||||
await status.check();
|
||||
await expect(status).toBeChecked();
|
||||
await expect(page.locator("#assignee-backlog")).not.toBeChecked();
|
||||
});
|
||||
}
|
||||
Reference in new issue
Block a user