From a497bae83a128feae8206f27e4930e6407a67630 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 3 Nov 2026 06:24:08 -0600 Subject: [PATCH] test(e2e): verify Cloud workspace access isolation (#18655) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip keeps agent work inside company boundaries. > - Cloud users switch workspaces through the tenant account menu. > - A company-bound credential must work inside its permitted company. > - A missing resource cannot prove an access denial. > - This journey uses real controls in two registered test workspaces. > - It checks reads and task creation in both directions and revokes both keys. ## Linked Issues or Issue Description Refs: #18728, #18729, #18652. **What existing behavior does this improve?** Product E2E coverage for Cloud workspace selection and company-bound agent API access. **Current behavior** The journey catalog lacks a two-workspace boundary check with real allowed and denied controls. **Proposed behavior** Use the visible Cloud menu to switch workspaces and return. Read a real task and create one task in each credential's own company. Require real foreign reads and the same task-creation operation to return 401 or 403 in both directions. Verify that no foreign task was created and that protected content was not returned. Revoke both credentials and require their earlier allowed reads to return 401. ## What Changed - Add a browser journey and a narrow target-provider fixture contract. - Require independent owner reads, exact agent authorship, zero-to-one task creation, stable run history, no unexpected work and no foreign creation. - Validate observed workspace entry links against the exact registered targets. - Add wrong, missing, leaked, foreign and invalid-entry evidence controls. Saved passes reject collected runs or nonempty model usage. - Register all seven required checks, definition-hash inputs, feature associations. The feature-map documents are unchanged, as requested. - Base the final tree on verified repository master `7b8aa60b6d79b6cd9e4d46ca1369515c33e2d5d4` plus only 15 access journey and catalog files. Preserve all other master files and modes. The held completion journey is outside this PR's diff. ## Verification - 851 catalog, journey, completion-helper, artifact and browser-action support tests passed across 40 files. - E2E TypeScript, credential-free discovery, feature-map validation, full workspace recursive typecheck, full build and `git diff --check` passed. - The broad local `pnpm test:run` attempt stopped in chat integration shard 1 before any assertion. Embedded Postgres initialization exited with code null. No chat tests or later unit shards ran. The startup cause is unresolved, the failure is retained, and the command was not retried. The database/chat/test-runner source is unchanged from the fixed master snapshot. The separate hosted full CI result is recorded below. - The first local E2E typecheck read stale compiled plugin SDK declarations after the merge. The SDK source already declared the field. Rebuilding its declarations cleared the diagnostic without a source edit. - The complete tree comparison proves exactly 17 changed paths before the documentation removal against the fixed master snapshot. Every other master file and mode is preserved. All four access flow, contract and test blobs match the previously passing source `fab9a1fb2fefb41a7248e4fc1aa239d77455f5e2` byte for byte. - The historical staging attempt on source `fab9a1fb2fefb41a7248e4fc1aa239d77455f5e2`, the earlier Core42 deployment and the earlier adapter passed all seven checks: owner resource controls, visible Cloud switch and return, scoped data, both restricted-agent positive controls, cross-workspace read/task-creation denials, credential revocation and no agent work. Both credentials were revoked, owned credential agents had terminal receipts, and evidence cleanup was complete. No agent or model work was recorded. - The earlier attempt failed before journey checks while reading the borrowed fixture agent's historical details. That failed result remains preserved. The provider then supplied a separately owned primary identity. The public history assertions remain unchanged. - The strict root catalog expectation first failed because it omitted the new access case. Adding only that expected entry made the test pass. The ordered list and default-selection assertions remain unchanged. - The first combined-head full CI run failed two unchanged attention-retention tests at their original 15-second limits. The catalog fix does not change those tests or their timing. The initial failures are retained, their cause is unresolved, and that failed run was not retried. - Normal full CI passed on the new commit in [run 41595755161](https://github.com/paperclipai/paperclip/actions/runs/41595755161), attempt 1: 48 successful jobs and one intended skip. The two unchanged attention-retention cases passed at about 12 seconds; the earlier timeout cause is still unknown. The fresh review on commit `4c0ea682444738ad6fa17017a6afd25a49037559` is 5/5 with no open findings. The fresh staging result is recorded below. This PR changes no daily plan, private configuration, model or live workspace. - Fresh staging qualification on commit `4c0ea682444738ad6fa17017a6afd25a49037559` passed all seven checks and all five phases. Both ordinary same-workspace task creations returned 201. Both cross-workspace task creations returned 401. Both credential agents reached terminal state. Both keys were revoked, and both revoked-key reads returned 401. The manifest records complete cleanup and no active operations. No model or task run was recorded. All three workflow jobs passed. - The coordinator verified TestOps case 719 and result 3761 as passed, then closed launch 1987. Its report shows model none and the current qualification run. The automatic Slack report recorded one pass and zero failures. - The original normalized result was passed. The first evidence collector left its own qualification false because its allowlist omitted the standard `dashboard.html` artifact. The adapter already declared that file, and the public launcher generated it. An exact-object read matched the 118,791-byte object, ETag and SHA256 archive receipt. This completed the three-artifact inventory. The original result, collector proof and seal remain unchanged. No HTML was executed or rendered, and no test was rerun or regraded. | Fresh staging provenance | Verified value | | --- | --- | | Public commit | `4c0ea682444738ad6fa17017a6afd25a49037559` | | Qualification workflow | `41597774599`, attempt 1 | | Serving Core commit | `09162fe893b464dd2ccdafd14bd3ef1611202a86` | | Target adapter commit | `7367cf6fd0ee1cf52c2651a8a6671e087728a9c1` | | Actual workflow bootstrap commit | `55351ffba2704a4e13c1ead3e50665acd51aef87` | | Exact 117-file QA contract SHA256 | `14d95c29059c82563ab0e1d65a01aa4ac6c4f60f77a236803cf06ad5d9e7d27b` | | Provider-pack configuration version | `9` | - At the user’s request, follow-up commit `7e0dbc4bdba8a25ff6486648b0ca0d0db2616a4c` removes both feature-map additions. The final diff contains 15 paths and no `maps/feature-map/` changes. An entire-tree comparison against the QA-passed commit proves these two Markdown files are the only changes; every executable file, test, fixture, registration and file mode is unchanged. `node scripts/check-maps.mjs` and `git diff --check` pass after the removal. The live QA run above was not repeated for this documentation-only change. Fresh current-head full CI passed in [run 41633385961](https://github.com/paperclipai/paperclip/actions/runs/41633385961), attempt 1. The current-head Greptile review is 5/5 with zero unresolved threads. The normal merge preserves every newer master file outside the 15 PR paths and keeps all 15 PR files byte-for-byte intact. ## Risks The private target provider must register two exact test workspaces and preserve their separate ownership and cleanup. The case verifies company-bound agent credentials. It does not certify human membership roles. It tests task creation because ordinary agent task edits require a real execution context. No execution context is forged and no product authorization guard changes. The local broad unit run could not start embedded Postgres, so it does not establish a full local unit pass. The earlier hosted attention-retention timeouts also remain unexplained. Passing a later run would not prove the source of those timeouts. The four case files retain their historical passing bytes. The fresh qualification verifies the executable test code retained in this PR against the deployed staging stack. Daily admission remains a coordinator action after the normal source merge. The case does not cover future deployments without another run. ## Model Used OpenAI GPT-6 through Codex with reasoning and code execution. The runtime does not expose a more specific model identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip Co-authored-by: Codex --- tests/runner-e2e/PRODUCT-JOURNEYS.md | 1 + tests/runner-e2e/catalog.test.ts | 2 +- tests/runner-e2e/journey-features.ts | 1 + .../product-journeys/COMPANY-ACCESS.md | 53 ++++++++ tests/runner-e2e/product-journeys/catalog.ts | 1 + .../company-access-contract.test.ts | 81 ++++++++++++ .../company-access-contract.ts | 104 +++++++++++++++ .../company-access-flow.test.ts | 108 ++++++++++++++++ .../product-journeys/company-access-flow.ts | 120 ++++++++++++++++++ .../product-journeys/contracts.test.ts | 11 ++ tests/runner-e2e/product-journeys/fixtures.ts | 2 +- tests/runner-e2e/product-journeys/launch.ts | 6 +- .../product-journeys/success-contract.ts | 1 + tests/runner-e2e/product-journeys/types.ts | 3 +- tests/runner-e2e/result-validation.ts | 3 +- 15 files changed, 491 insertions(+), 6 deletions(-) create mode 100644 tests/runner-e2e/product-journeys/COMPANY-ACCESS.md create mode 100644 tests/runner-e2e/product-journeys/company-access-contract.test.ts create mode 100644 tests/runner-e2e/product-journeys/company-access-contract.ts create mode 100644 tests/runner-e2e/product-journeys/company-access-flow.test.ts create mode 100644 tests/runner-e2e/product-journeys/company-access-flow.ts diff --git a/tests/runner-e2e/PRODUCT-JOURNEYS.md b/tests/runner-e2e/PRODUCT-JOURNEYS.md index 74463320bd9..42d8c6cde70 100644 --- a/tests/runner-e2e/PRODUCT-JOURNEYS.md +++ b/tests/runner-e2e/PRODUCT-JOURNEYS.md @@ -25,6 +25,7 @@ never start a server or depend on a tenant filesystem. | --- | --- | --- | | `repository-continuation` | Native agent; two work runs; a pinned public read-only repository | Verify independent Git bundle history and workspace bytes across two turns, preserve the original random file, and reopen registered outputs after reload. See [repository-continuation.md](product-journeys/repository-continuation.md) | | `routine-scheduled-run` | Native agent; one work run; an owned project and routine | Browser-create a routine, enable its minute schedule, verify one scheduled invocation and run-attributed saved output, then pause and observe the next schedule boundary with no new work | +| `company-access-isolation` | Two registered QA Cloud workspaces; disposable company-bound agent credentials; zero model runs | Switch through the real tenant Cloud menu and return. Require allowed task reads and task creation, foreign read/create denial in both directions, exact canonical creator evidence, and key revocation. See [COMPANY-ACCESS.md](product-journeys/COMPANY-ACCESS.md) | | `file-input-answer` | Native agent; one work run; synthetic CSV uploaded through the composer | Verify original uploaded bytes, independently calculated accepted-row count and units, a run-attributed task document after reload, and preservation of the input attachment. The task prompt contains neither the random file reference nor the answers | | `inbox-triage` | Signed-in board user; a separately authenticated QA human author; a fresh self-assigned task; zero model runs | Mark an unread task read, archive it through Mine, reload both views, verify unchanged work status, then observe a real Blocked transition resurface it through the normal archive policy | | `task-create` | Company and signed-in board user | Create a task through the shared production composer, assign it to yourself, reopen and reload it, verify the saved prompt and no agent work | diff --git a/tests/runner-e2e/catalog.test.ts b/tests/runner-e2e/catalog.test.ts index 13989a0efe5..f171e3c5b00 100644 --- a/tests/runner-e2e/catalog.test.ts +++ b/tests/runner-e2e/catalog.test.ts @@ -126,7 +126,7 @@ describe("runner E2E catalog", () => { }); it("exports deployment journeys separately from the model matrix and default selection", () => { - expect(productJourneys.map(journey => journey.id)).toEqual(["repository-continuation", "routine-scheduled-run", "inbox-triage", "file-input-answer", "task-create", "project-task-context", "ask-question", "question-answer-resume", "task-properties", "task-document-persistence", "task-search-filter", "plan-revise-accept", "build-revise", "protected-actions", "create-skill-studio", "stop-redirect", "followup-while-running", "hire-delegate-reuse", "cloud-onboarding"]); + expect(productJourneys.map(journey => journey.id)).toEqual(["company-access-isolation", "repository-continuation", "routine-scheduled-run", "inbox-triage", "file-input-answer", "task-create", "project-task-context", "ask-question", "question-answer-resume", "task-properties", "task-document-persistence", "task-search-filter", "plan-revise-accept", "build-revise", "protected-actions", "create-skill-studio", "stop-redirect", "followup-while-running", "hire-delegate-reuse", "cloud-onboarding"]); expect(runnerSuites.some(suite => suite.id === "product-journeys")).toBe(false); expect(runnerMatrix.some(execution => execution.suite.id === "product-journeys")).toBe(false); expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(execution => execution.suite.id === "product-journeys")).toBe(false); diff --git a/tests/runner-e2e/journey-features.ts b/tests/runner-e2e/journey-features.ts index 7da187c149e..962117f2ab1 100644 --- a/tests/runner-e2e/journey-features.ts +++ b/tests/runner-e2e/journey-features.ts @@ -19,6 +19,7 @@ function associate(suites: readonly string[], cases: Record { + it("accepts only an exact observed entry for the registered workspace", () => { + const portal = "https://portal.example.test"; + for (const path of ["/stacks/two/enter", "/orgs/qa-two/enter", "/v1/stacks/two/entry-redirect", "/v1/stacks/qa-two/entry-redirect?by=id"]) + expect(isolationEntryMatches(portal + path, portal, secondary)).toBe(true); + for (const href of [null, portal + "/stacks/one/enter", portal + "/v1/stacks/qa-two/entry-redirect", portal + "/v1/stacks/qa-two/entry-redirect?by=id&extra=1", "https://other.example.test/stacks/two/enter", portal + "/stacks/two/enter#fragment"]) + expect(isolationEntryMatches(href, portal, secondary)).toBe(false); + }); + it("requires a unique new canonical task with real agent creation attribution", () => { + const expected = { companyId: primary.companyId, title: "Owned create", description: "Fresh body", agentId: "agent-one" }; + const created = { ...expected, id: "new-task", identifier: "ONE-2", createdByAgentId: expected.agentId, createdByUserId: null, + status: "backlog", assigneeAgentId: null, assigneeUserId: null }; + const input = { response: { status: 201, contentLeaked: false, task: created }, expected, before: [], after: [created], canonical: created }; + expect(restrictedCreateAllowed(input)).toBe(true); + for (const patch of [{ createdByAgentId: "other" }, { createdByUserId: "human" }, { companyId: secondary.companyId }, { status: "in_progress" }, { assigneeAgentId: "agent-one" }, { description: "wrong" }]) + expect(restrictedCreateAllowed({ ...input, canonical: { ...created, ...patch } })).toBe(false); + expect(restrictedCreateAllowed({ ...input, canonical: null })).toBe(false); + expect(restrictedCreateAllowed({ ...input, before: [created] })).toBe(false); + expect(restrictedCreateAllowed({ ...input, after: [created, created] })).toBe(false); + expect(restrictedCreateAllowed({ ...input, response: { ...input.response, status: 403 } })).toBe(false); + }); + it("requires two different real testing workspaces", () => { + expect(workspacePairMatches(primary, secondary)).toBe(true); + for (const patch of [{ stackId: primary.stackId }, { companyId: primary.companyId }, { origin: primary.origin }, { origin: "http://two.example.test" }, { origin: "https://two.example.test/path" }, { origin: "https://user:password@two.example.test" }, { issuePrefix: "" }]) expect(workspacePairMatches(primary, { ...secondary, ...patch })).toBe(false); + }); + it("accepts only exact owner readback", () => { + expect(isolationTaskMatches(task, task as IsolationTask & { description: string })).toBe(true); + expect(isolationTaskMatches(undefined, task as IsolationTask & { description: string })).toBe(false); + for (const patch of [{ id: "different" }, { companyId: primary.companyId }, { description: "changed" }]) expect(isolationTaskMatches({ ...task, ...patch }, task as IsolationTask & { description: string })).toBe(false); + }); + it("requires an authenticated successful primary read and write control", () => { + const expected = { id: "task-one", companyId: primary.companyId, description: "written by restricted agent" }; + expect(restrictedPositiveControl({ status: 200, contentLeaked: false, task: expected }, expected)).toBe(true); + expect(restrictedPositiveControl({ status: 403, contentLeaked: false, task: null }, expected)).toBe(false); + expect(restrictedPositiveControl({ status: 200, contentLeaked: false, task: { ...expected, companyId: secondary.companyId } }, expected)).toBe(false); + expect(restrictedPositiveControl({ status: 200, contentLeaked: false, task: { ...expected, description: "old" } }, expected)).toBe(false); + }); + it("requires both foreign denials and unchanged owner state", () => { + expect(restrictedForeignDenied(denied, { status: 401, contentLeaked: false, task: null }, task, { ...task })).toBe(true); + for (const status of [0, 200, 302, 404, 500]) expect(restrictedForeignDenied({ status, contentLeaked: false, task: null }, denied, task, task)).toBe(false); + expect(restrictedForeignDenied(denied, { status: 200, contentLeaked: false, task }, task, task)).toBe(false); + expect(restrictedForeignDenied(denied, denied, task, { ...task, description: "unauthorized change" })).toBe(false); + expect(restrictedForeignDenied({ ...denied, contentLeaked: true }, denied, task, task)).toBe(false); + expect(restrictedForeignDenied({ status: 403, contentLeaked: false, task }, denied, task, task)).toBe(false); + }); +}); + +it("accepts only a complete, bound, settled actor run list", () => { + const actor = { id: "actor", companyId: "company" }; + const row = { id: "past", agentId: actor.id, companyId: actor.companyId, status: "succeeded" }; + expect(isolationAgentRunIds([row], actor)).toEqual(["past"]); + expect(isolationAgentRunIds([], actor)).toEqual([]); + for (const patch of [{ agentId: "other" }, { companyId: "other" }, { status: "running" }, { nativePhase: "retryable_failure" }, { runId: "different" }]) + expect(isolationAgentRunIds([{ ...row, ...patch }], actor)).toBeNull(); + expect(isolationAgentRunIds([row, row], actor)).toBeNull(); + expect(isolationAgentRunIds(Array.from({ length: 100 }, (_, id) => ({ ...row, id: String(id) })), actor)).toBeNull(); + expect(isolationAgentRunIds(null, actor)).toBeNull(); +}); + +it("requires detailed terminal native state and detects same-ID revisions", () => { + const actor = { id: "actor", companyId: "company" }, timestamp = "2026-11-01T12:00:00.000Z"; + const row = { id: "past", agentId: actor.id, companyId: actor.companyId, status: "succeeded", runtimeMode: "native", + nativePhase: "committed", nativePhaseUpdatedAt: timestamp, updatedAt: timestamp, startedAt: timestamp, finishedAt: timestamp }; + const before = isolationAgentRunSnapshot([row], actor); + expect(before).toHaveLength(1); + expect(isolationAgentRunSnapshot([], actor)).toEqual([]); + expect(isolationAgentRunSnapshot([{ ...row, runtimeMode: "legacy", nativePhase: null, nativePhaseUpdatedAt: null }], actor)).toHaveLength(1); + for (const patch of [{ nativePhase: "retryable_failure" }, { nativePhase: "result_accepted" }, { nativePhase: undefined }, + { nativePhaseUpdatedAt: undefined }, { runtimeMode: undefined }, { updatedAt: undefined }, { finishedAt: null }, { updatedAt: "invalid" }, { companyId: "other" }]) + expect(isolationAgentRunSnapshot([{ ...row, ...patch }], actor)).toBeNull(); + for (const field of ["nativePhaseUpdatedAt", "updatedAt", "startedAt", "finishedAt"]) + expect(isolationAgentRunSnapshot([{ ...row, [field]: "2026-11-02T12:00:00.000Z" }], actor)).not.toEqual(before); + expect(isolationAgentRunSnapshot(null, actor)).toBeNull(); +}); diff --git a/tests/runner-e2e/product-journeys/company-access-contract.ts b/tests/runner-e2e/product-journeys/company-access-contract.ts new file mode 100644 index 00000000000..12d54439e73 --- /dev/null +++ b/tests/runner-e2e/product-journeys/company-access-contract.ts @@ -0,0 +1,104 @@ +import type { ConnectionApi } from "../connection-target.js"; + +export interface IsolationTask { id: string; companyId: string; title: string; description: string | null; identifier: string } +export interface IsolationWorkspace { stackId: string; stackSlug: string; companyId: string; origin: string; displayName: string; issuePrefix: string } +export interface CreatedIsolationTask extends IsolationTask { + createdByAgentId: string | null; createdByUserId: string | null; status: string; + assigneeAgentId: string | null; assigneeUserId: string | null; +} +export interface IsolationCreateResponse { status: number; contentLeaked: boolean; task: CreatedIsolationTask | null } +export interface IsolationResponse { status: number; contentLeaked: boolean; task: Pick | null } +/** Authentication and exact secondary fixture ownership stay with the target provider. */ +export interface CompanyAccessFixture { + portalOrigin: string; + primary: IsolationWorkspace; + secondary: IsolationWorkspace; + secondaryApi: ConnectionApi; + restrictedActors: { primary: { kind: "agent"; id: string; companyId: string }; secondary: { kind: "agent"; id: string; companyId: string } }; + createSecondaryTask(title: string, description: string): Promise; + restrictedRequest(actor: "primary" | "secondary", scope: "primary" | "secondary", method: "GET", issueId: string): Promise; + restrictedCreate(actor: "primary" | "secondary", scope: "primary" | "secondary", title: string, description: string): Promise; + revokeCredentials(): Promise<{ revoked: boolean; deniedStatuses: { primary: number; secondary: number } }>; +} + +function validOrigin(value: string): boolean { + try { const url = new URL(value); return url.protocol === "https:" && url.origin === value && !url.username && !url.password; } + catch { return false; } +} +export function workspacePairMatches(primary: IsolationWorkspace, secondary: IsolationWorkspace): boolean { + return Boolean(primary.stackId && secondary.stackId && primary.stackSlug && secondary.stackSlug && primary.companyId && secondary.companyId && primary.issuePrefix && secondary.issuePrefix + && primary.displayName && secondary.displayName && primary.stackId !== secondary.stackId && primary.companyId !== secondary.companyId + && primary.origin !== secondary.origin && [primary.origin, secondary.origin].every(validOrigin)); +} +export function isolationTaskMatches(task: IsolationTask | null | undefined, expected: { id: string; companyId: string; title: string; description: string }): boolean { + return Boolean(task && task.id === expected.id && task.companyId === expected.companyId && task.title === expected.title && task.description === expected.description); +} +export function restrictedPositiveControl(response: IsolationResponse, expected: { id: string; companyId: string; description: string }): boolean { + return response.status === 200 && response.task?.id === expected.id && response.task.companyId === expected.companyId && response.task.description === expected.description; +} +/** Reject 404: owner controls must prove the requested foreign resource exists. */ +export function restrictedForeignDenied(read: IsolationResponse, write: IsolationResponse, before: IsolationTask, after: IsolationTask): boolean { + return [read, write].every(response => [401, 403].includes(response.status) && response.task === null && response.contentLeaked === false) + && before.id === after.id && before.companyId === after.companyId && before.title === after.title && before.description === after.description; +} + +/** Accept only shipped entry forms for the exact receipted workspace. */ +export function isolationEntryMatches(href: string | null, portalOrigin: string, workspace: IsolationWorkspace): boolean { + if (!href || !validOrigin(portalOrigin)) return false; + try { + const url = new URL(href, portalOrigin); + if (url.origin !== portalOrigin || url.username || url.password || url.hash) return false; + const slug = encodeURIComponent(workspace.stackSlug), id = encodeURIComponent(workspace.stackId); + return new Set([ + `${portalOrigin}/stacks/${slug}/enter`, `${portalOrigin}/orgs/${id}/enter`, + `${portalOrigin}/v1/stacks/${slug}/entry-redirect`, `${portalOrigin}/v1/stacks/${id}/entry-redirect?by=id`, + ]).has(url.href); + } catch { return false; } +} + +/** A successful write is a new canonical agent-created task, not an agent edit without a run. */ +export function restrictedCreateAllowed(input: { + response: IsolationCreateResponse; expected: { companyId: string; title: string; description: string; agentId: string }; + before: IsolationTask[]; after: IsolationTask[]; canonical: CreatedIsolationTask | null; +}): boolean { + const { response, expected, before, after, canonical } = input; + if (response.status !== 201 || response.contentLeaked !== false || !response.task?.id || before.length !== 0 || after.length !== 1) return false; + const correct = (task: CreatedIsolationTask | null) => Boolean(task && task.id === response.task!.id + && isolationTaskMatches(task, { ...expected, id: response.task!.id }) && task.createdByAgentId === expected.agentId + && task.createdByUserId === null && task.status === "backlog" && task.assigneeAgentId === null && task.assigneeUserId === null); + return correct(response.task) && correct(canonical) && after[0]?.id === response.task.id + && after[0].companyId === expected.companyId && after[0].title === expected.title; +} + +/** Validate the bounded list projection, which does not include native retry state. */ +export function isolationAgentRunIds(value: unknown, actor: { id: string; companyId: string }): string[] | null { + if (!Array.isArray(value) || value.length >= 100) return null; + const ids = new Set(); + for (const row of value) { + if (!row || typeof row !== "object" || Array.isArray(row)) return null; + const id = row.runId ?? row.id; + if (typeof id !== "string" || !/^[A-Za-z0-9][A-Za-z0-9_.:-]{0,255}$/.test(id) || (row.id != null && row.id !== id || row.runId != null && row.runId !== id) + || ids.has(id) || row.companyId !== actor.companyId || row.agentId !== actor.id + || !["succeeded", "failed", "cancelled", "timed_out"].includes(row.status) || row.nativePhase === "retryable_failure") return null; + ids.add(id); + } + return [...ids].sort(); +} + +/** Detailed terminal state and its persisted revision must stay unchanged. */ +export function isolationAgentRunSnapshot(value: unknown, actor: { id: string; companyId: string }): string[] | null { + const ids = isolationAgentRunIds(value, actor); + if (!ids) return null; + const timestamp = (field: unknown) => typeof field === "string" && Number.isFinite(Date.parse(field)) && new Date(field).toISOString() === field; + const states = new Map(); + for (const row of value as Record[]) { + if (typeof row.id !== "string" || !timestamp(row.updatedAt) || !timestamp(row.finishedAt) + || row.startedAt !== null && !timestamp(row.startedAt)) return null; + if (row.runtimeMode === "native") { + if (!["committed", "terminal_failure"].includes(row.nativePhase as string) || !timestamp(row.nativePhaseUpdatedAt)) return null; + } else if (row.runtimeMode !== "legacy" || row.nativePhase !== null || row.nativePhaseUpdatedAt !== null) return null; + states.set(row.id, JSON.stringify([row.id, row.status, row.runtimeMode, row.nativePhase, row.nativePhaseUpdatedAt, + row.startedAt, row.finishedAt, row.updatedAt])); + } + return ids.map(id => states.get(id)!); +} diff --git a/tests/runner-e2e/product-journeys/company-access-flow.test.ts b/tests/runner-e2e/product-journeys/company-access-flow.test.ts new file mode 100644 index 00000000000..4b52f09b351 --- /dev/null +++ b/tests/runner-e2e/product-journeys/company-access-flow.test.ts @@ -0,0 +1,108 @@ +import { describe, expect, it, vi } from "vitest"; +import { companyAccessIsolation } from "./company-access-flow.js"; +import type { CompanyAccessFixture, CreatedIsolationTask, IsolationWorkspace } from "./company-access-contract.js"; +import type { JourneyContext, JourneyEvidence, TargetSession } from "./types.js"; +vi.mock("@playwright/test", () => ({ expect: Object.assign( + (locator: { visible: boolean; text?: string }) => ({ + async toBeVisible() { expect(locator.visible).toBe(true); }, + async toContainText(value: string) { expect(locator.text).toContain(value); }, + }), { poll: (read: () => Promise) => ({ async toBe(value: unknown) { expect(await read()).toBe(value); } }) }, +) })); +function system(options: { newPrimaryRun?: boolean; retryableHistorical?: boolean; changedHistorical?: boolean; detailUnavailable?: boolean } = {}) { + const primary: IsolationWorkspace = { stackId: "one", stackSlug: "one", companyId: "company-one", origin: "https://one.example.test", displayName: "One", issuePrefix: "ONE" }; + const secondary: IsolationWorkspace = { stackId: "two", stackSlug: "two", companyId: "company-two", origin: "https://two.example.test", displayName: "Two", issuePrefix: "TWO" }; + const workspaces = { primary, secondary }, actors = { primary: { kind: "agent" as const, id: "agent-one", companyId: primary.companyId }, secondary: { kind: "agent" as const, id: "agent-two", companyId: secondary.companyId } }; + const tasks = new Map(); let next = 0, current = new URL(primary.origin), menu = false, created = false; + const task = (scope: "primary" | "secondary", title: string, description: string, agentId: string | null) => { + const value: CreatedIsolationTask = { id: `task-${++next}`, companyId: workspaces[scope].companyId, title, description, identifier: `${workspaces[scope].issuePrefix}-${next}`, + createdByAgentId: agentId, createdByUserId: agentId ? null : "owner", status: "backlog", assigneeAgentId: null, assigneeUserId: null }; + tasks.set(value.id, value); return structuredClone(value); + }; + const historical = { id: "historical", companyId: primary.companyId, agentId: actors.primary.id, + status: options.retryableHistorical ? "failed" : "succeeded", runtimeMode: "native", + nativePhase: options.retryableHistorical ? "retryable_failure" : "committed", + nativePhaseUpdatedAt: "2026-11-01T12:00:00.000Z", updatedAt: "2026-11-01T12:00:00.000Z", + startedAt: "2026-11-01T11:00:00.000Z", finishedAt: "2026-11-01T12:00:00.000Z" }; + // The real list projection omits runtimeMode, nativePhase and nativePhaseUpdatedAt. + const listed = (id = historical.id) => { + const { runtimeMode: _runtime, nativePhase: _phase, nativePhaseUpdatedAt: _phaseAt, ...row } = historical; + return { ...row, id }; + }; + const apis = Object.fromEntries((["primary", "secondary"] as const).map(scope => [scope, { + origin: workspaces[scope].origin, + async post(_route: string, body: { title: string; description: string }) { return task(scope, body.title, body.description, null); }, + async get(route: string) { + const url = new URL(route, workspaces[scope].origin); + if (url.pathname === "/api/companies") return [{ id: workspaces[scope].companyId }]; + if (url.pathname.endsWith("/heartbeat-runs")) return scope === "secondary" ? [] : [listed(), ...(created && options.newPrimaryRun ? [listed("new-outside-task")] : [])]; + if (url.pathname.startsWith("/api/heartbeat-runs/")) { + if (options.detailUnavailable) throw new Error("detail_unavailable"); + const id = url.pathname.split("/").at(-1)!; + return { ...historical, id, ...(created && options.changedHistorical ? { + nativePhaseUpdatedAt: "2026-11-02T12:00:00.000Z", updatedAt: "2026-11-02T12:00:00.000Z", finishedAt: "2026-11-02T12:00:00.000Z", + } : {}) }; + } + if (url.pathname.endsWith("/issues")) return [...tasks.values()].filter(row => row.companyId === workspaces[scope].companyId && row.title.includes(url.searchParams.get("q") ?? "")); + if (url.pathname.endsWith("/runs")) return []; + const row = tasks.get(url.pathname.split("/").at(-1)!); + if (!row || row.companyId !== workspaces[scope].companyId) throw new Error("wrong_fixture_scope"); + return structuredClone(row); + }, + }])) as unknown as { primary: JourneyContext["api"]; secondary: JourneyContext["api"] }; + const page = { + async goto(url: string) { current = new URL(url); }, url: () => current.href, async reload() {}, + async waitForURL(accept: (url: URL) => boolean) { expect(accept(current)).toBe(true); }, + getByRole(role: string, options: { name: string }) { + const workspace = current.origin === primary.origin ? primary : secondary; + return { visible: role === "button" ? options.name === `Open ${workspace.displayName} organization switcher` + : [...tasks.values()].some(row => row.companyId === workspace.companyId && row.title === options.name), async click() { menu = true; } }; + }, + locator() { return { async all() { return !menu ? [] : Object.values(workspaces).map(workspace => ({ visible: true, text: workspace.displayName, + async isVisible() { return true; }, async getAttribute() { return `https://portal.example.test/stacks/${workspace.stackSlug}/enter`; }, + async click() { current = new URL(workspace.origin); menu = false; }, + })); } }; }, + } as unknown as JourneyContext["page"]; + const fixture: CompanyAccessFixture = { + portalOrigin: "https://portal.example.test", primary, secondary, secondaryApi: apis.secondary, restrictedActors: actors, + async createSecondaryTask(title, description) { return task("secondary", title, description, null); }, + async restrictedRequest(actor, scope, _method, issueId) { + if (actor !== scope) return { status: 403, contentLeaked: false, task: null }; + const row = tasks.get(issueId)!; return { status: 200, contentLeaked: false, task: { id: row.id, companyId: row.companyId, description: row.description } }; + }, + async restrictedCreate(actor, scope, title, description) { + if (actor !== scope) return { status: 403, contentLeaked: false, task: null }; + created = true; return { status: 201, contentLeaked: false, task: task(scope, title, description, actors[actor].id) }; + }, + async revokeCredentials() { return { revoked: true, deniedStatuses: { primary: 401, secondary: 401 } }; }, + }; + const context = { page, api: apis.primary, company: { id: primary.companyId, issuePrefix: primary.issuePrefix }, fixtures: { companyAccess: fixture } } as JourneyContext; + const evidence = { checks: [] } as unknown as JourneyEvidence; + const session = { async own() {} } as unknown as TargetSession; + return { context, evidence, session }; +} +describe("isolation agent work observation", () => { + it("rejects a new primary run outside all four owned tasks", async () => { + const f = system({ newPrimaryRun: true }); + await expect(companyAccessIsolation(f.context, f.session, "journey-123-abcd1234", f.evidence)).rejects.toThrow("no_agent_work"); + expect(f.evidence.checks).toContainEqual({ id: "no-agent-work", passed: false }); + }); + it("accepts unchanged historical primary runs and zero work on owned tasks", async () => { + const f = system(); await companyAccessIsolation(f.context, f.session, "journey-123-abcd1234", f.evidence); + expect(f.evidence.checks).toContainEqual({ id: "no-agent-work", passed: true }); + }); + it("blocks retryable native history even when its terminal list row omits native phase", async () => { + const f = system({ retryableHistorical: true }); + await expect(companyAccessIsolation(f.context, f.session, "journey-123-abcd1234", f.evidence)).rejects.toThrow("isolation_primary_run_baseline_unproven"); + expect(f.evidence.checks).toEqual([]); + }); + it("rejects resumed work that finishes under the same historical run ID", async () => { + const f = system({ changedHistorical: true }); + await expect(companyAccessIsolation(f.context, f.session, "journey-123-abcd1234", f.evidence)).rejects.toThrow("no_agent_work"); + expect(f.evidence.checks).toContainEqual({ id: "no-agent-work", passed: false }); + }); + it("cannot qualify without the historical run detail", async () => { + const f = system({ detailUnavailable: true }); + await expect(companyAccessIsolation(f.context, f.session, "journey-123-abcd1234", f.evidence)).rejects.toThrow(); + expect(f.evidence.checks).toEqual([]); + }); +}); diff --git a/tests/runner-e2e/product-journeys/company-access-flow.ts b/tests/runner-e2e/product-journeys/company-access-flow.ts new file mode 100644 index 00000000000..308663282e0 --- /dev/null +++ b/tests/runner-e2e/product-journeys/company-access-flow.ts @@ -0,0 +1,120 @@ +import { expect } from "@playwright/test"; +import { isolationAgentRunIds, isolationAgentRunSnapshot, isolationEntryMatches, isolationTaskMatches, restrictedCreateAllowed, restrictedForeignDenied, restrictedPositiveControl, workspacePairMatches, type CreatedIsolationTask, type IsolationTask, type IsolationWorkspace } from "./company-access-contract.js"; +import { JourneyError, type JourneyContext, type JourneyEvidence, type TargetSession } from "./types.js"; + +/** Real Cloud workspace switching plus a company-scoped agent credential boundary. */ +export async function companyAccessIsolation(context: JourneyContext, session: TargetSession, campaignId: string, evidence: JourneyEvidence) { + const fixture = context.fixtures?.companyAccess; + if (!fixture || !workspacePairMatches(fixture.primary, fixture.secondary) || fixture.primary.companyId !== context.company.id + || fixture.primary.origin !== context.api.origin || fixture.restrictedActors.primary.kind !== "agent" + || fixture.restrictedActors.primary.companyId !== context.company.id || !fixture.restrictedActors.primary.id + || fixture.restrictedActors.secondary.kind !== "agent" || fixture.restrictedActors.secondary.companyId !== fixture.secondary.companyId + || !fixture.restrictedActors.secondary.id) throw new JourneyError("unsupported", "company_access_fixture_required"); + const { page, api, company } = context; + const primaryHistory = async () => { + const ids = isolationAgentRunIds(await api.get(`/api/companies/${company.id}/heartbeat-runs?agentId=${fixture.restrictedActors.primary.id}&limit=100`), fixture.restrictedActors.primary); + if (!ids) return null; + const details = await Promise.all(ids.map(id => api.get(`/api/heartbeat-runs/${encodeURIComponent(id)}`))); + const detailIds = isolationAgentRunIds(details, fixture.restrictedActors.primary); + if (!detailIds || ids.length !== detailIds.length || ids.some((id, index) => id !== detailIds[index])) return null; + return isolationAgentRunSnapshot(details, fixture.restrictedActors.primary); + }; + const primaryBefore = await primaryHistory(); + if (!primaryBefore) throw new JourneyError("blocked", "isolation_primary_run_baseline_unproven"); + const homeTitle = `Journey QA isolation primary ${campaignId}`, foreignTitle = `Journey QA isolation secondary ${campaignId}`; + const original = `Owned isolation sentinel ${campaignId}`; + // These tasks have no assignee, so neither setup nor the write control starts agent work. + await session.own({ kind: "issue", name: homeTitle }); + const home = await api.post(`/api/companies/${company.id}/issues`, { title: homeTitle, description: original, status: "backlog" }); + await session.own({ kind: "issue", name: homeTitle, id: home.id }); + const foreign = await fixture.createSecondaryTask(foreignTitle, original); + const check = (id: string, passed: boolean) => { + evidence.checks.push({ id, passed }); + if (!passed) throw new JourneyError("failed", id.replaceAll("-", "_")); + }; + const readHome = () => api.get(`/api/issues/${home.id}`); + const readForeign = () => fixture.secondaryApi.get(`/api/issues/${foreign.id}`); + const homeExpected = { id: home.id, companyId: company.id, title: homeTitle, description: original }; + const foreignExpected = { id: foreign.id, companyId: fixture.secondary.companyId, title: foreignTitle, description: original }; + check("isolation-owner-resource-controls", isolationTaskMatches(await readHome(), homeExpected) && isolationTaskMatches(await readForeign(), foreignExpected)); + + const switchTo = async (from: IsolationWorkspace, to: IsolationWorkspace, task: IsolationTask) => { + const switcher = page.getByRole("button", { name: `Open ${from.displayName} organization switcher`, exact: true }); + await expect(switcher).toBeVisible({ timeout: 30_000 }); + await switcher.click(); + const entries = async () => { + const matches = []; + for (const item of await page.locator('a[role="menuitem"]').all()) { + if (await item.isVisible() && isolationEntryMatches(await item.getAttribute("href"), fixture.portalOrigin, to)) matches.push(item); + } + return matches; + }; + await expect.poll(async () => (await entries()).length, { timeout: 30_000 }).toBe(1); + const matches = await entries(); + if (matches.length !== 1) throw new JourneyError("failed", "isolation_workspace_entry_unverified"); + await expect(matches[0]!).toContainText(to.displayName); + await matches[0]!.click(); + await page.waitForURL(url => url.origin === to.origin && !url.pathname.startsWith("/cloud/")); + await page.goto(`${to.origin}/${to.issuePrefix}/issues/${task.identifier}`, { waitUntil: "domcontentloaded" }); + await expect(page.getByRole("heading", { name: task.title, exact: true })).toBeVisible(); + await page.reload(); + await expect(page.getByRole("heading", { name: task.title, exact: true })).toBeVisible(); + }; + await switchTo(fixture.primary, fixture.secondary, foreign); + const secondaryCompanies = await fixture.secondaryApi.get>("/api/companies"); + await switchTo(fixture.secondary, fixture.primary, home); + const primaryCompanies = await api.get>("/api/companies"); + check("cloud-workspace-switch-and-return", new URL(page.url()).origin === api.origin); + check("workspace-data-scoped", primaryCompanies.length === 1 && primaryCompanies[0]?.id === company.id + && secondaryCompanies.length === 1 && secondaryCompanies[0]?.id === fixture.secondary.companyId + && isolationTaskMatches(await readHome(), homeExpected) && isolationTaskMatches(await readForeign(), foreignExpected)); + + const list = async (scope: "primary" | "secondary", title: string) => { + const selected = scope === "primary" ? api : fixture.secondaryApi; + const companyId = scope === "primary" ? company.id : fixture.secondary.companyId; + const rows = await selected.get(`/api/companies/${companyId}/issues?q=${encodeURIComponent(title)}&limit=100&offset=0&includeAgentCreatedTasks=true`); + if (!Array.isArray(rows) || rows.length >= 100) throw new JourneyError("failed", "isolation_task_listing_unproven"); + return rows.filter(row => row.title === title); + }; + const created: CreatedIsolationTask[] = []; + const positive = async (scope: "primary" | "secondary", task: IsolationTask) => { + const expected = scope === "primary" ? homeExpected : foreignExpected; + const read = await fixture.restrictedRequest(scope, scope, "GET", task.id); + const title = `Journey QA isolation ${scope} creation ${campaignId}`; + const description = `QA isolation owned create ${scope} ${campaignId}`; + const before = await list(scope, title); + const response = await fixture.restrictedCreate(scope, scope, title, description); + const after = await list(scope, title); + const selected = scope === "primary" ? api : fixture.secondaryApi; + const canonical = response.status === 201 && response.task?.id + ? await selected.get(`/api/issues/${response.task.id}`) : null; + if (canonical) created.push(canonical); + return restrictedPositiveControl(read, expected) && restrictedCreateAllowed({ response, before, after, canonical, + expected: { companyId: expected.companyId, title, description, agentId: fixture.restrictedActors[scope].id } }); + }; + check("restricted-agent-positive-controls", await positive("primary", home) && await positive("secondary", foreign)); + const deny = async (actor: "primary" | "secondary", scope: "primary" | "secondary", task: IsolationTask) => { + const selected = scope === "primary" ? api : fixture.secondaryApi; + const before = await selected.get(`/api/issues/${task.id}`); + // This attempted input differs from all protected foreign sentinel content. + const title = `Journey QA isolation ${scope} denied ${campaignId}`; + const description = `QA isolation denied create ${scope} ${campaignId}`; + const beforeCreates = await list(scope, title); + const deniedRead = await fixture.restrictedRequest(actor, scope, "GET", task.id); + const deniedWrite = await fixture.restrictedCreate(actor, scope, title, description); + const afterCreates = await list(scope, title); + return beforeCreates.length === 0 && afterCreates.length === 0 + && restrictedForeignDenied(deniedRead, deniedWrite, before, await selected.get(`/api/issues/${task.id}`)); + }; + check("cross-workspace-agent-read-write-denied", await deny("primary", "secondary", foreign) && await deny("secondary", "primary", home)); + const revoked = await fixture.revokeCredentials(); + check("isolation-credential-revoked", revoked.revoked && revoked.deniedStatuses.primary === 401 && revoked.deniedStatuses.secondary === 401); + const createdRuns = await Promise.all(created.map(task => (task.companyId === company.id ? api : fixture.secondaryApi).get(`/api/issues/${task.id}/runs`))); + const primaryAfter = await primaryHistory(); + const primaryUnchanged = primaryAfter !== null && primaryBefore.length === primaryAfter.length && primaryBefore.every((id, index) => id === primaryAfter[index]); + check("no-agent-work", primaryUnchanged && createdRuns.every(runs => runs.length === 0) && (await api.get(`/api/issues/${home.id}/runs`)).length === 0 + && (await fixture.secondaryApi.get(`/api/issues/${foreign.id}/runs`)).length === 0 + && (await fixture.secondaryApi.get(`/api/companies/${fixture.secondary.companyId}/heartbeat-runs?agentId=${fixture.restrictedActors.secondary.id}&limit=100`)).length === 0); + await page.reload(); + await expect(page.getByRole("heading", { name: homeTitle, exact: true })).toBeVisible(); +} diff --git a/tests/runner-e2e/product-journeys/contracts.test.ts b/tests/runner-e2e/product-journeys/contracts.test.ts index 8ddc2911323..b3d3bd7b441 100644 --- a/tests/runner-e2e/product-journeys/contracts.test.ts +++ b/tests/runner-e2e/product-journeys/contracts.test.ts @@ -233,6 +233,17 @@ describe("deployment journey contracts", () => { { runtimeMode: "native" }, { runtimeMode: "legacy" }]; for (const work of forbidden) expect(()=>validateRetainedRunnerResult({ ...browser, ...work })).toThrow("result.browserOnly"); }); + it("rejects collected runs and nonempty usage on a saved passing access result", () => { + const browser = result(); browser.caseId = "company-access-isolation"; + browser.executionId = "product-journeys.local-qa.local.company-access-isolation"; + browser.productJourney!.checks = journeyRequiredChecks[browser.caseId]!.map(id => ({ id, passed: true })); + expect(() => validateRetainedRunnerResult(browser)).not.toThrow(); + const collected = validateJourneyCollection({ journeyId: "company-access-isolation", runIds: ["collected-run"], usage: { runs: [{ runId: "collected-run", usage: { inputTokens: 1 } }] } }); + expect(() => validateRetainedRunnerResult({ ...browser, runIds: collected.runIds, usage: collected.usage })).toThrow("result.browserOnly"); + expect(() => validateRetainedRunnerResult({ ...browser, runIds: [], usage: collected.usage })).toThrow("result.browserOnly"); + expect(() => validateRetainedRunnerResult({ ...browser, usage: { inputTokens: 1 } })).toThrow("result.browserOnly"); + expect(() => validateRetainedRunnerResult({ ...browser, usage: { runs: [] } })).not.toThrow(); + }); it("keeps browser-only reporting free of invented model calls and rejects public publication", () => { const value=result(); expect(summarizeExecutionBilling(value)).toMatchObject({llm:{runCount:0,reportedCostUsd:0},runtime:{agentRunDurationMs:0}}); diff --git a/tests/runner-e2e/product-journeys/fixtures.ts b/tests/runner-e2e/product-journeys/fixtures.ts index b26ff61ddbb..3f33df99bac 100644 --- a/tests/runner-e2e/product-journeys/fixtures.ts +++ b/tests/runner-e2e/product-journeys/fixtures.ts @@ -11,7 +11,7 @@ export function journeyFixtures(session: TargetSession, campaignId?: string) { id: "ready-context", async setup() { return ready ??= await (preparation ??= session.prepare()); }, }); - for (const id of ["company", "board-user", "task-write", "agent", "skill-write", "review-service", "inbox-author"]) registry.register({ + for (const id of ["company", "board-user", "task-write", "agent", "skill-write", "review-service", "inbox-author", "company-access"]) registry.register({ id, dependencies: ["ready-context"], async setup(resolved) { const context = resolved.get("ready-context") as JourneyContext; diff --git a/tests/runner-e2e/product-journeys/launch.ts b/tests/runner-e2e/product-journeys/launch.ts index edfe84bbe75..6482fc9580a 100644 --- a/tests/runner-e2e/product-journeys/launch.ts +++ b/tests/runner-e2e/product-journeys/launch.ts @@ -20,6 +20,7 @@ import { runClarificationJourney } from "./clarification-flow.js"; import { runRepositoryContinuation } from "./repository-continuation-flow.js"; import { runScheduledRoutine } from "./routine-schedule-flow.js"; import { runInputFile } from "./input-file-flow.js"; +import { companyAccessIsolation } from "./company-access-flow.js"; import { inboxTriage } from "./inbox-triage-flow.js"; import { JourneyError, type JourneyContext, type JourneyEvidence, type JourneyOutcome, type TargetLifecycleInput, type TargetProvider, type TargetSession } from "./types.js"; import type { RunnerE2EResult } from "../types.js"; @@ -112,7 +113,7 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri try { sha = execFileSync("git", ["rev-parse", "HEAD"], { cwd: repositoryRoot, encoding: "utf8", stdio: ["ignore", "pipe", "ignore"] }).trim(); } catch { /* Packaged harnesses may not include Git metadata. */ } const source = resolveRunnerE2ESource({ sha, ref: null, workflowRunUrl: null }); const definitionHash = createHash("sha256").update(JSON.stringify(productJourneys)); - for (const name of ["observation-read.ts", "../connection-config.ts", "catalog.ts", "repository-continuation-flow.ts", "repository-continuation-contract.ts", "repository-continuation-evidence.ts", "../registered-artifact.ts", "../user-actions.ts", "../connection-target.ts", "../../../packages/shared/src/project-mentions.ts", "routine-schedule-flow.ts", "routine-schedule-evidence.ts", "input-file-flow.ts", "input-file-evidence.ts", "inbox-triage-flow.ts", "inbox-triage-evidence.ts", "inbox-triage-fixture.ts", "inbox-author-contract.ts", "task-properties.ts", "task-search-filter.ts", "task-search-evidence.ts", "basic-flows.ts", "basic-evidence.ts", "task-document-flow.ts", "task-document-evidence.ts", "project-context-flow.ts", "project-context-readback.ts", "project-context-evidence.ts", "agent-flows.ts", "hire-delegate-flow.ts", "hire-delegate-composer.ts", "hire-delegate-evidence.ts", "active-followup-flow.ts", "active-followup-evidence.ts", "stop-redirect-evidence.ts", "clarification-flow.ts", "clarification-evidence.ts", "../question-resume-ui.ts", "../question-resume-scoring.ts", "../runtime-question-readiness.ts", "../chat-flow.ts", "../continuation-readiness.ts", "../continuation-cases.ts", "success-contract.ts", "product-screenshot.ts", "cloud-onboarding.ts", "onboarding-connect.ts", "onboarding-first-task.ts", "workspace-deferral.ts", "onboarding-pre-acceptance.ts", "artifact-evidence.ts", "fixtures.ts", "launch.ts", "types.ts", "collection.ts", "local-target.ts", "../../../ui/src/lib/markdown-reply-content.ts", "../../../ui/src/lib/issue-reference.ts", "../../../ui/src/lib/company-routes.ts", "../../../ui/package.json"]) definitionHash.update(await readFile(path.join(import.meta.dirname, name))); + for (const name of ["observation-read.ts", "../connection-config.ts", "catalog.ts", "company-access-flow.ts", "company-access-contract.ts", "repository-continuation-flow.ts", "repository-continuation-contract.ts", "repository-continuation-evidence.ts", "../registered-artifact.ts", "../user-actions.ts", "../connection-target.ts", "../../../packages/shared/src/project-mentions.ts", "routine-schedule-flow.ts", "routine-schedule-evidence.ts", "input-file-flow.ts", "input-file-evidence.ts", "inbox-triage-flow.ts", "inbox-triage-evidence.ts", "inbox-triage-fixture.ts", "inbox-author-contract.ts", "task-properties.ts", "task-search-filter.ts", "task-search-evidence.ts", "basic-flows.ts", "basic-evidence.ts", "task-document-flow.ts", "task-document-evidence.ts", "project-context-flow.ts", "project-context-readback.ts", "project-context-evidence.ts", "agent-flows.ts", "hire-delegate-flow.ts", "hire-delegate-composer.ts", "hire-delegate-evidence.ts", "active-followup-flow.ts", "active-followup-evidence.ts", "stop-redirect-evidence.ts", "clarification-flow.ts", "clarification-evidence.ts", "../question-resume-ui.ts", "../question-resume-scoring.ts", "../runtime-question-readiness.ts", "../chat-flow.ts", "../continuation-readiness.ts", "../continuation-cases.ts", "success-contract.ts", "product-screenshot.ts", "cloud-onboarding.ts", "onboarding-connect.ts", "onboarding-first-task.ts", "workspace-deferral.ts", "onboarding-pre-acceptance.ts", "artifact-evidence.ts", "fixtures.ts", "launch.ts", "types.ts", "collection.ts", "local-target.ts", "../../../ui/src/lib/markdown-reply-content.ts", "../../../ui/src/lib/issue-reference.ts", "../../../ui/src/lib/company-routes.ts", "../../../ui/package.json"]) definitionHash.update(await readFile(path.join(import.meta.dirname, name))); for (const file of ["catalog.ts", "api.ts", "run-observations.ts", "native-task-reply.ts", "native-event-envelope.ts", "user-actions.ts", "plan-revision-flow.ts", "skill-studio-flow.ts", "protected-action-flow.ts", "story-artifact.ts", "stop-redirect-flow.ts", "everyday-cases.ts", "everyday-delivery.ts", "chat-hardening.ts", "everyday-decisions.ts", "connection-guidance-evidence.ts", "everyday-artifact.py"]) definitionHash.update(await readFile(path.join(import.meta.dirname, "..", file))); definitionHash.update(runnerSuites.flatMap(suite => suite.tasks).find(task => task.id === "ask-question")!.buildPrompt("definition")); const suiteDefinitionHash = definitionHash.digest("hex"); @@ -231,7 +232,8 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri await mkdir(directory, { recursive: true, mode: 0o700 }); await captureProductScreenshot(ready!.page, ready!.api.origin, path.join(directory, file), deadlineAt); } }); - } else if (id === "inbox-triage") await inboxTriage(ready, session!, campaignId, evidence); + } else if (id === "company-access-isolation") await companyAccessIsolation(ready, session!, campaignId, evidence); + else if (id === "inbox-triage") await inboxTriage(ready, session!, campaignId, evidence); else if (id === "task-create") await taskCreate(ready, session!, campaignId, evidence); else if (id === "task-document-persistence") await taskDocumentPersistence(ready, session!, campaignId, evidence); else if (id === "project-task-context") await projectTaskContext(ready, session!, campaignId, evidence); diff --git a/tests/runner-e2e/product-journeys/success-contract.ts b/tests/runner-e2e/product-journeys/success-contract.ts index 7141438d29c..32c2eca9b47 100644 --- a/tests/runner-e2e/product-journeys/success-contract.ts +++ b/tests/runner-e2e/product-journeys/success-contract.ts @@ -9,6 +9,7 @@ export function agentJourneyRunCountValid(id: string, count: unknown): boolean { return (count as number) >= minimum && (count as number) <= maximum; } export const journeyRequiredChecks: Record = { + "company-access-isolation": ["isolation-owner-resource-controls", "cloud-workspace-switch-and-return", "workspace-data-scoped", "restricted-agent-positive-controls", "cross-workspace-agent-read-write-denied", "isolation-credential-revoked", "no-agent-work"], "repository-continuation": ["repository-configured-source-bound", "repository-first-commit-verified", "repository-original-files-saved", "repository-followup-stored-once", "repository-same-workspace-retained", "repository-second-commit-verified", "repository-original-and-result-reopened", "configured-model-and-environment"], "routine-scheduled-run": ["routine-browser-created", "routine-scheduler-created-execution", "routine-result-attributed", "routine-history-preserved", "routine-pause-prevented-next-run", "routine-result-reopened", "configured-model-and-environment"], "inbox-triage": ["inbox-external-human-activity", "inbox-read-persisted", "inbox-archive-persisted", "inbox-archive-preserved-task-status", "inbox-relevant-activity-resurfaced", "no-agent-work"], diff --git a/tests/runner-e2e/product-journeys/types.ts b/tests/runner-e2e/product-journeys/types.ts index 0de951ff574..4c235990f6c 100644 --- a/tests/runner-e2e/product-journeys/types.ts +++ b/tests/runner-e2e/product-journeys/types.ts @@ -3,7 +3,7 @@ import type { ConnectionApi } from "../connection-target.js"; export type JourneyOutcome = "passed" | "failed" | "blocked" | "unsupported"; export type JourneyId = "task-properties" | "task-create" | "project-task-context" | "task-document-persistence" | "ask-question" | "cloud-onboarding" - | "question-answer-resume" | "task-search-filter" | "followup-while-running" | "plan-revise-accept" | "build-revise" | "protected-actions" | "create-skill-studio" | "stop-redirect" | "hire-delegate-reuse" | "file-input-answer" | "inbox-triage" | "routine-scheduled-run" | "repository-continuation"; + | "question-answer-resume" | "task-search-filter" | "followup-while-running" | "plan-revise-accept" | "build-revise" | "protected-actions" | "create-skill-studio" | "stop-redirect" | "hire-delegate-reuse" | "file-input-answer" | "inbox-triage" | "routine-scheduled-run" | "repository-continuation" | "company-access-isolation"; export type TargetMode = "managed-self-hosted" | "existing-cloud" | "new-cloud"; export interface DeploymentIdentity { kind: "self-hosted" | "cloud"; @@ -84,6 +84,7 @@ export interface JourneyContext { agent?: { id: string; name: string; provider: string; model: string; runtimeMode: "native" | "legacy" }; fixtures?: { backlogTask?: PreparedBacklogTask; + companyAccess?: import("./company-access-contract.js").CompanyAccessFixture; inboxAuthor?: import("./inbox-author-contract.js").InboxAuthorFixture; reviewService?: { connectionId: string; name: string; marker: string; invocationCount(): number }; }; diff --git a/tests/runner-e2e/result-validation.ts b/tests/runner-e2e/result-validation.ts index e709c7439ba..419e37a934d 100644 --- a/tests/runner-e2e/result-validation.ts +++ b/tests/runner-e2e/result-validation.ts @@ -434,7 +434,8 @@ export function validateRetainedRunnerResult( if (!evidence) return invalid("result.productJourney"); const required = journeyRequiredChecks[checked.caseId]; if (evidence.outcome !== "passed" || !evidence.deployment || !required || !required.every(id => evidence.checks.some(c => c.id === id && c.passed)) || evidence.checks.some(check => !check.passed) || evidence.phases.some(p => p.outcome !== "passed") || checked.cleanup !== "passed") invalid("result.productJourney"); - if (["inbox-triage", "task-properties", "task-create", "project-task-context", "task-document-persistence", "task-search-filter"].includes(checked.caseId) && (checked.runtimeMode !== "none" || checked.provider !== null || checked.model !== null || checked.runIds?.length)) invalid("result.browserOnly"); + if (["inbox-triage", "task-properties", "task-create", "project-task-context", "task-document-persistence", "task-search-filter", "company-access-isolation"].includes(checked.caseId) && (checked.runtimeMode !== "none" || checked.provider !== null || checked.model !== null || checked.runIds?.length)) invalid("result.browserOnly"); + if (checked.caseId === "company-access-isolation" && checked.usage != null && Object.entries(checked.usage).some(([key, value]) => key !== "runs" || !Array.isArray(value) || value.length > 0)) invalid("result.browserOnly"); if (Object.hasOwn(agentJourneyRuns, checked.caseId) && (!agentJourneyRunCountValid(checked.caseId, checked.runIds?.length) || !checked.model || !checked.provider || checked.runtimeMode !== "native")) invalid("result.agentJourneyRun"); if (checked.caseId === "cloud-onboarding" && (!checked.runIds?.length || !evidence.operations || !evidence.operations.requestCount || !evidence.operations.traceIds.length || evidence.operations.campaignId !== evidence.campaignId || evidence.operations.attemptId !== evidence.attemptId)) invalid("result.paidJourneyReceipt");