Files
PaperClipAI/tests/runner-e2e/connection-flow.ts
DottaandPaperclip a6306ba606 feat(runner): consolidate Cursor production integration (#15075)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The native Runner keeps provider sessions under company authority,
approvals, budgets and durable recovery.
> - Cursor work was spread across candidate branches. The published
branch lacked later plan, permission and cleanup fixes.
> - Production also needs public installation and matching runtime
assets for local and Daytona execution.
> - This pull request consolidates Cursor onto current mainline recovery
behavior and completes that installation path.
> - The installed v11 release passed focused local and Daytona
qualification after the generic mode and lifecycle cleanup. The later
model-selection correction and current mainline merge produce v14
artifacts that need matching release qualification.
> - Cursor admission is enabled in source; publish only an artifact
combination with matching qualification. Native AskQuestion and complete
per-run dollar accounting remain excluded.

## Linked Issues or Issue Description

Refs: #14435, #14631, #14669, #14699, #14724.

This completes the Cursor implementation by @cryppadotta from combined
source `22c78242a4e0c2369fecf0c2dc4e7600fbad6706`. It preserves newer
mainline recovery, completion and warm-directory behavior. Pi and
Copilot remain gated.

## What Changed

- Generate named Rust and TypeScript ACPX release profiles from one
manifest. Share runtime pins with packaging and server verification.
Preserve vendor runtime versions; bind the updated ACPX patch to Cursor
profile v14 and reject stale generated declarations at build/typecheck.
- Remove ACPX model allowlists, including the former Codex and Pi
restrictions and the duplicate developer test-drive gate. Send any
explicit model ID unchanged to its provider and verify the effective
selection before prompting. The bundled ACPX package forwards unlisted
IDs, rejects mismatched acknowledgements, and restores the exact
selection after session load. It does not expand Cursor model aliases.
Provider rejection, mismatch, or missing model controls fails without a
fallback. Model examples live in evaluation fixtures, outside runtime
declarations.

- Add pinned Cursor execution, contained instructions, exact model
verification and Agent/Plan/Ask modes.
- Carry an opaque generic `mode` identifier in shared native execution,
sidecar, Rust and recovery contracts. The provider adapter owns
supported modes, defaults, native translation and acknowledgement.
- Keep native RPC recognition, accepted-plan interpretation and
permission evidence behind provider adapters. Shared settlement and
recovery verify normalized facts and their committed evidence.
- Replace the Cursor-only warm-attachment branch with a runner-owned
capability. Only Cursor opts into it. Move profile compatibility and
optional usage parsing into provider metadata and adapters.
- Write generic plan-wait receipts. Read exact historical Cursor
receipts through a separate compatibility decoder. Reject mixed formats
and preserve existing authority checks.
- Carry native plans, semantic questions, todos, child activity,
permission identities and partial usage diagnostics through the Runner.
- Preserve durable response delivery, cancellation, warm ownership and
process retirement.
- Finish accepted planning runs successfully. Keep their tasks open for
explicit direction. Acceptance does not start implementation.
- Ship `paperclipai runtime setup cursor` and its provisioner through
the public package. npm installation does not download Cursor. Setup
uses the OS account's closure-keyed cache so system-wide npm packages
can remain read-only. Run it as the Paperclip service account.
- Include Cursor in normal provider packs and Daytona images for macOS
ARM64/x64 and Linux x64.
- Reject stale release packs by source revision and current ACPX/Cursor
pins before assembly writes files. Verify current Cursor
version/profile/closure again at runtime.
- Ship all three daemon targets and the expected Linux image-pack
identity. A macOS controller uses its packaged Linux daemon for Daytona.
Image mismatches fail before provider launch.
- Use the vendored Runner boundary for installed readiness probes.
Verify the actual installed Cursor probe.
- Verify compiled public Daytona plugins and their release versions in
installed smokes.
- Record exact artifacts, the acceptance matrix, retained failures,
supported capabilities and rollback behavior in the [readiness
report](https://github.com/paperclipai/paperclip/blob/codex/cursor-production-readiness/doc/plans/2026-10-03-cursor-production-readiness.md).

## Verification

- Current head `f7ec5cc1f0e30c62a829c016ff2013a2a9d79037` merges
mainline `faa8e452c73bae5e044dd6379179a00106abb131`. It keeps Cursor
plan and cancellation guards alongside mainline historical-question
filtering. The evaluation catalog includes both Cursor and expanded
adapter accounting cases (683 total). Recursive typecheck, full build,
696 lifecycle/recovery tests, 45 fixture tests and fixture typecheck
passed. Current-head CI passed: 56 successful checks, one neutral and
four skipped. [Complete
CI](https://github.com/paperclipai/paperclip/actions/runs/37557996535).
The fresh Base Greptile review is 5/5 on this exact head, with 304 files
reviewed, zero new comments and zero unresolved threads. The user
authorized overriding the CODEOWNER review gate after checks passed; no
failing checks are overridden. Prior results below retain their own head
identities.
- Corrective head `3d2b168366258036f6b6a6fccb382c49138cc601` fixes the
post-merge Apex finding. Automatic-review and new-evidence
reconciliation preserve pending child results and recheck delivery under
the status lock before completing. Account repair now excludes unrelated
secret consumers and requires the failed agent's identity. Regression
coverage includes the commit race, delivery statuses,
current-run/current-intent exclusions, repeated reconciliation, both
database reconciliation paths, and credential consumer boundaries. All
184 affected tests, server typecheck and server build passed.
Current-head Base Greptile review is 5/5, with 304 files reviewed, zero
new comments and zero unresolved threads. Current-head CI passed: 56
successful checks, one neutral and four skipped. [Complete
CI](https://github.com/paperclipai/paperclip/actions/runs/37535994724).
This Base review is distinct from the earlier Apex review.
- Merge head `5957c257a` reconciles mainline `b508a05c4`. It preserves
both accepted-plan waits and pending-child-completion checks, current
provider selectors, task-creation response identities, and mainline ACPX
missing-file handling. The combined patch is bound to Cursor profile
v14; historical records keep their original identities.
- Merge head `5957c257a` passed recursive typecheck, full build, 43
installed ACPX/package contracts, 107 provider UI and plan/recovery
tests, 593 database-backed lifecycle tests, 49 profile/native contract
tests, 45 Product E2E fixture tests, fixture typecheck, token gates,
three provider-free browser task-creation cases, and Runner
conformance/replay checks. Its complete CI passed (55 successful checks,
one neutral and four skipped), while Apex returned 2/5 with a
child-delivery finding addressed below.
- The local full-suite attempt again failed the unchanged Git streaming
test (360-second timeout) and was stopped. The concurrent local Rust
attempt failed four unchanged Codex process/deadline tests; all four
passed serially without code changes in 7.29 seconds after removing the
competing test load. These failed commands are retained and are not
reported as full-suite passes; the fresh Linux CI runs are tracked
separately.
- The previous head `907bdb2a2778c7ffeb4a662a91460c9d1ddfc9c5` earned
Apex 5/5 with zero comments after fixing all three findings: per-user
install cache, stale release-pack rejection, and public Linux smoke
account/home handling. Its real built installer passed from read-only
public packages on macOS ARM64 and Linux x64. All 137 release-registry
checks and 64 ACPX package contracts passed. That review does not cover
this mainline reconciliation.
- Prior `beadd3654` passed the full CI matrix; its one unchanged chat
test failure and successful single retry remain in the [CI
history](https://github.com/paperclipai/paperclip/actions/runs/37521449327).
Historical results below remain attributed to their original builds.

- Fixture follow-up `dd59d7e82b103a88b7cbd7d2c38b612c0fbbff7a` removes
provider-specific model choices from generic offline ACPX tests. The
fake sidecar preserves the model and session identity selected at open
through suspension. Affected verification passed: 106 Rust tests and 73
TypeScript tests. This commit changes test code only; the
production-code checks below retain their recorded identities. Its CI
and Greptile review later passed; those results belong to that
historical head.
- Model-selection cleanup `9a070808b48960a41fdfd369ae0636b95af82459`:
252 focused Runner tests passed (six platform skips), covering all six
ACPX agents, native model acknowledgement, rejected selections,
installation integrity and recovery identity. The merged branch passed
recursive typecheck, full build, token gates, server admission (19
tests), and the Product E2E catalog (45 tests). The acceptance catalog
passed all four tests. The full Rust suite passed: 643 tests, 2 ignored.
It verifies sidecar acknowledgement of unlisted models and rejection of
model mismatches. The final commits only update Rust tests; production
sources match the verified build at
`65ec3279ac50185e3cda109b5cfd9b4f56105de0`. No new paid provider calls
were made.
- The merge preserves both Cursor and the new mainline public-MCP
fixture cases. Auto-merge remains disabled; the latest follow-up status
is recorded above. The local `pnpm test:run` attempt hit the unchanged
Git streaming test's 300-second timeout and was interrupted before
merging mainline. The broad Runner attempt found obsolete single-model
assertions plus three macOS fixture-path failures caused by a
`/private/tmp` override. The assertions are corrected; affected
TypeScript checks passed with the standard macOS temporary directory,
and the complete Rust suite passed. Neither interrupted command is a
full-suite pass.
- Earlier declaration-cleanup head `6f4a5e9e2` passed recursive
typecheck, build, Rust and focused tests. Its CI later exposed a test
expecting duplicated Grok digest literals. The current source fixes that
assertion to compare launcher bytes with the shared manifest. Historical
successes and failed attempts are retained; no new live provider
qualification is claimed.
- Previous head `e75fde6098b0ddd8cec765bfb6ecaeecb88a26a6` passed
complete CI (56 successful checks, one neutral, four skipped) and
Greptile 5/5. [Historical complete
CI](https://github.com/paperclipai/paperclip/actions/runs/37489112305).
Those results are not claimed for the cleanup head.
- Frozen live application: `d7b696f9b8f79095233e9e3d56d23e6a6018dd48`.
Public package version: `0.0.0-cursor-verify.3d0c9b7761c6`. The
declaration cleanup preserves release pins and does not relabel that
tested artifact as a build of the new source. Mainline through
`e34abee670` was reconciled while preserving accepted-plan waits,
provider-capacity handling, and both Cursor and public-MCP fixtures.
- Clean normal installation, explicit Cursor setup and daemon resolution
passed on macOS ARM64, macOS x64 under Rosetta, and Linux x64. npm
lifecycle hooks ran without silently downloading Cursor.
- Historical v11 live matrix: **18/18 passed with cleanup** (nine local,
nine Daytona) after the generic mode and lifecycle cleanup. The campaign
has 23 attempts; all five failures and their diagnoses remain recorded.
Exact case identities, hashes and limits are in the readiness report.
All provider calls are real, use the explicit Luna model and
company-bound credentials, and run without qualification or
runtime-asset overrides.
- The immutable Daytona image is
`ghcr.io/paperclipai/paperclip-daytona-runner@sha256:d6259b6bba094702c13fc2283bd85550849c1c53145b656fb2746778f9fa1747`.
The public Daytona plugin is installed independently and its version is
checked.
- Recursive typecheck, full build, token gates and Runner
contract/conformance/replay checks passed on the frozen application. Its
complete Linux CI suite passed. The duplicate local full-suite command
was incomplete after timing failures; affected repeats passed, but that
command is not reported as a clean pass.
- Qualification fixtures passed typecheck, 1,675 Vitest tests (one
skip), 128 Node checks, three provider-free browser tests, and 150
focused lifecycle tests after the final diagnostic correction. The
affected legacy Cursor command file also passed all five tests after
removing its shorter 10-second override; it now inherits the suite’s
standard 15-second timeout. Greptile is 5/5 on `e75fde609` with no
unresolved review threads. CI results above are recorded separately from
historical build results.

## Risks

- Cursor v14 includes the updated ACPX dependency patch and release
identity. The v11 live matrix and image below remain historical
evidence. They do not certify new v14 package/image artifacts.

- ACPX accepts models beyond the qualification fixtures. Availability
and entitlement depend on the provider. Successful configuration is not
a claim of live qualification for every model.
- Shared mode is an opaque identifier. Provider adapters own its
meaning. Incompatible historical sessions remain fenced; exact committed
plan waits and task history remain inspectable.
- Native AskQuestion is excluded. Paperclip semantic questions are
supported. Authoritative per-run dollar accounting is unavailable;
partial counters remain diagnostics and unknown cost is not zero.
- Image input, detailed native diffs, deeper child transcripts and
native plan-file export remain follow-ups.
- macOS x64 has clean-install and daemon-startup proof under Rosetta,
not a separate live campaign on Intel hardware.
- Release only the tested package/image combination. Merging this PR
does not publish npm packages or deploy that image. Later builds need
their own release verification. Rollback disables new Cursor admission
while preserving records and recovery inspection.
- A model can fail an exact instruction: one cancelled-plan attempt
returned the wrong summary marker despite correct cancellation. The
unchanged repeat passed; both results remain in the report.

> ROADMAP.md was checked. This completes existing native Runner/Cursor
work; it does not add an independent core feature proposal.

## Model Used

OpenAI Codex, GPT-6. The exact serving variant and context window are
not exposed in this session. The agent used reasoning, repository
inspection, code execution, protocol tests and browser-backed Product
E2E tools. Cursor acceptance uses the explicit
`gpt-5.6-luna[context=272k,reasoning=medium,fast=false]` model. That is
the evaluated provider model.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass — affected suites passed;
full CI and the retained local failed attempts are recorded separately
above.
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green — 56 successful checks, one
neutral and four skipped on `f7ec5cc1f0e30c62a829c016ff2013a2a9d79037`
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups —
fresh Base review passed on `f7ec5cc1f0e30c62a829c016ff2013a2a9d79037`;
zero new comments and no unresolved threads. The earlier Apex finding
remains fixed.
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-06 20:48:15 -05:00

373 lines
27 KiB
TypeScript

import { createHash } from "node:crypto";
import { expect, type Page } from "@playwright/test";
import { aiRoutingModel } from "../../packages/shared/src/ai-provider-routing.js";
import { getConnectableAppDefinition } from "../../packages/shared/src/app-definitions.js";
import { connectionCell, type ConnectionHarness, type ConnectionMode } from "./connection-cases.js";
import { ConnectionBlock, ConnectionFailure, type ConnectionConfig, type resolveConnectionSettings } from "./connection-config.js";
import { ConnectionApi, connectionDelay } from "./connection-target.js";
import { createConnectionProof, verifyConnectionArtifact, verifyConnectionRun, completedConnectionArtifactRun, connectionProbeChecks, type ConnectionEvidence } from "./connection-evidence.js";
import { createTaskThroughUi, submitTaskReply } from "./user-actions.js";
import { collectRunEvents } from "./run-observations.js";
import type { MatrixExecution } from "./types.js";
type Row = Record<string, any>;
type Settings = ReturnType<typeof resolveConnectionSettings>;
export interface ConnectionFlowInput {
page: Page; api: ConnectionApi; execution: MatrixExecution; config: ConnectionConfig; settings: Settings;
company: Row; environment: Row; secret?: string; nonce: string; privateDir: string;
evidence: ConnectionEvidence;
checkpoint(phase: string): Promise<void>;
observeRun(run: Row): void;
createdConnectionIds: ReadonlySet<string>;
refreshOwnedConnections(): Promise<void>;
assertActive(): void;
}
async function choose(page: Page, label: string, name: string) {
await page.getByRole("combobox", { name: label, exact: true }).click();
await page.getByRole("option", { name, exact: true }).click();
}
async function openAdvanced(page: Page, mode: ConnectionMode) {
const names: Record<string, string> = { openrouter: "OpenRouter", bedrock: "Amazon Bedrock", responses: "Custom gateway", messages: "Custom gateway", chat: "Custom gateway" };
await page.getByRole("button", { name: new RegExp(`^${names[mode]}`) }).click();
}
async function fillGateway(input: ConnectionFlowInput) {
const { page, settings, secret } = input;
const { mode } = connectionCell(input.execution);
if (settings.route?.baseURL) {
await page.getByLabel("Provider URL", { exact: true }).fill(settings.route.baseURL);
const format = page.getByRole("combobox", { name: "API format", exact: true });
if (await format.isEnabled()) await choose(page, "API format", ({ responses: "OpenAI Responses · Codex", messages: "Anthropic Messages · Claude", chat: "Chat Completions · OpenCode, Hermes" } as Record<string, string>)[mode]!);
}
if (mode === "bedrock") await page.getByLabel("AWS region", { exact: true }).fill(settings.route!.region!);
if (settings.route?.auth !== "bearer") await choose(page, "Authentication", settings.route?.auth === "none" ? "No authentication" : "API key · x-api-key");
if (secret) await page.getByLabel("API key", { exact: true }).fill(secret);
// Normal defaults stay collapsed. Model alias is selected in Configure.
await page.getByRole("button", { name: "Connect", exact: true }).last().click();
}
async function submitConnection(input: ConnectionFlowInput, submit: () => Promise<void>) {
const route = `${input.api.origin}/api/companies/${input.company.id}/ai-connections`;
const response = input.page.waitForResponse(response => response.url() === route && response.request().method() === "POST", { timeout: 60_000 }).catch(() => null);
await submit();
const created = await response;
if (!created) throw new ConnectionFailure("connection_request_timeout");
if (created.ok()) return;
const payload = await created.json().catch(() => ({}));
const allowed = ["ai_connection_api_key_rejected", "ai_connection_verification_failed"];
const code = allowed.includes(payload?.details?.code) ? payload.details.code : "connection_request_rejected";
input.evidence.creationFailure = { status: created.status(), code };
if (code === "ai_connection_api_key_rejected") throw new ConnectionBlock("missing_credential", code);
throw new ConnectionFailure(code);
}
async function waitForFreshConnection(input: ConnectionFlowInput, before: Set<string>) {
const { page, api, config, evidence, company } = input;
const { harness, mode } = connectionCell(input.execution);
const deadline = Date.now() + (mode === "subscription" ? config.browser.loginTimeoutMs : 90_000);
let loginOpened = false;
let connected = false;
while (Date.now() < deadline) {
input.assertActive();
await input.refreshOwnedConnections();
const list = await api.get(`/api/companies/${company.id}/ai-connections`);
const fresh = list.connections.filter((c: Row) => !before.has(c.id) && input.createdConnectionIds.has(c.id));
if (fresh.length > 1) throw new ConnectionFailure("unexpected_multiple_connections");
if (fresh.length === 1) {
const connection = fresh[0];
if (connection.status !== "connected" || connection.method !== (mode === "subscription" ? "subscription" : "api_key")) throw new ConnectionFailure("saved_connection_method_or_status_mismatch");
return connection;
}
if (mode === "subscription") {
const signIn = page.getByRole("link", { name: /^Sign in to / }).first();
const cloudSignIn = page.getByRole("button", { name: /^Sign in to / }).first();
if (!loginOpened && (await signIn.isVisible() || await cloudSignIn.isVisible())) {
// Observe before clicking. No token, code, URL, or popup content is recorded.
const popup = page.context().waitForEvent("page", { timeout: 10_000 }).catch(() => undefined);
if (await signIn.isVisible()) await signIn.click(); else if (await cloudSignIn.isEnabled()) await cloudSignIn.click();
await popup;
loginOpened = true;
evidence.assisted = true;
evidence.waits.push({ kind: "provider_login", startedAt: new Date().toISOString() });
await input.checkpoint("awaiting_provider_login");
console.log(`[provider-connections] Sign in to ${harness.label} in the opened browser (${config.browser.accountAlias}); complete the code/consent step in Paperclip. This test resumes automatically.`);
}
// Grok currently exposes a terminal handoff; do not pretend it has the same browser flow.
if (!loginOpened && harness.id === "grok" && await page.getByRole("button", { name: "Copy sign-in command" }).isVisible()) {
loginOpened = true;
evidence.assisted = true;
evidence.waits.push({ kind: "provider_login", startedAt: new Date().toISOString() });
await input.checkpoint("awaiting_provider_login");
console.log("[provider-connections] Grok requires the sign-in command shown in Paperclip. Complete that production handoff; the test will resume automatically.");
}
const connect = page.getByRole("button", { name: "Connect", exact: true }).last();
if (!connected && await connect.isVisible() && await connect.isEnabled()) { await connect.click(); connected = true; }
if (await page.getByRole("alert").filter({ hasText: /expired|could not|failed|invalid/i }).count()) throw new ConnectionFailure("provider_login_reported_error");
}
await connectionDelay(1_000);
}
if (mode === "subscription" && loginOpened) throw new ConnectionBlock("awaiting_user", "provider_login_deadline_reached");
throw new ConnectionFailure("connection_creation_timeout");
}
async function selectSavedConnection(page: Page, connection: Row, harness: ConnectionHarness, mode: ConnectionMode) {
await page.getByRole("heading", { name: /^(Connect a model|Configure your agent)$/ }).waitFor();
const advanced = !["subscription", "api-key"].includes(mode);
const connectionType = page.getByRole("radiogroup", { name: "Connection type", exact: true });
if (["claude", "codex", "grok"].includes(harness.id)) {
await connectionType.waitFor({ state: "visible" });
if (advanced) {
await connectionType.getByRole("radio", { name: /Advanced/ }).click();
await choose(page, "Connection", connection.name);
await page.getByRole("button", { name: "Use connection", exact: true }).click();
} else {
await connectionType.getByRole("radio", { name: mode === "subscription" ? /Subscription/ : /API key/ }).click();
const saved = page.getByRole("combobox", { name: mode === "subscription" ? "Saved subscription" : "Saved API key", exact: true });
await saved.waitFor({ state: "visible" });
const option = await saved.locator("option").filter({ hasText: connection.name }).first().getAttribute("value");
if (!option) throw new ConnectionFailure("saved_connection_option_missing");
await saved.selectOption(option);
await page.getByRole("button", { name: mode === "subscription" ? "Use saved subscription" : "Use saved API key", exact: true }).click();
}
} else {
await choose(page, "Connection", connection.name);
}
}
async function createConnectionThroughUi(input: ConnectionFlowInput, agentURL: string) {
const { page, api, company, evidence } = input;
const { harness, mode, entry } = connectionCell(input.execution);
const priorConnections = (await api.get(`/api/companies/${company.id}/ai-connections`)).connections as Row[];
const before = new Set<string>(priorConnections.map(c => c.id));
// Fresh-creation proof cannot be satisfied by an existing personal default.
if (priorConnections.some(connection => connection.status !== "revoked")) throw new ConnectionBlock("blocked_target", "fresh_connection_requires_empty_qa_company");
const advanced = !["subscription", "api-key"].includes(mode);
await input.checkpoint("connection_navigation");
if (entry === "apps") {
await page.goto(`${api.origin}/${company.issuePrefix}/apps`, { waitUntil: "domcontentloaded" });
const catalogSlug = advanced ? ({ openrouter: "openrouter", bedrock: "bedrock", responses: "responses-api", messages: "messages-api", chat: "chat-completions-api" } as Record<string, string>)[mode]! : harness.provider;
const definition = getConnectableAppDefinition(catalogSlug);
if (!definition) throw new ConnectionFailure("connection_catalog_entry_missing");
await page.getByRole("button", { name: `Connect ${definition.name}`, exact: true }).click();
const next = page.getByRole("button", { name: /^(Save and continue|Continue)$/ }).last();
if (!advanced && await next.isVisible()) await next.click();
if (!advanced && harness.provider !== "google") {
// The credential step loads asynchronously. Wait for its provider tile
// before inspecting the mode switch, otherwise a still-loading page can
// skip the API-key switch and leave the subscription form selected.
await page.getByRole("radio", { name: new RegExp(harness.id === "grok" ? "Grok" : harness.label) }).click();
const switcher = page.getByRole("button", { name: mode === "subscription" ? "Use subscription instead" : "Use API key instead", exact: true });
if (await switcher.isVisible()) await switcher.click();
}
} else {
await page.goto(agentURL, { waitUntil: "domcontentloaded" });
const tiles = page.getByRole("radiogroup", { name: "Connection type", exact: true });
if (["claude", "codex", "grok"].includes(harness.id)) {
const choice = tiles.getByRole("radio", { name: advanced ? /Advanced/ : mode === "subscription" ? /Subscription/ : /API key/ });
await choice.click();
await expect(choice).toHaveAttribute("aria-checked", "true");
}
if (advanced || !["claude", "codex", "grok"].includes(harness.id)) {
await choose(page, "Connection", "Connect an account…");
if (advanced) {
const advancedProviders = page.locator("summary").filter({ hasText: "Advanced providers" });
if (await advancedProviders.isVisible()) {
await advancedProviders.click();
// The account dialog's disclosure leads to the provider chooser;
// the chooser's own disclosure already contains provider buttons.
const chooseProvider = page.getByRole("button", { name: "Choose another provider or gateway", exact: true });
if (await chooseProvider.isVisible()) await chooseProvider.click();
}
await openAdvanced(page, mode);
}
}
}
await input.checkpoint("connection_creation");
if (advanced) await submitConnection(input, () => fillGateway(input));
else if (mode === "api-key") {
const savedKey = page.getByRole("combobox", { name: "Saved API key", exact: true });
if (await savedKey.isVisible()) await savedKey.selectOption("");
await page.getByLabel("API key", { exact: true }).fill(input.secret!);
await submitConnection(input, () => page.getByRole("button", { name: "Connect", exact: true }).last().click());
}
const connection = await waitForFreshConnection(input, before);
evidence.connectionId = connection.id;
evidence.checkpoints.connection_created = true;
evidence.waits.forEach(wait => { wait.finishedAt ??= new Date().toISOString(); });
if (advanced) {
const expectedKind = ["bedrock", "openrouter"].includes(mode) ? mode : "gateway";
if (connection.routing?.kind !== expectedKind || (mode !== "openrouter" && connection.routing?.protocol !== mode) ||
(input.settings.route?.baseURL && connection.routing?.baseUrl !== input.settings.route.baseURL) ||
(mode === "bedrock" && connection.routing?.region !== input.settings.route?.region)) throw new ConnectionFailure("saved_routing_mismatch");
} else if (connection.provider !== harness.provider) throw new ConnectionFailure("saved_provider_mismatch");
// Navigate away and return through the normal list: proof includes durable UI state.
await page.goto(`${api.origin}/${company.issuePrefix}/apps`, { waitUntil: "domcontentloaded" });
await expect(page.getByText(connection.name, { exact: true }).first()).toBeVisible();
await page.reload({ waitUntil: "domcontentloaded" });
await expect(page.getByText(connection.name, { exact: true }).first()).toBeVisible();
evidence.checkpoints.connection_reloaded = true;
await input.checkpoint("connection_reloaded");
await page.goto(agentURL, { waitUntil: "domcontentloaded" });
await input.checkpoint("select_saved_connection");
await selectSavedConnection(page, connection, harness, mode);
await input.checkpoint("configure_agent");
return connection;
}
async function selectModel(page: Page, model: string) {
const select = page.getByRole("combobox", { name: "Model", exact: true });
await select.waitFor({ state: "visible" });
await expect(select).not.toHaveAttribute("aria-busy", "true", { timeout: 60_000 });
const values = await select.locator("option").evaluateAll(options => options.map(option => (option as HTMLOptionElement).value));
const matched = values.find(value => value === model || value === `openrouter/${model}`);
if (matched) await select.selectOption(matched);
else {
await select.selectOption({ label: "Enter custom model…" });
await page.getByLabel("Model ID", { exact: true }).fill(model);
}
}
export async function runConnectionFlow(input: ConnectionFlowInput) {
const { page, api, execution, evidence, settings, company, environment, config } = input;
const { harness, mode } = connectionCell(execution);
const agentName = `Connection QA ${input.nonce}`;
const agentURL = `${api.origin}/${company.issuePrefix}/agents/new?${new URLSearchParams({ name: agentName, adapterType: execution.profile.adapterType, runnerProvider: harness.id })}`;
const connection = await createConnectionThroughUi(input, agentURL);
await expect(page.getByRole("heading", { name: "Configure your agent", exact: true })).toBeVisible({ timeout: 120_000 });
const environmentSelect = page.getByRole("combobox", { name: "Environment", exact: true });
const selectedEnv = await environmentSelect.inputValue();
if (selectedEnv !== environment.id) {
if (!(await environmentSelect.isEnabled())) throw new ConnectionBlock("blocked_target", "requested_execution_environment_unavailable");
await environmentSelect.selectOption(environment.id);
if (["claude", "codex", "grok"].includes(harness.id)) await selectSavedConnection(page, connection, harness, mode);
}
if (await environmentSelect.inputValue() !== environment.id) throw new ConnectionBlock("blocked_target", "requested_execution_environment_unavailable");
await input.checkpoint("select_model");
await selectModel(page, settings.model);
await input.checkpoint("setup_probe");
const response = page.waitForResponse(response => response.url().startsWith(`${api.origin}/api/companies/${company.id}/adapters/`) && response.url().endsWith("/test-environment"), { timeout: 120_000 });
await page.getByRole("button", { name: /^(Run test|Test again)$/ }).click();
const probe = await response;
const result = await probe.json();
evidence.setupChecks = connectionProbeChecks(result);
if (evidence.setupChecks.some(check => check.level === "error" && (check.code === "hermes_cli_not_found" || /^(claude|codex|grok|opencode|gemini)_command_unresolvable$/.test(check.code))))
throw new ConnectionBlock("blocked_target", "runtime_cli_required");
if (!probe.ok() || result.status !== "pass") throw new ConnectionFailure("setup_probe_failed");
evidence.checkpoints.setup_probe = true;
await page.getByRole("button", { name: "Finish setup", exact: true }).click();
let agent: Row | undefined;
const deadline = Date.now() + 60_000;
while (!agent && Date.now() < deadline) {
agent = (await api.get<Row[]>(`/api/companies/${company.id}/agents`)).find(row => row.name === agentName);
if (!agent) await connectionDelay(500);
}
if (!agent) throw new ConnectionFailure("agent_creation_failed");
if (agent.defaultEnvironmentId !== environment.id) {
// A null override is valid only when the instance default is this exact environment.
const instance = await api.get<Row>("/api/instance/settings");
const effective = agent.defaultEnvironmentId ?? instance.defaultEnvironmentId
?? (environment.driver === "local" ? environment.id : null);
if (effective !== environment.id) throw new ConnectionFailure("saved_agent_environment_mismatch");
}
evidence.agentId = agent.id;
evidence.checkpoints.agent_created = true;
const expectedModel = connection.routing
? aiRoutingModel(connection.routing, harness.adapter, settings.model)
: settings.model;
if (agent.adapterType !== execution.profile.adapterType || agent.adapterConfig?.model !== expectedModel) throw new ConnectionFailure("agent_harness_or_model_mismatch");
const binding = agent.runtimeConfig?.aiConnection;
if (!binding || binding.method !== connection.method || (binding.mode !== "responsible_user" && binding.connectionId !== connection.id)) throw new ConnectionFailure("agent_binding_mismatch");
// Fixtures may constrain cost/scheduling, but cannot repair or replace the tested credentials/config.
await api.patch(`/api/agents/${agent.id}`, { budgetMonthlyCents: config.budgetCents });
evidence.checkpoints.binding = true;
await input.checkpoint("first_task");
const proof = createConnectionProof();
// Provider qualification must not depend on task-level attachment ingestion.
// Native runners admit chat attachments, while the new-task dialog uploads
// its files after scheduling. Supply identical oracle bytes in the task so
// every harness can decode them with its actual tools.
evidence.inputTransport = "prompt_base64";
const title = `Connection artifact ${input.nonce}`;
const createdTask = await createTaskThroughUi({ page, issuePrefix: company.issuePrefix, agentName, title, workMode: "standard", prompt: [
"Complete this bounded connection acceptance task. Using your actual tools, decode the following base64 into connection-input.json and read the file. Preserve the exact decoded bytes, including the final newline.",
`Input bytes (base64): ${Buffer.from(proof.input, "utf8").toString("base64")}`,
"Create the input and output files inside your current workspace using relative filenames. Do not put them in /tmp or another directory outside the workspace.",
'Write connection-proof.json with exactly this schema: {"nonce": string, "count": number, "total": number, "sha256": string}. Set nonce from the input, count to the number of values, total to their sum, and sha256 to the hash of the exact input file bytes. All four fields are required; use the literal field name "total", not "sum". Compute and validate with a tool; do not guess or paraphrase the data.',
"Deliver connection-proof.json as a downloadable attachment on this task. Use the normal Paperclip artifact/attachment workflow and include its download link in your final response. Then mark this task done. Do not create other tasks, read credentials, or inspect unrelated files.",
execution.profile.generation === "native" ? "Use your native register_deliverable tool with workspace-relative contentRef connection-proof.json and the exact byteSize and SHA-256 of that output file. Never submit an absolute contentRef." : "",
].join("\n") });
const issue = await api.get<Row>(`/api/issues/${createdTask.issueId}`);
if (issue.companyId !== company.id) throw new ConnectionFailure("task_creation_failed");
const runs = new Map<string, Row>();
async function completedRun(after: Set<string>, filename: string) {
const deadline = Date.now() + config.turnTimeoutMs;
while (Date.now() < deadline) {
input.assertActive();
const summaries = await api.get<Row[]>(`/api/issues/${issue!.id}/runs`);
for (const row of summaries) {
const id = row.runId ?? row.id;
if (id && !runs.has(id)) runs.set(id, await api.get(`/api/heartbeat-runs/${id}`));
else if (id && !["succeeded", "failed", "cancelled", "interrupted", "timed_out"].includes(runs.get(id)!.status)) runs.set(id, await api.get(`/api/heartbeat-runs/${id}`));
if (id) input.observeRun(runs.get(id)!);
}
if (runs.size > config.maxRuns) throw new ConnectionFailure("provider_run_limit_exceeded");
const next = [...runs.values()].filter(run => !after.has(run.id));
if (next.some(run => ["failed", "cancelled", "interrupted", "timed_out"].includes(run.status))) throw new ConnectionFailure("agent_run_failed");
const complete = next.find(run => verifyConnectionRun(run, { agentId: agent!.id, connectionId: connection.id, method: connection.method, runtimeMode: execution.profile.generation, environmentId: environment.id }));
if (complete) {
const status = (await api.get(`/api/issues/${issue!.id}`)).status;
if (status === "done") {
// A disposition repair may still be running after an earlier turn
// succeeded. Qualify the settled successful run that uploaded this
// artifact, preserving the configured run and time limits.
const attachments = await api.get<Row[]>(`/api/issues/${issue!.id}/attachments`);
const delivered = completedConnectionArtifactRun(next, attachments, filename,
{ agentId: agent!.id, connectionId: connection.id, method: connection.method, runtimeMode: execution.profile.generation, environmentId: environment.id });
if (delivered) return delivered;
if (next.every(run => ["succeeded", "failed", "cancelled", "interrupted", "timed_out"].includes(run.status))) throw new ConnectionFailure("run_attributed_artifact_missing");
}
if (status === "blocked") throw new ConnectionFailure("task_blocked_after_provider_run");
}
if (next.some(run => run.status === "succeeded" && !verifyConnectionRun(run, { agentId: agent!.id, connectionId: connection.id, method: connection.method, runtimeMode: execution.profile.generation, environmentId: environment.id }))) throw new ConnectionFailure("run_attribution_mismatch");
await connectionDelay(1500);
}
throw new ConnectionFailure("agent_run_or_attribution_timeout");
}
async function artifact(filename: string, expected: Record<string, unknown>, run: Row) {
const attachments = await api.get<Row[]>(`/api/issues/${issue!.id}/attachments`);
const attachment = attachments.find(row => row.originalFilename === filename && row.createdByAgentId === agent!.id && row.originatingRunId === run.id);
if (!attachment) throw new ConnectionFailure("run_attributed_artifact_missing");
const bytes = await api.bytes(`/api/attachments/${attachment.id}/content`);
let actual: Record<string, unknown> = {};
try { actual = JSON.parse(bytes.toString("utf8")); } catch { /* The independent verifier fails below. */ }
evidence.artifactChecks ??= [];
evidence.artifactChecks.push({ filename, runId: run.id, sha256: createHash("sha256").update(bytes).digest("hex"),
fields: Object.fromEntries(Object.entries(expected).map(([key, value]) => [key, actual?.[key] === value])), exactFields: Boolean(actual && Object.keys(actual).length === Object.keys(expected).length) });
if (!verifyConnectionArtifact(bytes, expected)) throw new ConnectionFailure("independent_artifact_verification_failed");
await page.goto(`${api.origin}/${company.issuePrefix}/issues/${issue!.identifier}`, { waitUntil: "domcontentloaded" });
await page.reload({ waitUntil: "domcontentloaded" });
await expect(page.locator(`a[href*="/api/attachments/${attachment.id}/content"]`).first()).toBeVisible({ timeout: 30_000 });
evidence.artifactSha256 = createHash("sha256").update(bytes).digest("hex");
}
const first = await completedRun(new Set(), "connection-proof.json");
evidence.checkpoints.first_run = true;
await artifact("connection-proof.json", proof.expected, first);
evidence.checkpoints.artifact = true;
// Artifact upload provenance is a server-owned tool-side effect in both runner generations.
const events = await collectRunEvents<Row>((afterSeq, limit) => api.get(`/api/heartbeat-runs/${first.id}/events?afterSeq=${afterSeq}&limit=${limit}`));
const toolReceipt = events.some(event => event.eventType === "tool.execution.completed" || (event.eventType === "item.completed" && ["tool_result", "command_execution"].includes(event.payload?.prpEvent?.payload?.item?.type)));
// Legacy adapters may expose their tool transcript only in the run log; upload attribution is still a durable successful tool effect.
evidence.checkpoints.tool_receipt = execution.profile.generation === "legacy" || toolReceipt;
if (!evidence.checkpoints.tool_receipt) throw new ConnectionFailure("native_tool_receipt_missing");
await input.checkpoint("followup");
const previousRuns = new Set(runs.keys());
await submitTaskReply(page, "Using the original connection-input.json decoded from this task, deliver connection-followup.json with exactly nonce, minimum, maximum, and total of the original values. Use exactly this JSON schema: {\"nonce\": string, \"minimum\": number, \"maximum\": number, \"total\": number}. All four fields are required. Compute with a tool and validate the file against all four fields before attaching it and its download link. Then mark this task done. Do not create other tasks or read credentials.");
const second = await completedRun(previousRuns, "connection-followup.json");
await artifact("connection-followup.json", proof.followup, second);
evidence.checkpoints.followup = true;
evidence.checkpoints.visible_result = true;
return { issueId: issue.id as string, issueIdentifier: issue.identifier as string };
}