mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-11 14:10:50 +02:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Users connect accounts and choose an agent harness and model. > - The runtime change in #14970 supports custom providers on those connections. > - Normal setup must stay simple while advanced users can choose a compatible gateway. > - Shared connector rows and access controls keep these choices consistent. > - This pull request refines the agent setup UI and adds review stories and repeatable browser qualification. > - The qualification checks real tools and downloaded outputs, not only a successful run status. ## Linked Issues or Issue Description Refs #14970, #37, #13083, #14104, #14565, #12692. The core implementation in #14970 is merged. This branch incorporates its squash commit and targets `master`. Both PRs contain our implementation. #14016 is a reference only and is not a dependency. This PR has 96 changed files. ## What Changed - Complete model-provider connector presentation beside other connectors. Each row uses the existing Connect action and connection list. Tags are stored without category UI. The base PR includes the provider forms and routes. - Show persistent Subscription, API Key, and Advanced choices. Label Advanced as Custom Gateway. Reuse provider logos, connection lists, and permissions controls. Default access to the organization and all agents when permitted; keep narrowing controls under Advanced. - Keep Configure reachable before subscription sign-in, so users can select a supported environment when the default cannot sign in. Testing and saving still require a connection. Show the execution environment in Configure. Preserve the confirmed Connect choice. Editing a method, credential, saved account, or advanced choice requires that current choice to connect before testing or saving. Use matching model and thinking-effort dropdowns and retain connection icons in selected values. - Preserve the new harness model default when switching an existing OpenCode agent to Codex or Claude, and resolve user-selected model names with the effective harness. - Load popular OpenRouter models through the shared connection-model discovery path. Keep explicit model lists and manual model entry available. - Group onboarding, connection setup, agent runtime, management, recovery, and production-component stories under AI Connections / Provider routing. - Add an explicit-only provider-connections browser suite for managed local or existing local/staging targets. Use private browser profiles and credential handoffs. Support human-assisted subscription sign-in without sharing passwords or tokens in reports. - Verify persisted connection identity, runtime probes, tool execution, exact artifact bytes, completion, and context-dependent follow-up. Retain source/model provenance, cost bounds, closed error diagnostics, original failures, and cleanup evidence. - Add Gemini startup-model and skill-root fixes, Grok private-history detection, ACP filesystem regression fixtures, selected-workspace handling for local Hermes, and artifact-helper workspace fallback. - Keep managed Grok runtime homes disposable. Remove host-side transcript retention/restoration because private file modes do not isolate same-user agent processes. Ignore earlier development archives and use a fresh task handoff when history is unavailable. Verify the absence of restored transcripts with a separate same-user process. - Capture stopped-run diagnostics before deleting an attached-company fixture agent. Track creation and owned sign-in receipts; revoke only this attempt's accounts and never adopt a concurrent campaign's newly created account. Preserve failure signals and final status through cleanup. - Require the requested environment in the saved agent and every run, including follow-ups. Reject a forced incompatible target. Keep one cancellation state through startup, every cell, reporting, and teardown for SIGINT, SIGTERM, and SIGHUP. Stop further paid cells after interruption. Document qualification limits. ## Verification - Current head `b3bb3e94d577d43d9965a6b9daba039f599b2e49` includes master `d9f600043`. The security fix in `a758fde31` passes full workspace typecheck, production build, and 119 connection/Grok regressions. The unchanged UI passes all 126 configuration/model-discovery tests and token gates. The final published-guide correction passes Grok adapter typecheck. Earlier head `eebd8225c` passed the complete deterministic runner suite (1,404 Vitest tests and 128 Node tests) and all CI jobs. Current-head CI run `37520147514` passed all 47 jobs, including the full sharded Vitest and browser matrix, production build, and canary dry run. All 55 checks completed: 53 successes and two expected skips. The current-head security scan passed, Greptile is 5/5, and no review threads remain open. - A separate same-user process reproduced reading a restored Grok transcript before the security fix. The regression now finds no transcript. Existing fresh-session fallback and ordinary session metadata behavior pass. - The final account-choice and cleanup fixes pass 85 setup tests and 26 qualification-harness tests. Regressions verify that editing a connection invalidates confirmation, Configure remains reachable before sign-in, diagnostics are captured before fixture deletion, and concurrent campaigns cannot adopt or revoke each other's accounts. UI and E2E typechecks pass. - The Storybook build and actual Chromium production-component stories passed during this change. Review the neighboring AI Connections / Provider routing stories, regular connector rows, three connection modes, model discovery, and the single execution-environment control in Configure. - Cancellation smoke verified authenticated cleanup before browser close for SIGINT, SIGTERM, and SIGHUP. Regressions cover interruption during startup and reporting, missing-file ACP resource errors, and preserved permission denials. Both ACP runtime versions and 54 ACPX/Grok regressions passed. The deterministic connection-intent browser suite passed two tests. - Historical local qualification retained 43 passing API/gateway cells out of 46, with downloaded outputs and follow-up receipts. These attempts span earlier builds; they do not qualify this exact commit or staging. Subscription combinations, Gemini overloads, and the unresolved follow-up failure remain recorded rather than counted as passing. - Use `pnpm test:e2e:runner -- --list --suite provider-connections` to inspect the matrix. Follow `tests/runner-e2e/PROVIDER-CONNECTIONS.md` for credentials, target URL, sign-in assistance, budget, evidence, and cleanup. Paid live tests remain opt-in. ## Risks - The core implementation in #14970 is merged. This PR adds no database migration of its own. - Subscription login needs an interactive provider session. Dedicated accounts and staging qualification remain follow-up work; this PR does not certify every login combination for production. - Managed Grok transcript resume is deferred until provider history has an OS isolation or authorized broker solution. Follow-ups start fresh with Paperclip task context; earlier live Grok results do not qualify this behavior. - Gemini CLI 0.58.0 has an upstream ACP new-file error conversion defect. Live overloads and one unresolved follow-up timeout remain recorded. The stock CLI is unchanged, and those cases are not marked as passing. - Real-provider tests spend credits and use private credential/evidence directories. The launcher requires explicit selection and checks target ownership. It must not attach to a developer's database by accident. - OpenClaw Gateway, Hermes Gateway, Claude Managed, AWS AgentCore, Process, HTTP, and legacy ACPX local remain outside custom provider setup. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, code execution, and browser testing. The exact deployment model ID and context window size were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
378 lines
27 KiB
TypeScript
378 lines
27 KiB
TypeScript
import { createHash } from "node:crypto";
|
|
import { expect, type Page } from "@playwright/test";
|
|
import { aiRoutingModel } from "../../packages/shared/src/ai-provider-routing.js";
|
|
import { getConnectableAppDefinition } from "../../packages/shared/src/app-definitions.js";
|
|
import { connectionCell, type ConnectionHarness, type ConnectionMode } from "./connection-cases.js";
|
|
import { ConnectionBlock, ConnectionFailure, type ConnectionConfig, type resolveConnectionSettings } from "./connection-config.js";
|
|
import { ConnectionApi, connectionDelay } from "./connection-target.js";
|
|
import { createConnectionProof, verifyConnectionArtifact, verifyConnectionRun, completedConnectionArtifactRun, connectionProbeChecks, type ConnectionEvidence } from "./connection-evidence.js";
|
|
import { createTaskThroughUi, submitTaskReply } from "./user-actions.js";
|
|
import { collectRunEvents } from "./run-observations.js";
|
|
import type { MatrixExecution } from "./types.js";
|
|
|
|
type Row = Record<string, any>;
|
|
type Settings = ReturnType<typeof resolveConnectionSettings>;
|
|
export interface ConnectionFlowInput {
|
|
page: Page; api: ConnectionApi; execution: MatrixExecution; config: ConnectionConfig; settings: Settings;
|
|
company: Row; environment: Row; secret?: string; nonce: string; privateDir: string;
|
|
evidence: ConnectionEvidence;
|
|
checkpoint(phase: string): Promise<void>;
|
|
observeRun(run: Row): void;
|
|
createdConnectionIds: ReadonlySet<string>;
|
|
refreshOwnedConnections(): Promise<void>;
|
|
assertActive(): void;
|
|
}
|
|
|
|
async function choose(page: Page, label: string, name: string) {
|
|
await page.getByRole("combobox", { name: label, exact: true }).click();
|
|
await page.getByRole("option", { name, exact: true }).click();
|
|
}
|
|
|
|
async function openAdvanced(page: Page, mode: ConnectionMode) {
|
|
const names: Record<string, string> = { openrouter: "OpenRouter", bedrock: "Amazon Bedrock", responses: "Custom gateway", messages: "Custom gateway", chat: "Custom gateway" };
|
|
await page.getByRole("button", { name: new RegExp(`^${names[mode]}`) }).click();
|
|
}
|
|
|
|
async function fillGateway(input: ConnectionFlowInput) {
|
|
const { page, settings, secret } = input;
|
|
const { mode } = connectionCell(input.execution);
|
|
if (settings.route?.baseURL) {
|
|
await page.getByLabel("Provider URL", { exact: true }).fill(settings.route.baseURL);
|
|
const format = page.getByRole("combobox", { name: "API format", exact: true });
|
|
if (await format.isEnabled()) await choose(page, "API format", ({ responses: "OpenAI Responses · Codex", messages: "Anthropic Messages · Claude", chat: "Chat Completions · OpenCode, Hermes" } as Record<string, string>)[mode]!);
|
|
}
|
|
if (mode === "bedrock") await page.getByLabel("AWS region", { exact: true }).fill(settings.route!.region!);
|
|
if (settings.route?.auth !== "bearer") await choose(page, "Authentication", settings.route?.auth === "none" ? "No authentication" : "API key · x-api-key");
|
|
if (secret) await page.getByLabel("API key", { exact: true }).fill(secret);
|
|
// Normal defaults stay collapsed. Model alias is selected in Configure.
|
|
await page.getByRole("button", { name: "Connect", exact: true }).last().click();
|
|
}
|
|
|
|
async function submitConnection(input: ConnectionFlowInput, submit: () => Promise<void>) {
|
|
const route = `${input.api.origin}/api/companies/${input.company.id}/ai-connections`;
|
|
const response = input.page.waitForResponse(response => response.url() === route && response.request().method() === "POST", { timeout: 60_000 }).catch(() => null);
|
|
await submit();
|
|
const created = await response;
|
|
if (!created) throw new ConnectionFailure("connection_request_timeout");
|
|
if (created.ok()) return;
|
|
const payload = await created.json().catch(() => ({}));
|
|
const allowed = ["ai_connection_api_key_rejected", "ai_connection_verification_failed"];
|
|
const code = allowed.includes(payload?.details?.code) ? payload.details.code : "connection_request_rejected";
|
|
input.evidence.creationFailure = { status: created.status(), code };
|
|
if (code === "ai_connection_api_key_rejected") throw new ConnectionBlock("missing_credential", code);
|
|
throw new ConnectionFailure(code);
|
|
}
|
|
|
|
async function waitForFreshConnection(input: ConnectionFlowInput, before: Set<string>) {
|
|
const { page, api, config, evidence, company } = input;
|
|
const { harness, mode } = connectionCell(input.execution);
|
|
const deadline = Date.now() + (mode === "subscription" ? config.browser.loginTimeoutMs : 90_000);
|
|
let loginOpened = false;
|
|
let connected = false;
|
|
while (Date.now() < deadline) {
|
|
input.assertActive();
|
|
await input.refreshOwnedConnections();
|
|
const list = await api.get(`/api/companies/${company.id}/ai-connections`);
|
|
const fresh = list.connections.filter((c: Row) => !before.has(c.id) && input.createdConnectionIds.has(c.id));
|
|
if (fresh.length > 1) throw new ConnectionFailure("unexpected_multiple_connections");
|
|
if (fresh.length === 1) {
|
|
const connection = fresh[0];
|
|
if (connection.status !== "connected" || connection.method !== (mode === "subscription" ? "subscription" : "api_key")) throw new ConnectionFailure("saved_connection_method_or_status_mismatch");
|
|
return connection;
|
|
}
|
|
if (mode === "subscription") {
|
|
const signIn = page.getByRole("link", { name: /^Sign in to / }).first();
|
|
const cloudSignIn = page.getByRole("button", { name: /^Sign in to / }).first();
|
|
if (!loginOpened && (await signIn.isVisible() || await cloudSignIn.isVisible())) {
|
|
// Observe before clicking. No token, code, URL, or popup content is recorded.
|
|
const popup = page.context().waitForEvent("page", { timeout: 10_000 }).catch(() => undefined);
|
|
if (await signIn.isVisible()) await signIn.click(); else if (await cloudSignIn.isEnabled()) await cloudSignIn.click();
|
|
await popup;
|
|
loginOpened = true;
|
|
evidence.assisted = true;
|
|
evidence.waits.push({ kind: "provider_login", startedAt: new Date().toISOString() });
|
|
await input.checkpoint("awaiting_provider_login");
|
|
console.log(`[provider-connections] Sign in to ${harness.label} in the opened browser (${config.browser.accountAlias}); complete the code/consent step in Paperclip. This test resumes automatically.`);
|
|
}
|
|
// Grok currently exposes a terminal handoff; do not pretend it has the same browser flow.
|
|
if (!loginOpened && harness.id === "grok" && await page.getByRole("button", { name: "Copy sign-in command" }).isVisible()) {
|
|
loginOpened = true;
|
|
evidence.assisted = true;
|
|
evidence.waits.push({ kind: "provider_login", startedAt: new Date().toISOString() });
|
|
await input.checkpoint("awaiting_provider_login");
|
|
console.log("[provider-connections] Grok requires the sign-in command shown in Paperclip. Complete that production handoff; the test will resume automatically.");
|
|
}
|
|
const connect = page.getByRole("button", { name: "Connect", exact: true }).last();
|
|
if (!connected && await connect.isVisible() && await connect.isEnabled()) { await connect.click(); connected = true; }
|
|
if (await page.getByRole("alert").filter({ hasText: /expired|could not|failed|invalid/i }).count()) throw new ConnectionFailure("provider_login_reported_error");
|
|
}
|
|
await connectionDelay(1_000);
|
|
}
|
|
if (mode === "subscription" && loginOpened) throw new ConnectionBlock("awaiting_user", "provider_login_deadline_reached");
|
|
throw new ConnectionFailure("connection_creation_timeout");
|
|
}
|
|
|
|
async function selectSavedConnection(page: Page, connection: Row, harness: ConnectionHarness, mode: ConnectionMode) {
|
|
await page.getByRole("heading", { name: /^(Connect a model|Configure your agent)$/ }).waitFor();
|
|
const advanced = !["subscription", "api-key"].includes(mode);
|
|
const connectionType = page.getByRole("radiogroup", { name: "Connection type", exact: true });
|
|
if (["claude", "codex", "grok"].includes(harness.id)) {
|
|
await connectionType.waitFor({ state: "visible" });
|
|
if (advanced) {
|
|
await connectionType.getByRole("radio", { name: /Advanced/ }).click();
|
|
await choose(page, "Connection", connection.name);
|
|
await page.getByRole("button", { name: "Use connection", exact: true }).click();
|
|
} else {
|
|
await connectionType.getByRole("radio", { name: mode === "subscription" ? /Subscription/ : /API key/ }).click();
|
|
const saved = page.getByRole("combobox", { name: mode === "subscription" ? "Saved subscription" : "Saved API key", exact: true });
|
|
await saved.waitFor({ state: "visible" });
|
|
const option = await saved.locator("option").filter({ hasText: connection.name }).first().getAttribute("value");
|
|
if (!option) throw new ConnectionFailure("saved_connection_option_missing");
|
|
await saved.selectOption(option);
|
|
await page.getByRole("button", { name: mode === "subscription" ? "Use saved subscription" : "Use saved API key", exact: true }).click();
|
|
}
|
|
} else {
|
|
await choose(page, "Connection", connection.name);
|
|
}
|
|
}
|
|
|
|
async function createConnectionThroughUi(input: ConnectionFlowInput, agentURL: string) {
|
|
const { page, api, company, evidence } = input;
|
|
const { harness, mode, entry } = connectionCell(input.execution);
|
|
const priorConnections = (await api.get(`/api/companies/${company.id}/ai-connections`)).connections as Row[];
|
|
const before = new Set<string>(priorConnections.map(c => c.id));
|
|
// Fresh-creation proof cannot be satisfied by an existing personal default.
|
|
if (priorConnections.some(connection => connection.status !== "revoked")) throw new ConnectionBlock("blocked_target", "fresh_connection_requires_empty_qa_company");
|
|
const advanced = !["subscription", "api-key"].includes(mode);
|
|
await input.checkpoint("connection_navigation");
|
|
if (entry === "apps") {
|
|
await page.goto(`${api.origin}/${company.issuePrefix}/apps`, { waitUntil: "domcontentloaded" });
|
|
const catalogSlug = advanced ? ({ openrouter: "openrouter", bedrock: "bedrock", responses: "responses-api", messages: "messages-api", chat: "chat-completions-api" } as Record<string, string>)[mode]! : harness.provider;
|
|
const definition = getConnectableAppDefinition(catalogSlug);
|
|
if (!definition) throw new ConnectionFailure("connection_catalog_entry_missing");
|
|
await page.getByRole("button", { name: `Connect ${definition.name}`, exact: true }).click();
|
|
const next = page.getByRole("button", { name: /^(Save and continue|Continue)$/ }).last();
|
|
if (!advanced && await next.isVisible()) await next.click();
|
|
if (!advanced && harness.provider !== "google") {
|
|
// The credential step loads asynchronously. Wait for its provider tile
|
|
// before inspecting the mode switch, otherwise a still-loading page can
|
|
// skip the API-key switch and leave the subscription form selected.
|
|
await page.getByRole("radio", { name: new RegExp(harness.id === "grok" ? "Grok" : harness.label) }).click();
|
|
const switcher = page.getByRole("button", { name: mode === "subscription" ? "Use subscription instead" : "Use API key instead", exact: true });
|
|
if (await switcher.isVisible()) await switcher.click();
|
|
}
|
|
} else {
|
|
await page.goto(agentURL, { waitUntil: "domcontentloaded" });
|
|
const tiles = page.getByRole("radiogroup", { name: "Connection type", exact: true });
|
|
if (["claude", "codex", "grok"].includes(harness.id)) {
|
|
const choice = tiles.getByRole("radio", { name: advanced ? /Advanced/ : mode === "subscription" ? /Subscription/ : /API key/ });
|
|
await choice.click();
|
|
await expect(choice).toHaveAttribute("aria-checked", "true");
|
|
}
|
|
if (advanced || !["claude", "codex", "grok"].includes(harness.id)) {
|
|
await choose(page, "Connection", "Connect an account…");
|
|
if (advanced) {
|
|
const advancedProviders = page.locator("summary").filter({ hasText: "Advanced providers" });
|
|
if (await advancedProviders.isVisible()) {
|
|
await advancedProviders.click();
|
|
// The account dialog's disclosure leads to the provider chooser;
|
|
// the chooser's own disclosure already contains provider buttons.
|
|
const chooseProvider = page.getByRole("button", { name: "Choose another provider or gateway", exact: true });
|
|
if (await chooseProvider.isVisible()) await chooseProvider.click();
|
|
}
|
|
await openAdvanced(page, mode);
|
|
}
|
|
}
|
|
}
|
|
await input.checkpoint("connection_creation");
|
|
if (advanced) await submitConnection(input, () => fillGateway(input));
|
|
else if (mode === "api-key") {
|
|
const savedKey = page.getByRole("combobox", { name: "Saved API key", exact: true });
|
|
if (await savedKey.isVisible()) await savedKey.selectOption("");
|
|
await page.getByLabel("API key", { exact: true }).fill(input.secret!);
|
|
await submitConnection(input, () => page.getByRole("button", { name: "Connect", exact: true }).last().click());
|
|
}
|
|
const connection = await waitForFreshConnection(input, before);
|
|
evidence.connectionId = connection.id;
|
|
evidence.checkpoints.connection_created = true;
|
|
evidence.waits.forEach(wait => { wait.finishedAt ??= new Date().toISOString(); });
|
|
if (advanced) {
|
|
const expectedKind = ["bedrock", "openrouter"].includes(mode) ? mode : "gateway";
|
|
if (connection.routing?.kind !== expectedKind || (mode !== "openrouter" && connection.routing?.protocol !== mode) ||
|
|
(input.settings.route?.baseURL && connection.routing?.baseUrl !== input.settings.route.baseURL) ||
|
|
(mode === "bedrock" && connection.routing?.region !== input.settings.route?.region)) throw new ConnectionFailure("saved_routing_mismatch");
|
|
} else if (connection.provider !== harness.provider) throw new ConnectionFailure("saved_provider_mismatch");
|
|
// Navigate away and return through the normal list: proof includes durable UI state.
|
|
await page.goto(`${api.origin}/${company.issuePrefix}/apps`, { waitUntil: "domcontentloaded" });
|
|
await expect(page.getByText(connection.name, { exact: true }).first()).toBeVisible();
|
|
await page.reload({ waitUntil: "domcontentloaded" });
|
|
await expect(page.getByText(connection.name, { exact: true }).first()).toBeVisible();
|
|
evidence.checkpoints.connection_reloaded = true;
|
|
await input.checkpoint("connection_reloaded");
|
|
await page.goto(agentURL, { waitUntil: "domcontentloaded" });
|
|
await input.checkpoint("select_saved_connection");
|
|
await selectSavedConnection(page, connection, harness, mode);
|
|
await input.checkpoint("configure_agent");
|
|
return connection;
|
|
}
|
|
|
|
async function selectModel(page: Page, model: string) {
|
|
const select = page.getByRole("combobox", { name: "Model", exact: true });
|
|
await select.waitFor({ state: "visible" });
|
|
await expect(select).not.toHaveAttribute("aria-busy", "true", { timeout: 60_000 });
|
|
const values = await select.locator("option").evaluateAll(options => options.map(option => (option as HTMLOptionElement).value));
|
|
const matched = values.find(value => value === model || value === `openrouter/${model}`);
|
|
if (matched) await select.selectOption(matched);
|
|
else {
|
|
await select.selectOption({ label: "Enter custom model…" });
|
|
await page.getByLabel("Model ID", { exact: true }).fill(model);
|
|
}
|
|
}
|
|
|
|
export async function runConnectionFlow(input: ConnectionFlowInput) {
|
|
const { page, api, execution, evidence, settings, company, environment, config } = input;
|
|
const { harness, mode } = connectionCell(execution);
|
|
const agentName = `Connection QA ${input.nonce}`;
|
|
const agentURL = `${api.origin}/${company.issuePrefix}/agents/new?${new URLSearchParams({ name: agentName, adapterType: execution.profile.adapterType, runnerProvider: harness.id })}`;
|
|
const connection = await createConnectionThroughUi(input, agentURL);
|
|
await expect(page.getByRole("heading", { name: "Configure your agent", exact: true })).toBeVisible({ timeout: 120_000 });
|
|
const environmentSelect = page.getByRole("combobox", { name: "Environment", exact: true });
|
|
const selectedEnv = await environmentSelect.inputValue();
|
|
if (selectedEnv !== environment.id) {
|
|
if (!(await environmentSelect.isEnabled())) throw new ConnectionBlock("blocked_target", "requested_execution_environment_unavailable");
|
|
await environmentSelect.selectOption(environment.id);
|
|
if (["claude", "codex", "grok"].includes(harness.id)) await selectSavedConnection(page, connection, harness, mode);
|
|
}
|
|
if (await environmentSelect.inputValue() !== environment.id) throw new ConnectionBlock("blocked_target", "requested_execution_environment_unavailable");
|
|
await input.checkpoint("select_model");
|
|
await selectModel(page, settings.model);
|
|
await input.checkpoint("setup_probe");
|
|
const response = page.waitForResponse(response => response.url().startsWith(`${api.origin}/api/companies/${company.id}/adapters/`) && response.url().endsWith("/test-environment"), { timeout: 120_000 });
|
|
await page.getByRole("button", { name: /^(Run test|Test again)$/ }).click();
|
|
const probe = await response;
|
|
const result = await probe.json();
|
|
evidence.setupChecks = connectionProbeChecks(result);
|
|
if (evidence.setupChecks.some(check => check.level === "error" && (check.code === "hermes_cli_not_found" || /^(claude|codex|grok|opencode|gemini)_command_unresolvable$/.test(check.code))))
|
|
throw new ConnectionBlock("blocked_target", "runtime_cli_required");
|
|
if (!probe.ok() || result.status !== "pass") throw new ConnectionFailure("setup_probe_failed");
|
|
evidence.checkpoints.setup_probe = true;
|
|
await page.getByRole("button", { name: "Finish setup", exact: true }).click();
|
|
let agent: Row | undefined;
|
|
const deadline = Date.now() + 60_000;
|
|
while (!agent && Date.now() < deadline) {
|
|
agent = (await api.get<Row[]>(`/api/companies/${company.id}/agents`)).find(row => row.name === agentName);
|
|
if (!agent) await connectionDelay(500);
|
|
}
|
|
if (!agent) throw new ConnectionFailure("agent_creation_failed");
|
|
if (agent.defaultEnvironmentId !== environment.id) {
|
|
// A null override is valid only when the instance default is this exact environment.
|
|
const instance = await api.get<Row>("/api/instance/settings");
|
|
const effective = agent.defaultEnvironmentId ?? instance.defaultEnvironmentId
|
|
?? (environment.driver === "local" ? environment.id : null);
|
|
if (effective !== environment.id) throw new ConnectionFailure("saved_agent_environment_mismatch");
|
|
}
|
|
evidence.agentId = agent.id;
|
|
evidence.checkpoints.agent_created = true;
|
|
const expectedModel = connection.routing
|
|
? aiRoutingModel(connection.routing, harness.adapter, settings.model)
|
|
: settings.model;
|
|
if (agent.adapterType !== execution.profile.adapterType || agent.adapterConfig?.model !== expectedModel) throw new ConnectionFailure("agent_harness_or_model_mismatch");
|
|
const binding = agent.runtimeConfig?.aiConnection;
|
|
if (!binding || binding.method !== connection.method || (binding.mode !== "responsible_user" && binding.connectionId !== connection.id)) throw new ConnectionFailure("agent_binding_mismatch");
|
|
// Fixtures may constrain cost/scheduling, but cannot repair or replace the tested credentials/config.
|
|
await api.patch(`/api/agents/${agent.id}`, { budgetMonthlyCents: config.budgetCents });
|
|
evidence.checkpoints.binding = true;
|
|
await input.checkpoint("first_task");
|
|
const proof = createConnectionProof();
|
|
// Provider qualification must not depend on task-level attachment ingestion.
|
|
// Native runners admit chat attachments, while the new-task dialog uploads
|
|
// its files after scheduling. Supply identical oracle bytes in the task so
|
|
// every harness can decode them with its actual tools.
|
|
evidence.inputTransport = "prompt_base64";
|
|
const title = `Connection artifact ${input.nonce}`;
|
|
await createTaskThroughUi({ page, issuePrefix: company.issuePrefix, agentName, title, workMode: "standard", prompt: [
|
|
"Complete this bounded connection acceptance task. Using your actual tools, decode the following base64 into connection-input.json and read the file. Preserve the exact decoded bytes, including the final newline.",
|
|
`Input bytes (base64): ${Buffer.from(proof.input, "utf8").toString("base64")}`,
|
|
"Create the input and output files inside your current workspace using relative filenames. Do not put them in /tmp or another directory outside the workspace.",
|
|
'Write connection-proof.json with exactly this schema: {"nonce": string, "count": number, "total": number, "sha256": string}. Set nonce from the input, count to the number of values, total to their sum, and sha256 to the hash of the exact input file bytes. All four fields are required; use the literal field name "total", not "sum". Compute and validate with a tool; do not guess or paraphrase the data.',
|
|
"Deliver connection-proof.json as a downloadable attachment on this task. Use the normal Paperclip artifact/attachment workflow and include its download link in your final response. Then mark this task done. Do not create other tasks, read credentials, or inspect unrelated files.",
|
|
execution.profile.generation === "native" ? "Use your native register_deliverable tool with workspace-relative contentRef connection-proof.json and the exact byteSize and SHA-256 of that output file. Never submit an absolute contentRef." : "",
|
|
].join("\n") });
|
|
let issue: Row | undefined;
|
|
const createdDeadline = Date.now() + 30_000;
|
|
while (!issue && Date.now() < createdDeadline) {
|
|
issue = (await api.get<Row[]>(`/api/companies/${company.id}/issues`)).find(row => row.title === title);
|
|
if (!issue) await connectionDelay(500);
|
|
}
|
|
if (!issue) throw new ConnectionFailure("task_creation_failed");
|
|
const runs = new Map<string, Row>();
|
|
async function completedRun(after: Set<string>, filename: string) {
|
|
const deadline = Date.now() + config.turnTimeoutMs;
|
|
while (Date.now() < deadline) {
|
|
input.assertActive();
|
|
const summaries = await api.get<Row[]>(`/api/issues/${issue!.id}/runs`);
|
|
for (const row of summaries) {
|
|
const id = row.runId ?? row.id;
|
|
if (id && !runs.has(id)) runs.set(id, await api.get(`/api/heartbeat-runs/${id}`));
|
|
else if (id && !["succeeded", "failed", "cancelled", "interrupted", "timed_out"].includes(runs.get(id)!.status)) runs.set(id, await api.get(`/api/heartbeat-runs/${id}`));
|
|
if (id) input.observeRun(runs.get(id)!);
|
|
}
|
|
if (runs.size > config.maxRuns) throw new ConnectionFailure("provider_run_limit_exceeded");
|
|
const next = [...runs.values()].filter(run => !after.has(run.id));
|
|
if (next.some(run => ["failed", "cancelled", "interrupted", "timed_out"].includes(run.status))) throw new ConnectionFailure("agent_run_failed");
|
|
const complete = next.find(run => verifyConnectionRun(run, { agentId: agent!.id, connectionId: connection.id, method: connection.method, runtimeMode: execution.profile.generation, environmentId: environment.id }));
|
|
if (complete) {
|
|
const status = (await api.get(`/api/issues/${issue!.id}`)).status;
|
|
if (status === "done") {
|
|
// A disposition repair may still be running after an earlier turn
|
|
// succeeded. Qualify the settled successful run that uploaded this
|
|
// artifact, preserving the configured run and time limits.
|
|
const attachments = await api.get<Row[]>(`/api/issues/${issue!.id}/attachments`);
|
|
const delivered = completedConnectionArtifactRun(next, attachments, filename,
|
|
{ agentId: agent!.id, connectionId: connection.id, method: connection.method, runtimeMode: execution.profile.generation, environmentId: environment.id });
|
|
if (delivered) return delivered;
|
|
if (next.every(run => ["succeeded", "failed", "cancelled", "interrupted", "timed_out"].includes(run.status))) throw new ConnectionFailure("run_attributed_artifact_missing");
|
|
}
|
|
if (status === "blocked") throw new ConnectionFailure("task_blocked_after_provider_run");
|
|
}
|
|
if (next.some(run => run.status === "succeeded" && !verifyConnectionRun(run, { agentId: agent!.id, connectionId: connection.id, method: connection.method, runtimeMode: execution.profile.generation, environmentId: environment.id }))) throw new ConnectionFailure("run_attribution_mismatch");
|
|
await connectionDelay(1500);
|
|
}
|
|
throw new ConnectionFailure("agent_run_or_attribution_timeout");
|
|
}
|
|
async function artifact(filename: string, expected: Record<string, unknown>, run: Row) {
|
|
const attachments = await api.get<Row[]>(`/api/issues/${issue!.id}/attachments`);
|
|
const attachment = attachments.find(row => row.originalFilename === filename && row.createdByAgentId === agent!.id && row.originatingRunId === run.id);
|
|
if (!attachment) throw new ConnectionFailure("run_attributed_artifact_missing");
|
|
const bytes = await api.bytes(`/api/attachments/${attachment.id}/content`);
|
|
let actual: Record<string, unknown> = {};
|
|
try { actual = JSON.parse(bytes.toString("utf8")); } catch { /* The independent verifier fails below. */ }
|
|
evidence.artifactChecks ??= [];
|
|
evidence.artifactChecks.push({ filename, runId: run.id, sha256: createHash("sha256").update(bytes).digest("hex"),
|
|
fields: Object.fromEntries(Object.entries(expected).map(([key, value]) => [key, actual?.[key] === value])), exactFields: Boolean(actual && Object.keys(actual).length === Object.keys(expected).length) });
|
|
if (!verifyConnectionArtifact(bytes, expected)) throw new ConnectionFailure("independent_artifact_verification_failed");
|
|
await page.goto(`${api.origin}/${company.issuePrefix}/issues/${issue!.identifier}`, { waitUntil: "domcontentloaded" });
|
|
await page.reload({ waitUntil: "domcontentloaded" });
|
|
await expect(page.locator(`a[href*="/api/attachments/${attachment.id}/content"]`).first()).toBeVisible({ timeout: 30_000 });
|
|
evidence.artifactSha256 = createHash("sha256").update(bytes).digest("hex");
|
|
}
|
|
const first = await completedRun(new Set(), "connection-proof.json");
|
|
evidence.checkpoints.first_run = true;
|
|
await artifact("connection-proof.json", proof.expected, first);
|
|
evidence.checkpoints.artifact = true;
|
|
// Artifact upload provenance is a server-owned tool-side effect in both runner generations.
|
|
const events = await collectRunEvents<Row>((afterSeq, limit) => api.get(`/api/heartbeat-runs/${first.id}/events?afterSeq=${afterSeq}&limit=${limit}`));
|
|
const toolReceipt = events.some(event => event.eventType === "tool.execution.completed" || (event.eventType === "item.completed" && ["tool_result", "command_execution"].includes(event.payload?.prpEvent?.payload?.item?.type)));
|
|
// Legacy adapters may expose their tool transcript only in the run log; upload attribution is still a durable successful tool effect.
|
|
evidence.checkpoints.tool_receipt = execution.profile.generation === "legacy" || toolReceipt;
|
|
if (!evidence.checkpoints.tool_receipt) throw new ConnectionFailure("native_tool_receipt_missing");
|
|
await input.checkpoint("followup");
|
|
const previousRuns = new Set(runs.keys());
|
|
await submitTaskReply(page, "Using the original connection-input.json decoded from this task, deliver connection-followup.json with exactly nonce, minimum, maximum, and total of the original values. Use exactly this JSON schema: {\"nonce\": string, \"minimum\": number, \"maximum\": number, \"total\": number}. All four fields are required. Compute with a tool and validate the file against all four fields before attaching it and its download link. Then mark this task done. Do not create other tasks or read credentials.");
|
|
const second = await completedRun(previousRuns, "connection-followup.json");
|
|
await artifact("connection-followup.json", proof.followup, second);
|
|
evidence.checkpoints.followup = true;
|
|
evidence.checkpoints.visible_result = true;
|
|
return { issueId: issue.id as string, issueIdentifier: issue.identifier as string };
|
|
}
|