From e38d6d16b6f234c29651f5ed501cc8d25cfc121a Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 6 Oct 2026 15:21:22 -0500 Subject: [PATCH] feat(connections): add advanced provider setup and live browser qualification (#15341) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Users connect accounts and choose an agent harness and model. > - The runtime change in #14970 supports custom providers on those connections. > - Normal setup must stay simple while advanced users can choose a compatible gateway. > - Shared connector rows and access controls keep these choices consistent. > - This pull request refines the agent setup UI and adds review stories and repeatable browser qualification. > - The qualification checks real tools and downloaded outputs, not only a successful run status. ## Linked Issues or Issue Description Refs #14970, #37, #13083, #14104, #14565, #12692. The core implementation in #14970 is merged. This branch incorporates its squash commit and targets `master`. Both PRs contain our implementation. #14016 is a reference only and is not a dependency. This PR has 96 changed files. ## What Changed - Complete model-provider connector presentation beside other connectors. Each row uses the existing Connect action and connection list. Tags are stored without category UI. The base PR includes the provider forms and routes. - Show persistent Subscription, API Key, and Advanced choices. Label Advanced as Custom Gateway. Reuse provider logos, connection lists, and permissions controls. Default access to the organization and all agents when permitted; keep narrowing controls under Advanced. - Keep Configure reachable before subscription sign-in, so users can select a supported environment when the default cannot sign in. Testing and saving still require a connection. Show the execution environment in Configure. Preserve the confirmed Connect choice. Editing a method, credential, saved account, or advanced choice requires that current choice to connect before testing or saving. Use matching model and thinking-effort dropdowns and retain connection icons in selected values. - Preserve the new harness model default when switching an existing OpenCode agent to Codex or Claude, and resolve user-selected model names with the effective harness. - Load popular OpenRouter models through the shared connection-model discovery path. Keep explicit model lists and manual model entry available. - Group onboarding, connection setup, agent runtime, management, recovery, and production-component stories under AI Connections / Provider routing. - Add an explicit-only provider-connections browser suite for managed local or existing local/staging targets. Use private browser profiles and credential handoffs. Support human-assisted subscription sign-in without sharing passwords or tokens in reports. - Verify persisted connection identity, runtime probes, tool execution, exact artifact bytes, completion, and context-dependent follow-up. Retain source/model provenance, cost bounds, closed error diagnostics, original failures, and cleanup evidence. - Add Gemini startup-model and skill-root fixes, Grok private-history detection, ACP filesystem regression fixtures, selected-workspace handling for local Hermes, and artifact-helper workspace fallback. - Keep managed Grok runtime homes disposable. Remove host-side transcript retention/restoration because private file modes do not isolate same-user agent processes. Ignore earlier development archives and use a fresh task handoff when history is unavailable. Verify the absence of restored transcripts with a separate same-user process. - Capture stopped-run diagnostics before deleting an attached-company fixture agent. Track creation and owned sign-in receipts; revoke only this attempt's accounts and never adopt a concurrent campaign's newly created account. Preserve failure signals and final status through cleanup. - Require the requested environment in the saved agent and every run, including follow-ups. Reject a forced incompatible target. Keep one cancellation state through startup, every cell, reporting, and teardown for SIGINT, SIGTERM, and SIGHUP. Stop further paid cells after interruption. Document qualification limits. ## Verification - Current head `b3bb3e94d577d43d9965a6b9daba039f599b2e49` includes master `d9f600043`. The security fix in `a758fde31` passes full workspace typecheck, production build, and 119 connection/Grok regressions. The unchanged UI passes all 126 configuration/model-discovery tests and token gates. The final published-guide correction passes Grok adapter typecheck. Earlier head `eebd8225c` passed the complete deterministic runner suite (1,404 Vitest tests and 128 Node tests) and all CI jobs. Current-head CI run `37520147514` passed all 47 jobs, including the full sharded Vitest and browser matrix, production build, and canary dry run. All 55 checks completed: 53 successes and two expected skips. The current-head security scan passed, Greptile is 5/5, and no review threads remain open. - A separate same-user process reproduced reading a restored Grok transcript before the security fix. The regression now finds no transcript. Existing fresh-session fallback and ordinary session metadata behavior pass. - The final account-choice and cleanup fixes pass 85 setup tests and 26 qualification-harness tests. Regressions verify that editing a connection invalidates confirmation, Configure remains reachable before sign-in, diagnostics are captured before fixture deletion, and concurrent campaigns cannot adopt or revoke each other's accounts. UI and E2E typechecks pass. - The Storybook build and actual Chromium production-component stories passed during this change. Review the neighboring AI Connections / Provider routing stories, regular connector rows, three connection modes, model discovery, and the single execution-environment control in Configure. - Cancellation smoke verified authenticated cleanup before browser close for SIGINT, SIGTERM, and SIGHUP. Regressions cover interruption during startup and reporting, missing-file ACP resource errors, and preserved permission denials. Both ACP runtime versions and 54 ACPX/Grok regressions passed. The deterministic connection-intent browser suite passed two tests. - Historical local qualification retained 43 passing API/gateway cells out of 46, with downloaded outputs and follow-up receipts. These attempts span earlier builds; they do not qualify this exact commit or staging. Subscription combinations, Gemini overloads, and the unresolved follow-up failure remain recorded rather than counted as passing. - Use `pnpm test:e2e:runner -- --list --suite provider-connections` to inspect the matrix. Follow `tests/runner-e2e/PROVIDER-CONNECTIONS.md` for credentials, target URL, sign-in assistance, budget, evidence, and cleanup. Paid live tests remain opt-in. ## Risks - The core implementation in #14970 is merged. This PR adds no database migration of its own. - Subscription login needs an interactive provider session. Dedicated accounts and staging qualification remain follow-up work; this PR does not certify every login combination for production. - Managed Grok transcript resume is deferred until provider history has an OS isolation or authorized broker solution. Follow-ups start fresh with Paperclip task context; earlier live Grok results do not qualify this behavior. - Gemini CLI 0.58.0 has an upstream ACP new-file error conversion defect. Live overloads and one unresolved follow-up timeout remain recorded. The stock CLI is unchanged, and those cases are not marked as passing. - Real-provider tests spend credits and use private credential/evidence directories. The launcher requires explicit selection and checks target ownership. It must not attach to a developer's database by accident. - OpenClaw Gateway, Hermes Gateway, Claude Managed, AWS AgentCore, Process, HTTP, and legacy ACPX local remain outside custom provider setup. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, code execution, and browser testing. The exact deployment model ID and context window size were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- README.md | 3 + doc/DOCKER.md | 11 +- doc/connections/AI-CONNECTIONS.md | 67 +- doc/evals.md | 4 + ...-live-provider-connection-qualification.md | 211 +++++++ .../src/acpx-engine/execute.test.ts | 62 +- .../adapter-utils/src/acpx-engine/execute.ts | 44 +- .../acpx-engine/filesystem-protocol.test.ts | 110 ++++ packages/adapters/gemini-local/src/index.ts | 4 + packages/adapters/grok-local/src/index.ts | 1 + .../grok-local/src/server/execute.test.ts | 25 + .../adapters/grok-local/src/server/execute.ts | 13 +- .../grok-local/src/server/grok-home.test.ts | 12 +- .../grok-local/src/server/grok-home.ts | 27 +- packages/adapters/hermes/README.md | 1 + packages/adapters/hermes/src/index.ts | 1 + .../hermes/src/server/execute.onspawn.test.ts | 12 + .../adapters/hermes/src/server/execute.ts | 6 +- patches/acpx@0.12.0.patch | 29 +- patches/acpx@0.13.1.patch | 86 +-- .../servers/acp-filesystem-agent.mjs | 43 ++ server/src/__tests__/ai-connections.test.ts | 44 +- .../__tests__/file-delivery-bridges.test.ts | 4 +- skills/paperclip/references/artifacts.md | 5 + .../scripts/paperclip-upload-artifact.sh | 17 +- tests/ai-connections-app/app.spec.ts | 27 +- tests/runner-e2e/FIXTURES.md | 6 + tests/runner-e2e/PROVIDER-CONNECTIONS.md | 498 +++++++++++++++ tests/runner-e2e/README.md | 6 + tests/runner-e2e/SECURITY.md | 14 + tests/runner-e2e/case-outcome.ts | 7 +- tests/runner-e2e/catalog.test.ts | 10 +- tests/runner-e2e/catalog.ts | 10 +- tests/runner-e2e/codex-ci-sandbox.test.ts | 4 +- tests/runner-e2e/connection-cancel.smoke.ts | 67 ++ tests/runner-e2e/connection-cases.ts | 88 +++ tests/runner-e2e/connection-config.ts | 162 +++++ tests/runner-e2e/connection-evidence.ts | 129 ++++ tests/runner-e2e/connection-flow.ts | 377 ++++++++++++ tests/runner-e2e/connection-launch.ts | 394 ++++++++++++ tests/runner-e2e/connection-support.test.ts | 282 +++++++++ tests/runner-e2e/connection-target.ts | 102 ++++ .../runner-e2e/gemini-acp-filesystem.smoke.ts | 82 +++ tests/runner-e2e/history.test.ts | 2 + tests/runner-e2e/launch.ts | 7 + tests/runner-e2e/report-assets.ts | 18 + tests/runner-e2e/report.ts | 15 +- tests/runner-e2e/result-validation.ts | 17 + tests/runner-e2e/selectors.ts | 6 + tests/runner-e2e/types.ts | 3 + tests/runner-e2e/user-actions.ts | 67 +- .../codex-local/config-fields.test.tsx | 76 ++- ui/src/adapters/codex-local/config-fields.tsx | 61 +- ui/src/adapters/config-sections.test.tsx | 67 +- ui/src/components/AdapterMark.tsx | 59 ++ .../AgentConfigForm.render.test.tsx | 55 +- ui/src/components/AgentConfigForm.tsx | 117 +++- .../AiConnectionAccountControls.tsx | 2 +- .../ManagedAiConnectionDetails.tsx | 12 +- .../useConnectionModels.test.tsx | 108 ++++ .../ai-connections/useConnectionModels.ts | 55 ++ .../ai-connections/useLocalAiLogin.ts | 2 +- .../new-agent/AgentBasicsDialog.tsx | 59 +- .../AgentProviderConnection.test.tsx | 84 ++- .../new-agent/AgentProviderConnection.tsx | 102 ++-- ui/src/components/new-agent/NewAgentSetup.tsx | 129 ++-- .../onboarding/CredentialModeLink.tsx | 22 +- .../onboarding/ModelSourceTiles.tsx | 18 +- ui/src/pages/NewAgent.test.tsx | 144 ++++- ui/src/pages/apps/Browse.test.tsx | 23 + ui/src/pages/apps/Browse.tsx | 22 +- ui/src/pages/apps/app-connect-policy.test.ts | 4 + ui/src/pages/apps/app-connect-policy.ts | 1 + ui/src/pages/apps/app-definition-display.ts | 6 +- .../agent-settings/AgentSettingsPreview.tsx | 211 ++++--- .../provider-routing/AgentSetup.tsx | 573 ++++++++++++++++++ .../provider-routing/ConnectionCatalog.tsx | 229 +++++++ .../provider-routing/ConnectionPicker.tsx | 67 ++ .../provider-routing/ConnectionSetup.tsx | 425 +++++++++++++ .../provider-routing/Management.tsx | 470 ++++++++++++++ .../prototypes/provider-routing/README.md | 118 ++++ .../prototypes/provider-routing/Recovery.tsx | 185 ++++++ .../provider-routing/ReviewIndex.tsx | 168 +++++ .../provider-routing/RoutingAccess.tsx | 71 +++ .../prototypes/provider-routing/model.ts | 250 ++++++++ .../prototypes/provider-routing/shared.tsx | 255 ++++++++ .../provider-routing/story-support.ts | 36 ++ .../stories/ai-connections.stories.tsx | 3 + .../stories/local-ai-subscription.stories.tsx | 61 ++ .../provider-routing-00-overview.stories.tsx | 19 + ...provider-routing-01-onboarding.stories.tsx | 93 +++ .../provider-routing-02-connect.stories.tsx | 116 ++++ .../provider-routing-03-agent.stories.tsx | 325 ++++++++++ .../provider-routing-04-manage.stories.tsx | 115 ++++ .../provider-routing-05-recovery.stories.tsx | 71 +++ ...provider-routing-06-production.stories.tsx | 274 +++++++++ 96 files changed, 8062 insertions(+), 543 deletions(-) create mode 100644 doc/plans/2026-10-03-live-provider-connection-qualification.md create mode 100644 packages/adapter-utils/src/acpx-engine/filesystem-protocol.test.ts create mode 100644 scripts/mcp-fixtures/servers/acp-filesystem-agent.mjs create mode 100644 tests/runner-e2e/PROVIDER-CONNECTIONS.md create mode 100644 tests/runner-e2e/connection-cancel.smoke.ts create mode 100644 tests/runner-e2e/connection-cases.ts create mode 100644 tests/runner-e2e/connection-config.ts create mode 100644 tests/runner-e2e/connection-evidence.ts create mode 100644 tests/runner-e2e/connection-flow.ts create mode 100644 tests/runner-e2e/connection-launch.ts create mode 100644 tests/runner-e2e/connection-support.test.ts create mode 100644 tests/runner-e2e/connection-target.ts create mode 100644 tests/runner-e2e/gemini-acp-filesystem.smoke.ts create mode 100644 tests/runner-e2e/report-assets.ts create mode 100644 ui/src/components/AdapterMark.tsx create mode 100644 ui/src/components/ai-connections/useConnectionModels.test.tsx create mode 100644 ui/src/components/ai-connections/useConnectionModels.ts create mode 100644 ui/storybook/prototypes/provider-routing/AgentSetup.tsx create mode 100644 ui/storybook/prototypes/provider-routing/ConnectionCatalog.tsx create mode 100644 ui/storybook/prototypes/provider-routing/ConnectionPicker.tsx create mode 100644 ui/storybook/prototypes/provider-routing/ConnectionSetup.tsx create mode 100644 ui/storybook/prototypes/provider-routing/Management.tsx create mode 100644 ui/storybook/prototypes/provider-routing/README.md create mode 100644 ui/storybook/prototypes/provider-routing/Recovery.tsx create mode 100644 ui/storybook/prototypes/provider-routing/ReviewIndex.tsx create mode 100644 ui/storybook/prototypes/provider-routing/RoutingAccess.tsx create mode 100644 ui/storybook/prototypes/provider-routing/model.ts create mode 100644 ui/storybook/prototypes/provider-routing/shared.tsx create mode 100644 ui/storybook/prototypes/provider-routing/story-support.ts create mode 100644 ui/storybook/stories/local-ai-subscription.stories.tsx create mode 100644 ui/storybook/stories/provider-routing-00-overview.stories.tsx create mode 100644 ui/storybook/stories/provider-routing-01-onboarding.stories.tsx create mode 100644 ui/storybook/stories/provider-routing-02-connect.stories.tsx create mode 100644 ui/storybook/stories/provider-routing-03-agent.stories.tsx create mode 100644 ui/storybook/stories/provider-routing-04-manage.stories.tsx create mode 100644 ui/storybook/stories/provider-routing-05-recovery.stories.tsx create mode 100644 ui/storybook/stories/provider-routing-06-production.stories.tsx diff --git a/README.md b/README.md index b3f130123d8..a0db4476643 100644 --- a/README.md +++ b/README.md @@ -417,6 +417,9 @@ This starts the UI and API at `http://localhost:3100`. An embedded PostgreSQL da > **Requirements:** Node.js 24.11+, pnpm 9.15+ +Local Claude and Codex subscription sign-in also needs Python 3 and the +corresponding provider CLI on the Paperclip host. The Docker image includes them. + Source development also builds the native Paperclip Runner when enabled (the self-hosted default). Install a Rust toolchain, or set `PAPERCLIP_RUNNER_BINARY` to a compatible prebuilt runner.
diff --git a/doc/DOCKER.md b/doc/DOCKER.md index 7d9bb1ea954..d59ddc102bd 100644 --- a/doc/DOCKER.md +++ b/doc/DOCKER.md @@ -212,24 +212,25 @@ The image pre-installs: - `claude` (Anthropic Claude Code CLI) - `codex` (OpenAI Codex CLI) +- `python3` (used to run local subscription sign-in in a pseudo-terminal) -If you want local adapter runs inside the container, pass API keys when starting the container: +Start the container, then open **Apps → Connections** to connect a Claude or +Codex subscription in your browser. You can also enter a provider API key there. +You do not run a shell command or configure a CLI home. ```sh docker run --name paperclip \ -p 3100:3100 \ -e HOST=0.0.0.0 \ -e PAPERCLIP_HOME=/paperclip \ - -e OPENAI_API_KEY=... \ - -e ANTHROPIC_API_KEY=... \ -v "$(pwd)/data/docker-paperclip:/paperclip" \ paperclip-local ``` Notes: -- Without API keys, the app still runs normally. -- Adapter environment checks in Paperclip will surface missing auth/CLI prerequisites. +- The image provides the CLI and Python prerequisites for local subscription sign-in. +- Adapter environment checks in Paperclip surface missing authentication or CLI prerequisites. ## Podman Quadlet (systemd) diff --git a/doc/connections/AI-CONNECTIONS.md b/doc/connections/AI-CONNECTIONS.md index ac56651ea2e..fe9e0584b3c 100644 --- a/doc/connections/AI-CONNECTIONS.md +++ b/doc/connections/AI-CONNECTIONS.md @@ -10,6 +10,11 @@ homes. Managed default/shared accounts are additional choices in that same selector. Selecting “Sign in to another account” survives background refreshes; Claude authorization paste keeps upstream's immediate Connecting feedback. +New-agent Connect offers three persistent tiles: the provider's subscription, +the provider's API key, and Advanced. All three stay visible while the selected +mode's form is shown below. Advanced opens the existing compatible-connection +picker and provider setup. Execution environment selection lives in Configure. + Storybook's simulated controllers and page annotations do not run in the app. ## Compatibility and selection @@ -379,8 +384,8 @@ onboarding, and agent setup share `LocalProviderLoginInstructions` and `useLocalAiLogin`. Claude and Codex start a local provider process behind the browser sign-in card. Claude accepts the authorization code in that card; Codex displays its device code there. The user does not run a shell command. Each -local runner requires Python 3 for its pseudo-terminal and the corresponding -provider CLI on the Paperclip host. Each +local runner requires Python 3 for its pseudo-terminal (included in the Docker +image) and the corresponding provider CLI on the Paperclip host. Each attempt retains a private credential home. The home is never seeded with the operator's existing login: copying a rotating refresh token would allow managed runs to invalidate credentials still used by legacy agents or the @@ -511,15 +516,24 @@ fingerprint. These checks do not relax current connection authorization. ## Advanced provider routing (2026-10-02) -The connection API and catalog support OpenRouter, Amazon Bedrock, Google Gemini, -Responses API, Messages API, Chat Completions API, and local endpoints. -The catalog tags these entries `model-provider`. Responses-compatible gateways -such as Emissary use the Responses API definition. +Use the regular rows on **Connectors** to add OpenRouter, Amazon Bedrock, +Google Gemini, a Responses API, Messages API, Chat Completions API, or local +endpoint connection. Each row has its own Connect action and saved accounts. +The catalog tags these entries `model-provider`; no category UI is shown. +Responses-compatible gateways such as Emissary use the Responses API row. +The native subscription/API-key onboarding remains the default. Provider choices +show the existing local brand artwork and reuse the existing access step. Custom +URLs, protocol, AWS region, and credential fields appear only after choosing the +corresponding connector. New connections default to everyone in the organization +and all agents when the actor has permission; the existing Advanced disclosure +contains the controls to narrow access, without a separate Access step. New-agent setup also offers the connection picker in its persistent +**Advanced / Custom Gateway** tile. -Provider routing belongs to the connection; the model belongs to the agent. -The API exposes compatible saved connections and optional model identifiers. -The advanced setup UI and review stories ship in the follow-up UI change. -Native subscription and API-key setup keep their existing controls. +At **Agents → [agent] → Harness / Runtime**, **Connection** is a dropdown of +compatible saved connections, including explicit personal accounts. The existing +model picker uses that connection’s optional model IDs and accepts manual IDs. +Changing connections preserves the model for explicit review. URLs and +credentials belong to the connection; the model belongs to the agent. | Harness | Implemented managed routes | | --- | --- | @@ -541,9 +555,10 @@ catalog discovery for custom gateways are not part of this implementation. OpenRouter connections without an explicit model list automatically load its public [model catalog](https://openrouter.ai/docs/api/api-reference/models/list-all-models-and-their-properties), -ordered with `sort=most-popular`. The company-scoped model discovery API preserves -the provider's ordering and adapts model IDs to the selected harness. Explicit -connection model lists take precedence. Catalog discovery sends no credentials. +ordered with `sort=most-popular`. New-agent setup and agent settings share this +discovery path, preserve the provider's ordering, and adapt model IDs to the selected +harness. Explicit connection model lists take precedence. Catalog discovery sends +no credentials; a failed request offers refresh and manual model entry. `config.ai.routing` stores only kind, protocol, URL, auth method, region, and optional model IDs/labels. The vault stores provider API keys, including Bedrock API keys. @@ -573,6 +588,15 @@ remote URLs; loopback endpoints may use HTTP. Localhost refers to the agent’s execution environment, including when it is a sandbox. URLs cannot contain user credentials, query parameters, or fragments. +Managed Grok uses a disposable runtime home. Paperclip does not retain or restore +Grok transcript files into host temporary directories: private file modes do not +isolate agents running as the same OS user. Archives from earlier development +builds are ignored. Provider session metadata still saves normally. If the selected +session has no history in its current execution environment, Grok starts a fresh +session with the Paperclip task handoff rather than attempting remote subscription +recovery. Transcript continuation requires an isolated provider history solution +and is not qualified by this change. + Runtime projection clears alternate provider credentials and routing overrides, uses disposable homes, and never falls back to host authentication. Codex probes retain the selected provider home. The new runner copies only the validated @@ -605,9 +629,10 @@ through PostgreSQL JSONB and verifies that credential rotation retains identity and agent access. General AWS access keys are not accepted or forwarded to Claude; use a Bedrock API key. Support for AWS roles requires a credential broker before it can ship. -A subsequent OpenCode tool-use check reached OpenRouter but was denied terminal -access; its follow-up ended with `process_lost`. Treat OpenCode tool-use acceptance -as unresolved rather than inferring it from a completed task status. +The initial OpenCode tool-use check was denied terminal access. The native driver +now includes the assigned workspace and its canonical path in the allowed +directories while preserving the configured tool policy. Subsequent local +qualification verified tool use and follow-up on the affected OpenCode route. Live Bedrock verification subsequently passed with a short-lived Bedrock API key, region `us-east-1`, and `us.anthropic.claude-sonnet-4-6`. The saved connection passed **Run test**. Claude legacy and Claude New Runner each ran a terminal @@ -616,6 +641,16 @@ tool output was verified for all four successful runs. Private gateways still have deterministic mapping and validation coverage but need live verification in the target deployment. Short-lived Bedrock keys must be rotated before expiry. +The repeatable provider connection campaign is documented in +[`tests/runner-e2e/PROVIDER-CONNECTIONS.md`](../../tests/runner-e2e/PROVIDER-CONNECTIONS.md). +It preserves actual tool outputs, downloaded artifacts, completion and follow-up +receipts, source provenance, cleanup, and provider failures. The retained local +qualification has 43 passing API/gateway cells out of 46; staging and all +subscription combinations remain unqualified. Gemini CLI 0.58.0 has an upstream +ACP new-file error conversion defect. The provider-free filesystem probe exposes +that defect without modifying the installed CLI. Provider overloads and one +follow-up timeout also remain live qualification limits. + ### Gateway completion compatibility Provider-facing `paperclip_finish` accepts an omitted or `null` continuation for diff --git a/doc/evals.md b/doc/evals.md index 709792c56a9..a242d214731 100644 --- a/doc/evals.md +++ b/doc/evals.md @@ -1,5 +1,9 @@ # Paperclip evaluation guide +The explicit-only [live provider connection suite](../tests/runner-e2e/PROVIDER-CONNECTIONS.md) +is a Product E2E workflow for fresh subscription/API-key/gateway connections, +with attended login and independent artifact checks against local or staging targets. + Paperclip has two live eval families with different questions, owners, and evidence. Choose the family before selecting a model, profile, or case. diff --git a/doc/plans/2026-10-03-live-provider-connection-qualification.md b/doc/plans/2026-10-03-live-provider-connection-qualification.md new file mode 100644 index 00000000000..a027a83d659 --- /dev/null +++ b/doc/plans/2026-10-03-live-provider-connection-qualification.md @@ -0,0 +1,211 @@ +# Live provider connection qualification + +Date: 2026-10-03 +Status: Runnable implementation delivered; operating contract and initial verification are in [PROVIDER-CONNECTIONS.md](../../tests/runner-e2e/PROVIDER-CONNECTIONS.md). Live qualification is reported per campaign, not implied by this plan. +Code inspected: `527e3bad2` (`codex/provider-routing-connections`). + +## Decision + +Extend the existing Playwright Product E2E harness with real connection-creation journeys, an explicit authentication handoff, and support for an existing deployment as the target. Run the subscription journeys in a visible browser on a developer's machine first. The same tests should target local Paperclip and staging. + +Playwright owns navigation, assertions, evidence, and the verdict. A person completes login challenges when necessary. A computer-use agent can assist with diagnosis and provider UI changes, but it must not silently change the steps or decide whether the test passed. An assisted success is useful evidence and must be labeled as assisted. + +The deliverable is a matrix answering **which configuration worked, against which deployment, with what proof**. A successful login, a saved connection, and a successful agent run are separate checkpoints. + +## Existing foundation and gaps + +- [`tests/runner-e2e`](../../tests/runner-e2e/README.md) already supplies real browser/server/runner/provider tests, catalog selection, isolated fixtures, independent grading, reports, cost accounting, and provenance. This is the appropriate family, as defined in [`doc/evals.md`](../evals.md). +- Its [Playwright configuration](../../tests/runner-e2e/playwright.config.ts) currently owns a fresh loopback server and database. Supporting staging requires a target lifecycle abstraction, not just substituting a URL. +- [`tests/ai-connections-app/app.spec.ts`](../../tests/ai-connections-app/app.spec.ts) includes simulated login sessions and popup destinations. It also retains local terminal-login assertions that predate this branch's browser-login changes. Keep fast simulated tests, update those stale expectations, and label their limited proof clearly. +- [`tests/hiring-ai-connections/hiring.live.spec.ts`](../../tests/hiring-ai-connections/hiring.live.spec.ts) already checks real runs, connection attribution, and legacy/native execution. It creates initial API-key connections through the API and requires a loopback `local_trusted` instance. Reuse the assertion patterns; it is not a complete connection-onboarding test or a staging harness. +- The existing Grok subscription qualification stages supplied `GROK_AUTH_JSON`. Its documentation correctly says this does not test interactive browser login. + +## Coverage contract + +Inventory harnesses separately from providers. Claude is a harness; Bedrock and OpenRouter are connection routes. A Paperclip deployment URL, an authentication host, and an agent execution environment are also separate dimensions. + +The initial inventory below comes from [managed connection capabilities](../../packages/shared/src/ai-connections.ts), [routing compatibility](../../packages/shared/src/ai-provider-routing.ts), and the [UI adapter registry](../../ui/src/adapters/registry.ts). It describes implementation eligibility, not proven live coverage. + +| Harness | Managed subscription | Direct API key | Advanced routes currently admitted | +| --- | --- | --- | --- | +| Claude Code | Yes | Anthropic | OpenRouter, Bedrock, Messages-compatible gateway | +| Codex | Yes | OpenAI | OpenRouter, Responses-compatible gateway | +| Grok | Yes; qualify its actual login UX separately | xAI | Not supported by the current routing compatibility table | +| Gemini | Not in the managed connection capability table | Google | Not in the current routing compatibility table | +| OpenCode | No managed subscription entry | OpenRouter | OpenRouter, Chat Completions-compatible gateway | +| Hermes local | No managed subscription entry | Inventory harness-specific behavior | OpenRouter, Chat Completions-compatible gateway | +| Cursor local, Kimi, Pi | Inventory required | Inventory required | Inventory required | + +“Not in the table” is a Paperclip managed-connection limitation, not a claim about everything the upstream CLI can do. Inventory native runner providers and installed first-party adapters as well as legacy UI adapters; any additional local provider must receive a manifest entry before the full campaign can qualify. + +Exclude remote OpenClaw and Hermes gateways, Cursor Cloud, managed Claude/AWS AgentCore, process/HTTP adapters, and the generic legacy ACPX integration from this campaign. Their remote systems own provider configuration. **Hermes local remains in scope.** Supported native Claude/Grok runner profiles that internally use ACPX remain in scope; that transport is distinct from the excluded generic legacy integration. + +For every supported harness/auth/route combination, enumerate the supported legacy/native runner profiles and target environments. Do not infer that an API-key pass covers subscription, that legacy covers native, or that local covers staging. Protocol routes need a real configured endpoint and credential; a mock gateway is useful for regression tests but does not supply live qualification. + +Keep three independent manifest fields: + +- Eligibility: supported, unsupported/product gap, needs inventory, or deliberately excluded with a reason. +- Execution: not run, awaiting user, missing credential, blocked target, running, complete. +- Qualification: passed or failed, with individual checkpoint results and assistance recorded. + +A credential-free catalog check must flag newly added local harnesses or changed capabilities without a reviewed coverage entry. Report required, selected, attempted, and passed counts separately. Unsupported configurations are visible product gaps, never green tests. + +## Authentication strategy + +Use three lanes, with separate results: + +| Lane | Starting state | What it proves | +| --- | --- | --- | +| Fresh connection, existing browser sign-in | Saved provider website/Google browser session; no Paperclip connection or runtime credential for this cell | The current Paperclip subscription flow obtains and saves a fresh usable credential | +| Fresh connection, signed-out browser | Empty provider browser state; no Paperclip connection or runtime credential | The complete first-time experience, including account selection and login | +| Existing connection | A connection previously created by the live UI journey | Selection, reuse, continuation, and reconnect behavior | + +Most developer campaigns use the first lane. The signed-out lane runs explicitly before releases or login changes. The existing-connection lane cannot substitute for either creation lane. + +There are three distinct identities/stores: Paperclip board authentication for the selected target, provider website authentication in the browser, and the resulting managed connection credential. Reusing provider website sign-in reduces friction while still exercising a fresh OAuth/device authorization and production credential persistence. Do not import `~/.codex`, `~/.claude`, or an existing token into a fresh-connection test. + +Initially, use the developer's existing subscription accounts through a dedicated QA browser profile, separate from their everyday Chrome profile. Keep account aliases and expected identities in private local configuration; do not put personal email addresses into the public catalog. Serialize tests sharing an account or refresh-token chain. Never copy the profile between CI workers. + +### Login checkpoint + +The flow starts the real production login session, arms popup/new-tab observation before clicking, and tracks the expected provider origin and Paperclip session. It handles ordinary redirects and device-code/code-entry steps using the actual UI. Provider and Google pages can require a person for account choice, passwords, passkeys, MFA, CAPTCHA, or consent. + +At such a boundary: + +1. Show the relevant window and mark the cell `awaiting_user`, with the provider, target, and expected account alias. +2. Pause only the affected account's queue. Keep a bounded deadline tied to the real login session's expiry. +3. Let the person complete the production browser flow. If the provider rejects the automated browser, support handoff to a normal browser using the product's existing device URL or code-return flow. Do not add authentication bypasses. +4. Resume from observed Paperclip login status and saved-connection state. Clicking “continue” is never sufficient proof of successful authorization. +5. If the session expires, retain that attempt and use the product's retry action to start a new session. Never replay an old callback code. + +Record which step needed assistance and its duration, without recording passwords, codes, callback query strings, or provider session IDs. Check the expected account using available provider/account metadata; where this cannot be machine-verified, explicitly record human identity confirmation rather than claiming an automated identity check. + +There is no promise that Google or provider sign-in will remain unattended indefinitely. Expired browser state is an operational login requirement, not a passing test. A bug that loses a completed callback is a product failure, not automatically “needs login.” + +## Local and staging target adapters + +Introduce two lifecycle modes while sharing the same journey: + +- **Managed local:** create and own an isolated Paperclip home, database, server, company, and runtime. Teardown may stop/delete only these owned resources. +- **Attach existing:** authenticate to a configured local/staging URL and create a dedicated QA company or use an explicitly designated test company. Never reset the deployment, change instance-wide feature settings, install plugins, or stop its server as fixture setup. + +The current local `3109` application can be an attach-existing target. Preserve its BOB company and user data. Full first-company onboarding belongs on a fresh managed target or a dedicated disposable staging tenant/instance; a shared staging deployment cannot prove that path merely by creating another agent. + +Each private target configuration supplies an exact origin, target mode, expected deployment identity/revision, board-auth account reference, QA company policy, and available execution environment. The staging URL is still to be supplied. Target validation precedes credential resolution. Verify redirects, health, authorization, installed harness versions, required capabilities, and environment availability; do not send credentials to a host introduced by a redirect or unreviewed target configuration. + +Record the tested server revision separately from the harness revision, plus runner/CLI version, immutable sandbox image when applicable, model ID, protocol, authentication host, and execution environment. Staging with a missing capability is blocked, not silently tested through a different route. Unknown build provenance prevents a release qualification claim. + +## Required browser journey and independent proof + +For each admitted cell: + +1. Create/select the isolated fixture company and a synthetic workspace through existing public surfaces. Ordinary bootstrap APIs may prepare the fixture; they must not create the connection or agent that the browser journey is supposed to test. +2. Enter the production new-agent flow, select the harness and Subscription/API Key/Advanced mode, and create the connection through the UI. Advanced cases supply the configured provider/protocol, endpoint or region, and selected credential reference. Verify that permissions default to everyone/all agents within the isolated test company and that advanced controls retain their values. +3. Verify the saved connection and compatible connection selector after reload. For a fresh-login cell, confirm that a new connection was actually created and that the selected connection is the one bound to the agent. +4. Choose the model and environment in Configure, run the product's test action, and finish setup. Verify the model list/custom model path and selected values survive reload. +5. Create a real task through the browser. Ask the agent to read a small synthetic input file, compute a deterministic result, and save a result artifact. Generate the input independently and keep the expected result out of the prompt. Seed the workspace using existing environment fixture facilities, not a private server hook. +6. Require the real tool receipt, persisted successful run, task completion, correct artifact bytes/hash, and a user-visible result link that opens after reload. “Provider routing works” or any other model-authored success sentence alone cannot pass. +7. Correlate the run with the selected connection, auth method, harness/runner, route, model, and execution environment using durable production evidence. Reuse the existing run-context assertions where available. Any missing attribution is an evidence gap to implement explicitly, not something to infer from the model's response. +8. Send one bounded follow-up that uses the saved result and produces a second independently checked output. This proves the saved credential remains usable beyond the setup probe. +9. Clean up only campaign-owned resources, retain sanitized results, and report cleanup failures separately. Failed cleanup prevents an entirely clean campaign result but does not erase successful functional evidence. + +Use a controlled runtime without ambient provider keys or personal CLI homes so a broken selected credential cannot fall back to another account. On staging, explicitly qualify the runtime's credential isolation. Add a negative case with a deliberately invalid selected credential and require an actionable failure with no successful fallback. For a controlled gateway, correlate sanitized request metadata when available; never retain authorization headers. + +Also cover the other product entry points: Apps connector row → Connect → connection list → agent selection; existing agent → switch compatible connection; and reconnect after an expired/invalid connection. Apply every supported auth method to both Apps and agent-creation entry points. Fresh company onboarding and destructive recovery cases are separately enumerated; their target requirements must be visible. A representative smoke selection is useful, but is not the full matrix. + +## Trustworthy results and safe evidence + +Extend the existing Product E2E result/report pipeline with authentication checkpoints, assistance, target identity, and coverage denominator. Preserve the existing failure classification: product, model/provider behavior, grading/evidence, or infrastructure. Operational blocks are separate from these failure causes. + +Keep first failures and all attempts. Permit only declared bounded infrastructure retries; do not let an agent change selectors or retry a failed product flow until the campaign looks green. A repaired test starts a new definition/revision measurement. + +Example report shape (illustrative, not results): + +| Cell | Target | Login | Saved | Tool/artifact | Follow-up | Outcome | +| --- | --- | --- | --- | --- | --- | --- | +| Claude subscription/native | Local | Assisted | Pass | Pass | Pass | Pass, assisted | +| Codex subscription/native | Staging | Needs reauthentication | — | — | — | Awaiting user | +| Grok custom gateway | Local | — | — | — | — | Unsupported/product gap | + +Disable automatic Playwright traces, HARs, screenshots, videos, request-body logging, and DOM dumps for the live authentication flow, including failure handlers. Masked password fields do not protect network traces, tokens, or callback URLs. Exception messages and reporters also need sanitized wrappers; do not serialize raw Playwright call logs containing filled values or navigated URLs. Calibrate redaction with synthetic sentinel credentials before using real accounts. + +Keep browser profiles and auth state outside the repository and artifact directories, using OS-protected storage or restrictive local permissions. Persist only selected provider state; never include it in reports, caches, or CI uploads. Resolve one cell's required credentials at a time from a private secret store; API-key fixtures may reference existing local secrets without sourcing the entire secrets file into the server environment. + +After authentication, allow only the existing synthetic-task screenshots and structured allowlisted evidence. The [current public evidence policy](../../tests/runner-e2e/SECURITY.md#public-evidence-boundary) explicitly excludes credential and setup pages; this feature must not widen that boundary. Auth diagnosis uses sanitized phase/status metadata, not screenshots of personal accounts. Read-only report regeneration must make no additional provider calls. + +## Implementation slices + +1. **Manifest and target support.** Add an explicit-only `provider-connections` Product E2E suite; register coverage and profiles in the existing catalog/fixture registry. Add owned-local/attach-existing target lifecycles without changing existing suites' defaults. Include credential-free tests for origin validation, coverage drift, selection, cleanup ownership, and blocked results. +2. **One complete subscription path.** Implement the authentication checkpoint and Claude subscription journey locally, then Codex. Prove each with a fresh UI-created connection, independent task artifact, reload, and follow-up. Prove both saved-browser-sign-in and signed-out lanes. Fix stale local-login browser assertions alongside this work. +3. **Same journey on staging.** Supply the staging target and board account; replay the Claude/Codex journeys against its actual login host and runner environment. Record deployment differences rather than building a separate staging script. +4. **API keys and advanced routes.** Add direct keys, OpenRouter, Bedrock, and each supported protocol gateway. Include wrong-key/no-fallback, mismatched protocol, model discovery, and custom alias cases. Bedrock has a selected region/model and an expiring credential; an expired credential is renewed before the case and never counted as a product pass. +5. **Complete the inventory.** Add Grok's actual subscription and key paths, Gemini, OpenCode, Hermes local, and supported Cursor/Kimi/Pi/native-provider cases. Resolve unsupported auth paths as explicit product decisions; do not invent a third mode for every harness. Expand entry points and runner/environment variants until every required manifest cell has evidence. +6. **Operationalize.** Start with a manually launched visible-browser campaign on the developer's machine. Later use dedicated subscription accounts on a trusted QA machine, plus unattended API-key/gateway jobs in the existing protected paid CI workflow. Keep subscription reauthentication an explicit attended operation whenever the provider requires it. + +The suite, target configuration, and authentication selectors are now implemented; use the commands in [PROVIDER-CONNECTIONS.md](../../tests/runner-e2e/PROVIDER-CONNECTIONS.md). Discovery, verification, and reports reuse `test:e2e:runner`. The authoritative README/FIXTURES/SECURITY docs and `doc/evals.md` link the operating contract. + +Each case declares a turn limit covering the setup probe and follow-up, a provider-run timeout, a separate login deadline, company/agent budget limits, and cleanup policy. Run one subscription case per account at a time. Present selected cells and estimated spend before paid execution; record actual usage and unknown cost separately. Do not call subscription usage “free,” and do not run the full matrix as a side effect of normal unit tests or `--all`. + +## Acceptance criteria + +- One catalog and journey implementation selects local or staging through configuration. +- At least one real fresh Claude and Codex subscription connection passes through each target before claiming the initial slice complete; no mocked popup/session or preinstalled credential substitutes for this proof. +- Human handoff, timeout, expired login, and account mismatch produce distinct, resumable outcomes without false passes. +- Passing requires the selected connection's persisted binding, real successful run, independently validated tool result/artifact, and a successful subsequent use. +- Credential-free grader tests reject a plausible success message without a tool result, a wrong artifact, wrong connection attribution, and absent evidence. +- Missing credentials, unsupported combinations, and unrun cells remain visible in the coverage denominator. Assisted and unattended results remain distinguishable. +- Existing local/staging user data survives; teardown owns only test resources. +- Secret sentinel tests prove that failure artifacts and report projections do not disclose login state or credentials. +- The full inventory is qualified only after all required supported cells have current evidence. Earlier slices are explicitly partial. + +## External references + +- [Playwright pages and popups](https://playwright.dev/docs/pages): popup/new-page events provide deterministic browser coordination. +- [Playwright authentication](https://playwright.dev/docs/auth): authenticated browser state can be reused, is sensitive, and expires. Reusing website state does not require bypassing Paperclip's connection flow. +- [Google OAuth policies](https://developers.google.com/identity/protocols/oauth2/policies): account authorization must use an appropriate browser flow. Google login restrictions and challenges are not something this harness should attempt to bypass. + + +## Verification update · 2026-10-05 + +Six previously failing local key/gateway journeys now pass: legacy Grok from +both entry points, legacy Codex/new-agent OpenRouter, legacy OpenCode/Apps +OpenRouter, and native OpenCode/OpenRouter from both entry points. Gemini +selected-model startup, current local skill-root guidance, and repair-run +artifact attribution have been corrected. Its final two browser retests still +failed the 300-second first-task deadline, so Gemini remains unqualified. +The affected ACP regression group passes 221 tests and Grok's two affected files +pass 54 tests. The ACP checks include a real missing-file protocol reproduction, +canonical local Gemini workspaces, preserved remote paths, and permission-denial +guards. The existing ACPX patch now returns resource-not-found for genuine +missing files; it keeps other errors and permissions intact. Workspace +typecheck/build and token gates pass; the full +Vitest suite has not produced a passing result after broad timeout failures. + +The pinned native Grok installation still requires administrator access. +Attended subscription completion, staging/Daytona, and qualification against a +single reviewed committed build remain outstanding. See +[the qualification runbook](../../tests/runner-e2e/PROVIDER-CONNECTIONS.md) for +retained attempts, exact evidence boundaries, and replay instructions. This +update does not mark the overall acceptance criteria complete. + +## Gemini investigation update · 2026-10-06 + +The new-agent/API-key workflow passes with `gemini-3.5-flash-lite`, including the +independent first artifact and follow-up checks. The Apps diagnostic retest +delivered its first artifact, then timed out on follow-up with no recognized +error signal; that cause remains unknown. Earlier deadline failures include provider HTTP 503 overload +retries; increasing the deadline would not resolve the underlying uncertainty. + +A new credential-free installed-CLI smoke distinguishes correct ACP missing-file +semantics from Gemini CLI 0.58.0's plain-error-object conversion bug. A temporary +CLI copy with that narrow conversion fixed passes creation, existing-file reads, +and external-path denial; the user's installed CLI remains unchanged. Explicit +thinking-effort configuration also needs a supported startup-settings path; +short comparisons do not show that default `HIGH` thinking caused the deadlines. + +The harness now projects terminal errors into closed diagnostic codes before +managed-instance teardown, retaining no error text or hidden reasoning. Fresh +managed targets preserve the original stopped onboarding database. Results, +cleanup, model/source provenance, and remaining qualification boundaries are +recorded in the private report and the runbook. The full acceptance criteria +remain open. diff --git a/packages/adapter-utils/src/acpx-engine/execute.test.ts b/packages/adapter-utils/src/acpx-engine/execute.test.ts index faa2b6e7a0d..571d07b81bf 100644 --- a/packages/adapter-utils/src/acpx-engine/execute.test.ts +++ b/packages/adapter-utils/src/acpx-engine/execute.test.ts @@ -52,6 +52,7 @@ import { const tempRoots: string[] = []; +let defaultGeminiTestHome: string | undefined; async function makeTempRoot() { const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-acpx-skills-")); @@ -60,6 +61,7 @@ async function makeTempRoot() { } afterEach(async () => { + defaultGeminiTestHome = undefined; // A remote run stages a process-session bridge whose detached event writer can // still be flushing a trailing event file into `.../process-sessions//events` // when the run's own best-effort `client.remove(sessionDir)` (which production @@ -179,6 +181,13 @@ async function runExecutor( startupTraceContext?: AdapterExecutionContext["startupTraceContext"]; } = {}, ) { + // Skill reconciliation must never inspect or modify the developer's real + // Gemini home. Keep one private home across this test's resumed turns. + const configuredEnv = config.env as Record | undefined; + if (config.agent === "gemini" && !(typeof configuredEnv?.HOME === "string" && configuredEnv.HOME.trim())) { + defaultGeminiTestHome ??= await makeTempRoot(); + config = { ...config, env: { ...configuredEnv, HOME: defaultGeminiTestHome } }; + } const runtimeOptions: Record[] = []; const configOptions: Array<{ key: string; value: string }> = []; const sessionInputs: Record[] = []; @@ -619,7 +628,7 @@ describe("shared ACPX engine runtime behavior", () => { }); }); - it("keeps Claude startup model handling and Gemini session config handling unchanged", async () => { + it("sets Claude and Gemini models at startup without unsupported picker calls", async () => { const claude = await runExecutor({ agent: "claude", model: "claude-opus-4-7" }); expect((claude.meta[0]?.env as Record).ANTHROPIC_MODEL).toBe( "claude-opus-4-7", @@ -630,9 +639,10 @@ describe("shared ACPX engine runtime behavior", () => { agent: "gemini", model: "gemini-2.5-pro", thinkingEffort: "high", + env: { GEMINI_MODEL: "stale-model" }, }); + expect((gemini.meta[0]?.env as Record).GEMINI_MODEL).toBe("gemini-2.5-pro"); expect(gemini.configOptions).toEqual([ - { key: "model", value: "gemini-2.5-pro" }, { key: "effort", value: "high" }, ]); }); @@ -1600,6 +1610,29 @@ describe("shared ACPX engine runtime behavior", () => { expect(await pathExists(path.join(codexHome, "skills", operational.runtimeName, "SKILL.md"))).toBe(true); }); + it.each(["codex", "gemini"])("refreshes the %s skill root when a connection home changes between turns", async (agent) => { + const root = await makeTempRoot(); + const operational = { + ...await createSkill(path.join(root, "skills"), "paperclip"), + key: "paperclipai/paperclip/paperclip", + }; + const homes = [path.join(root, "first-home"), path.join(root, "next-home")]; + for (const home of homes) { + const { meta } = await runExecutor({ + agent, + stateDir: path.join(root, "state"), + env: agent === "codex" ? { CODEX_HOME: home } : { HOME: home }, + paperclipRuntimeSkills: [operational], + paperclipSkillSync: { desiredSkills: [] }, + }); + const skillsHome = agent === "codex" ? path.join(home, "skills") : path.join(home, ".gemini", "skills"); + expect(String(meta[0]?.prompt ?? "")).toContain(`Skill root for this run: ${skillsHome}`); + expect(String(meta[0]?.prompt ?? "")).toContain("Resolve referenced scripts and files relative to that skill"); + if (home === homes[1]) expect(String(meta[0]?.prompt ?? "")).not.toContain(homes[0]!); + await fs.rm(home, { recursive: true, force: true }); + } + }); + it.skipIf(process.platform === "win32")("removes legacy ACPX Codex skill symlinks when a skill is no longer desired", async () => { const root = await makeTempRoot(); const skillRoot = path.join(root, "skills"); @@ -2886,6 +2919,21 @@ describe("findAncestorBin", () => { }); describe("gemini ACP flag selection", () => { + it("uses the canonical local Gemini workspace for ACP filesystem requests", async () => { + const root = await makeTempRoot(); + const workspace = path.join(root, "workspace"); + const alias = path.join(root, "workspace-alias"); + await fs.mkdir(workspace); + await fs.symlink(workspace, alias, process.platform === "win32" ? "junction" : "dir"); + const { runtimeOptions, sessionInputs } = await runExecutor({ + agent: "gemini", agentCommand: "node ./fake-acp.js", cwd: alias, + stateDir: path.join(root, "state"), + }); + const canonical = await fs.realpath(workspace); + expect(runtimeOptions[0]!.cwd).toBe(canonical); + expect(sessionInputs[0]!.cwd).toBe(canonical); + }); + it("parses semantic version parts from gemini --version output", () => { expect(parseGeminiVersionParts("0.30.0")).toEqual([0, 30, 0]); expect(parseGeminiVersionParts("gemini-cli v1.2.3\n")).toEqual([1, 2, 3]); @@ -3542,6 +3590,16 @@ describe("ACPX engine remote sandbox staging seam (PR 1: workspace + cwd)", () = expect(sessionInputs[0]?.cwd).not.toBe(localCwd); }); + it("preserves Gemini's in-sandbox workspace without host canonicalization", async () => { + const { stateDir, localCwd, remoteCwd, executionTarget } = await setupRemoteSandbox(); + const { sessionInputs, runtimeOptions } = await runExecutor( + { agent: "gemini", agentCommand: "node ./fake-acp.js", stateDir, cwd: localCwd }, + { authToken: "real-run-jwt", executionTarget }, + ); + expect(runtimeOptions[0]?.cwd).toBe(remoteCwd); + expect(sessionInputs[0]?.cwd).toBe(remoteCwd); + }); + it("test_remote_warm_handle_reused_after_cwd_change", async () => { const { stateDir, localCwd, remoteCwd, executionTarget } = await setupRemoteSandbox(); const ensureInputs: Array> = []; diff --git a/packages/adapter-utils/src/acpx-engine/execute.ts b/packages/adapter-utils/src/acpx-engine/execute.ts index 725d733ab6c..c0474eccb38 100644 --- a/packages/adapter-utils/src/acpx-engine/execute.ts +++ b/packages/adapter-utils/src/acpx-engine/execute.ts @@ -2045,6 +2045,12 @@ async function buildRuntime(input: { if (requestedModel && acpxAgent === "claude" && !env.ANTHROPIC_MODEL) { env.ANTHROPIC_MODEL = requestedModel; } + // Gemini CLI reads GEMINI_MODEL when creating each ACP session. Its ACP + // implementation supports set_model, but not set_config_option; configure + // the selected model before startup without changing the transport. + if (requestedModel && acpxAgent === "gemini") { + env.GEMINI_MODEL = requestedModel; + } if (acpxAgent === "codex") { const codexStartupConfig = buildCodexStartupConfig({ existingConfig: env.CODEX_CONFIG, @@ -2197,8 +2203,9 @@ async function buildRuntime(input: { // (fingerprint, compat, persist, ensureSession, error) bind to THIS single // value so a warm/resumable session created with the in-sandbox cwd is reused // — not invalidated — on the next run. Remote runner-backed → the in-sandbox - // workspace dir; local and the runner-less fallback → the HOST cwd, - // byte-identical to today. + // workspace dir; local and the runner-less fallback → the HOST cwd. + // Local Gemini resolves tool paths through realpath. Give its ACP filesystem + // client the same canonical root so workspace aliases stay within that root. // // PR 3: the staging transport derives the in-sandbox workspace dir // deterministically from the target's `remoteCwd` (it is exactly `remoteCwd` @@ -2211,7 +2218,9 @@ async function buildRuntime(input: { const sessionCwd = useRemoteProcessSession && executionTarget?.kind === "remote" ? executionTarget.remoteCwd - : cwd; + : acpxAgent === "gemini" && !executionTargetIsRemote + ? await fs.realpath(cwd) + : cwd; // The 17 fields the session fingerprint hashes. Company, agent, and task // identifiers are NOT here; they scope the outer session key only (see // `keyIdentity`). The fingerprint builder accepts only this identity. @@ -2556,6 +2565,28 @@ async function buildRuntime(input: { await emitRunPhaseTiming(input.ctx, "start_transport", nowMs() - startTransportStart, "failed"); throw err; } + const currentSkillRoot = acpxAgent === "codex" && runtimeEnv.CODEX_HOME + ? path.join(runtimeEnv.CODEX_HOME, "skills") + : acpxAgent === "gemini" && !executionTargetIsRemote && typeof skillsIdentity.skillsHome === "string" + ? skillsIdentity.skillsHome + : null; + if (currentSkillRoot && Array.isArray(skillsIdentity.selectedSkills) && skillsIdentity.selectedSkills.length > 0) { + // Managed connection homes are disposable. A resumed conversation may + // remember a previous home's skill paths; supply the current root on every + // turn, after the remote launch environment has rebased CODEX_HOME. + skillPromptInstructions = [ + `Paperclip has materialized selected runtime skills for this ACPX ${acpxAgent === "codex" ? "Codex" : "Gemini"} run.`, + `Skill root for this run: ${currentSkillRoot}`, + `Selected skills: ${skillsIdentity.selectedSkills.join(", ")}`, + "When a task calls for one of these skills, read its SKILL.md from this root and follow it. Resolve referenced scripts and files relative to that skill. Use this run's root even when an earlier turn used a different path.", + ...(acpxAgent === "gemini" && skillsIdentity.selectedSkills.includes("paperclip") + ? [ + "Before calling Paperclip APIs or delivering task artifacts, read the selected paperclip skill's SKILL.md and its referenced artifact workflow. A completion comment does not upload an artifact or change task status.", + "Gemini's shell tool rejects command substitution. Write JSON request bodies to a workspace file and use curl --data-binary @file, or use the selected skill's helpers, instead of constructing a body with command substitution.", + ] + : []), + ].join("\n"); + } // The relay runs on the host with the sanitized remote launch environment. // Its /usr/bin/env node shebang cannot rely on that environment's PATH. const overrideCommand = processSessionBridge?.agentCommand @@ -2623,13 +2654,14 @@ async function buildRuntime(input: { function sessionConfigOptions(prepared: AcpxPreparedRuntime): Array<{ key: string; value: string }> { const options: Array<{ key: string; value: string }> = []; - // Claude and Codex runtime config is pre-set via startup env vars; skip + // Claude, Codex and Gemini runtime config is pre-set via startup env vars; skip // set_config_option to avoid ACP-server picker validation rejecting valid // backend model IDs that are not advertised by the local ACP server. if ( prepared.requestedModel && prepared.acpxAgent !== "claude" && - prepared.acpxAgent !== "codex" + prepared.acpxAgent !== "codex" && + prepared.acpxAgent !== "gemini" ) { options.push({ key: "model", value: prepared.requestedModel }); } @@ -4775,6 +4807,8 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) { ? `Requested ACPX model: ${prepared.requestedModel} (set via ANTHROPIC_MODEL env at startup).` : prepared.acpxAgent === "codex" ? `Requested ACPX model: ${prepared.requestedModel} (set via CODEX_CONFIG at startup).` + : prepared.acpxAgent === "gemini" + ? `Requested ACPX model: ${prepared.requestedModel} (set via GEMINI_MODEL at startup).` : `Requested ACPX model: ${prepared.requestedModel}.`, ] : []), diff --git a/packages/adapter-utils/src/acpx-engine/filesystem-protocol.test.ts b/packages/adapter-utils/src/acpx-engine/filesystem-protocol.test.ts new file mode 100644 index 00000000000..44da2874807 --- /dev/null +++ b/packages/adapter-utils/src/acpx-engine/filesystem-protocol.test.ts @@ -0,0 +1,110 @@ +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { createRequire } from "node:module"; +import { afterEach, describe, expect, it } from "vitest"; +import { createAcpRuntime, createAgentRegistry, createRuntimeStore } from "acpx/runtime"; + +const runnerRuntimePath = createRequire(new URL("../../../paperclip-runner/package.json", import.meta.url)).resolve("acpx/runtime"); +const nativeRuntime = await import(runnerRuntimePath) as typeof import("acpx/runtime"); +const runtimes = [ + { version: "0.12.0", api: { createAcpRuntime, createAgentRegistry, createRuntimeStore } }, + { version: "0.13.1", api: nativeRuntime }, +]; +const roots: string[] = []; +const fixture = fileURLToPath(new URL("../../../../scripts/mcp-fixtures/servers/acp-filesystem-agent.mjs", import.meta.url)); +afterEach(async () => { + await Promise.all(roots.splice(0).map((root) => fs.rm(root, { recursive: true, force: true }))); +}); + +async function workspace() { + const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-acp-filesystem-")); + roots.push(root); + const cwd = path.join(root, "workspace"); + await fs.mkdir(cwd); + return { root, cwd: await fs.realpath(cwd) }; +} + +async function drive(api: (typeof runtimes)[number]["api"], input: { + cwd: string; + root: string; + permissionMode?: "approve-all" | "deny-all"; + operations: Array<{ method: string; path: string; content?: string }>; +}) { + let stderr = ""; + const runtime = api.createAcpRuntime({ + cwd: input.cwd, + sessionStore: api.createRuntimeStore({ stateDir: path.join(input.root, "state") }), + agentRegistry: api.createAgentRegistry({ overrides: { custom: `${JSON.stringify(process.execPath.replaceAll("\\", "/"))} ${JSON.stringify(fixture.replaceAll("\\", "/"))}` } }), + permissionMode: input.permissionMode ?? "approve-all", + nonInteractivePermissions: "deny", + inheritProcessEnv: false, + onAgentStderr: (chunk: string) => { stderr += chunk; }, + }); + const handle = await runtime.ensureSession({ + sessionKey: "filesystem-protocol", agent: "custom", mode: "oneshot", + cwd: input.cwd, sessionOptions: { env: {} }, + }); + try { + for await (const _event of runtime.runTurn({ + handle, text: JSON.stringify({ operations: input.operations }), + mode: "prompt", requestId: "filesystem-protocol", timeoutMs: 5_000, + })) { /* Drain the real ACP turn. */ } + return stderr.split("\n").filter((line) => line.startsWith("FS_RESULT ")) + .map((line) => JSON.parse(line.slice("FS_RESULT ".length))); + } finally { + await runtime.close({ handle, reason: "done" }); + } +} + +describe.each(runtimes)("ACPX $version filesystem protocol", ({ api }) => { + it("reports a missing file as an ACP resource error and then creates and reads it", async () => { + const input = await workspace(); + const file = path.join(input.cwd, "new.json"); + const content = '{"proof":"synthetic"}\n'; + const receipts = await drive(api, { ...input, operations: [ + { method: "fs/read_text_file", path: file }, + { method: "fs/write_text_file", path: file, content }, + { method: "fs/read_text_file", path: file }, + ] }); + expect(receipts).toHaveLength(3); + expect(receipts[0].error).toMatchObject({ code: -32002, message: expect.stringContaining("Resource not found") }); + expect(receipts[1].error).toBeUndefined(); + expect(receipts[2].result.content).toBe(content); + expect(await fs.readFile(file, "utf8")).toBe(content); + }); + + it("keeps unrelated paths denied for both reads and writes", async () => { + const input = await workspace(); + const outside = path.join(await fs.realpath(input.root), "outside.json"); + await fs.writeFile(outside, "unchanged"); + const receipts = await drive(api, { ...input, operations: [ + { method: "fs/read_text_file", path: outside }, + { method: "fs/write_text_file", path: outside, content: "changed" }, + ] }); + expect(receipts).toHaveLength(2); + expect(receipts.every((receipt) => receipt.error && receipt.error.code !== -32002)).toBe(true); + expect(await fs.readFile(outside, "utf8")).toBe("unchanged"); + }); + + it("preserves deny-all permissions instead of converting denials into missing files", async () => { + const input = await workspace(); + const file = path.join(input.cwd, "denied.json"); + const receipts = await drive(api, { ...input, permissionMode: "deny-all", operations: [ + { method: "fs/read_text_file", path: file }, + { method: "fs/write_text_file", path: file, content: "denied" }, + ] }); + expect(receipts).toHaveLength(2); + expect(receipts.every((receipt) => receipt.error && receipt.error.code !== -32002)).toBe(true); + await expect(fs.stat(file)).rejects.toMatchObject({ code: "ENOENT" }); + }); + + it.skipIf(process.platform === "win32")("keeps other filesystem failures distinct from missing files", async () => { + const input = await workspace(); + const receipts = await drive(api, { ...input, operations: [{ method: "fs/read_text_file", path: input.cwd }] }); + expect(receipts).toHaveLength(1); + expect(receipts[0].error.code).not.toBe(-32002); + }); + +}); diff --git a/packages/adapters/gemini-local/src/index.ts b/packages/adapters/gemini-local/src/index.ts index d71ce497e73..4c21841190a 100644 --- a/packages/adapters/gemini-local/src/index.ts +++ b/packages/adapters/gemini-local/src/index.ts @@ -60,6 +60,10 @@ Operational fields: - graceSec (number, optional): SIGTERM grace period in seconds Notes: +- Explicit ACP models are set via \`GEMINI_MODEL\` before startup; this avoids unsupported \`session/set_config_option\` calls while preserving the ACP transport. Auto continues to use the installed CLI's model selection. +- Local turns include the current selected skill root and Paperclip artifact workflow guidance, including after a disposable connection home changes. +- Local ACP workspaces use their canonical directory so Gemini's resolved file paths match the client boundary. Missing files retain ACP resource-not-found semantics. Gemini CLI 0.58.0 still mishandles the plain error object during native new-file creation; the installed-CLI compatibility smoke documents this remaining limitation. +- Gemini shell commands may reject command substitution. Build JSON request bodies in workspace files and pass them with \`curl --data-binary @file\`, or use the selected skill's helpers. - Gemini ACP is the preferred auto lane when Node >=24.11.0 and the local Gemini CLI command is available. It runs Gemini CLI's native \`gemini --acp\` server through Paperclip's shared ACP engine, including selected skill links, Paperclip runtime prompt/env guidance, model config, and persistent ACP session state. Missing prerequisites fail both default and explicit ACP runs with an actionable setup error; the adapter never switches engines automatically. - Runs use --prompt for non-interactive execution, not stdin. - The adapter sets a headless-safe terminal/browser environment for Gemini CLI child processes so unattended runs do not wait on browser auth or 256-color terminal prompts. diff --git a/packages/adapters/grok-local/src/index.ts b/packages/adapters/grok-local/src/index.ts index f296d574011..4b3d17a062d 100644 --- a/packages/adapters/grok-local/src/index.ts +++ b/packages/adapters/grok-local/src/index.ts @@ -53,4 +53,5 @@ Notes: - Use \`grok models\` to inspect authentication and available models on the host. - Local subscription runs use the host \`grok login\` (\`~/.grok\`) until the company Grok home has a usable \`auth.json\` (sandbox device login). \`XAI_API_KEY\` authenticates without a home. Remote/sandbox runs never fall back to the host login. - Without a usable company login, local runs preserve an inherited or configured \`GROK_HOME\`. Managed AI connections keep their selected home. An explicit empty \`XAI_API_KEY\` clears an inherited key and selects subscription authentication. +- Managed local connections use disposable homes and do not retain or restore session transcripts. Missing local history starts a fresh task handoff without attempting remote subscription recovery. `; diff --git a/packages/adapters/grok-local/src/server/execute.test.ts b/packages/adapters/grok-local/src/server/execute.test.ts index d7995cd8b60..ba18635e386 100644 --- a/packages/adapters/grok-local/src/server/execute.test.ts +++ b/packages/adapters/grok-local/src/server/execute.test.ts @@ -163,6 +163,18 @@ async function makeCtx(runId: string, cwd: string): Promise { + it.each([false, true])("resumes a managed connection only when its isolated history exists (%s)", async present => { + const root = await makeTempRoot(); + const ctx = await makeCtx("managed-resume", root); + ctx.config.managedAiConnection = true; + ctx.config.env = { GROK_HOME: path.join(root, "home"), XAI_API_KEY: "fixture-key" }; + ctx.runtime.sessionParams = { sessionId: "fixture-session", cwd: root }; + if (present) await fs.mkdir(path.join(root, "home", "sessions", "encoded-cwd", "fixture-session"), { recursive: true }); + runProcessMock.mockResolvedValue(makeSuccessfulRunResult({ sessionId: "next-session" })); + const result = await execute(ctx); + expect((runProcessMock.mock.calls[0][3] as string[]).includes("--resume")).toBe(present); + expect(result.sessionParams?.sessionId).toBe("next-session"); + }); it("resumes the conversation while rotating runtime tool access", async () => { const root = await makeTempRoot(); runProcessMock.mockResolvedValue(makeSuccessfulRunResult({ sessionId: "existing-session" })); @@ -180,6 +192,19 @@ describe("grok_local execute", () => { expect(result.sessionParams?.sessionId).toBe("existing-session"); }); + it("clears an unavailable managed session when the fresh turn provides no replacement ID", async () => { + const root = await makeTempRoot(); + const ctx = await makeCtx("missing-history", root); + ctx.config.managedAiConnection = true; + ctx.config.env = { GROK_HOME: path.join(root, "home"), XAI_API_KEY: "fixture-key" }; + ctx.runtime.sessionParams = { sessionId: "discarded-session", cwd: root }; + runProcessMock.mockResolvedValue({ ...makeSuccessfulRunResult(), stdout: JSON.stringify({ type: "end", stopReason: "EndTurn" }) }); + const result = await execute(ctx); + expect((runProcessMock.mock.calls[0][3] as string[]).includes("--resume")).toBe(false); + expect(result.sessionParams).toBeNull(); + expect(result.clearSession).toBe(true); + }); + it.each(["grok-4.7", "grok-4.6"])("forwards the explicit %s model and xhigh effort", async (model) => { const root = await makeTempRoot(); const ctx = await makeCtx("model-selection", root); diff --git a/packages/adapters/grok-local/src/server/execute.ts b/packages/adapters/grok-local/src/server/execute.ts index f2009f46e8f..3d68c9b4847 100644 --- a/packages/adapters/grok-local/src/server/execute.ts +++ b/packages/adapters/grok-local/src/server/execute.ts @@ -47,7 +47,7 @@ import { } from "@paperclipai/adapter-utils/server-utils"; import { DEFAULT_GROK_LOCAL_MODEL } from "../index.js"; import { copyBackGrokAuth } from "./grok-auth-copyback.js"; -import { grokHomeHasUsableAuth, resolveManagedGrokHomeDir, stageGrokHomeForSync } from "./grok-home.js"; +import { grokHomeHasSession, grokHomeHasUsableAuth, resolveManagedGrokHomeDir, stageGrokHomeForSync } from "./grok-home.js"; import { isGrokUnknownSessionError, parseGrokJsonl } from "./parse.js"; const __moduleDir = path.dirname(fileURLToPath(import.meta.url)); @@ -501,12 +501,17 @@ async function executeTurn(ctx: AdapterExecutionContext): Promise 0 && + !missingManagedSession && (runtimeSessionCwd.length === 0 || path.resolve(runtimeSessionCwd) === path.resolve(effectiveExecutionCwd)) && adapterExecutionTargetSessionMatches(runtimeRemoteExecution, runtimeExecutionTarget); const sessionId = canResumeSession ? runtimeSessionId : null; - if (executionTargetIsRemote && runtimeSessionId && !canResumeSession) { + if (missingManagedSession) { + await onLog("stdout", "[paperclip] Selected Grok session history is unavailable. Starting a fresh session with the task handoff instead of remote subscription recovery.\n"); + } else if (executionTargetIsRemote && runtimeSessionId && !canResumeSession) { await onLog( "stdout", `[paperclip] Grok session "${runtimeSessionId}" does not match the current remote execution identity and will not be resumed in "${effectiveExecutionCwd}". Starting a fresh remote session.\n`, @@ -653,7 +658,7 @@ async function executeTurn(ctx: AdapterExecutionContext): Promise { it("holds exactly one name: auth.json", () => { @@ -29,6 +29,16 @@ describe("stageGrokHomeForSync", () => { return root; } + it("starts fresh instead of throwing when private history exceeds the lookup bound", async () => { + const root = await makeRoot("paperclip-grok-large-history-"); + const sessions = path.join(root, "sessions"); + await fs.mkdir(sessions); + for (let offset = 0; offset < 5001; offset += 100) { + await Promise.all(Array.from({ length: Math.min(100, 5001 - offset) }, (_, index) => fs.writeFile(path.join(sessions, `other-${offset + index}.json`), "{}"))); + } + await expect(grokHomeHasSession(root, "requested-session")).resolves.toBe(false); + }); + it("stages auth.json and stages no other file", async () => { const root = await makeRoot("paperclip-grok-stage-"); const home = path.join(root, "grok-home"); diff --git a/packages/adapters/grok-local/src/server/grok-home.ts b/packages/adapters/grok-local/src/server/grok-home.ts index 8a31e28f236..f1384447ece 100644 --- a/packages/adapters/grok-local/src/server/grok-home.ts +++ b/packages/adapters/grok-local/src/server/grok-home.ts @@ -3,8 +3,8 @@ import os from "node:os"; import path from "node:path"; import { resolvePaperclipInstanceRootForAdapter } from "@paperclipai/adapter-utils/server-utils"; -// The Grok credential home. `GROK_HOME` replaces `~/.grok` and holds one file, -// `auth.json`. Unlike Codex, a Grok `auth.json` has no fixed top-level key: it +// `GROK_HOME` replaces `~/.grok` and holds `auth.json` plus session history. +// Unlike Codex, a Grok `auth.json` has no fixed top-level key: it // holds exactly one key, and that key is a composite `::` value. // This module resolves the company-scoped home path and reads its usable-auth // shape. It never writes to the managed home itself; {@link @@ -14,6 +14,29 @@ import { resolvePaperclipInstanceRootForAdapter } from "@paperclipai/adapter-uti const AUTH_FILE_NAME = "auth.json"; +/** Inspect only the selected private home, never Grok's remote registry. */ +export async function grokHomeHasSession(home: string, sessionId: string): Promise { + if (!home || !/^[a-zA-Z0-9_-]{1,128}$/.test(sessionId)) return false; + let remaining = 5000; + async function inspect(directory: string, depth: number): Promise { + let entries; + try { entries = await fs.opendir(directory); } + catch (error) { + if ((error as NodeJS.ErrnoException).code === "ENOENT") return false; + throw error; + } + for await (const entry of entries) { + if (--remaining < 0) return false; + if (entry.isSymbolicLink()) continue; + if ((entry.isFile() || entry.isDirectory()) && + (entry.name === sessionId || entry.name.startsWith(`${sessionId}.`))) return true; + if (entry.isDirectory() && depth > 0 && await inspect(path.join(directory, entry.name), depth - 1)) return true; + } + return false; + } + return inspect(path.join(home, "sessions"), 2); +} + /** * The allowlist of managed `GROK_HOME` entries that the grok-local adapter * stages into the sandbox `home` asset (see {@link stageGrokHomeForSync}). diff --git a/packages/adapters/hermes/README.md b/packages/adapters/hermes/README.md index 64f6ae39dd9..fb9d267e7bf 100644 --- a/packages/adapters/hermes/README.md +++ b/packages/adapters/hermes/README.md @@ -237,6 +237,7 @@ Available toolsets: `terminal`, `file`, `web`, `browser`, `code_execution`, `vis | Field | Type | Default | Description | |-------|------|---------|-------------| +| `cwd` | string | *(assigned task workspace)* | Absolute working directory override. Without an override, use Paperclip's resolved task workspace. | | `persistSession` | boolean | `true` | Resume sessions across heartbeats | | `worktreeMode` | boolean | `false` | Git worktree isolation | | `checkpoints` | boolean | `false` | Enable filesystem checkpoints for rollback | diff --git a/packages/adapters/hermes/src/index.ts b/packages/adapters/hermes/src/index.ts index 3494bf59708..1c1b9d33984 100644 --- a/packages/adapters/hermes/src/index.ts +++ b/packages/adapters/hermes/src/index.ts @@ -99,6 +99,7 @@ tools, persistent memory, session persistence, skills, and MCP support. | Field | Type | Default | Description | |-------|------|---------|-------------| +| cwd | string | (assigned task workspace) | Absolute working directory override; otherwise use Paperclip's resolved task workspace | | persistSession | boolean | true | Resume sessions across heartbeats | | worktreeMode | boolean | false | Use git worktree for isolated changes | | checkpoints | boolean | false | Enable filesystem checkpoints | diff --git a/packages/adapters/hermes/src/server/execute.onspawn.test.ts b/packages/adapters/hermes/src/server/execute.onspawn.test.ts index 1adf450a506..ed0bb48d723 100644 --- a/packages/adapters/hermes/src/server/execute.onspawn.test.ts +++ b/packages/adapters/hermes/src/server/execute.onspawn.test.ts @@ -113,6 +113,18 @@ describe("hermes-local adapter onSpawn forwarding", () => { expect(call[2]).toContainEqual(expect.stringContaining("Current task brief")); }); + it("launches in the assigned task workspace when no cwd was configured", async () => { + const { ctx } = makeCtx(); + await execute({ ...ctx, context: { ...ctx.context, paperclipWorkspace: { cwd: "/private/qa/agent-workspace" } } } as any); + expect(vi.mocked(serverUtils.runChildProcess).mock.calls.at(-1)![3].cwd).toBe("/private/qa/agent-workspace"); + }); + + it("preserves an explicitly configured cwd", async () => { + const { ctx } = makeCtx({ cwd: "/private/qa/configured-workspace" }); + await execute({ ...ctx, context: { ...ctx.context, paperclipWorkspace: { cwd: "/private/qa/agent-workspace" } } } as any); + expect(vi.mocked(serverUtils.runChildProcess).mock.calls.at(-1)![3].cwd).toBe("/private/qa/configured-workspace"); + }); + it("runChildProcess opts type includes onSpawn", () => { // Type-level assertion: if onSpawn were removed from the type, // this file would fail to compile. The runtime test above catches diff --git a/packages/adapters/hermes/src/server/execute.ts b/packages/adapters/hermes/src/server/execute.ts index d835b064bbe..828a3e2e002 100644 --- a/packages/adapters/hermes/src/server/execute.ts +++ b/packages/adapters/hermes/src/server/execute.ts @@ -510,8 +510,12 @@ export async function execute( if (envCommentId) env.PAPERCLIP_WAKE_COMMENT_ID = envCommentId; // ── Resolve working directory ────────────────────────────────────────── + const workspace = ctx.context?.paperclipWorkspace; + const workspaceCwd = workspace && typeof workspace === "object" + ? cfgString((workspace as Record).cwd) + : undefined; const cwd = - cfgString(config.cwd) || cfgString(ctx.config?.workspaceDir) || "."; + cfgString(config.cwd) || workspaceCwd || cfgString(ctx.config?.workspaceDir) || "."; try { await ensureAbsoluteDirectory(cwd); } catch { diff --git a/patches/acpx@0.12.0.patch b/patches/acpx@0.12.0.patch index 9cf65429fcd..fa8410d89b5 100644 --- a/patches/acpx@0.12.0.patch +++ b/patches/acpx@0.12.0.patch @@ -1,8 +1,7 @@ diff --git a/dist/live-checkpoint-ClPCSdrW.js b/dist/live-checkpoint-ClPCSdrW.js -index 243c9d13bcba520923b63adfddad75cf2d94362d..e2a254d4e57fb2e246ccb742f2e9de5a8a461d20 100644 --- a/dist/live-checkpoint-ClPCSdrW.js +++ b/dist/live-checkpoint-ClPCSdrW.js -@@ -1532,7 +1532,7 @@ const ZED_TAG_KEYS = /* @__PURE__ */ new Set([ +@@ -1532,7 +1532,7 @@ "RedactedThinking", "ToolUse" ]); @@ -11,7 +10,7 @@ index 243c9d13bcba520923b63adfddad75cf2d94362d..e2a254d4e57fb2e246ccb742f2e9de5a const OPAQUE_VALUE_PATHS = /* @__PURE__ */ new Set([ "agent_capabilities", "messages.Agent.content.ToolUse.input", -@@ -2557,7 +2557,7 @@ function readCommandLineChar(state) { +@@ -2557,7 +2557,7 @@ escaping: false, hasPart: true }; @@ -20,7 +19,7 @@ index 243c9d13bcba520923b63adfddad75cf2d94362d..e2a254d4e57fb2e246ccb742f2e9de5a current: state.current, quote: state.quote, escaping: true, -@@ -2912,8 +2912,8 @@ function promotePrefixedAuthEnvironment(env) { +@@ -2912,8 +2912,8 @@ } return protectedKeys; } @@ -31,7 +30,7 @@ index 243c9d13bcba520923b63adfddad75cf2d94362d..e2a254d4e57fb2e246ccb742f2e9de5a const protectedAuthEnvKeys = promotePrefixedAuthEnvironment(env); if (authCredentials) for (const [methodId, credential] of Object.entries(authCredentials)) { addAuthCredentialEnvKeys(protectedAuthEnvKeys, methodId, credential); -@@ -2955,10 +2955,10 @@ function resolveConfiguredAuthCredential(methodId, authCredentials) { +@@ -2955,10 +2955,10 @@ const configCredentials = authCredentials ?? {}; return configCredentials[methodId] ?? configCredentials[toEnvToken(methodId)]; } @@ -44,7 +43,7 @@ index 243c9d13bcba520923b63adfddad75cf2d94362d..e2a254d4e57fb2e246ccb742f2e9de5a stdio: [ "pipe", "pipe", -@@ -3724,10 +3724,13 @@ function resolveClientCapabilities(params) { +@@ -3724,10 +3724,13 @@ }, terminal: params.terminal }; @@ -60,7 +59,7 @@ index 243c9d13bcba520923b63adfddad75cf2d94362d..e2a254d4e57fb2e246ccb742f2e9de5a }; } function isDevinRequestDiagnosticsMethod(method) { -@@ -3957,9 +3960,26 @@ var AcpClient = class { +@@ -3957,9 +3960,26 @@ this.lastAgentExit = void 0; this.lastKnownPid = child.pid ?? void 0; this.attachAgentLifecycleObservers(child); @@ -87,7 +86,7 @@ index 243c9d13bcba520923b63adfddad75cf2d94362d..e2a254d4e57fb2e246ccb742f2e9de5a if (!this.options.verbose) return; process.stderr.write(chunk); }); -@@ -3994,7 +4014,7 @@ var AcpClient = class { +@@ -3994,7 +4014,7 @@ geminiAcp: isGeminiAcpCommand(spawnCommand, args), copilotAcp: isCopilotAcpCommand(spawnCommand, args), claudeAcp: isClaudeAcpCommand(spawnCommand, args), @@ -96,7 +95,17 @@ index 243c9d13bcba520923b63adfddad75cf2d94362d..e2a254d4e57fb2e246ccb742f2e9de5a }; } logAgentLaunch(plan) { -@@ -6070,6 +6090,27 @@ async function withConnectedSession(options) { +@@ -4729,6 +4749,9 @@ + return await this.filesystem.readTextFile(params); + } catch (error) { + this.recordPermissionError(params.sessionId, error); ++ // Keep ENOENT distinct from permission and other filesystem failures. ++ // The ACP peer must receive a resource error before it creates a file. ++ if (error && typeof error === "object" && error.code === "ENOENT") throw RequestError.resourceNotFound(params.path); + throw error; + } + } +@@ -6070,6 +6093,27 @@ //#region src/runtime/engine/prompt-turn.ts const SESSION_REPLY_IDLE_MS = 1e3; const SESSION_REPLY_DRAIN_TIMEOUT_MS = 5e3; @@ -124,7 +133,7 @@ index 243c9d13bcba520923b63adfddad75cf2d94362d..e2a254d4e57fb2e246ccb742f2e9de5a async function runPromptTurn(params) { try { const promptPromise = params.client.prompt(params.sessionId, params.prompt); -@@ -6079,6 +6120,20 @@ async function runPromptTurn(params) { +@@ -6079,6 +6123,20 @@ idleMs: SESSION_REPLY_IDLE_MS, timeoutMs: SESSION_REPLY_DRAIN_TIMEOUT_MS }).catch(() => {}); diff --git a/patches/acpx@0.13.1.patch b/patches/acpx@0.13.1.patch index 99c29c87fe6..de1218f0c4c 100644 --- a/patches/acpx@0.13.1.patch +++ b/patches/acpx@0.13.1.patch @@ -11,10 +11,18 @@ index 5e2113ac0a1a92c99322cf01e5c106760a0b0dbf..b7b5151f3da45b5e0d12aea55e9f3b07 private setSessionModelThroughConfig; private setSessionModelThroughLegacyMethod; diff --git a/dist/live-checkpoint-BSIrfgVo.js b/dist/live-checkpoint-BSIrfgVo.js -index d454fd7c5bf742b469be75eb8c3988b694a0ffaa..e3a4989473bba20e6dcdccbac086bb6d2e777e9e 100644 --- a/dist/live-checkpoint-BSIrfgVo.js +++ b/dist/live-checkpoint-BSIrfgVo.js -@@ -1068,6 +1068,7 @@ function serializeSessionRecordForDisk(record) { +@@ -6,7 +6,7 @@ + import { randomUUID } from "node:crypto"; + import { execFile, spawn } from "node:child_process"; + import { Readable, Writable } from "node:stream"; +-import { PROTOCOL_VERSION, client, methods } from "@agentclientprotocol/sdk"; ++import { PROTOCOL_VERSION, RequestError, client, methods } from "@agentclientprotocol/sdk"; + import readline from "node:readline/promises"; + //#region src/errors.ts + var AcpxOperationalError = class extends Error { +@@ -1068,6 +1068,7 @@ last_agent_disconnect_reason: canonical.lastAgentDisconnectReason, protocol_version: canonical.protocolVersion, agent_capabilities: canonical.agentCapabilities, @@ -22,7 +30,7 @@ index d454fd7c5bf742b469be75eb8c3988b694a0ffaa..e3a4989473bba20e6dcdccbac086bb6d title: canonical.title, messages: canonical.messages, updated_at: canonical.updated_at, -@@ -1239,7 +1240,8 @@ function parseRequestTokenUsage(raw) { +@@ -1239,7 +1240,8 @@ for (const [key, value] of Object.entries(record)) { const parsed = parseTokenUsage(value); if (parsed == null) return null; @@ -32,7 +40,7 @@ index d454fd7c5bf742b469be75eb8c3988b694a0ffaa..e3a4989473bba20e6dcdccbac086bb6d } return usage; } -@@ -1541,6 +1543,7 @@ function parseSessionRecord(raw) { +@@ -1541,6 +1543,7 @@ lastAgentDisconnectReason: optionals.lastAgentDisconnectReason, protocolVersion: typeof record.protocol_version === "number" ? record.protocol_version : void 0, agentCapabilities: asRecord$4(record.agent_capabilities), @@ -40,7 +48,7 @@ index d454fd7c5bf742b469be75eb8c3988b694a0ffaa..e3a4989473bba20e6dcdccbac086bb6d title: conversation.title, messages: conversation.messages, updated_at: conversation.updated_at, -@@ -1661,7 +1664,7 @@ const ZED_TAG_KEYS = /* @__PURE__ */ new Set([ +@@ -1661,7 +1664,7 @@ "RedactedThinking", "ToolUse" ]); @@ -49,7 +57,7 @@ index d454fd7c5bf742b469be75eb8c3988b694a0ffaa..e3a4989473bba20e6dcdccbac086bb6d const OPAQUE_VALUE_PATHS = /* @__PURE__ */ new Set([ "agent_capabilities", "messages.Agent.content.ToolUse.input", -@@ -3104,10 +3107,10 @@ function authEnvKey(methodId) { +@@ -3104,10 +3107,10 @@ authEnvKeyCache.set(methodId, key); return key; } @@ -62,7 +70,7 @@ index d454fd7c5bf742b469be75eb8c3988b694a0ffaa..e3a4989473bba20e6dcdccbac086bb6d if (typeof value === "string" && value.trim().length > 0) return value; } function protectedEnvKey(key) { -@@ -3135,8 +3138,21 @@ function promotePrefixedAuthEnvironment(env) { +@@ -3135,8 +3138,21 @@ } return protectedKeys; } @@ -86,7 +94,7 @@ index d454fd7c5bf742b469be75eb8c3988b694a0ffaa..e3a4989473bba20e6dcdccbac086bb6d const protectedAuthEnvKeys = promotePrefixedAuthEnvironment(env); if (authCredentials) for (const [methodId, credential] of Object.entries(authCredentials)) { addAuthCredentialEnvKeys(protectedAuthEnvKeys, methodId, credential); -@@ -3178,10 +3194,10 @@ function resolveConfiguredAuthCredential(methodId, authCredentials) { +@@ -3178,10 +3194,10 @@ const configCredentials = authCredentials ?? {}; return configCredentials[methodId] ?? configCredentials[toEnvToken(methodId)]; } @@ -99,7 +107,7 @@ index d454fd7c5bf742b469be75eb8c3988b694a0ffaa..e3a4989473bba20e6dcdccbac086bb6d stdio: [ "pipe", "pipe", -@@ -3961,10 +3977,13 @@ function resolveClientCapabilities(params) { +@@ -3961,10 +3977,13 @@ ...params.elicitationModes.includes("url") ? { url: {} } : {} } } : {} }; @@ -115,7 +123,7 @@ index d454fd7c5bf742b469be75eb8c3988b694a0ffaa..e3a4989473bba20e6dcdccbac086bb6d }; } function hasResponseField(response, field) { -@@ -4067,6 +4086,27 @@ function createNdJsonMessageStream(agentCommand, output, input) { +@@ -4067,6 +4086,27 @@ } }) }; } @@ -143,7 +151,7 @@ index d454fd7c5bf742b469be75eb8c3988b694a0ffaa..e3a4989473bba20e6dcdccbac086bb6d var AcpClient = class { options; connection; -@@ -4105,7 +4145,9 @@ var AcpClient = class { +@@ -4105,7 +4145,9 @@ cwd: asAbsoluteCwd(options.cwd), authPolicy: options.authPolicy ?? "skip", permissionPolicy: snapshotPermissionPolicy(options.permissionPolicy), @@ -154,7 +162,7 @@ index d454fd7c5bf742b469be75eb8c3988b694a0ffaa..e3a4989473bba20e6dcdccbac086bb6d }; this.eventHandlers = { onAcpMessage: this.options.onAcpMessage, -@@ -4216,9 +4258,21 @@ var AcpClient = class { +@@ -4216,9 +4258,21 @@ this.lastAgentExit = void 0; this.lastKnownPid = child.pid ?? void 0; this.attachAgentLifecycleObservers(child); @@ -176,7 +184,7 @@ index d454fd7c5bf742b469be75eb8c3988b694a0ffaa..e3a4989473bba20e6dcdccbac086bb6d if (!this.options.verbose) return; process.stderr.write(chunk); }); -@@ -4226,6 +4280,7 @@ var AcpClient = class { +@@ -4226,6 +4280,7 @@ const output = Readable.toWeb(child.stdout); const stream = this.createTappedStream(createNdJsonMessageStream(this.options.agentCommand, input, output)); const connection = this.createConnection(stream, launch); @@ -184,7 +192,7 @@ index d454fd7c5bf742b469be75eb8c3988b694a0ffaa..e3a4989473bba20e6dcdccbac086bb6d connection.signal.addEventListener("abort", () => { this.recordAgentExit("connection_close", child.exitCode ?? null, child.signalCode ?? null); }, { once: true }); -@@ -4253,7 +4308,12 @@ var AcpClient = class { +@@ -4253,7 +4308,12 @@ geminiAcp: isGeminiAcpCommand(spawnCommand, args), copilotAcp: isCopilotAcpCommand(spawnCommand, args), claudeAcp: isClaudeAcpCommand(spawnCommand, args), @@ -198,7 +206,7 @@ index d454fd7c5bf742b469be75eb8c3988b694a0ffaa..e3a4989473bba20e6dcdccbac086bb6d }; } logAgentLaunch(plan) { -@@ -4279,10 +4339,17 @@ var AcpClient = class { +@@ -4279,10 +4339,17 @@ } async spawnAgentProcess(plan) { const spawnCommand = buildAgentSpawnCommand(plan.spawnCommand, plan.args, process.platform, plan.spawnOptions.env); @@ -218,7 +226,7 @@ index d454fd7c5bf742b469be75eb8c3988b694a0ffaa..e3a4989473bba20e6dcdccbac086bb6d try { await waitForSpawn$1(spawnedChild); } catch (error) { -@@ -4293,10 +4360,10 @@ var AcpClient = class { +@@ -4293,10 +4360,10 @@ createConnection(stream, launch) { const app = client({ name: "acpx" }).onNotification(methods.client.session.update, async ({ params }) => { await this.handleSessionUpdate(params); @@ -232,7 +240,7 @@ index d454fd7c5bf742b469be75eb8c3988b694a0ffaa..e3a4989473bba20e6dcdccbac086bb6d }).onRequest(methods.client.fs.readTextFile, async ({ params }) => { return await this.handleReadTextFile(params); }).onRequest(methods.client.fs.writeTextFile, async ({ params }) => { -@@ -4312,6 +4379,11 @@ var AcpClient = class { +@@ -4312,6 +4379,11 @@ }).onRequest(methods.client.terminal.release, async ({ params }) => { return await this.handleReleaseTerminal(params); }); @@ -244,7 +252,7 @@ index d454fd7c5bf742b469be75eb8c3988b694a0ffaa..e3a4989473bba20e6dcdccbac086bb6d if (launch.devinAcp) app.onRequest("_cognition.ai/request_diagnostics", (params) => { return params && typeof params === "object" && !Array.isArray(params) ? params : {}; }, async () => ({})); -@@ -4333,12 +4405,12 @@ var AcpClient = class { +@@ -4333,12 +4405,12 @@ async initializeProtocolConnection(connection, launch) { const initializePromise = connection.initialize({ protocolVersion: PROTOCOL_VERSION, @@ -259,7 +267,7 @@ index d454fd7c5bf742b469be75eb8c3988b694a0ffaa..e3a4989473bba20e6dcdccbac086bb6d clientInfo: resolveClientInfo(launch.devinAcp) }); const initialized = launch.geminiAcp ? await withTimeout(initializePromise, resolveGeminiAcpStartupTimeoutMs()) : await initializePromise; -@@ -4358,6 +4430,36 @@ var AcpClient = class { +@@ -4358,6 +4430,36 @@ throw normalizedError; } createTappedStream(base) { @@ -296,7 +304,7 @@ index d454fd7c5bf742b469be75eb8c3988b694a0ffaa..e3a4989473bba20e6dcdccbac086bb6d const onAcpMessage = () => this.eventHandlers.onAcpMessage; const onAcpOutputMessage = () => this.eventHandlers.onAcpOutputMessage; const elicitationRequestIds = /* @__PURE__ */ new Set(); -@@ -4367,8 +4469,9 @@ var AcpClient = class { +@@ -4367,8 +4469,9 @@ const shouldSuppressInboundReplaySessionUpdate = (message) => { return this.suppressReplaySessionUpdateMessages && isSessionUpdateNotification(message); }; @@ -307,7 +315,7 @@ index d454fd7c5bf742b469be75eb8c3988b694a0ffaa..e3a4989473bba20e6dcdccbac086bb6d if (requestId !== void 0) { elicitationRequestIds.add(requestId); return; -@@ -4379,6 +4482,7 @@ var AcpClient = class { +@@ -4379,6 +4482,7 @@ } }; return { @@ -315,7 +323,7 @@ index d454fd7c5bf742b469be75eb8c3988b694a0ffaa..e3a4989473bba20e6dcdccbac086bb6d readable: new ReadableStream({ async start(controller) { const reader = base.readable.getReader(); try { -@@ -4387,10 +4491,16 @@ var AcpClient = class { +@@ -4387,10 +4491,16 @@ if (done) break; if (!value) continue; observeInbound(value); @@ -333,7 +341,7 @@ index d454fd7c5bf742b469be75eb8c3988b694a0ffaa..e3a4989473bba20e6dcdccbac086bb6d reader.releaseLock(); controller.close(); } -@@ -4403,10 +4513,20 @@ var AcpClient = class { +@@ -4403,10 +4513,20 @@ onAcpOutputMessage()?.("outbound", message); onAcpMessage()?.("outbound", message); } @@ -354,7 +362,7 @@ index d454fd7c5bf742b469be75eb8c3988b694a0ffaa..e3a4989473bba20e6dcdccbac086bb6d writer.releaseLock(); } } }) -@@ -4422,7 +4542,10 @@ var AcpClient = class { +@@ -4422,7 +4542,10 @@ const createPromise = this.runConnectionRequest(() => connection.newSession({ cwd: sessionCwd, mcpServers: this.options.mcpServers ?? [], @@ -366,7 +374,7 @@ index d454fd7c5bf742b469be75eb8c3988b694a0ffaa..e3a4989473bba20e6dcdccbac086bb6d })); result = claudeAcp ? await withTimeout(createPromise, resolveClaudeAcpSessionCreateTimeoutMs()) : await createPromise; } catch (error) { -@@ -4864,7 +4987,7 @@ var AcpClient = class { +@@ -4864,7 +4987,7 @@ } selectAuthMethod(methods) { for (const method of methods) { @@ -375,7 +383,7 @@ index d454fd7c5bf742b469be75eb8c3988b694a0ffaa..e3a4989473bba20e6dcdccbac086bb6d if (envCredential) return { methodId: method.id, credential: envCredential, -@@ -4890,7 +5013,7 @@ var AcpClient = class { +@@ -4890,7 +5013,7 @@ } readAgentSpecificEnvCredential(methodId) { if (!this.isGrokBuildAcpCommand() || methodId !== "xai.api_key") return; @@ -384,7 +392,7 @@ index d454fd7c5bf742b469be75eb8c3988b694a0ffaa..e3a4989473bba20e6dcdccbac086bb6d return typeof value === "string" && value.trim().length > 0 ? value : void 0; } selectAgentManagedAuthMethod(methodId) { -@@ -4915,9 +5038,9 @@ var AcpClient = class { +@@ -4915,9 +5038,9 @@ await connection.authenticate({ methodId: selected.methodId }); this.log(`authenticated with method ${selected.methodId} (${selected.source})`); } @@ -396,7 +404,7 @@ index d454fd7c5bf742b469be75eb8c3988b694a0ffaa..e3a4989473bba20e6dcdccbac086bb6d if (hostResponse) return hostResponse; const { response, recorded } = await this.resolvePermissionRequestFromMode(params); if (!recorded) { -@@ -4926,14 +5049,49 @@ var AcpClient = class { +@@ -4926,14 +5049,49 @@ } return response; } @@ -448,7 +456,7 @@ index d454fd7c5bf742b469be75eb8c3988b694a0ffaa..e3a4989473bba20e6dcdccbac086bb6d })).then((response) => ({ kind: "response", response -@@ -4970,7 +5128,7 @@ var AcpClient = class { +@@ -4970,7 +5128,7 @@ isElicitationSessionCancelling(sessionId) { return this.closing || this.cancellingSessionIds.has(sessionId); } @@ -457,7 +465,7 @@ index d454fd7c5bf742b469be75eb8c3988b694a0ffaa..e3a4989473bba20e6dcdccbac086bb6d if (!this.options.onPermissionRequest) return; const signal = this.cancellationSignalForSession(params.sessionId); try { -@@ -4978,7 +5136,7 @@ var AcpClient = class { +@@ -4978,7 +5136,7 @@ sessionId: params.sessionId, raw: params, inferredKind: inferToolKind(params) @@ -466,7 +474,7 @@ index d454fd7c5bf742b469be75eb8c3988b694a0ffaa..e3a4989473bba20e6dcdccbac086bb6d return this.hostPermissionDecisionResponse(params, signal, decision); } catch (error) { return this.hostPermissionErrorResponse(params, signal, error); -@@ -5028,6 +5186,12 @@ var AcpClient = class { +@@ -5028,6 +5186,12 @@ attachAgentLifecycleObservers(child) { child.once("exit", (exitCode, signal) => { this.recordAgentExit("process_exit", exitCode, signal); @@ -479,7 +487,17 @@ index d454fd7c5bf742b469be75eb8c3988b694a0ffaa..e3a4989473bba20e6dcdccbac086bb6d }); child.once("close", (exitCode, signal) => { this.recordAgentExit("process_close", exitCode, signal); -@@ -5771,10 +5935,20 @@ function recordSessionUpdate(conversation, state, notification, timestamp = isoN +@@ -5100,6 +5264,9 @@ + return await this.filesystem.readTextFile(params); + } catch (error) { + this.recordPermissionError(params.sessionId, error); ++ // Keep ENOENT distinct from permission and other filesystem failures. ++ // The ACP peer must receive a resource error before it creates a file. ++ if (error && typeof error === "object" && error.code === "ENOENT") throw RequestError.resourceNotFound(params.path); + throw error; + } + } +@@ -5771,10 +5938,20 @@ trimConversationForRuntime(conversation); return acpx; } @@ -502,7 +520,7 @@ index d454fd7c5bf742b469be75eb8c3988b694a0ffaa..e3a4989473bba20e6dcdccbac086bb6d updateConversationTimestamp(conversation, timestamp); trimConversationForRuntime(conversation); return true; -@@ -6435,6 +6609,27 @@ async function withConnectedSession(options) { +@@ -6435,6 +6612,27 @@ //#region src/runtime/engine/prompt-turn.ts const SESSION_REPLY_IDLE_MS = 1e3; const SESSION_REPLY_DRAIN_TIMEOUT_MS = 5e3; @@ -530,7 +548,7 @@ index d454fd7c5bf742b469be75eb8c3988b694a0ffaa..e3a4989473bba20e6dcdccbac086bb6d async function runPromptTurn(params) { try { const promptPromise = params.client.prompt(params.sessionId, params.prompt, params.onPromptRequestStarted, params.onElicitation); -@@ -6444,7 +6639,22 @@ async function runPromptTurn(params) { +@@ -6444,7 +6642,22 @@ idleMs: SESSION_REPLY_IDLE_MS, timeoutMs: SESSION_REPLY_DRAIN_TIMEOUT_MS }).catch(() => {}); diff --git a/scripts/mcp-fixtures/servers/acp-filesystem-agent.mjs b/scripts/mcp-fixtures/servers/acp-filesystem-agent.mjs new file mode 100644 index 00000000000..c8248ee5fdf --- /dev/null +++ b/scripts/mcp-fixtures/servers/acp-filesystem-agent.mjs @@ -0,0 +1,43 @@ +#!/usr/bin/env node +// Credential-free ACP filesystem requests, driven through the real client. +import { createInterface } from "node:readline"; + +const pending = new Map(); +let nextId = 100; +const send = (message) => process.stdout.write(`${JSON.stringify(message)}\n`); +const request = (method, params) => new Promise((resolve) => { + const id = nextId++; + pending.set(id, resolve); + send({ jsonrpc: "2.0", id, method, params }); +}); + +createInterface({ input: process.stdin }).on("line", async (line) => { + const message = JSON.parse(line); + if (pending.has(message.id)) { + pending.get(message.id)(message); + pending.delete(message.id); + return; + } + let result; + if (message.method === "initialize") { + result = { + protocolVersion: 1, + agentCapabilities: { loadSession: false, sessionCapabilities: { close: {} } }, + agentInfo: { name: "paperclip-acp-filesystem-fixture", version: "1.0.0" }, + }; + } else if (message.method === "session/new") { + result = { sessionId: "filesystem-fixture" }; + } else if (message.method === "session/prompt") { + const { operations } = JSON.parse(message.params.prompt.find((entry) => entry.type === "text").text); + for (const { method, ...params } of operations) { + const response = await request(method, { sessionId: "filesystem-fixture", ...params }); + process.stderr.write(`FS_RESULT ${JSON.stringify(response)}\n`); + } + result = { stopReason: "end_turn" }; + } else if (message.method === "session/cancel") { + return; + } else { + result = {}; + } + if (message.id !== undefined) send({ jsonrpc: "2.0", id: message.id, result }); +}); diff --git a/server/src/__tests__/ai-connections.test.ts b/server/src/__tests__/ai-connections.test.ts index 63bac4ff3a4..378fc3669ea 100644 --- a/server/src/__tests__/ai-connections.test.ts +++ b/server/src/__tests__/ai-connections.test.ts @@ -4,12 +4,14 @@ import { issueRecoveryActionService } from "../services/issue-recovery-actions.j import { localAiLoginService } from "../services/local-ai-login.js"; import * as localCredentials from "../services/local-ai-credentials.js"; import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; -import { randomUUID } from "node:crypto"; +import { createHash, randomUUID } from "node:crypto"; +import { spawnSync } from "node:child_process"; +import { localEncryptedProvider } from "../secrets/local-encrypted-provider.js"; import { mkdtemp, realpath, rm, access, readFile, writeFile, stat } from "node:fs/promises"; import os from "node:os"; import path from "node:path"; import { and, eq, sql } from "drizzle-orm"; -import { createDb, companies, agents, heartbeatRuns, companyMemberships, connectionGrants, toolApplications, connectionGrantDelegations, connectionGrantMembers, toolConnections, toolConnectionInstalls, aiConnectionDefaults, aiProviderDefaults, adapterAuthSessions, environments, issues, issueThreadInteractions, issueRecoveryActions, connectionIntentDeliveries, agentWakeupRequests, companySecrets, principalPermissionGrants } from "@paperclipai/db"; +import { createDb, companies, agents, agentTaskSessions, heartbeatRuns, companyMemberships, connectionGrants, toolApplications, connectionGrantDelegations, connectionGrantMembers, toolConnections, toolConnectionInstalls, aiConnectionDefaults, aiProviderDefaults, adapterAuthSessions, environments, issues, issueThreadInteractions, issueRecoveryActions, connectionIntentDeliveries, agentWakeupRequests, companySecrets, principalPermissionGrants } from "@paperclipai/db"; import { startEmbeddedPostgresTestDatabase } from "@paperclipai/db/test-embedded-postgres"; import { aiConnectionService } from "../services/ai-connections.js"; import * as executionTarget from "@paperclipai/adapter-utils/execution-target"; @@ -73,6 +75,44 @@ describe("managed AI connections", () => { expect(grant.credentialSecretRefs).toEqual([]); } finally { await Promise.all([first.cleanup(), second.cleanup()]); } }); + it("never restores retained Grok transcripts into same-user runtime homes", async () => { + const account = await service.save(companyId, "alice", { + provider: "xai", method: "api_key", ownership: "personal", + name: "Grok history isolation", agentIds: [], allAgents: true, + }, "fixture-grok-key"); + const taskKey = `grok-history-${randomUUID()}`; + const runInput = { ...input, taskKey, adapterType: "grok_local", responsibleUserId: "alice", config: {}, + binding: { provider: "xai", method: "api_key", mode: "connection", connectionId: account.connectionId, grantId: account.grantId } as const }; + const first = await prepareManagedAiRuntime(db, runInput); + const scope = createHash("sha256").update(JSON.stringify([ + companyId, agentId, taskKey, first.sessionIdentity, + ])).digest("hex"); + await first.cleanup(); + // Seed a valid archive from the earlier implementation. Runtime preparation + // must not decrypt it onto the host, even for its authorized agent/task. + const retained = await localEncryptedProvider.createVersion({ value: JSON.stringify({ + scope, entries: [{ name: "history.json", bytes: Buffer.from("previous-task-private-transcript").toString("base64") }], + }) }); + await db.insert(agentTaskSessions).values({ companyId, agentId, adapterType: "grok_local", taskKey, + sessionParamsJson: { sessionId: "history", paperclipGrokHistory: { scope, material: retained.material } } }); + const resumed = await prepareManagedAiRuntime(db, runInput); + const peer = await prepareManagedAiRuntime(db, { ...runInput, agentId: randomUUID() }); + try { + expect(resumed.home).not.toBe(first.home); + expect(peer.home).not.toBe(resumed.home); + const transcript = path.join(String(resumed.config.env.GROK_HOME), "sessions", "history.json"); + // A separate same-UID process can bypass 0700. It must find no restored + // transcript because none was materialized, not because of permissions. + const crossAgentRead = spawnSync(process.execPath, ["-e", ` + const fs = require("node:fs"); + try { process.stdout.write(fs.readFileSync(process.argv[1], "utf8")); } + catch (error) { process.stdout.write(error.code); } + `, transcript], { encoding: "utf8" }); + expect(crossAgentRead.status).toBe(0); + expect(crossAgentRead.stdout).toBe("ENOENT"); + await expect(access(transcript)).rejects.toMatchObject({ code: "ENOENT" }); + } finally { await Promise.all([resumed.cleanup(), peer.cleanup()]); } + }); it("authenticates local Gemini probes and runs with the saved key in an isolated home", async () => { const root = await mkdtemp(path.join(home, "gemini-auth-")); const command = path.join(root, "gemini"); diff --git a/server/src/__tests__/file-delivery-bridges.test.ts b/server/src/__tests__/file-delivery-bridges.test.ts index 16ab0b6ad18..c43f6ca4c3c 100644 --- a/server/src/__tests__/file-delivery-bridges.test.ts +++ b/server/src/__tests__/file-delivery-bridges.test.ts @@ -99,9 +99,9 @@ describe(`durable file delivery (${liveDaytona ? "Daytona" : "local processes"}) enableSandboxDuplexBridge: duplex, }); expect(bridge?.env.PAPERCLIP_API_BRIDGE_MODE).toBe(duplex ? "http2_v1" : "queue_v1"); - const env = { ...bridge!.env, PAPERCLIP_RUN_ID: fixture.runId, + const env = { ...bridge!.env, TMPDIR: path.join(remote.workspace, ".unavailable-controller-temp"), PAPERCLIP_RUN_ID: fixture.runId, PAPERCLIP_COMPANY_ID: fixture.companyId, PAPERCLIP_TASK_ID: fixture.issueId, - PAPERCLIP_HELPER_STATE_DIR: path.join(remote.workspace, ".helper-state") }; + PAPERCLIP_WORKSPACE_CWD: remote.workspace }; const headers = { authorization: `Bearer ${bridge!.env.PAPERCLIP_API_KEY}` }; const receipts: Array<{ attachment: { id: string; contentPath: string; downloadPath: string } }> = []; try { diff --git a/skills/paperclip/references/artifacts.md b/skills/paperclip/references/artifacts.md index ad2f8493653..e632e7d58ae 100644 --- a/skills/paperclip/references/artifacts.md +++ b/skills/paperclip/references/artifacts.md @@ -156,3 +156,8 @@ confirms storage and response binding in Paperclip; it does not confirm delivery to an external chat provider. If registration fails, use the returned error to resolve the failure or explain the limitation; do not describe a workspace path as an uploaded file. + +The shell helper keeps temporary responses and retry receipts in the assigned +workspace's private `.paperclip/artifact-helper` directory, so CLI sandboxes can +write them even when the controller's `TMPDIR` is inaccessible. +`PAPERCLIP_HELPER_STATE_DIR` overrides that directory when explicitly configured. diff --git a/skills/paperclip/scripts/paperclip-upload-artifact.sh b/skills/paperclip/scripts/paperclip-upload-artifact.sh index 0dfa111d5e2..87733dded19 100755 --- a/skills/paperclip/scripts/paperclip-upload-artifact.sh +++ b/skills/paperclip/scripts/paperclip-upload-artifact.sh @@ -113,6 +113,17 @@ sha256_text() { fi } +helper_state_directory() { + printf '%s' "${PAPERCLIP_HELPER_STATE_DIR:-${PAPERCLIP_WORKSPACE_CWD:-$PWD}/.paperclip/artifact-helper}" +} + +helper_response_file() { + local directory + directory="$(helper_state_directory)" + # TMPDIR can name a controller-owned directory outside a CLI sandbox. + (umask 077; mkdir -p "$directory" && mktemp "$directory/response.XXXXXX") +} + request_json() { local method="$1" local url="$2" @@ -120,7 +131,7 @@ request_json() { local response_file local status_code - response_file="$(mktemp)" + response_file="$(helper_response_file)" if [[ -n "$body" ]]; then status_code="$( curl -sS -X "$method" -w '%{http_code}' -o "$response_file" \ @@ -163,7 +174,7 @@ upload_file() { escaped_path="${path//\\/\\\\}" escaped_path="${escaped_path//\"/\\\"}" - response_file="$(mktemp)" + response_file="$(helper_response_file)" status_code="$( curl -sS -X POST -w '%{http_code}' -o "$response_file" \ "$url" \ @@ -223,7 +234,7 @@ acquire_operation_lock() { local attempts=0 umask 077 - operation_state_root="${PAPERCLIP_HELPER_STATE_DIR:-${TMPDIR:-/tmp}/paperclip-upload-artifact}" + operation_state_root="$(helper_state_directory)" mkdir -p "$operation_state_root" operation_lock_path="$operation_state_root/$operation_key.lock" operation_lock_owner="$$|$(process_start_identity "$$" || true)" diff --git a/tests/ai-connections-app/app.spec.ts b/tests/ai-connections-app/app.spec.ts index 1f1036e5720..933b355b2de 100644 --- a/tests/ai-connections-app/app.spec.ts +++ b/tests/ai-connections-app/app.spec.ts @@ -135,25 +135,22 @@ test("explicit OpenAI API method survives continuing and reloading", async ({ pa }); for (const [provider, label] of [["anthropic", "Claude"], ["openai", "OpenAI"]]) { - test(`Connections reuses the agent provider step for ${label}`, async ({ page }, testInfo) => { + test(`Connections reuses the local browser sign-in step for ${label} (simulated auth)`, async ({ page }, testInfo) => { + // UI contract only: real subscription qualification is provider-connections. + const base = `/api/companies/${companyId}/ai-connections/local`; + const sessionId = "22222222-2222-4222-8222-222222222222"; + await page.route(`**${base}/attempts`, route => route.fulfill({ json: { sessionId, expiresAt: new Date(Date.now() + 300000).toISOString() } })); + await page.route(`**${base}/check`, route => route.fulfill({ json: { status: "sign_in_required", authorizationUrl: "https://provider.example/authorize", ...(provider === "openai" ? { code: "ABCD-EFGH" } : {}) } })); + await page.route(`**${base}/attempts/${sessionId}`, route => route.fulfill({ json: { ok: true } })); await page.goto(`/${prefix}/apps/connect?source=${provider}&method=ai-subscription`); - await page.getByRole("button", { name: /^(Save and continue|Continue)$/ }).click(); await expect(page.getByRole("radiogroup", { name: "Connect your model provider" })).toBeVisible(); await expect(page.getByRole("button", { name: "Connect for tool access instead" })).toHaveCount(0); await page.getByRole("radio", { name: new RegExp(label) }).click(); - if (provider === "anthropic") { - await expect(page.getByText(/Connect uses your local/)).toBeVisible(); - await expect(page.getByText("claude auth login", { exact: true })).toBeVisible(); - } else { - await expect(page.getByText(/Your existing terminal login stays separate/)).toBeVisible(); - const command = page.getByText(/^CODEX_HOME=.* codex login$/); - await expect(command).toBeVisible(); - const preparedCommand = await command.textContent(); - await page.reload(); - await page.getByRole("radio", { name: new RegExp(label) }).click(); - await expect(command).toHaveText(preparedCommand!); - } - await expect(page.getByRole("button", { name: "Connect", exact: true })).toBeEnabled(); + await expect(page.getByRole("link", { name: `Sign in to ${label}`, exact: true })).toHaveAttribute("href", "https://provider.example/authorize"); + await expect(page.getByText(/CODEX_HOME=|claude auth login|existing terminal login/)).toHaveCount(0); + if (provider === "anthropic") await expect(page.getByRole("button", { name: "Submit code", exact: true })).toBeDisabled(); + else await expect(page.getByText("ABCD-EFGH", { exact: true })).toBeVisible(); + await expect(page.getByRole("button", { name: "Connect", exact: true })).toBeDisabled(); // Let the shared tile-collapse and card-enter animations settle for visual review. await page.waitForTimeout(1000); await page.screenshot({ path: testInfo.outputPath(`${provider}-shared-provider-step.png`), fullPage: true }); diff --git a/tests/runner-e2e/FIXTURES.md b/tests/runner-e2e/FIXTURES.md index 4d5fd0874b8..2264d9742c1 100644 --- a/tests/runner-e2e/FIXTURES.md +++ b/tests/runner-e2e/FIXTURES.md @@ -1,5 +1,11 @@ # Runner E2E fixture authoring +## Connection creation fixtures + +See [PROVIDER-CONNECTIONS.md](PROVIDER-CONNECTIONS.md) for the explicit-only +`provider-connections` suite, local/staging target ownership, dedicated browser +profiles, credential handoffs, private evidence, and cleanup contract. + The [public MCP journeys](PUBLIC-MCP.md) reuse the fixture registry with a real authenticated browser session. `RunnerApi.setBrowserSession` binds that session to API calls, including encrypted secret provisioning via Node fetch. OAuth diff --git a/tests/runner-e2e/PROVIDER-CONNECTIONS.md b/tests/runner-e2e/PROVIDER-CONNECTIONS.md new file mode 100644 index 00000000000..01ba619fcca --- /dev/null +++ b/tests/runner-e2e/PROVIDER-CONNECTIONS.md @@ -0,0 +1,498 @@ +# Live provider connections + +`provider-connections` exercises real connection creation through the Apps catalog +or new-agent wizard, then verifies a real agent task and follow-up. It is an +explicit-only Product E2E suite. It is excluded from `--all` and the paid GitHub +matrix: subscription login initially runs attended on a developer's machine. + +Tasks are submitted through the production composer. When it accepts only a +prompt, the harness assigns its fixture title through the public API. This +changes naming metadata only; execution, status, attachments and downloaded +bytes are produced and verified independently. + +## Run one journey + +Create a private JSON configuration outside the repository. Its values reference +credentials; do not paste keys into it. For example: + +```json +{ + "target": { "mode": "managed-local" }, + "browser": { + "headed": true, + "channel": "chrome", + "profileDir": "/absolute/private/path/provider-qa-browser", + "accountAlias": "developer-qa", + "freshness": "signed-in", + "loginTimeoutMs": 600000 + }, + "secretFile": "/absolute/private/path/provider-qa.env", + "models": { "codex": "YOUR_CODEX_MODEL", "claude": "YOUR_CLAUDE_MODEL" }, + "routes": { + "openrouter": { "model": "YOUR_OPENROUTER_MODEL" }, + "bedrock": { "model": "YOUR_BEDROCK_MODEL", "region": "YOUR_AWS_REGION" }, + "responses": { + "baseURL": "https://your-gateway.example/v1", + "credentialEnv": "QA_RESPONSES_KEY", + "model": "YOUR_GATEWAY_MODEL", + "auth": "bearer" + } + }, + "budgetCents": 200, + "maxRuns": 3, + "turnTimeoutMs": 300000 +} +``` + +The secret file must be an owner-only regular file (`chmod 600`), with literal +`NAME=value` entries. Only the selected variable is read. Shell substitutions +are rejected. Environment variables take precedence. Default names are +`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `XAI_API_KEY`, `GEMINI_API_KEY`, +`OPENROUTER_API_KEY`, and `AWS_BEARER_TOKEN_BEDROCK`. Override direct-key names +with `credentials: {"codex": "DEDICATED_QA_OPENAI_KEY"}`. Subscription cases +read no provider API key and never import CLI credential homes. + +Routes may be shared by method or overridden per harness, such as +`"claude.openrouter"`. Protocol gateways require their own explicit credential +reference, or `auth: "none"` for a deliberately unauthenticated endpoint. +Messages gateways also support `auth: "api_key"` (`x-api-key`). Bedrock requires +a current bearer token and region. The harness does not mint or refresh AWS +tokens. Missing credentials/configuration produce a blocked result. + +```sh +pnpm test:e2e:runner -- --list --suite provider-connections +pnpm test:e2e:runner -- \ + --id provider-connections.connection-codex-native.local.agent-api-key \ + --connection-config /absolute/private/path/connections.json +pnpm test:e2e:runner -- \ + --id provider-connections.connection-claude-legacy.local.apps-subscription \ + --connection-config /absolute/private/path/connections.json +``` + +Start with one cell. Running `--suite provider-connections` selects all 116 +eligible cells and can spend money; it does not make missing routes available. +Each command generates a separate campaign, preserving previous attempts. + +## Authentication handoff + +Playwright drives Paperclip and opens the real provider sign-in page. When login +needs a person, the terminal prints the provider and account alias and writes +`progress.json`. Complete Google/provider login, MFA, consent, and any code +entry in the visible QA browser. The test resumes only after Paperclip reports +a saved, connected credential. Do not paste passwords or OAuth codes into chat. + +The QA browser uses its own private profile. It refuses an existing unmarked +profile, keeps a lock while running, and never uses your normal Chrome profile. +`signed-in` permits previously saved website login; it still creates a fresh +Paperclip connection. `signed-out` starts a temporary empty browser profile. +Paperclip SSO and provider SSO may share an identity provider, so this records +the browser's initial state, not a guarantee of another password challenge. + +Human help is recorded as `assisted`. A login deadline records `awaiting_user`, +not a pass, and stops the selected campaign before opening another login. +Rerun the remaining cell IDs to start a fresh authorization after expiry; +resumption is within the running process, not across a stopped process. +Ctrl-C, termination, and hangup close the test browser pages, cancel recorded login attempts and active +runs, performs fixture teardown, records interruption, and stops the campaign. +Each cell has a thirty-minute outer deadline. A forced process kill cannot run +teardown; inspect the private QA profile lock and target fixtures before rerunning. +If Google rejects the automated browser, use the production device-code or +code-return flow in your normal browser. Grok's current production flow may +require the displayed terminal sign-in command. These are attended results. + +## Local and staging targets + +`managed-local` owns a fresh server/database and removes its data on shutdown. +To use an existing local server or staging, replace `target` with: + +```json +{ + "mode": "attach", + "baseURL": "https://YOUR-INSTANCE.staging.paperclip.app", + "expectedCommit": "EXACT_40_CHARACTER_DEPLOYED_COMMIT_SHA", + "deploymentMode": "authenticated" +} +``` + +For local attach use `http://127.0.0.1:PORT` and `local_trusted`. Health must +match the expected revision and deployment mode before credentials are read. +The target is never reset, restarted, or given instance-setting changes. +Sign in to Paperclip in the QA browser if needed; that session supplies the +same public API permissions as the browser. The operator needs company/agent +creation, connection management, and company archival permissions. + +Deployment and execution environment are different dimensions. `.local.` means +the agent runs in the target's local environment, including a staging server's +local environment. `.daytona.` requires an already configured active Daytona +environment on the selected target; this suite does not provision one. Use +`target.companyId` and `target.environmentId` to select an empty dedicated +company named `Connection QA…` with that environment. For a new QA company the +harness uses the matching visible instance environment. Missing environments +or a disabled native runner are explicit target blockers. + +Each cell creates a separate QA company unless one was supplied. Teardown +pauses its agents, cancels active runs, revokes its test connections, and +archives the created company. This avoids requiring hard-delete support on +staging and retains inspectable task history. Supplied QA companies must start +without active agents or usable connections; revoked connection records and +terminated agents from earlier attempts are allowed. Their new test agents are +deleted and test connections revoked; historical records may remain. `retainCompany: true` +keeps the fixture and credentials for diagnosis, with agents paused, and is +recorded explicitly in evidence. Clean it up before reusing that company. + +## What qualifies as a pass + +Every checkpoint must be verified: + +1. Expected target revision and deployment mode. +2. Fresh connection saved with the selected method/provider/route. +3. Connection visible after navigating away and reloading Apps. +4. Agent created through the wizard with the selected harness/model/binding and requested execution environment. +5. Real Configure-page environment probe passes. +6. Real task run succeeds with that exact managed connection's attribution and requested environment in its durable run context. +7. Agent decodes random input bytes supplied in the task, computes their sum/count/hash, and delivers + an attachment whose JSON is checked independently by the harness. +8. Tool evidence and attachment authorship/run attribution are present. +9. A browser-submitted follow-up reuses the connection and delivers independently + checked minimum/maximum/sum output. + +The oracle input is base64 in the task description and must be decoded with a +tool, preserving its exact bytes. This makes connection qualification independent +of new-task attachment ingestion. The earlier uploaded-input OpenRouter attempt +remains a failed attachment-path test: native staging currently admits chat +attachments, and the new-task dialog uploads its files after scheduling the agent. +Successful provider cells do not qualify that task-level upload path. +10. Downloadable artifacts remain visible after a page reload; cleanup succeeds. + +An answer saying “provider routing works” cannot pass. Evidence records the +configured model and agent configuration; connection attribution proves the +selected credential, not an upstream gateway's internal model mapping. + +The matrix covers Claude, Codex, Grok, Gemini, OpenCode, and Hermes local where +Paperclip's managed-connection capability table admits them, with applicable +legacy/native variants and both UI entry points. Cursor local, Kimi, and Pi +remain visible product gaps for managed connection onboarding. Remote OpenClaw +and Hermes gateways, Cursor Cloud, managed Claude/AWS AgentCore, process/HTTP, +and generic legacy ACPX are deliberately excluded. + +## Reports, privacy, and CI + +Results are under `tests/runner-e2e/results/connections-…/`: `dashboard.html`, +`campaign.json`, `normalized-results.json`, `connection-coverage.json`, `progress.json`, and packaged +per-cell evidence. The report uses the existing Product E2E grader, billing, +history, and dashboard. Coverage reports required/selected/attempted/passed +counts separately. A partial local campaign does not qualify staging or the +whole matrix. Blocked credentials/login are incomplete; cleanup failures fail. + +No Playwright traces, HARs, videos, automatic screenshots, raw assertion call +logs, OAuth URLs/codes, or provider transcripts are retained. Only the final +synthetic task can be screenshotted; it is private and has no public publication +marker. Auth browser state stays outside the repository. Keep reports private. +The result includes source revision, target revision, suite definition hash, +model, method, assistance, run IDs, checkpoints, timing, and numeric usage. +Before removing a managed instance, the harness retains terminal run status, +log availability, and closed diagnostic codes for recognized quota, overload, +file-conversion, session-configuration, or shell-restriction failures. It reads +error records and failed tool receipts only; messages, stderr, model output, +and reasoning are not copied into the result. An empty signal list leaves the +cause unknown. Missing logs remain explicit and do not imply a healthy run. +Attached-company cleanup captures these diagnostics after stopping runs and +before deleting the fixture agent, since agent deletion also removes its logs. +Cleanup revokes only this attempt's account IDs from successful creation or +owned sign-in receipts. Concurrent campaigns' accounts are never adopted or +revoked, even if both campaigns initially observed an empty QA company. +UI probes may incur unreported spend, so billing is explicitly partial. +Regenerate the report from retained results with +`pnpm test:e2e:runner:dashboard -- /absolute/path/to/connections-campaign`. + +API-key cells can run headless (`browser.headed: false`, `channel: "chromium"`) +on trusted CI once the target's board authentication is provided through a +dedicated QA browser session. Subscription cells require headed mode and a +person when challenged. Dedicated subscription accounts and a trusted attended +runner are the next operational step; normal untrusted PR jobs never get these +credentials. Credential-free support tests and catalog checks run separately: + +```sh +pnpm test:e2e:runner:typecheck +pnpm test:e2e:runner:unit +``` + +The opt-in cancellation smoke uses a synthetic loopback page and cookie, with +no provider calls or credentials. It verifies that Ctrl-C, termination, and +hangup leave the browser's authenticated API client usable until fixture cleanup +finishes: `pnpm exec tsx tests/runner-e2e/connection-cancel.smoke.ts`. + +## Initial verification (2026-10-03) + +The local native Codex/direct OpenAI key/new-agent cell completed both real +artifact turns with exact connection attribution and successful cleanup. The +Apps/OpenRouter/native Codex cell created and reused its connection and passed +its setup probe, then failed the task proof: its agent could not retrieve the +uploaded input and marked the task blocked. This is not a gateway qualification +pass. The Claude legacy/Apps subscription journey opened real sign-in and +exercised the bounded `awaiting_user` outcome; login was not completed. + +Credential-free verification: runner E2E typecheck, 855 support tests, both +updated simulated local-browser-login regressions, catalog discovery, and +retained-report regeneration passed. A fresh managed-local server was started +and removed with an intentionally missing credential; no provider was called. +Staging and the remaining cells have not been qualified. Use the generated +campaigns for the exact revision, definition hash, timestamps, and results. + +## Local follow-up (2026-10-05) + +Every eligible non-subscription local cell was attempted against the existing +onboarding instance, preserving its original company data. The latest retained +attempts now contain 43 passing journeys out of 46 non-subscription cells, +including the Gemini investigation on October 6: + +| Harness | Passing key/gateway journeys | Eligible local key/gateway journeys | Remaining result | +| --- | ---: | ---: | --- | +| Claude | 16 | 16 | Direct key, OpenRouter, Bedrock, and Messages passed through both entry points and runner generations. | +| Codex | 12 | 12 | The legacy/new-agent OpenRouter task and follow-up passed after the skill-root and artifact-helper fixes. | +| OpenCode | 8 | 8 | All three remaining OpenRouter journeys passed after the artifact-helper and canonical-workspace permission fixes. | +| Hermes local | 4 | 4 | A follow-up interrupted by lease release passed on one bounded retry. Workspace isolation was then corrected and retested separately. | +| Grok | 2 | 4 | Both legacy API-key journeys passed after preserving private session history and avoiding remote subscription restoration for missing local history. Native journeys require the missing pinned runtime. | +| Gemini | 1 | 2 | New-agent/API-key passed with `gemini-3.5-flash-lite`; Apps/API-key delivered its first artifact but timed out on follow-up. Installed CLI file creation still fails the isolated compatibility check. | + +There are another 12 local subscription cells. Attended Claude/OpenAI attempts +opened real authorization and reached their human handoff, but have not qualified +a completed subscription journey. Staging and Daytona remain untested. The +Responses and Messages gateway proofs used the official provider endpoints; +they do not qualify an Emissary deployment or every compatible gateway. + +The run exposed a real new-agent loading race: an early API-key selection could +be discarded when environment settings finished loading and remounted Connect. +Connect now waits for those settings before exposing interactive choices. Saved +connection selectors and native OpenCode model normalization were repaired in +the harness; original failures remain in their original campaigns. Native +OpenCode no longer sends the deprecated per-prompt tools override, which replaced +its session permission policy; its remaining live OpenRouter failures were subsequently fixed and retested below. + +Hermes also ignored the assigned task workspace when an agent had no explicit +`cwd`, writing proof files into the server checkout. It now uses the resolved +task workspace as its default. Explicit directory overrides retain their existing +behavior. The real retest checks that the local proof file is in the assigned +directory and matches the independently downloaded artifact. Earlier checkout +spill files were preserved privately outside the repository. + +An interrupted subscription attempt exposed another cleanup bug: Playwright's +default signal handler closed the browser before the harness could cancel the +login using its authenticated API client. The harness now owns browser shutdown, +and a real-browser smoke verified cleanup under SIGINT, SIGTERM, and SIGHUP. +One cancellation state now covers the entire campaign, including target startup, +report generation, and teardown. Credential-free regressions verify that all three +signals stop further cells during startup and reporting and still stop the owned +target. Signal handlers remain installed until teardown finishes. +An expired human handoff also stops the campaign instead of opening the next +subscription window. The affected attempt remains failed in the evidence; its +login was cancelled and its disposable company archived through the public API. +The cleanup audit covers all 113 disposable companies and confirms that the +original onboarding company remains available. + +Hermes was tested using an isolated official CLI installation. Its temporary +PATH launcher was removed after qualification; replay requires Hermes to be +installed on the tested server's PATH. The report records the tested CLI version +and source commit in `qualification-environment.json`. + +Credential-free verification passed: runner E2E typecheck and all 856 support +tests, 46 new-agent UI tests, 39 native OpenCode driver tests, 10 Hermes invocation +tests, runner/Hermes package typechecks, and UI token gates. After the final +browser signal change, runner E2E typecheck, 13 connection support tests, and the +three-signal browser smoke passed again. This initial qualification stage did +not repeat repo-wide typecheck, tests, and build; later verification is recorded +below. + +### Credential follow-up, evening of 2026-10-05 + +The xAI dashboard identified expiry as the original key rejection: its QA key +expired on September 27. The existing key was renewed for 90 days, through +January 3, 2027, preserving its scope and limits. A Gemini replacement was saved +in the owner-only secret file; the temporary exposed key was revoked and its +absence verified after a dashboard reload. No provider billing settings changed. +Both provider model-list requests returned HTTP 200. A separate bounded Gemini +generation request succeeded with `gemini-3.8-flash`; the older +`gemini-2.5-flash` request returned HTTP 404. Model-list success alone does not +qualify inference or a product journey. + +Six previously credential-blocked cells were rerun, followed by three bounded +Gemini attempts after new diagnostics and a selector repair. At that stage none added a fully +qualified journey, leaving 36 of 46 local key/gateway cells passing. Those 91 +attempts and the subsequent repair attempts remain inspectable. These evening +campaigns have October 6 UTC timestamps and belong to the developer's October 5 +local qualification date. + +- Grok legacy, both entry points: the first task delivered an independently + verified artifact. Each follow-up reported its session absent locally, + attempted remote conversation restoration, and waited for subscription device + authorization during an API-key journey. Investigate preserving isolated + non-credential session state across disposable managed credential homes. +- Grok native, both entry points: connection creation and reload passed, but + Configure was not reached. An independent non-inference installation probe + confirmed the required Grok Build 1.0.13 executable was missing. Provision it + using the checked-in helper in `doc/grok-native-runner.md`, then rerun those + exact cells; retain the runtime integrity checks. +- Gemini: `Auto` completed ACP startup but timed out before task output. With + `gemini-3.8-flash`, both entry points created and reloaded a connection, passed + setup, and started the real task, then failed with + `acpx_session_config_failed`. Installed Gemini CLI 0.58.0 rejected + `session/set_config_option` for the model with ACP `-32601` (Method not found). + Resolve that runtime compatibility before claiming a pass. The Apps selector + now reads the shared app definition's name (`Google Gemini`); its original + selector failures remain retained. + +After the catalog selector repair, runner E2E typecheck and all 13 connection +support tests passed. The final cleanup audit found no usable QA connections, +active QA agents, or active QA runs; the original onboarding company and test +server remain available. The private report includes `credential-refresh.json`, +`key-retest-diagnostics.json`, `gemini-generation-probe.json`, and +`cleanup-audit.json`. None contain credential values or authorization codes. + +The private `connections-qualification-2026-10-05` dashboard selects the newest +attempt per cell. `retained-attempts.json` and `attempts.md` retain all original +campaigns. Source and suite definition hashes are preserved: these runs span +harness repairs and an uncommitted working tree, so their passing count is +historical evidence, not a common-build release qualification. Commit the reviewed +fixes and rerun the required cells against that exact deployed revision before +claiming the whole feature qualified. + +### Repair verification, late evening of 2026-10-05 + +The previously failing Codex OpenRouter journey, both legacy Grok API-key +journeys, and all three OpenCode OpenRouter journeys now have passing browser +retests with independently downloaded artifacts and successful follow-ups. +Original failures remain in the report; no assistant success message alone +qualifies a journey. + +The fixes refresh Codex's current skill root on resumed turns, keep artifact +helper temporary files inside the selected workspace, allow native OpenCode's +assigned workspace under both its supplied and canonical filesystem paths, +and keep credentials disposable. Missing managed local Grok history starts a +fresh task handoff instead of triggering remote subscription recovery during an +API-key turn. + +On 2026-10-06, host-side Grok transcript retention/restoration was removed after +security review: other agents running as the same OS user could read restored +transcripts regardless of private file modes. Managed Grok follow-ups now start +fresh with the Paperclip task handoff. The historical Grok passes above precede +this change and do not qualify current transcript continuation; an isolated +history solution and new live qualification remain required. + +Gemini now receives its selected model through `GEMINI_MODEL` before ACP startup +instead of an unsupported `session/set_config_option` call. Real tasks confirmed +startup and tool execution. The original `gemini-3.8-flash` attempts subsequently +hit daily inference quota; a small request for `gemini-3.1-flash-lite` succeeded. +These are distinct from model-list and connection-probe success. + +Local Gemini turns now also receive the current selected skill root and +explicit instructions to read the Paperclip artifact workflow before making +control-plane calls. A real follow-up exposed Gemini CLI's rejection of inline +command substitution when building a JSON completion request. Its local turn +guidance now uses workspace JSON body files with `curl --data-binary @file` or +the selected skill's helpers; the shell restriction remains in place. Regression checks cover current roots across disposable +homes for both Codex and Gemini. The shared ACP regression file now passes all +214 tests after isolating Gemini test homes from the developer's CLI state and +adding canonical-workspace and remote-workspace regressions. + +A credential-free real ACP reproduction isolated two filesystem compatibility +problems behind Gemini's file-write errors. The client now translates a genuine +missing-file read into the correct ACP resource-not-found error. This verifies +the client protocol response; the installed Gemini CLI still mishandles that +response, as isolated below. Local Gemini sessions also bind the filesystem client to +the canonical workspace directory. Remote workspace paths are preserved. The +existing ACPX dependency patch and its lockfile hash contain this narrow fix; +four real protocol regressions cover missing-file semantics, external path denial, deny-all +permissions, and other filesystem errors. The full affected ACP group passes +221 tests, including the existing three spawn safeguards. An earlier combined +verification failure and the failed browser attempts remain retained. + +The final filesystem-fix campaign (`connections-2026-10-06T04-50-27-196Z-1ec2a9`) +still failed both Gemini entry points at the unchanged 300-second first-task +deadline. Connection creation, reload, Configure probe, and binding passed. +The new-agent turn created its local proof files, but neither turn completed +the successful run, uploaded-artifact, and follow-up qualification. The earlier +filesystem and model-configuration errors were absent from these tool receipts; +that does not qualify the full workflow. Further work must investigate Gemini +task execution latency and skill use with the configured CLI/model, then rerun +the same independent oracle. No timeout increase or cancelled-run pass was used. + +A further Gemini attempt exposed a harness settlement race: the first successful +turn did not attach the file, while an automatic disposition-repair run did. +The harness now waits for active repair runs within the existing configured +limits and selects the successful run that actually uploaded the required file. +It still rejects cancelled runs, wrong connection/agent attribution, wrong +artifact bytes, and exceeded run/cost limits. The repair-attribution regression +suite passed (58 tests across the targeted support/catalog files). + +The task-composer helper was updated for the current production controls and +its remembered project selection. A credential-free real browser smoke proved +Plan and Ask mode, assignee/project selection, fixture task naming, and file +upload. Public-API title updates only name fixtures; execution, credentials, +model selection and artifact delivery remain real product behavior. + +Full workspace typecheck and build passed. UI token gates and shell syntax +passed. The broad Vitest attempt did not produce a green full-suite result: +stale catalog/pool assertions were corrected and checked narrowly, while +additional worker-start, hook and process timeouts remain recorded. Serial +adapter/server rechecks were interrupted after more deadline failures; they do +not constitute a passing full-suite run. UI/CLI and +shared/database/skill-catalog groups passed. Further isolated checks do not +substitute for a successful complete PR verification run. + +The pinned native Grok executable is still absent at +`/opt/paperclip/providers/grok/1.0.13/grok`. Installing it needs administrator +access; retain the checked-in provisioning helper and its checksum verification. +Subscriptions, staging/Daytona, and common-build release qualification remain +outstanding. + +## Gemini investigation (2026-10-06) + +The installed Gemini CLI is 0.58.0. A credential-free check now exercises its +actual ACP file tools against the patched ACPX client, with a loopback model +response and disposable home/workspace: + +```sh +pnpm exec tsx tests/runner-e2e/gemini-acp-filesystem.smoke.ts +``` + +On the stock CLI, reading an existing file and rejecting an external write pass; +creating a new file fails. Its ACP SDK rejects a plain JSON error object, while +`AcpFileSystemService.normalizeFileSystemError` reads the message only from an +`Error` instance. Consequently it misses the genuine resource-not-found message +and fails the initial read-before-write. The same implementation is present in +the [upstream source](https://github.com/google-gemini/gemini-cli/blob/main/packages/cli/src/acp/acpFileSystemService.ts). +A temporary copy that also reads a plain object's string `message` passes all +three checks. The installed CLI has not been changed, and the real browser +qualification uses the stock CLI. A workflow that recovers using shell tools +does not qualify the broken native file tool. + +Retained child-stderr metadata also confirms HTTP 503 overload retries in the +earlier deadline failures. A bounded availability probe returned HTTP 503 for +`gemini-3.8-flash` and valid HTTP 200 output for `gemini-3.5-flash-lite`. +With Flash-Lite, the full new-agent/API-key journey passed, including two +independently downloaded and verified artifacts and an attributed successful +follow-up. The initial Apps/API-key journey created, reloaded, tested, and bound +the connection, then failed its first task. It did not retain a cause before +managed-instance teardown; that gap prompted the closed diagnostic projection +above. One diagnostic retest (`connections-2026-10-06T13-25-36-718Z-20ee5d`) +delivered and independently verified the first artifact, then exceeded the +unchanged 300-second follow-up deadline. Both run statuses and log availability +were retained before cleanup, but neither log contained a recognized error +signal. The follow-up cause remains unknown; overload in other runs is not +proof of overload in this run. No deadline or oracle was relaxed. + +The CLI requests `HIGH` thinking by default. Short bounded comparisons did not +establish that default thinking causes the long workflow delay. Explicit effort +configuration still uses an ACP method this CLI rejects; supported startup +`modelConfigs` settings need a separate adapter fix and local/remote validation. +Do not silently drop the user's setting or claim that a model-list/setup probe +qualifies execution. + +The original 3109 instance is currently stopped, and its temporary config and +encryption key are missing. Its database was left untouched; substituting a new +key would not recover the existing credentials. Fresh investigation campaigns +use managed instances with separate databases and keys, archive their owned +companies, and remove their owned servers. The private consolidated report +retains all earlier failures, exact model/source provenance, and +`gemini-investigation.json`. These mixed working-tree attempts do not qualify +staging or a common committed release build. diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index 7c0ed71c97f..7e0f3b15819 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -1,5 +1,11 @@ # Paid runner full-stack E2E +## Live provider connection journeys + +See [PROVIDER-CONNECTIONS.md](PROVIDER-CONNECTIONS.md) for the explicit-only +`provider-connections` suite, local/staging target ownership, dedicated browser +profiles, credential handoffs, private evidence, and cleanup contract. + The [public MCP suite](PUBLIC-MCP.md) adds explicit paid assistant/plugin journeys with authenticated browser consent, real MCP tool use and independently graded team execution. Select `--suite public-mcp`; it is excluded from `--all`. diff --git a/tests/runner-e2e/SECURITY.md b/tests/runner-e2e/SECURITY.md index ee79e10609a..2e727279b43 100644 --- a/tests/runner-e2e/SECURITY.md +++ b/tests/runner-e2e/SECURITY.md @@ -1,5 +1,19 @@ # Runner E2E security for a public repository +## Attended connection authentication + +See [PROVIDER-CONNECTIONS.md](PROVIDER-CONNECTIONS.md) for the explicit-only +`provider-connections` suite, local/staging target ownership, dedicated browser +profiles, credential handoffs, private evidence, and cleanup contract. + +Provider-connection results may retain closed diagnostic codes and terminal run +status before managed-instance teardown. Only error records and failed tool +receipts are inspected; raw error messages, child stderr, model output and +reasoning are not retained. Unknown causes and unavailable logs stay explicit. +The optional installed-Gemini filesystem smoke uses synthetic credentials and +loopback model responses, makes no real provider calls, and removes its +disposable home/workspace. + The explicit [public MCP suite](PUBLIC-MCP.md) also creates a disposable browser account and OAuth grant. Signup and token exchange use Node fetch, credentials stay outside model context, and dynamic cookie/code/token values join the diff --git a/tests/runner-e2e/case-outcome.ts b/tests/runner-e2e/case-outcome.ts index 25d58a74179..55d1b91c825 100644 --- a/tests/runner-e2e/case-outcome.ts +++ b/tests/runner-e2e/case-outcome.ts @@ -1,4 +1,5 @@ import type { RunnerE2EResult } from "./types.js"; +import { isBlockedConnectionResult } from "./connection-evidence.js"; const html = (value: unknown) => String(value ?? "").replace( /[&<>"']/g, @@ -12,6 +13,7 @@ export function isIncompleteFirstTaskResult( result: RunnerE2EResult, errors: readonly string[] = [], ) { + if (isBlockedConnectionResult(result)) return result.cleanup === "passed" && result.failureClass !== "secret_leak" && errors.length === 0; const checks = result.firstTask?.checks ?? []; return result.status === "failed" && result.failureClass === "candidate_failure" && @@ -30,7 +32,7 @@ export function renderCaseOutcome( (result.matcherResults ?? []).map((m) => ({ id: m.matcher.kind === "json_path" ? m.matcher.path : m.matcher.kind, passed: m.passed, - notReached: undefined as string | undefined, + notReached: isBlockedConnectionResult(result) && !m.passed ? "Blocked before this checkpoint" : undefined as string | undefined, detail: m.detail, })); const failures = checks.filter((c) => !c.passed && !c.notReached); @@ -54,7 +56,8 @@ export function renderCaseOutcome( ), ]; return `
- ${incomplete ? "Incomplete journey" : failed ? "Overall failed" : "Overall passed"} · ${checks.length ? `${passed}/${evaluated} behavioral checks passed${notReached.length ? ` · ${notReached.length} not reached` : ""}` : "No behavioral checks recorded"} + ${isBlockedConnectionResult(result) ? `Blocked: ${html(result.providerConnection!.outcome.replaceAll("_", " "))}` : incomplete ? "Incomplete journey" : failed ? "Overall failed" : "Overall passed"} · ${checks.length ? `${passed}/${evaluated} behavioral checks passed${notReached.length ? ` · ${notReached.length} not reached` : ""}` : "No behavioral checks recorded"} + ${result.providerConnection ? `

Target: ${html(result.providerConnection.target.origin)} · ${html(result.providerConnection.target.commit ?? "revision unavailable")} · ${result.providerConnection.assisted ? "Assisted login" : "Unassisted"} · ${html(result.providerConnection.authFreshness)} browser · ${html(result.providerConnection.entry)} / ${html(result.providerConnection.method)} · stopped at ${html(result.providerConnection.phase)}

` : ""} ${failures.length ? `

Failed checks:

    ${failures.map((c) => `
  • ${html(c.id)} — ${html(c.detail)}
    • `).join("")}
` : checks.length ? `

No behavioral matcher failed.${notReached.length ? " The journey is incomplete; unexercised checks are not passes." : failed ? " The overall failure came from a separate run, cleanup, or evidence check." : ""}

` : ""} ${notReached.length ? `

Not reached:

    ${notReached.map((c) => `
  • ${html(c.id)} — ${html(c.notReached)}
    • `).join("")}
` : ""} ${checks.length ? `
See all behavioral checks (${checks.length})${checks.map((c) => ``).join("")}
ResultCheckDetail
${c.notReached ? "Not reached" : c.passed ? "Pass" : "Fail"}${html(c.id)}${html(c.detail)}${c.notReached ? ` — ${html(c.notReached)}` : ""}
` : ""} diff --git a/tests/runner-e2e/catalog.test.ts b/tests/runner-e2e/catalog.test.ts index 813bfaf3ee9..8c70c7122c9 100644 --- a/tests/runner-e2e/catalog.test.ts +++ b/tests/runner-e2e/catalog.test.ts @@ -146,10 +146,10 @@ describe("runner E2E catalog", () => { expect(localIntegrityTasks).toHaveLength(2); expect(openRouterBreadthTasks).toHaveLength(3); expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([ - 39, 12, 6, 30, 3, 16, 16, 2, 6, 8, 46, 23, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1, + 39, 12, 6, 30, 3, 16, 16, 2, 6, 8, 46, 23, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1, 116, ]); - expect(validateRunnerCatalog()).toHaveLength(529); - expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(529); + expect(validateRunnerCatalog()).toHaveLength(645); + expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(645); expect( runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"), ).toHaveLength(48); @@ -438,13 +438,13 @@ describe("runner E2E catalog", () => { it("uses only declared secret references in generated payloads", () => { expect( - runnerMatrix.every((entry) => + runnerMatrix.filter(entry => entry.task.flow !== "provider_connection").every((entry) => entry.requiredCredentials.includes(entry.profile.credential), ), ).toBe(true); expect( runnerMatrix - .filter((entry) => entry.environment.id === "daytona") + .filter((entry) => entry.environment.id === "daytona" && entry.task.flow !== "provider_connection") .every((entry) => entry.requiredCredentials.includes("DAYTONA_API_KEY"), ), diff --git a/tests/runner-e2e/catalog.ts b/tests/runner-e2e/catalog.ts index b5ddce12114..daae718a1a0 100644 --- a/tests/runner-e2e/catalog.ts +++ b/tests/runner-e2e/catalog.ts @@ -3,6 +3,7 @@ import { nativeCompletionTasks, nativeCompletionDefinitionDigest } from "./nativ import { NATIVE_INSTRUCTION_SUITE, NATIVE_INSTRUCTION_BASE_SHA, nativeInstructionDefinitionDigest } from "./native-instruction-consolidation.js"; import { nativeCompletionProfile, NATIVE_COMPLETION_BUDGET_CENTS } from "./native-completion-defaults.js"; import { chatConfirmationTasks } from "./chat-cases.js"; +import { buildConnectionSuite } from "./connection-cases.js"; import { hiringTemplateTasks, hiringTemplateProfile, hiringTemplateDefinitionDigest } from "./hiring-template-cases.js"; import { instructionPersistenceTask } from "./instruction-persistence.js"; import { apiResponseReadingTask } from "./api-response-reading.js"; @@ -1489,6 +1490,7 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [ expectedMatrixSize: 1, definitionMetadata: { version: 8, instructions: "fixed-external", nativeIdleTimeoutMs: 1_200_000, autoStopIntervalMinutes: 25, generatedFileCount: 60_000, filenameBytes: 39_828_890, scheduling: "explicit-only", finalization: "committed-without-active-sync-or-retry", copyback: "all-generated-file-contents-change-each-turn" }, }, + buildConnectionSuite(runnerProfiles, runnerEnvironments), ] as const; export function suiteDefinitionHash(suite: RunnerSuiteFixture) { @@ -1548,8 +1550,8 @@ export function buildRunnerMatrix( ]), ], requiredCredentials: [ - profile.credential, - ...(environment.credential ? [environment.credential] : []), + ...(task.flow === "provider_connection" ? [] : [profile.credential]), + ...(task.flow !== "provider_connection" && environment.credential ? [environment.credential] : []), ], })) .filter((execution) => !excludedExecutionIds.has(execution.id)), @@ -1595,8 +1597,10 @@ function assertNoRawSecretValues(value: unknown, label: string) { } export function validateRunnerCatalog(): MatrixExecution[] { - const allProfiles = [...extendedHarnessProfiles, ...runnerProfiles, ...legacyAcpxProfiles, ...pendingContextIntegrityProfiles, ...openRouterBreadthProfiles, ...everydayProfiles.filter(p => !runnerProfiles.some(existing => existing.id === p.id))]; + const connectionSuite = runnerSuites.find(suite => suite.id === "provider-connections")!; + const allProfiles = [...connectionSuite.profiles, ...extendedHarnessProfiles, ...runnerProfiles, ...legacyAcpxProfiles, ...pendingContextIntegrityProfiles, ...openRouterBreadthProfiles, ...everydayProfiles.filter(p => !runnerProfiles.some(existing => existing.id === p.id))]; const allTasks = [ + ...connectionSuite.tasks, extendedHarnessFileTask, ...contextIntegrityTasks, ...blockerTasks, diff --git a/tests/runner-e2e/codex-ci-sandbox.test.ts b/tests/runner-e2e/codex-ci-sandbox.test.ts index b41dc24d895..7ab0ab021ba 100644 --- a/tests/runner-e2e/codex-ci-sandbox.test.ts +++ b/tests/runner-e2e/codex-ci-sandbox.test.ts @@ -37,9 +37,9 @@ describe("Codex CI sandbox trust boundary", () => { [...condition!.matchAll(/matrix\.profileId == '([^']+)'/g)].map((match) => match[1]), ); expect([...provisionedProfiles].sort()).toEqual( - [...new Set(runnerMatrix.filter(requiresCodexCiSandbox).map((cell) => cell.profile.id))].sort(), + [...new Set(runnerMatrix.filter(cell => cell.task.flow !== "provider_connection").filter(requiresCodexCiSandbox).map((cell) => cell.profile.id))].sort(), ); - for (const cell of runnerMatrix) { + for (const cell of runnerMatrix.filter(cell => cell.task.flow !== "provider_connection")) { expect( cell.environment.id === "local" && provisionedProfiles.has(cell.profile.id), cell.id, diff --git a/tests/runner-e2e/connection-cancel.smoke.ts b/tests/runner-e2e/connection-cancel.smoke.ts new file mode 100644 index 00000000000..374b18dc9d6 --- /dev/null +++ b/tests/runner-e2e/connection-cancel.smoke.ts @@ -0,0 +1,67 @@ +/** Opt-in, credential-free browser smoke: teardown retains its authenticated API client. */ +import { spawn } from "node:child_process"; +import { createServer } from "node:http"; +import path from "node:path"; +import { parseConnectionConfig } from "./connection-config.js"; +import { openConnectionBrowser } from "./connection-launch.js"; +import { runnerE2ETypeScriptProcessArgs } from "./web-server-command.js"; + +const repositoryRoot = path.resolve(import.meta.dirname, "../.."); + +if (process.argv[2] === "--worker") { + const origin = process.argv[3]!; + const browser = await openConnectionBrowser(parseConnectionConfig({ browser: { + headed: false, channel: "chromium", freshness: "signed-out", + } }), repositoryRoot, origin); + await browser.context.pages()[0]!.goto(origin); + let stopping = false; + for (const signal of ["SIGINT", "SIGTERM", "SIGHUP"] as const) process.once(signal, () => { + if (stopping) return; + stopping = true; + void (async () => { + let ok = false; + try { ok = (await browser.context.request.get(`${origin}/cleanup`)).ok(); } + finally { await browser.close(); } + process.send?.({ kind: "cleanup", ok }); + process.disconnect(); + process.exitCode = ok ? 0 : 1; + })().catch(() => { process.exitCode = 1; process.disconnect(); }); + }); + process.send?.({ kind: "ready" }); +} else { + let cleanupRequests = 0; + const server = createServer((req, res) => { + if (req.url === "/cleanup") { + const authenticated = req.headers.cookie === "qa-cleanup=synthetic"; + if (authenticated) cleanupRequests++; + res.writeHead(authenticated ? 200 : 401).end(); + } else res.writeHead(200, { "Content-Type": "text/html", "Set-Cookie": "qa-cleanup=synthetic; HttpOnly; Path=/" }).end("

Synthetic cleanup fixture

"); + }); + await new Promise(resolve => server.listen(0, "127.0.0.1", resolve)); + const origin = `http://127.0.0.1:${(server.address() as { port: number }).port}`; + try { + for (const signal of ["SIGINT", "SIGTERM", "SIGHUP"] as const) { + const env = Object.fromEntries(["PATH", "HOME", "USER", "LOGNAME", "TMPDIR", "LANG"].flatMap(key => process.env[key] ? [[key, process.env[key]!]] : [])); + await new Promise((resolve, reject) => { + const child = spawn(process.execPath, [...runnerE2ETypeScriptProcessArgs(repositoryRoot, import.meta.filename), "--worker", origin], { + cwd: repositoryRoot, env, stdio: ["ignore", "ignore", "ignore", "ipc"], + }); + let cleaned = false; + const timeout = setTimeout(() => { child.kill("SIGKILL"); reject(new Error(`${signal}: browser cleanup timed out`)); }, 30_000); + child.on("message", message => { + const event = message as { kind?: string; ok?: boolean }; + if (event.kind === "ready") child.kill(signal); + if (event.kind === "cleanup") cleaned = event.ok === true; + }); + child.once("error", error => { clearTimeout(timeout); reject(error); }); + child.once("exit", code => { + clearTimeout(timeout); + if (code === 0 && cleaned) resolve(); + else reject(new Error(`${signal}: browser closed before authenticated fixture cleanup`)); + }); + }); + console.log(`${signal}: authenticated cleanup completed before browser close`); + } + if (cleanupRequests !== 3) throw new Error("Missing authenticated cleanup requests"); + } finally { await new Promise((resolve, reject) => server.close(error => error ? reject(error) : resolve())); } +} diff --git a/tests/runner-e2e/connection-cases.ts b/tests/runner-e2e/connection-cases.ts new file mode 100644 index 00000000000..f2dd011cfd1 --- /dev/null +++ b/tests/runner-e2e/connection-cases.ts @@ -0,0 +1,88 @@ +import { createHash } from "node:crypto"; +import { readFileSync } from "node:fs"; +import { AI_CONNECTION_CAPABILITIES } from "../../packages/shared/src/ai-connections.js"; +import { isAiRoutingCompatible } from "../../packages/shared/src/ai-provider-routing.js"; +import type { AiProviderRouting } from "../../packages/shared/src/ai-provider-routing.js"; +import type { EnvironmentFixture, MatrixExecution, RunnerProfileFixture, RunnerSuiteFixture, RunnerTaskFixture } from "./types.js"; + +export const connectionModes = ["subscription", "api-key", "openrouter", "bedrock", "responses", "messages", "chat"] as const; +export type ConnectionMode = typeof connectionModes[number]; +export type ConnectionEntry = "agent" | "apps"; +export const connectionHarnesses = [ + { id: "claude", adapter: "claude_local", provider: "anthropic", label: "Claude", key: "ANTHROPIC_API_KEY", native: true }, + { id: "codex", adapter: "codex_local", provider: "openai", label: "OpenAI", key: "OPENAI_API_KEY", native: true }, + { id: "grok", adapter: "grok_local", provider: "xai", label: "Grok", key: "XAI_API_KEY", native: true }, + { id: "gemini", adapter: "gemini_local", provider: "google", label: "Google", key: "GEMINI_API_KEY", native: false }, + { id: "opencode", adapter: "opencode_local", provider: "openrouter", label: "OpenRouter", key: "OPENROUTER_API_KEY", native: true }, + { id: "hermes", adapter: "hermes_local", provider: "openrouter", label: "OpenRouter", key: "OPENROUTER_API_KEY", native: false }, +] as const; +export type ConnectionHarness = typeof connectionHarnesses[number]; + +/** Inventory entries remain visible even when no live journey can be selected. */ +export const connectionScopeGaps = [ + { adapter: "cursor", status: "unsupported", reason: "No managed connection flow; qualify the harness-specific login separately." }, + { adapter: "kimi_local", status: "unsupported", reason: "No managed connection flow; harness-specific credentials remain outside this suite." }, + { adapter: "pi_local", status: "unsupported", reason: "No managed connection flow or qualified model source." }, + { adapter: "openclaw_gateway", status: "excluded", reason: "Remote gateway owns provider configuration." }, + { adapter: "hermes_gateway", status: "excluded", reason: "Remote gateway owns provider configuration." }, + { adapter: "cursor_cloud", status: "excluded", reason: "Remote integration owns provider configuration." }, + { adapter: "process", status: "excluded", reason: "External process owns provider configuration." }, + { adapter: "http", status: "excluded", reason: "Remote integration owns provider configuration." }, + { adapter: "acpx_local", status: "excluded", reason: "Generic legacy integration; native Claude/Grok ACPX profiles remain covered." }, + { adapter: "claude_managed", status: "excluded", reason: "Managed remote integration owns credentials." }, + { adapter: "aws_agentcore", status: "excluded", reason: "Remote integration owns credentials." }, +] as const; + +export function routingForMode(mode: ConnectionMode): AiProviderRouting | undefined { + if (mode === "subscription" || mode === "api-key") return undefined; + return { kind: mode === "openrouter" || mode === "bedrock" ? mode : "gateway", protocol: mode === "openrouter" ? "responses" : mode, auth: "bearer", models: [], ...(mode === "bedrock" ? { region: "us-east-1" } : {}) }; +} + +export function supportsConnection(harness: ConnectionHarness, mode: ConnectionMode) { + const routing = routingForMode(mode); + if (routing) return isAiRoutingCompatible(routing, harness.adapter); + // OpenRouter's normal key is represented by its explicit routing case. + if (harness.provider === "openrouter") return false; + const methods = AI_CONNECTION_CAPABILITIES[harness.provider].methods; + return Boolean(methods[mode === "subscription" ? "subscription" : "api_key"]?.adapters.includes(harness.adapter)); +} + +export function connectionCell(execution: MatrixExecution) { + const match = /^(agent|apps)-(subscription|api-key|openrouter|bedrock|responses|messages|chat)$/.exec(execution.task.id); + const harness = connectionHarnesses.find(h => execution.profile.id === `connection-${h.id}-${execution.profile.generation}`); + if (execution.suite.id !== "provider-connections" || !harness || !match) throw new Error("Not a provider connection execution"); + return { harness, entry: match[1] as ConnectionEntry, mode: match[2] as ConnectionMode }; +} + +export function buildConnectionSuite(profiles: readonly RunnerProfileFixture[], environments: readonly EnvironmentFixture[]): RunnerSuiteFixture { + const selectedProfiles = connectionHarnesses.flatMap(h => (["legacy", ...(h.native ? ["native"] : [])] as Array<"legacy" | "native">).map(generation => { + const existing = profiles.find(p => p.generation === generation && (p.adapterType === h.adapter || (generation === "native" && p.id === (h.id === "claude" || h.id === "grok" ? `runner-acpx-${h.id}` : `runner-${h.id}`)))); + const id = `connection-${h.id}-${generation}`; + return { + id, label: `${h.label} ${generation}`, generation, groups: [generation], adapterType: generation === "native" ? "paperclip_runner" : h.adapter, + provider: h.id, model: existing?.model ?? "configured-in-connection-file", + modelQualification: existing?.modelQualification ?? { source: "candidate_runner_profile", qualificationId: "operator-configured-model" }, + credential: h.key, + supportedEnvironments: ["local", "daytona"], expectedRuntimeMode: generation, + expectedRuntimeMetadata: { adapterType: generation === "native" ? "paperclip_runner" : h.adapter, provider: generation === "native" && ["claude", "grok"].includes(h.id) ? "acpx" : h.id }, + buildAgent: () => ({ name: id, role: "qa", adapterType: generation === "native" ? "paperclip_runner" : h.adapter, adapterConfig: {}, runtimeConfig: {} }), + } satisfies RunnerProfileFixture; + })); + const tasks: RunnerTaskFixture[] = (["agent", "apps"] as const).flatMap(entry => connectionModes.map(mode => ({ + id: `${entry}-${mode}`, label: `${entry === "apps" ? "Apps connector" : "New agent"} · ${mode}`, groups: [], flow: "provider_connection", workMode: "standard", + expectedRunCount: 3, minimumExpectedRunCount: 2, + attemptTimeoutMs: { local: 30 * 60_000, daytona: 30 * 60_000 }, turnTimeoutMs: 5 * 60_000, + expectedTerminalState: { issue: "done", run: "succeeded" }, + buildTitle: nonce => `Connection proof ${nonce}`, buildPrompt: () => "Runtime-generated independent attachment proof.", buildVisibleMarker: nonce => nonce, + buildMatchers: () => [], + }))); + const excludedExecutionIds = selectedProfiles.flatMap(profile => environments.flatMap(environment => tasks.flatMap(task => { + const harness = connectionHarnesses.find(h => profile.id === `connection-${h.id}-${profile.generation}`)!; + const mode = task.id.replace(/^(agent|apps)-/, "") as ConnectionMode; + return supportsConnection(harness, mode) ? [] : [`provider-connections.${profile.id}.${environment.id}.${task.id}`]; + }))); + const hash = createHash("sha256"); + for (const file of ["connection-cases.ts", "connection-config.ts", "connection-flow.ts", "connection-evidence.ts", "connection-launch.ts", "connection-target.ts", "user-actions.ts"]) hash.update(file).update(readFileSync(new URL(`./${file}`, import.meta.url))); + const digest = hash.digest("hex"); + return { id: "provider-connections", label: "Live provider connections", description: "Fresh production UI connections, attended subscription login, independently verified tasks and reuse on a selected deployment.", groups: [], manualOnly: true, profiles: selectedProfiles, environments, tasks, excludedExecutionIds, expectedMatrixSize: selectedProfiles.length * environments.length * tasks.length - excludedExecutionIds.length, definitionMetadata: { version: 1, digest, authentication: "fresh-ui-connection", traces: "disabled", scheduling: "explicit-only" } }; +} diff --git a/tests/runner-e2e/connection-config.ts b/tests/runner-e2e/connection-config.ts new file mode 100644 index 00000000000..c38fbc14820 --- /dev/null +++ b/tests/runner-e2e/connection-config.ts @@ -0,0 +1,162 @@ +import { readFile, lstat, realpath } from "node:fs/promises"; +import path from "node:path"; +import { connectionCell, type ConnectionMode } from "./connection-cases.js"; +import type { MatrixExecution } from "./types.js"; + +export interface ConnectionRoute { + model: string; credentialEnv?: string; baseURL?: string; region?: string; auth: "bearer" | "api_key" | "none"; +} +export interface ConnectionConfig { + target: { mode: "managed-local" } | { mode: "attach"; baseURL: string; expectedCommit: string; deploymentMode: "local_trusted" | "authenticated"; companyId?: string; environmentId?: string }; + browser: { headed: boolean; channel: "chrome" | "chromium"; profileDir?: string; accountAlias: string; freshness: "signed-in" | "signed-out"; loginTimeoutMs: number }; + secretFile?: string; + models: Record; + routes: Record; + credentials: Record; + budgetCents: number; maxRuns: number; turnTimeoutMs: number; retainCompany: boolean; +} + +/** Strict, dependency-free private configuration parser. Never echo bad input. */ +export function parseConnectionConfig(input: unknown): ConnectionConfig { + const invalid = (): never => { throw new ConnectionBlock("blocked_target", "invalid_connection_configuration"); }; + const object = (value: unknown, allowed?: string[]): Record => { + if (!value || typeof value !== "object" || Array.isArray(value)) return invalid(); + if (allowed && Object.keys(value).some(key => !allowed.includes(key))) return invalid(); + return value as Record; + }; + const text = (value: unknown, fallback?: string): string => { + if (value === undefined && fallback !== undefined) return fallback; + if (typeof value !== "string" || !value.trim() || value.length > 2048) return invalid(); + return value; + }; + const choice = (value: unknown, options: readonly T[], fallback: T): T => value === undefined ? fallback : options.includes(value as T) ? value as T : invalid(); + const flag = (value: unknown, fallback: boolean): boolean => value === undefined ? fallback : typeof value === "boolean" ? value : invalid(); + const number = (value: unknown, fallback: number, min: number, max: number): number => value === undefined ? fallback : typeof value === "number" && Number.isSafeInteger(value) && value >= min && value <= max ? value : invalid(); + const name = (value: unknown) => { const result = text(value); if (!/^[A-Z][A-Z0-9_]+$/.test(result)) return invalid(); return result; }; + const uuid = (value: unknown) => { const result = text(value); if (!/^[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}$/i.test(result)) return invalid(); return result; }; + const root = object(input, ["target", "browser", "secretFile", "models", "routes", "credentials", "budgetCents", "maxRuns", "turnTimeoutMs", "retainCompany"]); + const t = object(root.target ?? { mode: "managed-local" }); + let target: ConnectionConfig["target"]; + if (t.mode === "managed-local") { object(t, ["mode"]); target = { mode: "managed-local" }; } + else if (t.mode === "attach") { + object(t, ["mode", "baseURL", "expectedCommit", "deploymentMode", "companyId", "environmentId"]); + const expectedCommit = text(t.expectedCommit); + if (!/^[a-f0-9]{40}$/.test(expectedCommit)) return invalid(); + target = { mode: "attach", baseURL: targetOrigin(text(t.baseURL)), expectedCommit, deploymentMode: choice(t.deploymentMode, ["local_trusted", "authenticated"], "authenticated"), + ...(t.companyId === undefined ? {} : { companyId: uuid(t.companyId) }), ...(t.environmentId === undefined ? {} : { environmentId: uuid(t.environmentId) }) }; + } else return invalid(); + const b = object(root.browser ?? {}, ["headed", "channel", "profileDir", "accountAlias", "freshness", "loginTimeoutMs"]); + const accountAlias = text(b.accountAlias, "local-qa"); + if (!/^[a-zA-Z0-9_-]{1,60}$/.test(accountAlias)) return invalid(); + const routes = Object.fromEntries(Object.entries(object(root.routes ?? {})).map(([key, value]) => { + const r = object(value, ["model", "credentialEnv", "baseURL", "region", "auth"]); + return [key, { model: text(r.model), auth: choice(r.auth, ["bearer", "api_key", "none"], "bearer"), + ...(r.credentialEnv === undefined ? {} : { credentialEnv: name(r.credentialEnv) }), ...(r.baseURL === undefined ? {} : { baseURL: text(r.baseURL) }), ...(r.region === undefined ? {} : { region: text(r.region) }) }]; + })); + return { target, browser: { headed: flag(b.headed, true), channel: choice(b.channel, ["chrome", "chromium"], "chrome"), accountAlias, freshness: choice(b.freshness, ["signed-in", "signed-out"], "signed-in"), loginTimeoutMs: number(b.loginTimeoutMs, 600_000, 1000, 1200_000), ...(b.profileDir === undefined ? {} : { profileDir: text(b.profileDir) }) }, + ...(root.secretFile === undefined ? {} : { secretFile: text(root.secretFile) }), + models: Object.fromEntries(Object.entries(object(root.models ?? {})).map(([key, value]) => [key, text(value)])), routes, + credentials: Object.fromEntries(Object.entries(object(root.credentials ?? {})).map(([key, value]) => [key, name(value)])), + budgetCents: number(root.budgetCents, 200, 1, 10_000), maxRuns: number(root.maxRuns, 3, 2, 5), turnTimeoutMs: number(root.turnTimeoutMs, 300_000, 10_000, 600_000), retainCompany: flag(root.retainCompany, false) }; +} + +/** Only harness-owned error codes may enter a retained report. */ +export class ConnectionFailure extends Error {} + +export class ConnectionBlock extends Error { + constructor(readonly reason: "awaiting_user" | "missing_credential" | "blocked_target", readonly code: string) { + super(code); + } +} + +export function targetOrigin(value: string): string { + const url = new URL(value); + const loopback = ["127.0.0.1", "localhost", "[::1]"].includes(url.hostname); + if (url.username || url.password || url.search || url.hash || url.pathname !== "/" || + !(url.protocol === "https:" || (url.protocol === "http:" && loopback))) { + throw new ConnectionBlock("blocked_target", "target_requires_https_or_loopback_origin"); + } + return url.origin; +} + +export async function loadConnectionConfig(file?: string): Promise { + try { + const config = parseConnectionConfig(file ? JSON.parse(await readFile(file, "utf8")) : {}); + if (config.target.mode === "attach") config.target.baseURL = targetOrigin(config.target.baseURL); + return config; + } catch (error) { + if (error instanceof ConnectionBlock) throw error; + // Parsing errors can contain configuration values. Never echo input. + throw new ConnectionBlock("blocked_target", "invalid_connection_configuration"); + } +} + +export async function assertPrivateFile(file: string) { + const stat = await lstat(file); + if (!stat.isFile() || stat.isSymbolicLink() || (stat.mode & 0o077) !== 0 || (process.getuid && stat.uid !== process.getuid())) { + throw new ConnectionBlock("missing_credential", "credential_file_requires_owner_only_regular_file"); + } +} + +/** Literal dotenv only: never evaluate shell expansions or source an entire file. */ +export function selectedSecret(text: string, name: string): string | undefined { + let found: string | undefined; + for (const raw of text.split(/\r?\n/)) { + const line = raw.replace(/^\s*export\s+/, ""); + const match = /^\s*([A-Z][A-Z0-9_]+)\s*=\s*(.*?)\s*$/.exec(line); + if (match?.[1] !== name) continue; + let value = match[2]!; + if ((value.startsWith('"') && value.endsWith('"')) || (value.startsWith("'") && value.endsWith("'"))) value = value.slice(1, -1); + if (!value || /[`\r\n]|\$\(/.test(value)) throw new ConnectionBlock("missing_credential", "credential_must_be_a_literal_value"); + found = value; + } + return found; +} + +export async function resolveConnectionSecret(config: ConnectionConfig, name?: string, env: NodeJS.ProcessEnv = process.env) { + if (!name) return undefined; + if (env[name]?.trim()) return env[name]!.trim(); + if (config.secretFile) { + await assertPrivateFile(config.secretFile); + const value = selectedSecret(await readFile(config.secretFile, "utf8"), name); + if (value) return value; + } + throw new ConnectionBlock("missing_credential", `missing_${name}`); +} + +export function resolveConnectionSettings(execution: MatrixExecution, config: ConnectionConfig) { + const { harness, mode } = connectionCell(execution); + const route = config.routes[`${harness.id}.${mode}`] ?? config.routes[mode]; + const advanced = mode !== "subscription" && mode !== "api-key"; + if (advanced && !route) throw new ConnectionBlock("missing_credential", "route_configuration_required"); + const selectedModel = route?.model ?? config.models[harness.id] ?? execution.profile.model; + if (selectedModel === "configured-in-connection-file") throw new ConnectionBlock("blocked_target", "explicit_model_required"); + if (["responses", "messages", "chat"].includes(mode)) { + if (!route?.baseURL) throw new ConnectionBlock("blocked_target", "gateway_url_required"); + const url = new URL(route.baseURL); + if (url.username || url.password || url.search || url.hash || !["https:", "http:"].includes(url.protocol)) throw new ConnectionBlock("blocked_target", "invalid_gateway_url"); + } + if (mode === "bedrock" && (!route?.region || route.auth !== "bearer")) throw new ConnectionBlock("blocked_target", "bedrock_region_and_bearer_required"); + if (route?.auth === "api_key" && mode !== "messages") throw new ConnectionBlock("blocked_target", "api_key_header_requires_messages"); + if (mode === "openrouter" && route?.auth !== "bearer") throw new ConnectionBlock("blocked_target", "openrouter_requires_bearer"); + const defaultKey = mode === "bedrock" ? "AWS_BEARER_TOKEN_BEDROCK" : mode === "openrouter" ? "OPENROUTER_API_KEY" : harness.key; + const credentialEnv = mode === "subscription" || route?.auth === "none" ? undefined : route?.credentialEnv ?? config.credentials[harness.id] ?? (advanced && !["bedrock", "openrouter"].includes(mode) ? undefined : defaultKey); + if (mode !== "subscription" && route?.auth !== "none" && !credentialEnv) throw new ConnectionBlock("missing_credential", "gateway_credential_reference_required"); + return { model: selectedModel, route, credentialEnv }; +} + +export function connectionCatalogSource(mode: ConnectionMode, provider: string) { + return ({ responses: "responses-api", messages: "messages-api", chat: "chat-completions-api", bedrock: "bedrock", openrouter: "openrouter" } as Record)[mode] ?? provider; +} + +export async function assertOutsideRepository(file: string, repositoryRoot: string) { + // Resolve an existing ancestor too, so a missing child below a symlink cannot evade containment. + let existing = path.resolve(file); + while (true) { + try { existing = await realpath(existing); break; } + catch (error) { if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; const parent = path.dirname(existing); if (parent === existing) throw error; existing = parent; } + } + const root = await realpath(repositoryRoot); + const relative = path.relative(root, existing); + if (relative === "" || (!relative.startsWith(`..${path.sep}`) && relative !== ".." && !path.isAbsolute(relative))) throw new ConnectionBlock("blocked_target", "browser_profile_must_be_outside_repository"); +} diff --git a/tests/runner-e2e/connection-evidence.ts b/tests/runner-e2e/connection-evidence.ts new file mode 100644 index 00000000000..9ccb4c4d105 --- /dev/null +++ b/tests/runner-e2e/connection-evidence.ts @@ -0,0 +1,129 @@ +import { createHash, randomBytes } from "node:crypto"; +import type { RunnerE2EResult } from "./types.js"; + +export const connectionCheckpoints = ["target", "connection_created", "connection_reloaded", "agent_created", "binding", "setup_probe", "first_run", "artifact", "tool_receipt", "followup", "visible_result"] as const; +export type ConnectionCheckpoint = typeof connectionCheckpoints[number]; +export const connectionRunSignals = ["provider_overloaded", "provider_quota", "missing_file_conversion", "unsupported_session_configuration", "command_substitution_denied"] as const; +export interface ConnectionRunDiagnostic { + runId: string; + status: "succeeded" | "failed" | "cancelled" | "interrupted" | "timed_out" | "unknown"; + signals: Array; + logAvailable: boolean; +} +export interface ConnectionEvidence { + version: 1; + outcome: "running" | "passed" | "failed" | "awaiting_user" | "missing_credential" | "blocked_target"; + phase: string; + assisted: boolean; + authFreshness: "signed-in" | "signed-out"; + target: { mode: "managed-local" | "attach"; origin: string; commit: string | null; deploymentMode: string | null }; + entry: "apps" | "agent"; + method: string; + checkpoints: Partial>; + waits: Array<{ kind: "board_login" | "provider_login"; startedAt: string; finishedAt?: string }>; + connectionId?: string; + agentId?: string; + companyId?: string; + artifactSha256?: string; + model?: string; + inputTransport?: "prompt_base64"; + setupChecks?: Array<{ code: string; level: "info" | "warn" | "error" }>; + creationFailure?: { status: number; code: "ai_connection_api_key_rejected" | "ai_connection_verification_failed" | "connection_request_rejected" }; + cleanupRetained?: boolean; + companyArchived?: boolean; + artifactChecks?: Array<{ filename: string; runId: string; sha256: string; fields: Record; exactFields: boolean }>; + runDiagnostics?: ConnectionRunDiagnostic[]; +} + +/** Project terminal errors to closed diagnostic codes; never retain messages or reasoning. */ +export function connectionRunDiagnostic(run: Record, log?: unknown): ConnectionRunDiagnostic { + const signals = new Set(); + const detect = (text: unknown) => { + if (typeof text !== "string") return; + if (/"code"\s*:\s*503\b|\bHTTP\s*503\b/i.test(text) && /overloaded|high demand/i.test(text)) signals.add("provider_overloaded"); + if (/"code"\s*:\s*429\b|\bHTTP\s*429\b|\bRESOURCE_EXHAUSTED\b|\bTerminalQuotaError\b/.test(text)) signals.add("provider_quota"); + if (/Error checking existing file: (?:Resource not found|Internal error)/.test(text)) signals.add("missing_file_conversion"); + if (/set_config_option/.test(text) && /[Mm]ethod not found|-32601/.test(text)) signals.add("unsupported_session_configuration"); + if (/Blocked: command substitution detected/.test(text)) signals.add("command_substitution_denied"); + }; + detect(run.error); + detect(run.stderrExcerpt); + const content = log && typeof log === "object" && typeof (log as Record).content === "string" + ? (log as { content: string }).content : undefined; + if (content !== undefined) { + let inner = ""; + for (const line of content.split("\n")) { + try { const row = JSON.parse(line); if (typeof row.chunk === "string") inner += row.chunk; } + catch { /* Non-JSON application lines do not classify provider failures. */ } + } + for (const line of inner.split("\n")) { + try { + const row = JSON.parse(line); + if (row.type === "acpx.error") { detect(row.message); detect(row.childStderrTail); } + if (row.type === "acpx.tool_call" && row.status === "failed") detect(row.text); + } catch { /* Model text, partial records, and raw output are not retained. */ } + } + } + const statuses = ["succeeded", "failed", "cancelled", "interrupted", "timed_out"]; + return { runId: run.id, status: statuses.includes(run.status) ? run.status : "unknown", signals: [...signals], logAvailable: content !== undefined }; +} + +export function gradeConnectionEvidence(evidence: ConnectionEvidence) { + return connectionCheckpoints.map(id => ({ matcher: { kind: "json_path" as const, path: `providerConnection.checkpoints.${id}`, expected: true }, passed: evidence.checkpoints[id] === true, detail: `${id}: ${evidence.checkpoints[id] === true ? "verified" : "not verified"}` })); +} + +/** Retain stable product check codes, never provider messages or command output. */ +export function connectionProbeChecks(value: unknown): NonNullable { + if (!value || typeof value !== "object" || !Array.isArray((value as Record).checks)) return []; + return ((value as { checks: unknown[] }).checks).slice(0, 50).flatMap(entry => { + if (!entry || typeof entry !== "object") return []; + const { code, level } = entry as Record; + return typeof code === "string" && /^[a-z][a-z0-9_]{0,95}$/.test(code) && ["info", "warn", "error"].includes(String(level)) + ? [{ code, level: level as "info" | "warn" | "error" }] : []; + }); +} + +export function connectionEvidencePasses(evidence: ConnectionEvidence) { + return evidence.outcome === "passed" && Boolean(evidence.target.commit) && gradeConnectionEvidence(evidence).every(check => check.passed); +} + +export function isBlockedConnectionResult(result: RunnerE2EResult) { + return Boolean(result.providerConnection && ["awaiting_user", "missing_credential", "blocked_target"].includes(result.providerConnection.outcome)); +} + +export function createConnectionProof() { + const nonce = randomBytes(12).toString("hex"); + const values = Array.from({ length: 7 }, () => randomBytes(2).readUInt16BE()); + const input = `${JSON.stringify({ nonce, values })}\n`; + const expected = { nonce, count: values.length, total: values.reduce((sum, n) => sum + n, 0), sha256: createHash("sha256").update(input).digest("hex") }; + const followup = { nonce, minimum: Math.min(...values), maximum: Math.max(...values), total: expected.total }; + return { input, expected, followup }; +} + +export function verifyConnectionArtifact(bytes: Buffer, expected: Record) { + try { + const value = JSON.parse(bytes.toString("utf8")); + return Object.keys(value).length === Object.keys(expected).length && Object.entries(expected).every(([key, entry]) => value[key] === entry); + } catch { return false; } +} + +/** Only durable run attribution is accepted; assistant text cannot provide it. */ +export function verifyConnectionRun(run: Record, expected: { agentId: string; connectionId: string; method: string; runtimeMode: string; environmentId: string }) { + const connection = run.contextSnapshot?.aiConnection; + return run.status === "succeeded" && run.agentId === expected.agentId && run.runtimeMode === expected.runtimeMode && + connection?.connectionId === expected.connectionId && connection?.method === expected.method && + run.contextSnapshot?.paperclipEnvironment?.id === expected.environmentId; +} + +/** Wait for scheduled repairs to settle, then select the run that actually delivered the file. */ +export function completedConnectionArtifactRun( + runs: Record[], + attachments: Record[], + filename: string, + expected: Parameters[1], +) { + const terminal = ["succeeded", "failed", "cancelled", "interrupted", "timed_out"]; + if (runs.some(run => !terminal.includes(run.status))) return undefined; + return runs.find(run => verifyConnectionRun(run, expected) && attachments.some(attachment => + attachment.originalFilename === filename && attachment.createdByAgentId === expected.agentId && attachment.originatingRunId === run.id)); +} diff --git a/tests/runner-e2e/connection-flow.ts b/tests/runner-e2e/connection-flow.ts new file mode 100644 index 00000000000..58ba835b7ac --- /dev/null +++ b/tests/runner-e2e/connection-flow.ts @@ -0,0 +1,377 @@ +import { createHash } from "node:crypto"; +import { expect, type Page } from "@playwright/test"; +import { aiRoutingModel } from "../../packages/shared/src/ai-provider-routing.js"; +import { getConnectableAppDefinition } from "../../packages/shared/src/app-definitions.js"; +import { connectionCell, type ConnectionHarness, type ConnectionMode } from "./connection-cases.js"; +import { ConnectionBlock, ConnectionFailure, type ConnectionConfig, type resolveConnectionSettings } from "./connection-config.js"; +import { ConnectionApi, connectionDelay } from "./connection-target.js"; +import { createConnectionProof, verifyConnectionArtifact, verifyConnectionRun, completedConnectionArtifactRun, connectionProbeChecks, type ConnectionEvidence } from "./connection-evidence.js"; +import { createTaskThroughUi, submitTaskReply } from "./user-actions.js"; +import { collectRunEvents } from "./run-observations.js"; +import type { MatrixExecution } from "./types.js"; + +type Row = Record; +type Settings = ReturnType; +export interface ConnectionFlowInput { + page: Page; api: ConnectionApi; execution: MatrixExecution; config: ConnectionConfig; settings: Settings; + company: Row; environment: Row; secret?: string; nonce: string; privateDir: string; + evidence: ConnectionEvidence; + checkpoint(phase: string): Promise; + observeRun(run: Row): void; + createdConnectionIds: ReadonlySet; + refreshOwnedConnections(): Promise; + assertActive(): void; +} + +async function choose(page: Page, label: string, name: string) { + await page.getByRole("combobox", { name: label, exact: true }).click(); + await page.getByRole("option", { name, exact: true }).click(); +} + +async function openAdvanced(page: Page, mode: ConnectionMode) { + const names: Record = { openrouter: "OpenRouter", bedrock: "Amazon Bedrock", responses: "Custom gateway", messages: "Custom gateway", chat: "Custom gateway" }; + await page.getByRole("button", { name: new RegExp(`^${names[mode]}`) }).click(); +} + +async function fillGateway(input: ConnectionFlowInput) { + const { page, settings, secret } = input; + const { mode } = connectionCell(input.execution); + if (settings.route?.baseURL) { + await page.getByLabel("Provider URL", { exact: true }).fill(settings.route.baseURL); + const format = page.getByRole("combobox", { name: "API format", exact: true }); + if (await format.isEnabled()) await choose(page, "API format", ({ responses: "OpenAI Responses · Codex", messages: "Anthropic Messages · Claude", chat: "Chat Completions · OpenCode, Hermes" } as Record)[mode]!); + } + if (mode === "bedrock") await page.getByLabel("AWS region", { exact: true }).fill(settings.route!.region!); + if (settings.route?.auth !== "bearer") await choose(page, "Authentication", settings.route?.auth === "none" ? "No authentication" : "API key · x-api-key"); + if (secret) await page.getByLabel("API key", { exact: true }).fill(secret); + // Normal defaults stay collapsed. Model alias is selected in Configure. + await page.getByRole("button", { name: "Connect", exact: true }).last().click(); +} + +async function submitConnection(input: ConnectionFlowInput, submit: () => Promise) { + const route = `${input.api.origin}/api/companies/${input.company.id}/ai-connections`; + const response = input.page.waitForResponse(response => response.url() === route && response.request().method() === "POST", { timeout: 60_000 }).catch(() => null); + await submit(); + const created = await response; + if (!created) throw new ConnectionFailure("connection_request_timeout"); + if (created.ok()) return; + const payload = await created.json().catch(() => ({})); + const allowed = ["ai_connection_api_key_rejected", "ai_connection_verification_failed"]; + const code = allowed.includes(payload?.details?.code) ? payload.details.code : "connection_request_rejected"; + input.evidence.creationFailure = { status: created.status(), code }; + if (code === "ai_connection_api_key_rejected") throw new ConnectionBlock("missing_credential", code); + throw new ConnectionFailure(code); +} + +async function waitForFreshConnection(input: ConnectionFlowInput, before: Set) { + const { page, api, config, evidence, company } = input; + const { harness, mode } = connectionCell(input.execution); + const deadline = Date.now() + (mode === "subscription" ? config.browser.loginTimeoutMs : 90_000); + let loginOpened = false; + let connected = false; + while (Date.now() < deadline) { + input.assertActive(); + await input.refreshOwnedConnections(); + const list = await api.get(`/api/companies/${company.id}/ai-connections`); + const fresh = list.connections.filter((c: Row) => !before.has(c.id) && input.createdConnectionIds.has(c.id)); + if (fresh.length > 1) throw new ConnectionFailure("unexpected_multiple_connections"); + if (fresh.length === 1) { + const connection = fresh[0]; + if (connection.status !== "connected" || connection.method !== (mode === "subscription" ? "subscription" : "api_key")) throw new ConnectionFailure("saved_connection_method_or_status_mismatch"); + return connection; + } + if (mode === "subscription") { + const signIn = page.getByRole("link", { name: /^Sign in to / }).first(); + const cloudSignIn = page.getByRole("button", { name: /^Sign in to / }).first(); + if (!loginOpened && (await signIn.isVisible() || await cloudSignIn.isVisible())) { + // Observe before clicking. No token, code, URL, or popup content is recorded. + const popup = page.context().waitForEvent("page", { timeout: 10_000 }).catch(() => undefined); + if (await signIn.isVisible()) await signIn.click(); else if (await cloudSignIn.isEnabled()) await cloudSignIn.click(); + await popup; + loginOpened = true; + evidence.assisted = true; + evidence.waits.push({ kind: "provider_login", startedAt: new Date().toISOString() }); + await input.checkpoint("awaiting_provider_login"); + console.log(`[provider-connections] Sign in to ${harness.label} in the opened browser (${config.browser.accountAlias}); complete the code/consent step in Paperclip. This test resumes automatically.`); + } + // Grok currently exposes a terminal handoff; do not pretend it has the same browser flow. + if (!loginOpened && harness.id === "grok" && await page.getByRole("button", { name: "Copy sign-in command" }).isVisible()) { + loginOpened = true; + evidence.assisted = true; + evidence.waits.push({ kind: "provider_login", startedAt: new Date().toISOString() }); + await input.checkpoint("awaiting_provider_login"); + console.log("[provider-connections] Grok requires the sign-in command shown in Paperclip. Complete that production handoff; the test will resume automatically."); + } + const connect = page.getByRole("button", { name: "Connect", exact: true }).last(); + if (!connected && await connect.isVisible() && await connect.isEnabled()) { await connect.click(); connected = true; } + if (await page.getByRole("alert").filter({ hasText: /expired|could not|failed|invalid/i }).count()) throw new ConnectionFailure("provider_login_reported_error"); + } + await connectionDelay(1_000); + } + if (mode === "subscription" && loginOpened) throw new ConnectionBlock("awaiting_user", "provider_login_deadline_reached"); + throw new ConnectionFailure("connection_creation_timeout"); +} + +async function selectSavedConnection(page: Page, connection: Row, harness: ConnectionHarness, mode: ConnectionMode) { + await page.getByRole("heading", { name: /^(Connect a model|Configure your agent)$/ }).waitFor(); + const advanced = !["subscription", "api-key"].includes(mode); + const connectionType = page.getByRole("radiogroup", { name: "Connection type", exact: true }); + if (["claude", "codex", "grok"].includes(harness.id)) { + await connectionType.waitFor({ state: "visible" }); + if (advanced) { + await connectionType.getByRole("radio", { name: /Advanced/ }).click(); + await choose(page, "Connection", connection.name); + await page.getByRole("button", { name: "Use connection", exact: true }).click(); + } else { + await connectionType.getByRole("radio", { name: mode === "subscription" ? /Subscription/ : /API key/ }).click(); + const saved = page.getByRole("combobox", { name: mode === "subscription" ? "Saved subscription" : "Saved API key", exact: true }); + await saved.waitFor({ state: "visible" }); + const option = await saved.locator("option").filter({ hasText: connection.name }).first().getAttribute("value"); + if (!option) throw new ConnectionFailure("saved_connection_option_missing"); + await saved.selectOption(option); + await page.getByRole("button", { name: mode === "subscription" ? "Use saved subscription" : "Use saved API key", exact: true }).click(); + } + } else { + await choose(page, "Connection", connection.name); + } +} + +async function createConnectionThroughUi(input: ConnectionFlowInput, agentURL: string) { + const { page, api, company, evidence } = input; + const { harness, mode, entry } = connectionCell(input.execution); + const priorConnections = (await api.get(`/api/companies/${company.id}/ai-connections`)).connections as Row[]; + const before = new Set(priorConnections.map(c => c.id)); + // Fresh-creation proof cannot be satisfied by an existing personal default. + if (priorConnections.some(connection => connection.status !== "revoked")) throw new ConnectionBlock("blocked_target", "fresh_connection_requires_empty_qa_company"); + const advanced = !["subscription", "api-key"].includes(mode); + await input.checkpoint("connection_navigation"); + if (entry === "apps") { + await page.goto(`${api.origin}/${company.issuePrefix}/apps`, { waitUntil: "domcontentloaded" }); + const catalogSlug = advanced ? ({ openrouter: "openrouter", bedrock: "bedrock", responses: "responses-api", messages: "messages-api", chat: "chat-completions-api" } as Record)[mode]! : harness.provider; + const definition = getConnectableAppDefinition(catalogSlug); + if (!definition) throw new ConnectionFailure("connection_catalog_entry_missing"); + await page.getByRole("button", { name: `Connect ${definition.name}`, exact: true }).click(); + const next = page.getByRole("button", { name: /^(Save and continue|Continue)$/ }).last(); + if (!advanced && await next.isVisible()) await next.click(); + if (!advanced && harness.provider !== "google") { + // The credential step loads asynchronously. Wait for its provider tile + // before inspecting the mode switch, otherwise a still-loading page can + // skip the API-key switch and leave the subscription form selected. + await page.getByRole("radio", { name: new RegExp(harness.id === "grok" ? "Grok" : harness.label) }).click(); + const switcher = page.getByRole("button", { name: mode === "subscription" ? "Use subscription instead" : "Use API key instead", exact: true }); + if (await switcher.isVisible()) await switcher.click(); + } + } else { + await page.goto(agentURL, { waitUntil: "domcontentloaded" }); + const tiles = page.getByRole("radiogroup", { name: "Connection type", exact: true }); + if (["claude", "codex", "grok"].includes(harness.id)) { + const choice = tiles.getByRole("radio", { name: advanced ? /Advanced/ : mode === "subscription" ? /Subscription/ : /API key/ }); + await choice.click(); + await expect(choice).toHaveAttribute("aria-checked", "true"); + } + if (advanced || !["claude", "codex", "grok"].includes(harness.id)) { + await choose(page, "Connection", "Connect an account…"); + if (advanced) { + const advancedProviders = page.locator("summary").filter({ hasText: "Advanced providers" }); + if (await advancedProviders.isVisible()) { + await advancedProviders.click(); + // The account dialog's disclosure leads to the provider chooser; + // the chooser's own disclosure already contains provider buttons. + const chooseProvider = page.getByRole("button", { name: "Choose another provider or gateway", exact: true }); + if (await chooseProvider.isVisible()) await chooseProvider.click(); + } + await openAdvanced(page, mode); + } + } + } + await input.checkpoint("connection_creation"); + if (advanced) await submitConnection(input, () => fillGateway(input)); + else if (mode === "api-key") { + const savedKey = page.getByRole("combobox", { name: "Saved API key", exact: true }); + if (await savedKey.isVisible()) await savedKey.selectOption(""); + await page.getByLabel("API key", { exact: true }).fill(input.secret!); + await submitConnection(input, () => page.getByRole("button", { name: "Connect", exact: true }).last().click()); + } + const connection = await waitForFreshConnection(input, before); + evidence.connectionId = connection.id; + evidence.checkpoints.connection_created = true; + evidence.waits.forEach(wait => { wait.finishedAt ??= new Date().toISOString(); }); + if (advanced) { + const expectedKind = ["bedrock", "openrouter"].includes(mode) ? mode : "gateway"; + if (connection.routing?.kind !== expectedKind || (mode !== "openrouter" && connection.routing?.protocol !== mode) || + (input.settings.route?.baseURL && connection.routing?.baseUrl !== input.settings.route.baseURL) || + (mode === "bedrock" && connection.routing?.region !== input.settings.route?.region)) throw new ConnectionFailure("saved_routing_mismatch"); + } else if (connection.provider !== harness.provider) throw new ConnectionFailure("saved_provider_mismatch"); + // Navigate away and return through the normal list: proof includes durable UI state. + await page.goto(`${api.origin}/${company.issuePrefix}/apps`, { waitUntil: "domcontentloaded" }); + await expect(page.getByText(connection.name, { exact: true }).first()).toBeVisible(); + await page.reload({ waitUntil: "domcontentloaded" }); + await expect(page.getByText(connection.name, { exact: true }).first()).toBeVisible(); + evidence.checkpoints.connection_reloaded = true; + await input.checkpoint("connection_reloaded"); + await page.goto(agentURL, { waitUntil: "domcontentloaded" }); + await input.checkpoint("select_saved_connection"); + await selectSavedConnection(page, connection, harness, mode); + await input.checkpoint("configure_agent"); + return connection; +} + +async function selectModel(page: Page, model: string) { + const select = page.getByRole("combobox", { name: "Model", exact: true }); + await select.waitFor({ state: "visible" }); + await expect(select).not.toHaveAttribute("aria-busy", "true", { timeout: 60_000 }); + const values = await select.locator("option").evaluateAll(options => options.map(option => (option as HTMLOptionElement).value)); + const matched = values.find(value => value === model || value === `openrouter/${model}`); + if (matched) await select.selectOption(matched); + else { + await select.selectOption({ label: "Enter custom model…" }); + await page.getByLabel("Model ID", { exact: true }).fill(model); + } +} + +export async function runConnectionFlow(input: ConnectionFlowInput) { + const { page, api, execution, evidence, settings, company, environment, config } = input; + const { harness, mode } = connectionCell(execution); + const agentName = `Connection QA ${input.nonce}`; + const agentURL = `${api.origin}/${company.issuePrefix}/agents/new?${new URLSearchParams({ name: agentName, adapterType: execution.profile.adapterType, runnerProvider: harness.id })}`; + const connection = await createConnectionThroughUi(input, agentURL); + await expect(page.getByRole("heading", { name: "Configure your agent", exact: true })).toBeVisible({ timeout: 120_000 }); + const environmentSelect = page.getByRole("combobox", { name: "Environment", exact: true }); + const selectedEnv = await environmentSelect.inputValue(); + if (selectedEnv !== environment.id) { + if (!(await environmentSelect.isEnabled())) throw new ConnectionBlock("blocked_target", "requested_execution_environment_unavailable"); + await environmentSelect.selectOption(environment.id); + if (["claude", "codex", "grok"].includes(harness.id)) await selectSavedConnection(page, connection, harness, mode); + } + if (await environmentSelect.inputValue() !== environment.id) throw new ConnectionBlock("blocked_target", "requested_execution_environment_unavailable"); + await input.checkpoint("select_model"); + await selectModel(page, settings.model); + await input.checkpoint("setup_probe"); + const response = page.waitForResponse(response => response.url().startsWith(`${api.origin}/api/companies/${company.id}/adapters/`) && response.url().endsWith("/test-environment"), { timeout: 120_000 }); + await page.getByRole("button", { name: /^(Run test|Test again)$/ }).click(); + const probe = await response; + const result = await probe.json(); + evidence.setupChecks = connectionProbeChecks(result); + if (evidence.setupChecks.some(check => check.level === "error" && (check.code === "hermes_cli_not_found" || /^(claude|codex|grok|opencode|gemini)_command_unresolvable$/.test(check.code)))) + throw new ConnectionBlock("blocked_target", "runtime_cli_required"); + if (!probe.ok() || result.status !== "pass") throw new ConnectionFailure("setup_probe_failed"); + evidence.checkpoints.setup_probe = true; + await page.getByRole("button", { name: "Finish setup", exact: true }).click(); + let agent: Row | undefined; + const deadline = Date.now() + 60_000; + while (!agent && Date.now() < deadline) { + agent = (await api.get(`/api/companies/${company.id}/agents`)).find(row => row.name === agentName); + if (!agent) await connectionDelay(500); + } + if (!agent) throw new ConnectionFailure("agent_creation_failed"); + if (agent.defaultEnvironmentId !== environment.id) { + // A null override is valid only when the instance default is this exact environment. + const instance = await api.get("/api/instance/settings"); + const effective = agent.defaultEnvironmentId ?? instance.defaultEnvironmentId + ?? (environment.driver === "local" ? environment.id : null); + if (effective !== environment.id) throw new ConnectionFailure("saved_agent_environment_mismatch"); + } + evidence.agentId = agent.id; + evidence.checkpoints.agent_created = true; + const expectedModel = connection.routing + ? aiRoutingModel(connection.routing, harness.adapter, settings.model) + : settings.model; + if (agent.adapterType !== execution.profile.adapterType || agent.adapterConfig?.model !== expectedModel) throw new ConnectionFailure("agent_harness_or_model_mismatch"); + const binding = agent.runtimeConfig?.aiConnection; + if (!binding || binding.method !== connection.method || (binding.mode !== "responsible_user" && binding.connectionId !== connection.id)) throw new ConnectionFailure("agent_binding_mismatch"); + // Fixtures may constrain cost/scheduling, but cannot repair or replace the tested credentials/config. + await api.patch(`/api/agents/${agent.id}`, { budgetMonthlyCents: config.budgetCents }); + evidence.checkpoints.binding = true; + await input.checkpoint("first_task"); + const proof = createConnectionProof(); + // Provider qualification must not depend on task-level attachment ingestion. + // Native runners admit chat attachments, while the new-task dialog uploads + // its files after scheduling. Supply identical oracle bytes in the task so + // every harness can decode them with its actual tools. + evidence.inputTransport = "prompt_base64"; + const title = `Connection artifact ${input.nonce}`; + await createTaskThroughUi({ page, issuePrefix: company.issuePrefix, agentName, title, workMode: "standard", prompt: [ + "Complete this bounded connection acceptance task. Using your actual tools, decode the following base64 into connection-input.json and read the file. Preserve the exact decoded bytes, including the final newline.", + `Input bytes (base64): ${Buffer.from(proof.input, "utf8").toString("base64")}`, + "Create the input and output files inside your current workspace using relative filenames. Do not put them in /tmp or another directory outside the workspace.", + 'Write connection-proof.json with exactly this schema: {"nonce": string, "count": number, "total": number, "sha256": string}. Set nonce from the input, count to the number of values, total to their sum, and sha256 to the hash of the exact input file bytes. All four fields are required; use the literal field name "total", not "sum". Compute and validate with a tool; do not guess or paraphrase the data.', + "Deliver connection-proof.json as a downloadable attachment on this task. Use the normal Paperclip artifact/attachment workflow and include its download link in your final response. Then mark this task done. Do not create other tasks, read credentials, or inspect unrelated files.", + execution.profile.generation === "native" ? "Use your native register_deliverable tool with workspace-relative contentRef connection-proof.json and the exact byteSize and SHA-256 of that output file. Never submit an absolute contentRef." : "", + ].join("\n") }); + let issue: Row | undefined; + const createdDeadline = Date.now() + 30_000; + while (!issue && Date.now() < createdDeadline) { + issue = (await api.get(`/api/companies/${company.id}/issues`)).find(row => row.title === title); + if (!issue) await connectionDelay(500); + } + if (!issue) throw new ConnectionFailure("task_creation_failed"); + const runs = new Map(); + async function completedRun(after: Set, filename: string) { + const deadline = Date.now() + config.turnTimeoutMs; + while (Date.now() < deadline) { + input.assertActive(); + const summaries = await api.get(`/api/issues/${issue!.id}/runs`); + for (const row of summaries) { + const id = row.runId ?? row.id; + if (id && !runs.has(id)) runs.set(id, await api.get(`/api/heartbeat-runs/${id}`)); + else if (id && !["succeeded", "failed", "cancelled", "interrupted", "timed_out"].includes(runs.get(id)!.status)) runs.set(id, await api.get(`/api/heartbeat-runs/${id}`)); + if (id) input.observeRun(runs.get(id)!); + } + if (runs.size > config.maxRuns) throw new ConnectionFailure("provider_run_limit_exceeded"); + const next = [...runs.values()].filter(run => !after.has(run.id)); + if (next.some(run => ["failed", "cancelled", "interrupted", "timed_out"].includes(run.status))) throw new ConnectionFailure("agent_run_failed"); + const complete = next.find(run => verifyConnectionRun(run, { agentId: agent!.id, connectionId: connection.id, method: connection.method, runtimeMode: execution.profile.generation, environmentId: environment.id })); + if (complete) { + const status = (await api.get(`/api/issues/${issue!.id}`)).status; + if (status === "done") { + // A disposition repair may still be running after an earlier turn + // succeeded. Qualify the settled successful run that uploaded this + // artifact, preserving the configured run and time limits. + const attachments = await api.get(`/api/issues/${issue!.id}/attachments`); + const delivered = completedConnectionArtifactRun(next, attachments, filename, + { agentId: agent!.id, connectionId: connection.id, method: connection.method, runtimeMode: execution.profile.generation, environmentId: environment.id }); + if (delivered) return delivered; + if (next.every(run => ["succeeded", "failed", "cancelled", "interrupted", "timed_out"].includes(run.status))) throw new ConnectionFailure("run_attributed_artifact_missing"); + } + if (status === "blocked") throw new ConnectionFailure("task_blocked_after_provider_run"); + } + if (next.some(run => run.status === "succeeded" && !verifyConnectionRun(run, { agentId: agent!.id, connectionId: connection.id, method: connection.method, runtimeMode: execution.profile.generation, environmentId: environment.id }))) throw new ConnectionFailure("run_attribution_mismatch"); + await connectionDelay(1500); + } + throw new ConnectionFailure("agent_run_or_attribution_timeout"); + } + async function artifact(filename: string, expected: Record, run: Row) { + const attachments = await api.get(`/api/issues/${issue!.id}/attachments`); + const attachment = attachments.find(row => row.originalFilename === filename && row.createdByAgentId === agent!.id && row.originatingRunId === run.id); + if (!attachment) throw new ConnectionFailure("run_attributed_artifact_missing"); + const bytes = await api.bytes(`/api/attachments/${attachment.id}/content`); + let actual: Record = {}; + try { actual = JSON.parse(bytes.toString("utf8")); } catch { /* The independent verifier fails below. */ } + evidence.artifactChecks ??= []; + evidence.artifactChecks.push({ filename, runId: run.id, sha256: createHash("sha256").update(bytes).digest("hex"), + fields: Object.fromEntries(Object.entries(expected).map(([key, value]) => [key, actual?.[key] === value])), exactFields: Boolean(actual && Object.keys(actual).length === Object.keys(expected).length) }); + if (!verifyConnectionArtifact(bytes, expected)) throw new ConnectionFailure("independent_artifact_verification_failed"); + await page.goto(`${api.origin}/${company.issuePrefix}/issues/${issue!.identifier}`, { waitUntil: "domcontentloaded" }); + await page.reload({ waitUntil: "domcontentloaded" }); + await expect(page.locator(`a[href*="/api/attachments/${attachment.id}/content"]`).first()).toBeVisible({ timeout: 30_000 }); + evidence.artifactSha256 = createHash("sha256").update(bytes).digest("hex"); + } + const first = await completedRun(new Set(), "connection-proof.json"); + evidence.checkpoints.first_run = true; + await artifact("connection-proof.json", proof.expected, first); + evidence.checkpoints.artifact = true; + // Artifact upload provenance is a server-owned tool-side effect in both runner generations. + const events = await collectRunEvents((afterSeq, limit) => api.get(`/api/heartbeat-runs/${first.id}/events?afterSeq=${afterSeq}&limit=${limit}`)); + const toolReceipt = events.some(event => event.eventType === "tool.execution.completed" || (event.eventType === "item.completed" && ["tool_result", "command_execution"].includes(event.payload?.prpEvent?.payload?.item?.type))); + // Legacy adapters may expose their tool transcript only in the run log; upload attribution is still a durable successful tool effect. + evidence.checkpoints.tool_receipt = execution.profile.generation === "legacy" || toolReceipt; + if (!evidence.checkpoints.tool_receipt) throw new ConnectionFailure("native_tool_receipt_missing"); + await input.checkpoint("followup"); + const previousRuns = new Set(runs.keys()); + await submitTaskReply(page, "Using the original connection-input.json decoded from this task, deliver connection-followup.json with exactly nonce, minimum, maximum, and total of the original values. Use exactly this JSON schema: {\"nonce\": string, \"minimum\": number, \"maximum\": number, \"total\": number}. All four fields are required. Compute with a tool and validate the file against all four fields before attaching it and its download link. Then mark this task done. Do not create other tasks or read credentials."); + const second = await completedRun(previousRuns, "connection-followup.json"); + await artifact("connection-followup.json", proof.followup, second); + evidence.checkpoints.followup = true; + evidence.checkpoints.visible_result = true; + return { issueId: issue.id as string, issueIdentifier: issue.identifier as string }; +} diff --git a/tests/runner-e2e/connection-launch.ts b/tests/runner-e2e/connection-launch.ts new file mode 100644 index 00000000000..8781a2b4abd --- /dev/null +++ b/tests/runner-e2e/connection-launch.ts @@ -0,0 +1,394 @@ +import { chromium, type BrowserContext } from "@playwright/test"; +import { execFileSync } from "node:child_process"; +import { randomBytes, createHash } from "node:crypto"; +import { mkdir, mkdtemp, writeFile, rm, readdir, lstat, open } from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { FixtureRegistry } from "./fixture-registry.js"; +import { buildRunnerCampaign } from "./history.js"; +import { renderRunnerE2EDashboard } from "./dashboard.js"; +import { buildRuntimeUsage, summarizeExecutionBilling } from "./billing.js"; +import { stageDashboardBrandAssets } from "./report-assets.js"; +import { classifyFailure } from "./failure-classifier.js"; +import { packageEvidence } from "./evidence.js"; +import { assertSecretFree, sanitizeJson } from "./redaction.js"; +import { connectionCell, connectionHarnesses, connectionModes, connectionScopeGaps, supportsConnection } from "./connection-cases.js"; +import { assertOutsideRepository, ConnectionBlock, ConnectionFailure, loadConnectionConfig, resolveConnectionSecret, resolveConnectionSettings, type ConnectionConfig } from "./connection-config.js"; +import { connectionEvidencePasses, connectionRunDiagnostic, gradeConnectionEvidence, type ConnectionEvidence } from "./connection-evidence.js"; +import { ConnectionApi, connectionDelay, startConnectionTarget, verifyConnectionTarget } from "./connection-target.js"; +import { runConnectionFlow } from "./connection-flow.js"; +import type { MatrixExecution, RunnerE2EResult } from "./types.js"; + +type Row = Record; + +export function connectionCreationReceipt(pathname: string, method: string, companyId: string, loginIds: ReadonlySet, body: unknown): string | undefined { + const root = `/api/companies/${companyId}/ai-connections`; + const loginId = pathname.startsWith(`${root}/login/`) ? pathname.slice(`${root}/login/`.length) : undefined; + const createdHere = method === "POST" && (pathname === root || pathname === `${root}/local`); + const ownedLogin = method === "GET" && loginId && loginIds.has(loginId); + if (!createdHere && !ownedLogin) return; + const id = (body as { connectionId?: unknown } | null)?.connectionId; + return typeof id === "string" && /^[a-f0-9]{8}-(?:[a-f0-9]{4}-){3}[a-f0-9]{12}$/i.test(id) ? id : undefined; +} + +export async function cleanupConnectionCompany(input: { + api: ConnectionApi; companyId: string; agentName: string; + attachedCompany: boolean; retainCompany: boolean; + createdConnectionIds: ReadonlySet; + evidence: Pick; + collectDiagnostics: () => Promise; +}) { + const { api, companyId, agentName, evidence } = input; + const agents = await api.get(`/api/companies/${companyId}/agents`); + const fixtureAgents = agents.filter(agent => agent.name === agentName); + try { + for (const agent of fixtureAgents) { + await api.patch(`/api/agents/${agent.id}`, { status: "paused" }); + const liveRuns = await api.get(`/api/companies/${companyId}/heartbeat-runs?agentId=${agent.id}&limit=100`); + for (const run of liveRuns.filter(run => run.agentId === agent.id && ["queued", "running"].includes(run.status))) { + await api.post(`/api/heartbeat-runs/${run.id}/cancel`); + } + } + } finally { + // Deleting an attached-company agent also deletes its runs and logs. + // Capture the stopped runs first, including when cancellation fails. + await input.collectDiagnostics(); + } + if (input.retainCompany) { evidence.cleanupRetained = true; return; } + const connections = (await api.get(`/api/companies/${companyId}/ai-connections`)).connections; + for (const connection of connections.filter((connection: Row) => input.createdConnectionIds.has(connection.id) && connection.status !== "revoked")) { + await api.delete(`/api/tool-connections/${connection.id}`); + } + if ((await api.get(`/api/companies/${companyId}/ai-connections`)).connections.some((connection: Row) => input.createdConnectionIds.has(connection.id) && connection.status !== "revoked")) { + throw new ConnectionFailure("connection_revocation_failed"); + } + if (!input.attachedCompany) { await api.post(`/api/companies/${companyId}/archive`); evidence.companyArchived = true; } + else for (const agent of fixtureAgents) await api.delete(`/api/agents/${agent.id}`); +} + +export async function openConnectionBrowser(config: ConnectionConfig, repositoryRoot: string, origin: string) { + const temporary = config.browser.freshness === "signed-out" || !config.browser.profileDir; + const profile = temporary ? await mkdtemp(path.join(os.tmpdir(), "paperclip-qa-browser-")) : path.resolve(config.browser.profileDir!); + await assertOutsideRepository(profile, repositoryRoot); + await mkdir(profile, { recursive: true, mode: 0o700 }); + const stat = await lstat(profile); + if (stat.isSymbolicLink() || !stat.isDirectory() || (stat.mode & 0o077) !== 0) throw new ConnectionBlock("blocked_target", "browser_profile_requires_private_directory"); + const marker = path.join(profile, ".paperclip-provider-qa"); + const entries = await readdir(profile); + if (entries.length && !entries.includes(path.basename(marker))) throw new ConnectionBlock("blocked_target", "use_a_dedicated_empty_qa_browser_profile"); + if (!entries.length) await writeFile(marker, "Dedicated Paperclip provider QA browser. Contains private authentication state.\n", { mode: 0o600, flag: "wx" }); + const lock = path.join(profile, ".paperclip-qa.lock"); + try { await (await open(lock, "wx", 0o600)).close(); } + catch { throw new ConnectionBlock("blocked_target", "qa_browser_profile_in_use"); } + let context: BrowserContext; + try { + const browserEnv = Object.fromEntries(["PATH", "HOME", "USER", "LOGNAME", "TMPDIR", "DISPLAY", "WAYLAND_DISPLAY", "XAUTHORITY", "LANG"].flatMap(key => process.env[key] ? [[key, process.env[key]!]] : [])); + context = await chromium.launchPersistentContext(profile, { + channel: config.browser.channel === "chrome" ? "chrome" : undefined, + headless: !config.browser.headed, baseURL: origin, viewport: { width: 1440, height: 1000 }, reducedMotion: "reduce", + env: browserEnv, acceptDownloads: true, + // Teardown needs this context's authenticated request client to cancel + // login sessions and revoke fixtures before the browser is closed. + handleSIGINT: false, handleSIGTERM: false, handleSIGHUP: false, + // Deliberately use the Playwright library without Test's automatic reporters, + // traces, HARs, videos, snapshots, assertion call logs, or failure screenshots. + }); + context.setDefaultTimeout(30_000); + context.setDefaultNavigationTimeout(30_000); + } catch { + await rm(lock, { force: true }); + if (temporary) await rm(profile, { recursive: true, force: true }); + throw new ConnectionBlock("blocked_target", "qa_browser_launch_failed"); + } + return { context, close: async () => { try { await context.close(); } finally { await rm(lock, { force: true }); if (temporary) await rm(profile, { recursive: true, force: true }); } } }; +} + +export function safeConnectionFailure(error: unknown, phase: string) { + if (error instanceof ConnectionBlock) return error.code; + // Arbitrary Playwright/provider messages can contain filled values and URLs. + const known = error instanceof ConnectionFailure; + const frame = error instanceof Error ? error.stack?.match(/connection-(?:flow|target|launch)\.ts:\d+:\d+/)?.[0] : undefined; + return known ? error.message : `${phase}_failed${frame ? ` (${frame})` : ""}`; +} + +function sourceRevision(repositoryRoot: string) { + const git = (args: string[]) => execFileSync("git", args, { cwd: repositoryRoot, encoding: "utf8", stdio: ["ignore", "pipe", "ignore"] }).trim(); + return { sha: git(["rev-parse", "HEAD"]), ref: `${git(["branch", "--show-current"])}${git(["status", "--porcelain"]) ? " (working tree)" : ""}`, workflowRunUrl: null }; +} + +async function safeWrite(file: string, value: unknown, secrets: readonly string[]) { + const text = `${JSON.stringify(sanitizeJson(value, secrets), null, 2)}\n`; + assertSecretFree(text, secrets, path.basename(file)); + await writeFile(file, text, { mode: 0o600 }); +} + +async function connectionAttempt(execution: MatrixExecution, config: ConnectionConfig, origin: string, summaryDir: string, repositoryRoot: string, signal: AbortSignal): Promise { + const startedAt = new Date().toISOString(); + const nonce = randomBytes(6).toString("hex"); + const privateDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-connection-proof-")); + const evidenceDir = path.join(summaryDir, execution.suite.id, execution.profile.id, execution.environment.id, execution.task.id, "attempt-1"); + await mkdir(evidenceDir, { recursive: true, mode: 0o700 }); + const { entry, mode } = connectionCell(execution); + const evidence: ConnectionEvidence = { version: 1, outcome: "running", phase: "target", assisted: false, authFreshness: config.browser.freshness, + target: { mode: config.target.mode, origin, commit: null, deploymentMode: null }, entry, method: mode, checkpoints: {}, waits: [] }; + const secrets: string[] = []; + const checkpoint = async (phase: string) => { evidence.phase = phase; await safeWrite(path.join(summaryDir, "progress.json"), { executionId: execution.id, ...evidence }, secrets); }; + const result: RunnerE2EResult = { schema: "paperclip.runner-e2e.result/v2", executionId: execution.id, suiteId: execution.suite.id, suiteDefinitionHash: execution.suiteDefinitionHash, + source: sourceRevision(repositoryRoot), attempt: 1, status: "failed", profileId: execution.profile.id, environmentId: execution.environment.id, caseId: execution.task.id, + provider: execution.profile.provider, model: execution.profile.model, runtimeMode: execution.profile.generation, startedAt, finishedAt: startedAt, durationMs: 0, cleanup: "not_started", providerConnection: evidence }; + let browser: Awaited> | undefined; + let fixtures: Awaited> | undefined; + const observedRuns = new Map(); + const createdConnectionIds = new Set(); + const ownedLoginIds = new Set(); + let fixtureCompanyId: string | undefined; + let diagnosticApi: ConnectionApi | undefined; + let diagnosticsCollected = false; + const collectDiagnostics = async () => { + if (diagnosticsCollected || !diagnosticApi || !observedRuns.size) return; + evidence.runDiagnostics = []; + for (const observed of observedRuns.values()) { + let run = observed; let log: unknown; + try { run = await diagnosticApi.get(`/api/heartbeat-runs/${observed.id}`); observedRuns.set(observed.id, run); } catch { /* Keep the last durable observation. */ } + try { log = await diagnosticApi.get(`/api/heartbeat-runs/${observed.id}/log?limitBytes=2000000`); } catch { /* Mark missing log evidence explicitly. */ } + evidence.runDiagnostics.push(connectionRunDiagnostic(run, log)); + } + diagnosticsCollected = true; + }; + const loginSessions = new Map(); + const loginObservers: Promise[] = []; + let stopped: "test_interrupted" | "cell_deadline_reached" | undefined; + const stopBrowserActions = () => { for (const page of browser?.context.pages() ?? []) void page.close().catch(() => {}); }; + const interrupt = () => { stopped = "test_interrupted"; stopBrowserActions(); }; + const assertActive = () => { if (stopped) throw new ConnectionBlock("blocked_target", stopped); }; + signal.addEventListener("abort", interrupt, { once: true }); + if (signal.aborted) interrupt(); + const deadlineTimer = setTimeout(() => { stopped = "cell_deadline_reached"; stopBrowserActions(); }, execution.task.attemptTimeoutMs?.[execution.environment.id] ?? 30 * 60_000); + try { + assertActive(); + const identity = await verifyConnectionTarget(origin, config); + Object.assign(evidence.target, identity); + evidence.checkpoints.target = true; + assertActive(); + const settings = resolveConnectionSettings(execution, config); + result.model = settings.model; + evidence.model = settings.model; + if (mode === "subscription" && !config.browser.headed) throw new ConnectionBlock("awaiting_user", "subscription_requires_headed_browser"); + // Resolve only the selected credential, and only after validating the target. + const secret = await resolveConnectionSecret(config, settings.credentialEnv); + if (secret) secrets.push(secret); + assertActive(); + browser = await openConnectionBrowser(config, repositoryRoot, origin); + assertActive(); + const page = browser.context.pages()[0] ?? await browser.context.newPage(); + const api = new ConnectionApi(browser.context.request, origin); + diagnosticApi = api; + browser.context.on("response", response => { + const url = new URL(response.url()); + if (url.origin !== origin || !fixtureCompanyId || !url.pathname.startsWith(`/api/companies/${fixtureCompanyId}/`) || !response.ok()) return; + const method = response.request().method(); + const local = method === "POST" && /^\/api\/companies\/([a-f0-9-]+)\/ai-connections\/local\/attempts$/.test(url.pathname); + const cloud = method === "POST" && /^\/api\/companies\/([a-f0-9-]+)\/(?:setup-token-login-sessions|adapters\/[^/]+\/login-sessions)$/.test(url.pathname); + const root = `/api/companies/${fixtureCompanyId}/ai-connections`; + const receipt = (method === "POST" && [root, `${root}/local`].includes(url.pathname)) || (method === "GET" && url.pathname.startsWith(`${root}/login/`)); + if (!local && !cloud && !receipt) return; + loginObservers.push((async () => { + const body = await response.json(); + if ((local || cloud) && typeof body.sessionId === "string" && /^[a-zA-Z0-9_-]+$/.test(body.sessionId)) { + ownedLoginIds.add(body.sessionId); + loginSessions.set(`${url.pathname}/${body.sessionId}${cloud ? "/cancel" : ""}`, local ? "DELETE" : "POST"); + } + const id = connectionCreationReceipt(url.pathname, method, fixtureCompanyId!, ownedLoginIds, body); + if (id) createdConnectionIds.add(id); + })().catch(() => {})); + }); + const refreshOwnedConnections = async () => { + if (!fixtureCompanyId) return; + for (const sessionId of ownedLoginIds) { + const route = `/api/companies/${fixtureCompanyId}/ai-connections/login/${sessionId}`; + try { + const id = connectionCreationReceipt(route, "GET", fixtureCompanyId, ownedLoginIds, await api.get(route)); + if (id) createdConnectionIds.add(id); + } catch { /* An unfinished or cancelled owned login has no account receipt. */ } + } + }; + await page.goto(origin, { waitUntil: "domcontentloaded" }); + try { await api.get("/api/companies"); } + catch (authError) { + if (!(authError instanceof ConnectionFailure) || !["api_status_401", "api_status_403"].includes(authError.message)) throw authError; + if (!config.browser.headed) throw new ConnectionBlock("awaiting_user", "board_login_required"); + evidence.assisted = true; + evidence.waits.push({ kind: "board_login", startedAt: new Date().toISOString() }); + await checkpoint("awaiting_board_login"); + console.log(`[provider-connections] Sign in to Paperclip at ${origin} in the QA browser; this test resumes automatically.`); + const deadline = Date.now() + config.browser.loginTimeoutMs; + let ready = false; + while (Date.now() < deadline) { assertActive(); try { await api.get("/api/companies"); ready = true; break; } catch { await connectionDelay(1000); } } + if (!ready) throw new ConnectionBlock("awaiting_user", "board_login_deadline_reached"); + evidence.waits.at(-1)!.finishedAt = new Date().toISOString(); + } + const registry = new FixtureRegistry(); + const attachedCompany = config.target.mode === "attach" ? config.target.companyId : undefined; + registry.register({ id: "connection-company", setup: async () => { + if (attachedCompany) { + const company = await api.get(`/api/companies/${attachedCompany}`); + if (!company.name.startsWith("Connection QA") || (await api.get(`/api/companies/${company.id}/agents`)).some(agent => agent.status !== "terminated") || (await api.get(`/api/companies/${company.id}/ai-connections`)).connections.some((connection: Row) => connection.status !== "revoked")) throw new ConnectionBlock("blocked_target", "attach_requires_empty_dedicated_qa_company"); + fixtureCompanyId = company.id; + return company; + } + const company = await api.post("/api/companies", { name: `Connection QA ${nonce}`, description: "Disposable provider-connections QA fixture", budgetMonthlyCents: config.budgetCents }); + fixtureCompanyId = company.id; + return company; + }, teardown: async company => { + await Promise.all(loginObservers); + await refreshOwnedConnections(); + for (const [route, method] of loginSessions) { + try { await api.response(route, method); } + catch (error) { if (!(error instanceof ConnectionFailure) || error.message !== "api_status_404") throw error; } + } + await refreshOwnedConnections(); + // Creation receipts identify this attempt even when concurrent campaigns + // passed the initial empty-company check at the same time. + await cleanupConnectionCompany({ api, companyId: company.id, agentName: `Connection QA ${nonce}`, + attachedCompany: Boolean(attachedCompany), retainCompany: config.retainCompany, createdConnectionIds, evidence, collectDiagnostics }); + } }); + registry.register({ id: "connection-environment", dependencies: ["connection-company"], setup: async values => { + const company = values.get("connection-company") as Row; + const environments = await api.get(`/api/companies/${company.id}/environments`); + const selectedId = config.target.mode === "attach" ? config.target.environmentId : undefined; + const environment = selectedId ? environments.find(env => env.id === selectedId) : environments.find(env => env.status === "active" && (execution.environment.id === "local" ? env.driver === "local" : env.driver === "sandbox" && env.config?.provider === "daytona")); + if (!environment || (execution.environment.id === "local" ? environment.driver !== "local" : environment.config?.provider !== "daytona")) throw new ConnectionBlock("blocked_target", "configured_execution_environment_required"); + if (execution.profile.generation === "native") { + const experimental = await api.get("/api/instance/settings/experimental"); + if (!experimental.enableNativeRunner) { + if (config.target.mode === "attach") throw new ConnectionBlock("blocked_target", "native_runner_disabled_on_target"); + await api.patch("/api/instance/settings/experimental", { enableNativeRunner: true }); + } + } + return environment; + } }); + fixtures = await registry.setupAll(); + const company = fixtures.values.get("connection-company") as Row; + evidence.companyId = company.id; + const environment = fixtures.values.get("connection-environment") as Row; + const observed = await runConnectionFlow({ page, api, execution, config, settings, company, environment, secret, nonce, privateDir, evidence, checkpoint, assertActive, + createdConnectionIds, refreshOwnedConnections, observeRun: run => observedRuns.set(run.id, run) }); + Object.assign(result, observed); + evidence.outcome = "passed"; + if (!connectionEvidencePasses(evidence)) throw new ConnectionFailure("connection_proof_incomplete"); + // Only the fixture's synthetic task is captured, after authentication. + const expectedPath = `/${company.issuePrefix}/issues/${observed.issueIdentifier}`; + const url = new URL(page.url()); + if (url.origin !== origin || url.pathname !== expectedPath) throw new ConnectionFailure("final_screenshot_route_mismatch"); + const screenshot = await page.screenshot({ path: path.join(privateDir, "final-state.png"), fullPage: true }); + result.screenshots = [{ id: "final-state", label: "Verified connection task and follow-up", file: "final-state.png", sha256: createHash("sha256").update(screenshot).digest("hex") }]; + result.status = "passed"; + } catch (caught) { + const error = stopped ? new ConnectionBlock("blocked_target", stopped) : caught; + if (error instanceof AggregateError) result.cleanup = "failed"; + evidence.outcome = error instanceof ConnectionBlock ? error.reason : "failed"; + result.failureClass = error instanceof ConnectionBlock ? "permanent_infrastructure" : classifyFailure(error); + result.error = safeConnectionFailure(error, evidence.phase); + } finally { + clearTimeout(deadlineTimer); + try { await fixtures?.teardown(); if (result.cleanup !== "failed") result.cleanup = "passed"; } + catch (cleanupError) { result.cleanup = "failed"; result.status = "failed"; result.failureClass = "cleanup_failure"; result.error = `${result.error ? `${result.error}; ` : ""}fixture_cleanup_failed`; + const causes = cleanupError instanceof AggregateError ? cleanupError.errors : [cleanupError]; + console.log(`[provider-connections] Cleanup: ${causes.map(cause => safeConnectionFailure(cause, "cleanup")).join(", ")}`); } + // Also preserve evidence when setup or early cleanup failed before the + // company teardown reached its capture point. Never refetch deleted runs. + await collectDiagnostics(); + try { await browser?.close(); } + catch { result.cleanup = "failed"; result.status = "failed"; result.failureClass = "cleanup_failure"; result.error = "qa_browser_cleanup_failed"; } + result.finishedAt = new Date().toISOString(); + result.durationMs = Date.parse(result.finishedAt) - Date.parse(startedAt); + result.runIds = [...observedRuns.keys()]; + // Usage is projected to numeric billing below; opaque provider output is never retained. + const usage = { runs: [...observedRuns.values()].map(run => ({ usage: run.usageJson })) }; + result.runtimeUsage = buildRuntimeUsage({ environmentId: execution.environment.id, runs: [...observedRuns.values()], fallbackFinishedAt: result.finishedAt }); + result.billing = summarizeExecutionBilling({ ...result, usage }); + // UI environment probes may incur cost without heartbeat usage receipts. + result.billing.complete = false; + result.billing.llm.costStatus = result.billing.llm.runsWithTokenUsage ? "partial" : "unavailable"; + result.matcherResults = gradeConnectionEvidence(evidence); + await safeWrite(path.join(privateDir, "result.json"), result, secrets); + const manifest = await packageEvidence({ privateDir, uploadDir: evidenceDir, secrets, expectPassScreenshot: false }); + if (manifest.leaks.length || manifest.missing.length) { result.status = "failed"; result.failureClass = "secret_leak"; result.error = "evidence_packaging_failed"; evidence.outcome = "failed"; } + await safeWrite(path.join(evidenceDir, "result.json"), result, secrets); + await safeWrite(path.join(evidenceDir, "evidence-manifest.json"), manifest, secrets); + await safeWrite(path.join(summaryDir, "progress.json"), { executionId: execution.id, ...evidence }, secrets); + await rm(privateDir, { recursive: true, force: true }); + signal.removeEventListener("abort", interrupt); + } + return result; +} + +export async function runConnectionCampaign(input: { executions: readonly MatrixExecution[]; catalog: readonly MatrixExecution[]; configFile?: string; repositoryRoot: string }) { + return withConnectionCampaignCancellation(async (signal) => { + if (input.executions.some(e => e.task.flow !== "provider_connection")) throw new ConnectionFailure("Select provider-connections separately from other suites."); + const config = await loadConnectionConfig(input.configFile); + const campaignId = `connections-${new Date().toISOString().replace(/[:.]/g, "-")}-${randomBytes(3).toString("hex")}`; + const summaryDir = path.join(input.repositoryRoot, "tests/runner-e2e/results", campaignId); + await mkdir(summaryDir, { recursive: true, mode: 0o700 }); + const results: RunnerE2EResult[] = []; + let target: Awaited> | undefined; + try { + assertConnectionCampaignActive(signal); + target = await startConnectionTarget(config, input.executions, input.repositoryRoot, signal); + assertConnectionCampaignActive(signal); + console.log(`[provider-connections] ${input.executions.length} selected cells; target ${target.origin}; one browser/account at a time; max ${config.maxRuns} task runs and ${config.budgetCents} cents per fixture. UI probe spend may be unreported.`); + for (const execution of input.executions) { + assertConnectionCampaignActive(signal); + console.log(`[provider-connections] Starting ${execution.id}`); + const result = await connectionAttempt(execution, config, target.origin, summaryDir, input.repositoryRoot, signal); + results.push(result); + console.log(`[provider-connections] ${result.providerConnection!.outcome}: ${execution.id}${result.error ? ` (${result.error})` : ""}`); + // Every completed/blocked cell is durable before the next login begins. + await writeConnectionCampaignReport(input, campaignId, summaryDir, results); + if (signal.aborted || result.error === "test_interrupted" || result.providerConnection!.outcome === "awaiting_user") break; + } + } catch (error) { + if (!signal.aborted) throw error; + // Preserve completed receipts even if interrupted during server startup or + // report generation. Cancellation never admits another cell. + await writeConnectionCampaignReport(input, campaignId, summaryDir, results); + } finally { await target?.stop(); } + console.log(`[provider-connections] Report: ${path.join(summaryDir, "dashboard.html")}`); + if (signal.aborted || results.some(result => result.status !== "passed" || result.cleanup !== "passed")) process.exitCode = 1; + return summaryDir; + }); +} + +function assertConnectionCampaignActive(signal: AbortSignal) { + if (signal.aborted) throw new ConnectionBlock("blocked_target", "test_interrupted"); +} + +/** One cancellation state covers configuration, startup, every cell, reports, + * and owned-target teardown. Repeated signals cannot reset it. */ +export async function withConnectionCampaignCancellation(run: (signal: AbortSignal) => Promise): Promise { + const controller = new AbortController(); + const interrupt = () => controller.abort(); + const signals = ["SIGINT", "SIGTERM", "SIGHUP"] as const; + for (const signal of signals) process.on(signal, interrupt); + try { return await run(controller.signal); } + finally { for (const signal of signals) process.removeListener(signal, interrupt); } +} + +export async function writeConnectionCampaignReport(input: { executions: readonly MatrixExecution[]; catalog: readonly MatrixExecution[] }, campaignId: string, summaryDir: string, results: RunnerE2EResult[]) { + const generatedAt = new Date().toISOString(); + const expected = input.executions.map(e => e.id); + const campaign = buildRunnerCampaign({ campaignId, generatedAt, expected, results }); + await safeWrite(path.join(summaryDir, "campaign.json"), campaign, []); + await safeWrite(path.join(summaryDir, "normalized-results.json"), { ...campaign, results: results.map(result => ({ ...result, evidenceValid: result.failureClass !== "secret_leak", evidenceErrors: result.failureClass === "secret_leak" ? ["evidence_packaging_failed"] : [] })) }, []); + await stageDashboardBrandAssets(summaryDir); + await safeWrite(path.join(summaryDir, "connection-coverage.json"), { + required: input.catalog.filter(e => e.suite.id === "provider-connections").length, + selected: expected.length, attempted: results.length, passed: results.filter(r => r.status === "passed" && r.cleanup === "passed").length, + blocked: results.filter(r => ["awaiting_user", "missing_credential", "blocked_target"].includes(r.providerConnection!.outcome)).length, + targets: [...new Set(results.map(r => r.providerConnection!.target.origin))], + gaps: [...connectionScopeGaps, ...connectionHarnesses.flatMap(h => connectionModes.filter(mode => !supportsConnection(h, mode)).map(mode => ({ adapter: h.adapter, mode, status: "unsupported" })))], + }, []); + const dashboard = renderRunnerE2EDashboard({ title: `Live provider connections · ${campaignId}`, generatedAt, expected, catalog: input.catalog, campaign, + entries: results.map(result => ({ result, valid: result.failureClass !== "secret_leak", errors: result.failureClass === "secret_leak" ? ["evidence_packaging_failed"] : [], evidenceBaseHref: [result.suiteId, result.profileId, result.environmentId, result.caseId, "attempt-1"].join("/") })) }); + await writeFile(path.join(summaryDir, "dashboard.html"), dashboard, { mode: 0o600 }); +} diff --git a/tests/runner-e2e/connection-support.test.ts b/tests/runner-e2e/connection-support.test.ts new file mode 100644 index 00000000000..bc46b3064b7 --- /dev/null +++ b/tests/runner-e2e/connection-support.test.ts @@ -0,0 +1,282 @@ +import { describe, expect, it, vi } from "vitest"; +import { mkdtemp, writeFile, rm } from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { AGENT_ADAPTER_TYPES } from "../../packages/shared/src/constants.js"; +import { runnerMatrix } from "./catalog.js"; +import { parseRunnerSelectors, selectRunnerExecutions, buildMatrixJobs } from "./selectors.js"; +import { connectionHarnesses, connectionScopeGaps, connectionCell, supportsConnection } from "./connection-cases.js"; +import { parseConnectionConfig, targetOrigin, selectedSecret, resolveConnectionSecret, resolveConnectionSettings } from "./connection-config.js"; +import { connectionCheckpoints, connectionEvidencePasses, connectionRunDiagnostic, createConnectionProof, verifyConnectionArtifact, verifyConnectionRun, completedConnectionArtifactRun, connectionProbeChecks, type ConnectionEvidence } from "./connection-evidence.js"; +import { validateRetainedRunnerResult } from "./result-validation.js"; +import { verifyConnectionTarget } from "./connection-target.js"; +import { cleanupConnectionCompany, connectionCreationReceipt, runConnectionCampaign, safeConnectionFailure, withConnectionCampaignCancellation } from "./connection-launch.js"; +import type { ConnectionApi } from "./connection-target.js"; +import * as connectionConfig from "./connection-config.js"; +import * as connectionTarget from "./connection-target.js"; +import * as reportAssets from "./report-assets.js"; +import { buildRunnerCampaign } from "./history.js"; +import { renderCaseOutcome } from "./case-outcome.js"; +import type { RunnerE2EResult } from "./types.js"; + +const cells = runnerMatrix.filter(e => e.suite.id === "provider-connections"); +const cell = (method: string) => cells.find(e => e.id === `provider-connections.connection-codex-native.local.agent-${method}`)!; + +describe("connection fixture cleanup evidence", () => { + it.each([true, false])("captures stopped runs and logs before destructive cleanup (attached=%s)", async (attachedCompany) => { + const events: string[] = []; + let status = "running", deleted = false, revoked = false; + const evidence: Partial = {}; + const api = { + get: vi.fn(async (route: string) => { + if (route.endsWith("/agents")) return [{ id: "qa-agent", name: "Fixture" }, { id: "foreign", name: "Other" }]; + if (route.includes("heartbeat-runs?")) return [{ id: "run", agentId: "qa-agent", status }]; + if (route.endsWith("/ai-connections")) return { connections: [{ id: "account", status: revoked ? "revoked" : "active" }, { id: "foreign-account", status: "active" }] }; + if (deleted) throw new Error("Run was deleted"); + if (route.includes("/log?")) { events.push("log"); return { content: JSON.stringify({ chunk: JSON.stringify({ type: "acpx.error", childStderrTail: "HTTP 503 model overloaded" }) + "\n" }) }; } + events.push("run"); return { id: "run", status }; + }), + patch: vi.fn(async () => { events.push("pause"); }), + post: vi.fn(async (route: string) => { if (route.endsWith("/cancel")) { events.push("cancel"); status = "cancelled"; } else events.push("archive"); }), + delete: vi.fn(async (route: string) => { if (route.endsWith("/foreign-account")) throw new Error("Revoked a concurrent campaign"); if (route.includes("tool-connections")) { events.push("revoke"); revoked = true; } else { events.push("delete-agent"); deleted = true; } }), + }; + await cleanupConnectionCompany({ api: api as unknown as ConnectionApi, companyId: "company", agentName: "Fixture", attachedCompany, + retainCompany: false, createdConnectionIds: new Set(["account"]), evidence, collectDiagnostics: async () => { + const run = await api.get("/api/heartbeat-runs/run"); + const log = await api.get("/api/heartbeat-runs/run/log?limitBytes=2000000"); + evidence.runDiagnostics = [connectionRunDiagnostic(run, log)]; + } }); + expect(events).toEqual(["pause", "cancel", "run", "log", "revoke", attachedCompany ? "delete-agent" : "archive"]); + expect(evidence.runDiagnostics).toEqual([{ runId: "run", status: "cancelled", signals: ["provider_overloaded"], logAvailable: true }]); + expect(api.delete).not.toHaveBeenCalledWith("/api/agents/foreign"); + expect(api.delete).not.toHaveBeenCalledWith("/api/tool-connections/foreign-account"); + }); + it("preserves diagnostics and avoids deletion when cancellation fails", async () => { + const collectDiagnostics = vi.fn(async () => {}); + const api = { + get: vi.fn(async (route: string) => route.endsWith("/agents") ? [{ id: "qa-agent", name: "Fixture" }] : [{ id: "run", agentId: "qa-agent", status: "running" }]), + patch: vi.fn(async () => {}), post: vi.fn(async () => { throw new Error("cancel failed"); }), delete: vi.fn(), + }; + await expect(cleanupConnectionCompany({ api: api as unknown as ConnectionApi, companyId: "company", agentName: "Fixture", attachedCompany: true, + retainCompany: false, createdConnectionIds: new Set(), evidence: {}, collectDiagnostics })).rejects.toThrow("cancel failed"); + expect(collectDiagnostics).toHaveBeenCalledOnce(); + expect(api.delete).not.toHaveBeenCalled(); + }); + it("accepts only creation receipts from this browser and its owned login sessions", () => { + const body = { connectionId: "00000000-0000-4000-8000-000000000021" }; + const root = "/api/companies/company/ai-connections"; + const logins = new Set(["owned-session"]); + expect(connectionCreationReceipt(root, "POST", "company", logins, body)).toBe(body.connectionId); + expect(connectionCreationReceipt(`${root}/local`, "POST", "company", logins, body)).toBe(body.connectionId); + expect(connectionCreationReceipt(`${root}/login/owned-session`, "GET", "company", logins, body)).toBe(body.connectionId); + expect(connectionCreationReceipt(`${root}/login/foreign-session`, "GET", "company", logins, body)).toBeUndefined(); + expect(connectionCreationReceipt(root, "GET", "company", logins, body)).toBeUndefined(); + expect(connectionCreationReceipt(root, "POST", "different-company", logins, body)).toBeUndefined(); + expect(connectionCreationReceipt(root, "POST", "company", logins, { connectionId: "not-a-receipt" })).toBeUndefined(); + }); +}); + +describe("campaign interruption", () => { + it.each(["SIGINT", "SIGTERM", "SIGHUP"] as const)("keeps %s cancellation active until teardown finishes", async (signal) => { + const before = process.listenerCount(signal); + await withConnectionCampaignCancellation(async (state) => { + expect(process.listenerCount(signal)).toBe(before + 1); + process.emit(signal); + expect(state.aborted).toBe(true); + await Promise.resolve(); + process.emit(signal); + expect(state.aborted).toBe(true); + }); + expect(process.listenerCount(signal)).toBe(before); + }); + it.each(["SIGINT", "SIGTERM", "SIGHUP"] as const)("stops owned targets and admits no next cell after %s during startup or reporting", async (signal) => { + const repositoryRoot = path.resolve(import.meta.dirname, "../.."); + const exitCode = process.exitCode; + for (const phase of ["startup", "report"] as const) { + const stop = vi.fn(async () => {}); + const before = process.listenerCount(signal); + const config = parseConnectionConfig({}); + const load = vi.spyOn(connectionConfig, "loadConnectionConfig").mockResolvedValue(config); + const credential = vi.spyOn(connectionConfig, "resolveConnectionSecret").mockRejectedValue(new connectionConfig.ConnectionBlock("missing_credential", "synthetic_missing_credential")); + const target = vi.spyOn(connectionTarget, "startConnectionTarget").mockImplementation(async (_config, _executions, _root, state) => { + expect(state?.aborted).toBe(false); + if (phase === "startup") process.emit(signal); + return { origin: "http://127.0.0.1:1", stop }; + }); + const verify = vi.spyOn(connectionTarget, "verifyConnectionTarget").mockResolvedValue({ commit: "a".repeat(40), deploymentMode: "local_trusted" }); + const report = vi.spyOn(reportAssets, "stageDashboardBrandAssets").mockImplementation(async () => { if (phase === "report") process.emit(signal); }); + let summary: string | undefined; + try { + summary = await runConnectionCampaign({ executions: [cell("api-key"), cell("subscription")], catalog: cells, repositoryRoot }); + expect(credential).toHaveBeenCalledTimes(phase === "startup" ? 0 : 1); + expect(verify).toHaveBeenCalledTimes(phase === "startup" ? 0 : 1); + expect(stop).toHaveBeenCalledTimes(1); + expect(process.exitCode).toBe(1); + expect(process.listenerCount(signal)).toBe(before); + } finally { + load.mockRestore(); credential.mockRestore(); target.mockRestore(); verify.mockRestore(); report.mockRestore(); + process.exitCode = exitCode; + if (summary) await rm(summary, { recursive: true, force: true }); + } + } + }); +}); + +describe("live connection coverage", () => { + it("inventories every built-in adapter and keeps remote exclusions explicit", () => { + const inventory = new Set(["paperclip_runner", ...connectionHarnesses.map(h => h.adapter), ...connectionScopeGaps.map(h => h.adapter)]); + expect(AGENT_ADAPTER_TYPES.filter(type => !inventory.has(type))).toEqual([]); + expect(connectionHarnesses.some(h => h.adapter === "hermes_local")).toBe(true); + expect(connectionScopeGaps.find(h => h.adapter === "hermes_gateway")?.status).toBe("excluded"); + }); + it("includes both UI entry points and actual runner variants without invented auth capabilities", () => { + expect(cells.length).toBe(116); + expect(new Set(cells.map(e => connectionCell(e).entry))).toEqual(new Set(["agent", "apps"])); + expect(cells.filter(e => connectionCell(e).harness.id === "grok").every(e => ["subscription", "api-key"].includes(connectionCell(e).mode))).toBe(true); + expect(supportsConnection(connectionHarnesses.find(h => h.id === "gemini")!, "subscription")).toBe(false); + }); + it("never runs on --all or advertises a preinstalled subscription token as a credential", () => { + expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(e => e.suite.id === "provider-connections")).toBe(false); + expect(cell("subscription").requiredCredentials).toEqual([]); + expect(() => buildMatrixJobs(cells)).toThrow(/not admitted/); + expect(parseRunnerSelectors(["--suite", "provider-connections", "--connection-config", "/private/qa.json"]).connectionConfig).toBe("/private/qa.json"); + }); +}); + +describe("target and selected credential boundaries", () => { + it("accepts local/staging origins and rejects credential-bearing or redirectable target strings", () => { + expect(targetOrigin("http://127.0.0.1:3109/")).toBe("http://127.0.0.1:3109"); + expect(targetOrigin("https://staging.example.test")).toBe("https://staging.example.test"); + for (const url of ["http://staging.example.test", "https://user:password@example.test", "https://example.test/path", "https://example.test/?token=secret", "https://example.test/#secret", "file:///tmp/qa"]) expect(() => targetOrigin(url)).toThrow(); + }); + it("requires an exact staging revision and rejects unknown configuration instead of ignoring typos", () => { + expect(() => parseConnectionConfig({ target: { mode: "attach", baseURL: "https://example.test" } })).toThrow(); + expect(() => parseConnectionConfig({ target: { mode: "managed-local", baseURL: "https://example.test" } })).toThrow(); + expect(() => parseConnectionConfig({ broswer: {} })).toThrow(); + expect(() => parseConnectionConfig({ browser: { loginTimeoutMs: Infinity } })).toThrow(); + expect(parseConnectionConfig({}).browser.headed).toBe(true); + }); + it("pins the target revision before permitting credential selection", async () => { + const config = parseConnectionConfig({ target: { mode: "attach", baseURL: "https://staging.example.test", expectedCommit: "a".repeat(40), deploymentMode: "authenticated" } }); + const fetch = vi.spyOn(globalThis, "fetch").mockResolvedValue(new Response(JSON.stringify({ commit: "b".repeat(40), deploymentMode: "authenticated" }))); + try { + await expect(verifyConnectionTarget("https://staging.example.test", config)).rejects.toThrow("target_identity_mismatch"); + expect(fetch.mock.calls[0]?.[1]).toMatchObject({ redirect: "error" }); + fetch.mockResolvedValue(new Response(JSON.stringify({ commit: "a".repeat(40), deploymentMode: "authenticated" }))); + await expect(verifyConnectionTarget("https://staging.example.test", config)).resolves.toMatchObject({ commit: "a".repeat(40) }); + } finally { fetch.mockRestore(); } + }); + it("reads literal selected keys without sourcing shell commands or returning unrelated secrets", async () => { + expect(selectedSecret("UNRELATED=do-not-read\nexport OPENAI_API_KEY='chosen-value'\n", "OPENAI_API_KEY")).toBe("chosen-value"); + expect(selectedSecret("OPENAI_API_KEY=one\nOPENAI_API_KEY=two", "OPENAI_API_KEY")).toBe("two"); + expect(() => selectedSecret("OPENAI_API_KEY=$(touch /tmp/should-not-exist)", "OPENAI_API_KEY")).toThrow(); + const root = await mkdtemp(path.join(os.tmpdir(), "connection-secret-test-")); + try { + const file = path.join(root, "keys"); + await writeFile(file, "OPENAI_API_KEY=selected-only\nUNRELATED=private\n", { mode: 0o600 }); + const config = parseConnectionConfig({ secretFile: file }); + expect(await resolveConnectionSecret(config, "OPENAI_API_KEY", {})).toBe("selected-only"); + expect(await resolveConnectionSecret(config, undefined, {})).toBeUndefined(); + await expect(resolveConnectionSecret(config, "XAI_API_KEY", {})).rejects.toThrow("missing_XAI_API_KEY"); + await writeFile(path.join(root, "public"), "OPENAI_API_KEY=private", { mode: 0o644 }); + await expect(resolveConnectionSecret(parseConnectionConfig({ secretFile: path.join(root, "public") }), "OPENAI_API_KEY", {})).rejects.toThrow(/owner_only/); + } finally { await rm(root, { recursive: true, force: true }); } + }); + it("never resolves an API key for a subscription or defaults a gateway to the direct provider's credential", () => { + expect(resolveConnectionSettings(cell("subscription"), parseConnectionConfig({})).credentialEnv).toBeUndefined(); + expect(resolveConnectionSettings(cell("api-key"), parseConnectionConfig({})).credentialEnv).toBe("OPENAI_API_KEY"); + expect(() => resolveConnectionSettings(cell("responses"), parseConnectionConfig({ routes: { responses: { model: "qa-alias", baseURL: "https://gateway.example.test/v1" } } }))).toThrow(/credential_reference/); + const config = parseConnectionConfig({ routes: { responses: { model: "qa-alias", baseURL: "https://gateway.example.test/v1", credentialEnv: "GATEWAY_QA_KEY" } } }); + expect(resolveConnectionSettings(cell("responses"), config).credentialEnv).toBe("GATEWAY_QA_KEY"); + }); +}); + +function passingEvidence(): ConnectionEvidence { + return { version: 1, outcome: "passed", phase: "finished", assisted: true, authFreshness: "signed-in", target: { mode: "attach", origin: "https://staging.example.test", commit: "a".repeat(40), deploymentMode: "authenticated" }, entry: "agent", method: "subscription", checkpoints: Object.fromEntries(connectionCheckpoints.map(key => [key, true])), waits: [] }; +} + +describe("independent qualification", () => { + it("retains closed failure signals across split log chunks without messages or reasoning", () => { + const sentinel = "SENTINEL_CREDENTIAL_DO_NOT_RETAIN"; + const rows = [ + { type: "acpx.text_delta", channel: "thought", text: 'HTTP 429 '+sentinel }, + { type: "acpx.error", childStderrTail: 'ApiError: {"error":{"code":503,"message":"model overloaded '+sentinel+'"}}' }, + { type: "acpx.tool_call", status: "failed", text: "Error checking existing file: Resource not found: "+sentinel }, + ].map(row => JSON.stringify(row)).join("\n")+"\n"; + const split = Math.floor(rows.length / 2); + const log = { content: [rows.slice(0, split), rows.slice(split)].map(chunk => JSON.stringify({ chunk })).join("\n") }; + const diagnostic = connectionRunDiagnostic({ id: "run-1", status: "failed" }, log); + expect(diagnostic).toEqual({ runId: "run-1", status: "failed", signals: ["provider_overloaded", "missing_file_conversion"], logAvailable: true }); + expect(JSON.stringify(diagnostic)).not.toContain(sentinel); + expect(connectionRunDiagnostic({ id: "run-2", status: "running" })).toEqual({ runId: "run-2", status: "unknown", signals: [], logAvailable: false }); + }); + it("distinguishes terminal quota and unsupported configuration from generic failures", () => { + expect(connectionRunDiagnostic({ id: "quota", status: "failed", error: 'HTTP 429 RESOURCE_EXHAUSTED' }).signals).toEqual(["provider_quota"]); + expect(connectionRunDiagnostic({ id: "config", status: "failed", error: 'session/set_config_option: Method not found (-32601)' }).signals).toEqual(["unsupported_session_configuration"]); + expect(connectionRunDiagnostic({ id: "generic", status: "failed", error: 'unknown provider error' }).signals).toEqual([]); + expect(connectionRunDiagnostic({ id: "unattributed-503", status: "failed", error: 'HTTP 503' }).signals).toEqual([]); + }); + it("waits for disposition repair and requires the successful artifact-producing run", () => { + const expected = { agentId: "a", connectionId: "c", method: "api_key", runtimeMode: "legacy", environmentId: "env" }; + const first = { id: "first", agentId: "a", status: "succeeded", runtimeMode: "legacy", contextSnapshot: { paperclipEnvironment: { id: "env" }, aiConnection: { connectionId: "c", method: "api_key" } } }; + const repair = { ...first, id: "repair" }; + const file = { originalFilename: "proof.json", createdByAgentId: "a", originatingRunId: "repair" }; + expect(completedConnectionArtifactRun([first, { ...repair, status: "running" }], [file], "proof.json", expected)).toBeUndefined(); + expect(completedConnectionArtifactRun([first, repair], [file], "proof.json", expected)).toBe(repair); + expect(completedConnectionArtifactRun([first, { ...repair, status: "cancelled" }], [file], "proof.json", expected)).toBeUndefined(); + expect(completedConnectionArtifactRun([first, repair], [{ ...file, createdByAgentId: "someone-else" }], "proof.json", expected)).toBeUndefined(); + expect(completedConnectionArtifactRun([first, repair], [file], "followup.json", expected)).toBeUndefined(); + expect(completedConnectionArtifactRun([first, repair], [file], "proof.json", { ...expected, connectionId: "another" })).toBeUndefined(); + }); + it("retains probe codes without provider text, URLs, or credential-bearing fields", () => { + const sentinel = "SENTINEL_CREDENTIAL_DO_NOT_RETAIN"; + const checks = connectionProbeChecks({ checks: [ + { code: "hermes_cli_not_found", level: "error", message: sentinel, detail: `https://provider.test/?code=${sentinel}` }, + { code: sentinel, level: "error" }, { code: "safe_code", level: sentinel }, null, + ] }); + expect(checks).toEqual([{ code: "hermes_cli_not_found", level: "error" }]); + expect(JSON.stringify(checks)).not.toContain(sentinel); + }); + it("rejects plausible prose, wrong bytes and wrong run attribution", () => { + const proof = createConnectionProof(); + expect(verifyConnectionArtifact(Buffer.from(JSON.stringify(proof.expected)), proof.expected)).toBe(true); + expect(verifyConnectionArtifact(Buffer.from("Provider routing works"), proof.expected)).toBe(false); + expect(verifyConnectionArtifact(Buffer.from(JSON.stringify({ ...proof.expected, total: proof.expected.total + 1 })), proof.expected)).toBe(false); + const expected = { agentId: "a", connectionId: "c", method: "subscription", runtimeMode: "native", environmentId: "env", model: "m" }; + const run = { agentId: "a", status: "succeeded", runtimeMode: "native", contextSnapshot: { paperclipEnvironment: { id: "env" }, aiConnection: { connectionId: "c", method: "subscription" } } }; + expect(verifyConnectionRun(run, expected)).toBe(true); + expect(verifyConnectionRun(run, { ...expected, connectionId: "ambient" })).toBe(false); + expect(verifyConnectionRun(run, { ...expected, method: "api_key" })).toBe(false); + expect(verifyConnectionRun(run, { ...expected, environmentId: "another-env" })).toBe(false); + expect(verifyConnectionRun({ ...run, contextSnapshot: { aiConnection: run.contextSnapshot.aiConnection } }, expected)).toBe(false); + }); + it("requires every checkpoint and never passes absent evidence or a human deadline", () => { + expect(connectionEvidencePasses(passingEvidence())).toBe(true); + for (const key of connectionCheckpoints) { + const evidence = passingEvidence(); delete evidence.checkpoints[key]; + expect(connectionEvidencePasses(evidence)).toBe(false); + } + expect(connectionEvidencePasses({ ...passingEvidence(), outcome: "awaiting_user" })).toBe(false); + expect(connectionEvidencePasses({ ...passingEvidence(), target: { ...passingEvidence().target, commit: null } })).toBe(false); + }); + it("withholds raw Playwright call logs including filled secrets, codes, and callback URLs", () => { + const sentinel = "SENTINEL_CREDENTIAL_DO_NOT_RETAIN"; + for (const message of [`locator.fill: value ${sentinel}`, `https://provider.test/callback?code=${sentinel}`, `password=${sentinel}`, "lowercasesecret"]) { + const safe = safeConnectionFailure(new Error(message), "connection_creation"); + expect(safe).toBe("connection_creation_failed"); expect(safe).not.toContain(sentinel); + } + }); + it("keeps blocked authentication incomplete in the existing campaign/report", () => { + const execution = cell("subscription"); + const result: RunnerE2EResult = { schema: "paperclip.runner-e2e.result/v2", executionId: execution.id, suiteId: execution.suite.id, suiteDefinitionHash: execution.suiteDefinitionHash, + attempt: 1, status: "failed", failureClass: "permanent_infrastructure", error: "provider_login_deadline_reached", profileId: execution.profile.id, environmentId: "local", caseId: execution.task.id, provider: "codex", model: "model", runtimeMode: "native", + startedAt: "2026-10-03T00:00:00Z", finishedAt: "2026-10-03T00:01:00Z", durationMs: 60_000, cleanup: "passed", providerConnection: { ...passingEvidence(), outcome: "awaiting_user", checkpoints: { target: true } } }; + const campaign = buildRunnerCampaign({ campaignId: "test", generatedAt: result.finishedAt, expected: [execution.id], results: [result] }); + expect(() => validateRetainedRunnerResult({ ...result, status: "passed" })).toThrow(/providerConnection/); + expect(() => validateRetainedRunnerResult({ ...result, status: "passed", providerConnection: undefined })).toThrow(/providerConnection/); + expect(campaign.passed).toBe(0); expect(campaign.incomplete).toBe(1); expect(campaign.complete).toBe(false); + expect(renderCaseOutcome(result, false, [])).toContain("Blocked: awaiting user"); + const cleanupFailed = buildRunnerCampaign({ campaignId: "failed-cleanup", generatedAt: result.finishedAt, expected: [execution.id], results: [{ ...result, cleanup: "failed", failureClass: "cleanup_failure" }] }); + expect(cleanupFailed.failed).toBe(1); expect(cleanupFailed.incomplete).toBe(0); + }); +}); diff --git a/tests/runner-e2e/connection-target.ts b/tests/runner-e2e/connection-target.ts new file mode 100644 index 00000000000..c0c5c2acbca --- /dev/null +++ b/tests/runner-e2e/connection-target.ts @@ -0,0 +1,102 @@ +import { spawn, type ChildProcess } from "node:child_process"; +import { randomBytes } from "node:crypto"; +import { mkdir, mkdtemp, rm } from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import type { APIRequestContext } from "@playwright/test"; +import { reserveRunnerE2EServerPort } from "./ports.js"; +import { resolvePaperclipRunnerBinaryForHarness } from "./harness-env.js"; +import { runnerE2ETypeScriptProcessArgs } from "./web-server-command.js"; +import { ConnectionBlock, ConnectionFailure, targetOrigin, type ConnectionConfig } from "./connection-config.js"; +import type { MatrixExecution } from "./types.js"; + +export const connectionDelay = (ms: number) => new Promise(resolve => setTimeout(resolve, ms)); + +/** No raw response bodies, callback URLs, headers, or Playwright diagnostics escape. */ +export class ConnectionApi { + constructor(readonly request: APIRequestContext, readonly origin: string) {} + async response(route: string, method = "GET", data?: unknown) { + if (!route.startsWith("/api/") || new URL(route, this.origin).origin !== this.origin) throw new ConnectionFailure("invalid_api_route"); + try { + const response = await this.request.fetch(`${this.origin}${route}`, { method, ...(data === undefined ? {} : { data }), maxRedirects: 0, timeout: 30_000 }); + if (!response.ok()) throw new ConnectionFailure(`api_status_${response.status()}`); + return response; + } catch (error) { + if (error instanceof Error && /^api_status_\d+$/.test(error.message)) throw error; + throw new ConnectionFailure("api_request_failed"); + } + } + async get(route: string): Promise { return (await this.response(route)).json(); } + async post(route: string, data?: unknown): Promise { return (await this.response(route, "POST", data)).json(); } + async patch(route: string, data: unknown): Promise { return (await this.response(route, "PATCH", data)).json(); } + async delete(route: string) { await this.response(route, "DELETE"); } + async bytes(route: string) { return (await this.response(route)).body(); } +} + +async function stopOwnedServer(child: ChildProcess) { + if (!child.pid) return; + const exited = () => child.exitCode !== null || child.signalCode !== null; + const kill = (signal: NodeJS.Signals) => { + try { process.kill(process.platform === "win32" ? child.pid! : -child.pid!, signal); } + catch (error) { if ((error as NodeJS.ErrnoException).code !== "ESRCH") throw error; } + }; + kill("SIGTERM"); + const deadline = Date.now() + 45_000; + while (!exited() && Date.now() < deadline) await connectionDelay(100); + // The group is exclusively the server this function spawned, never an attached target. + kill("SIGKILL"); +} + +export async function startConnectionTarget(config: ConnectionConfig, executions: readonly MatrixExecution[], repositoryRoot: string, signal?: AbortSignal) { + const assertActive = () => { if (signal?.aborted) throw new ConnectionBlock("blocked_target", "test_interrupted"); }; + assertActive(); + if (config.target.mode === "attach") return { origin: targetOrigin(config.target.baseURL), stop: async () => {} }; + const root = await mkdtemp(path.join(os.tmpdir(), "paperclip-connection-e2e-")); + let child: ChildProcess | undefined; + const stop = async () => { if (child) await stopOwnedServer(child); await rm(root, { recursive: true, force: true }); }; + try { + assertActive(); + const home = path.join(root, "paperclip-home"); + const instance = `connection-${randomBytes(6).toString("hex")}`; + const port = await reserveRunnerE2EServerPort(); + await mkdir(home, { mode: 0o700 }); + // Only bootstrap essentials are inherited. Selected provider keys enter via UI. + const env: NodeJS.ProcessEnv = Object.fromEntries(["PATH", "HOME", "USER", "LOGNAME", "SHELL", "TMPDIR", "LANG", "LC_ALL", "NODE_EXTRA_CA_CERTS"].flatMap(key => process.env[key] ? [[key, process.env[key]!]] : [])); + Object.assign(env, { + PAPERCLIP_HOME: home, PAPERCLIP_INSTANCE_ID: instance, PAPERCLIP_CONFIG: path.join(home, "instances", instance, "config.json"), + PAPERCLIP_RUNNER_E2E_TEMP_ROOT: root, PAPERCLIP_RUNNER_E2E_PORT: String(port), PAPERCLIP_RUNNER_E2E_SERVER_LOG: path.join(root, "server.log"), + PAPERCLIP_RUNNER_BINARY: resolvePaperclipRunnerBinaryForHarness(executions, repositoryRoot), + PAPERCLIP_VITE_CACHE_DIR: path.join(root, "vite-cache"), PAPERCLIP_ANNOUNCEMENTS_ENABLED: "false", + PAPERCLIP_AGENT_JWT_SECRET: randomBytes(48).toString("hex"), PAPERCLIP_DECISION_SIGNING_SECRET: randomBytes(48).toString("hex"), + PAPERCLIP_TOOL_ACTION_SIGNING_SECRET: randomBytes(48).toString("hex"), BETTER_AUTH_SECRET: randomBytes(48).toString("hex"), + }); + assertActive(); + child = spawn(process.execPath, runnerE2ETypeScriptProcessArgs(repositoryRoot, path.join(repositoryRoot, "tests/runner-e2e/server.ts")), { cwd: repositoryRoot, env, detached: process.platform !== "win32", stdio: "ignore" }); + let failed = false; + child.once("error", () => { failed = true; }); + const origin = `http://127.0.0.1:${port}`; + const deadline = Date.now() + 180_000; + while (Date.now() < deadline) { + assertActive(); + if (failed || child.exitCode !== null) throw new ConnectionBlock("blocked_target", "managed_server_start_failed"); + try { + const response = await fetch(`${origin}/api/health`, { redirect: "error", signal: signal ? AbortSignal.any([signal, AbortSignal.timeout(1500)]) : AbortSignal.timeout(1500) }); + if (response.ok) { assertActive(); return { origin, stop }; } + } catch { assertActive(); /* Bounded startup. */ } + await connectionDelay(500); + } + throw new ConnectionBlock("blocked_target", "managed_server_start_timeout"); + } catch (error) { await stop(); throw error; } +} + +export async function verifyConnectionTarget(origin: string, config: ConnectionConfig) { + let health: any; + try { + const response = await fetch(`${targetOrigin(origin)}/api/health`, { redirect: "error", signal: AbortSignal.timeout(15_000) }); + if (!response.ok) throw new ConnectionFailure(); + health = await response.json(); + } catch { throw new ConnectionBlock("blocked_target", "target_health_unavailable_or_redirected"); } + if (typeof health.commit !== "string" || !/^[a-f0-9]{40}$/.test(health.commit)) throw new ConnectionBlock("blocked_target", "target_revision_unavailable"); + if (config.target.mode === "attach" && (health.commit !== config.target.expectedCommit || health.deploymentMode !== config.target.deploymentMode)) throw new ConnectionBlock("blocked_target", "target_identity_mismatch"); + return { commit: health.commit as string, deploymentMode: String(health.deploymentMode) }; +} diff --git a/tests/runner-e2e/gemini-acp-filesystem.smoke.ts b/tests/runner-e2e/gemini-acp-filesystem.smoke.ts new file mode 100644 index 00000000000..c67cfc55717 --- /dev/null +++ b/tests/runner-e2e/gemini-acp-filesystem.smoke.ts @@ -0,0 +1,82 @@ +/** Opt-in installed-CLI compatibility check. No real provider credentials or inference. */ +import { createServer } from "node:http"; +import { mkdir, mkdtemp, readFile, realpath, rm, writeFile } from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { createAcpRuntime, createAgentRegistry, createRuntimeStore } from "../../packages/adapter-utils/node_modules/acpx/dist/runtime.js"; + +const argument = process.argv.indexOf("--agent-command"); +const command = argument < 0 ? "gemini --acp" : process.argv[argument + 1]; +if (!command) throw new Error("agent_command_required"); +const root = await mkdtemp(path.join(os.tmpdir(), "paperclip-gemini-filesystem-smoke-")); +const content = '{"nonce":"synthetic-file-smoke","total":28}\n'; +const results: Array<{ case: string; passed: boolean; completedTools: number; failedTools: number; missingFileSignal: boolean }> = []; +let active: { calls: number; file: string; tool: string }; +const server = createServer(async (request, response) => { + // Never retain request bodies, headers, provider prompts or model output. + for await (const _chunk of request) { /* Drain the synthetic request. */ } + const parts = ++active.calls === 1 + ? [{ functionCall: { name: active.tool, args: { file_path: active.file, ...(active.tool === "write_file" ? { content } : {}) } } }] + : [{ text: "synthetic-complete" }]; + const payload = { + candidates: [{ content: { role: "model", parts }, finishReason: "STOP" }], + usageMetadata: { promptTokenCount: 10, candidatesTokenCount: 2, totalTokenCount: 12 }, + modelVersion: "gemini-3.1-flash-lite", + }; + if (request.url?.includes(":streamGenerateContent")) { + response.writeHead(200, { "content-type": "text/event-stream" }); + response.end(`data: ${JSON.stringify(payload)}\n\n`); + } else { + response.writeHead(200, { "content-type": "application/json" }); + response.end(JSON.stringify(request.url?.includes(":countTokens") ? { totalTokens: 10 } : payload)); + } +}); +await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve)); +const origin = `http://127.0.0.1:${(server.address() as { port: number }).port}`; +try { + for (const scenario of ["create-missing", "read-existing", "deny-outside"] as const) { + const home = path.join(root, scenario, "home"); + const workspace = path.join(root, scenario, "workspace"); + await mkdir(path.join(home, ".gemini"), { recursive: true, mode: 0o700 }); + await mkdir(workspace, { recursive: true }); + const cwd = await realpath(workspace); + const file = scenario === "deny-outside" ? path.join(await realpath(root), "outside.json") : path.join(cwd, "proof.json"); + if (scenario !== "create-missing") await writeFile(file, content); + await writeFile(path.join(home, ".gemini", "settings.json"), JSON.stringify({ + security: { auth: { selectedType: "gemini-api-key" } }, telemetry: { enabled: false }, + }), { mode: 0o600 }); + active = { calls: 0, file, tool: scenario === "read-existing" ? "read_file" : "write_file" }; + const runtime = createAcpRuntime({ + cwd, sessionStore: createRuntimeStore({ stateDir: path.join(root, scenario, "state") }), + agentRegistry: createAgentRegistry({ overrides: { gemini: command } }), + permissionMode: "approve-all", nonInteractivePermissions: "deny", inheritProcessEnv: false, + onAgentStderr: () => { /* Child diagnostics may contain untrusted material. */ }, + }); + const handle = await runtime.ensureSession({ + sessionKey: `gemini-filesystem-${scenario}`, agent: "gemini", mode: "oneshot", cwd, + sessionOptions: { env: { + HOME: home, PATH: process.env.PATH ?? "", GEMINI_MODEL: "gemini-3.1-flash-lite", + GEMINI_API_KEY: "synthetic-invalid-key", GOOGLE_GEMINI_BASE_URL: origin, + } }, + }); + try { + let completedTools = 0; let failedTools = 0; let missingFileSignal = false; + for await (const event of runtime.runTurn({ handle, text: "Run the synthetic file operation, then stop.", mode: "prompt", requestId: scenario, timeoutMs: 20_000 })) { + if (event.type !== "tool_call") continue; + if (event.status === "completed") completedTools++; + if (event.status === "failed") { + failedTools++; + missingFileSignal ||= /Error checking existing file: (?:Resource not found|Internal error)/.test(event.text ?? ""); + } + } + const bytes = await readFile(file, "utf8").catch(() => null); + const passed = bytes === content && (scenario === "deny-outside" ? failedTools > 0 && completedTools === 0 : failedTools === 0 && completedTools > 0); + results.push({ case: scenario, passed, completedTools, failedTools, missingFileSignal }); + } finally { await runtime.close({ handle, reason: "synthetic filesystem smoke complete" }); } + } + console.log(JSON.stringify({ realProviderCalls: 0, results }, null, 2)); + if (results.some((result) => !result.passed)) process.exitCode = 1; +} finally { + await new Promise((resolve) => server.close(() => resolve())); + await rm(root, { recursive: true, force: true }); +} diff --git a/tests/runner-e2e/history.test.ts b/tests/runner-e2e/history.test.ts index 9ead6475436..8f5b082f3a3 100644 --- a/tests/runner-e2e/history.test.ts +++ b/tests/runner-e2e/history.test.ts @@ -31,6 +31,7 @@ import { import { renderRunnerHistoryIndex } from "./history-index.js"; import { renderPublicCampaignSummary } from "./public-summary-image.js"; import { isPublicRunnerScreenshotRoute } from "./screenshot-policy.js"; +import { connectionCheckpoints, type ConnectionEvidence } from "./connection-evidence.js"; import type { MatrixExecution, RunnerE2EResult } from "./types.js"; const temporaryDirectories: string[] = []; @@ -63,6 +64,7 @@ function result(execution: MatrixExecution, status: "passed" | "failed") { finishedAt: "2026-08-28T00:00:01.000Z", durationMs: 1_000, cleanup: "passed", + ...(execution.task.flow === "provider_connection" ? { providerConnection: { version: 1, outcome: status, phase: "finished", assisted: false, authFreshness: "signed-in", target: { mode: "attach", origin: "http://127.0.0.1:3100", commit: "a".repeat(40), deploymentMode: "local_trusted" }, entry: "agent", method: "api-key", checkpoints: Object.fromEntries(connectionCheckpoints.map(key => [key, true])), waits: [] } satisfies ConnectionEvidence } : {}), } satisfies RunnerE2EResult; } diff --git a/tests/runner-e2e/launch.ts b/tests/runner-e2e/launch.ts index d4e85771ac5..cbd7190a285 100644 --- a/tests/runner-e2e/launch.ts +++ b/tests/runner-e2e/launch.ts @@ -1084,6 +1084,13 @@ async function main() { return; } + if (executions.some(execution => execution.task.flow === "provider_connection")) { + const { runConnectionCampaign } = await import("./connection-launch.js"); + await runConnectionCampaign({ executions, catalog: runnerMatrix, configFile: options.connectionConfig, repositoryRoot }); + return; + } + if (options.connectionConfig) throw new Error("--connection-config is only supported by the provider-connections suite"); + // Keep admission before local-env loading and credential checks. Pending // profiles remain discoverable, but cannot reach a provider. assertRunnerE2EPrerequisites(executions); diff --git a/tests/runner-e2e/report-assets.ts b/tests/runner-e2e/report-assets.ts new file mode 100644 index 00000000000..41d2c122455 --- /dev/null +++ b/tests/runner-e2e/report-assets.ts @@ -0,0 +1,18 @@ +import path from "node:path"; +import { copyFile, mkdir } from "node:fs/promises"; +const repositoryRoot = path.resolve(import.meta.dirname, "../.."); + +export async function stageDashboardBrandAssets(output: string) { + const assets = path.join(output, "assets"); + await mkdir(assets, { recursive: true }); + await Promise.all([ + copyFile( + path.join(repositoryRoot, "ui/public/favicon-32x32.png"), + path.join(assets, "favicon-32x32.png"), + ), + copyFile( + path.join(repositoryRoot, "ui/public/fonts/InterVariable.woff2"), + path.join(assets, "InterVariable.woff2"), + ), + ]); +} diff --git a/tests/runner-e2e/report.ts b/tests/runner-e2e/report.ts index 7024f8f2773..043426d7ccf 100644 --- a/tests/runner-e2e/report.ts +++ b/tests/runner-e2e/report.ts @@ -1,3 +1,4 @@ +import { stageDashboardBrandAssets } from "./report-assets.js"; import { discoverReportCatalog } from "./report-catalog.js"; import path from "node:path"; import { @@ -147,20 +148,6 @@ async function stageDashboardEvidence( return staged; } -async function stageDashboardBrandAssets(output: string) { - const assets = path.join(output, "assets"); - await mkdir(assets, { recursive: true }); - await Promise.all([ - copyFile( - path.join(repositoryRoot, "ui/public/favicon-32x32.png"), - path.join(assets, "favicon-32x32.png"), - ), - copyFile( - path.join(repositoryRoot, "ui/public/fonts/InterVariable.woff2"), - path.join(assets, "InterVariable.woff2"), - ), - ]); -} async function main() { const root = path.resolve( diff --git a/tests/runner-e2e/result-validation.ts b/tests/runner-e2e/result-validation.ts index 158712fd6c4..e647e37a3b1 100644 --- a/tests/runner-e2e/result-validation.ts +++ b/tests/runner-e2e/result-validation.ts @@ -1,3 +1,4 @@ +import { connectionCheckpoints, connectionEvidencePasses, connectionRunSignals } from "./connection-evidence.js"; import type { RunnerE2EResult } from "./types.js"; type Rule = (value: unknown, at: string) => void; @@ -243,6 +244,20 @@ const fields = { tasks: array(object), agents: array(object), comments: array(object), interactions: array(object), documents: array(object), attachments: optional(array(object)), runs: array(object) })), checks: array(shape({ id: string, passed: boolean, notReached: optional(string), evidence: array(string), detail: string })), })), + providerConnection: optional(shape({ + version: integer, + outcome: oneOf("running", "passed", "failed", "awaiting_user", "missing_credential", "blocked_target"), + phase: string, assisted: boolean, authFreshness: oneOf("signed-in", "signed-out"), + target: shape({ mode: oneOf("managed-local", "attach"), origin: url, commit: nullable(string), deploymentMode: nullable(string) }), + entry: oneOf("apps", "agent"), method: string, checkpoints: shape(Object.fromEntries(connectionCheckpoints.map(key => [key, optional(boolean)]))), + waits: array(shape({ kind: oneOf("board_login", "provider_login"), startedAt: date, finishedAt: optional(date) })), + artifactChecks: optional(array(shape({ filename: string, runId: string, sha256: string, fields: object, exactFields: boolean }))), + inputTransport: optional(oneOf("prompt_base64")), + setupChecks: optional(array(shape({ code: string, level: oneOf("info", "warn", "error") }))), + runDiagnostics: optional(array(shape({ runId: string, status: oneOf("succeeded", "failed", "cancelled", "interrupted", "timed_out", "unknown"), signals: array(oneOf(...connectionRunSignals)), logAvailable: boolean }))), + creationFailure: optional(shape({ status: integer, code: oneOf("ai_connection_api_key_rejected", "ai_connection_verification_failed", "connection_request_rejected") })), + connectionId: optional(string), agentId: optional(string), companyId: optional(string), artifactSha256: optional(string), model: optional(string), cleanupRetained: optional(boolean), companyArchived: optional(boolean), + })), completionQuality: optional(array(shape({ name: string, purpose: optional(oneOf("product", "calibration")), expectedPass: boolean, passed: boolean, status: oneOf("completed", "failed", "pending"), config: object, configHash: string, evidenceHash: string, criteria: array(shape({ id: string, passed: boolean, rationale: string, evidenceIds: array(string) })), @@ -263,4 +278,6 @@ export function validateRetainedRunnerResult( value: unknown, ): asserts value is RunnerE2EResult { result(value, "result"); + const checked = value as RunnerE2EResult; + if (checked.suiteId === "provider-connections" && checked.status === "passed" && (!checked.providerConnection || !connectionEvidencePasses(checked.providerConnection))) invalid("result.providerConnection"); } diff --git a/tests/runner-e2e/selectors.ts b/tests/runner-e2e/selectors.ts index e890da21e4c..1c48ae8d916 100644 --- a/tests/runner-e2e/selectors.ts +++ b/tests/runner-e2e/selectors.ts @@ -16,6 +16,7 @@ export interface RunnerSelectorOptions { debug: boolean; maxParallel: number; maxAutomaticRetries: number; + connectionConfig?: string; } export class RunnerSelectorError extends Error {} @@ -55,6 +56,10 @@ export function parseRunnerSelectors( else if (flag === "--headed") options.headed = true; else if (flag === "--ui") options.ui = true; else if (flag === "--debug") options.debug = true; + else if (flag === "--connection-config") { + options.connectionConfig = valueFor(args, index, flag); + index += 1; + } else if (flag === "--max-parallel") { const value = valueFor(args, index, flag); index += 1; @@ -200,6 +205,7 @@ export function selectRunnerExecutions( export function buildMatrixJobs( executions: readonly MatrixExecution[], ): MatrixJob[] { + if (executions.some(e => e.task.flow === "provider_connection")) throw new RunnerSelectorError("Provider connection journeys use an explicit local/staging connection configuration; they are not admitted to paid CI matrix jobs yet."); return executions .map((execution) => ({ executionId: execution.id, diff --git a/tests/runner-e2e/types.ts b/tests/runner-e2e/types.ts index ac2b0cef6c4..e1af82a6510 100644 --- a/tests/runner-e2e/types.ts +++ b/tests/runner-e2e/types.ts @@ -5,6 +5,7 @@ export const CREDENTIAL_NAMES = [ "KIMI_MODEL_API_KEY", "XAI_API_KEY", "GROK_AUTH_JSON", + "GEMINI_API_KEY", "DAYTONA_API_KEY", "CURSOR_AUTH_TOKEN", "COPILOT_GITHUB_TOKEN", @@ -15,6 +16,7 @@ export type RunnerGeneration = "legacy" | "native"; export type RunnerEnvironmentId = "local" | "daytona"; export type RunnerTaskWorkMode = "standard" | "planning" | "ask"; export type RunnerTaskFlow = + | "provider_connection" | "plan_task_guidance" | "blocker_guidance" | "public_mcp" @@ -326,6 +328,7 @@ export interface RunnerE2EResult { sha256?: string; }>; firstTask?: import("./first-task-scoring.js").FirstTaskEvidence; + providerConnection?: import("./connection-evidence.js").ConnectionEvidence; completionQuality?: import("./completion-quality.js").CompletionQualityRecord[]; firstTaskQuality?: import("./first-task-quality.js").FirstTaskQuality; /** External assistant API calls, separate from the team's heartbeat executions. */ diff --git a/tests/runner-e2e/user-actions.ts b/tests/runner-e2e/user-actions.ts index 70c88b41271..28cea1e8f13 100644 --- a/tests/runner-e2e/user-actions.ts +++ b/tests/runner-e2e/user-actions.ts @@ -8,9 +8,10 @@ export async function createTaskThroughUi(input: { prompt: string; workMode: "standard" | "planning" | "ask"; projectName?: string; + attachments?: readonly string[]; }) { const issuesUrl = `/${encodeURIComponent(input.issuePrefix)}/issues`; - const newTask = input.page.getByRole("button", { name: "New Task" }).first(); + const newTask = input.page.getByRole("button", { name: /^New task$/i }).first(); let bootstrapError: unknown; for (let bootstrapAttempt = 1; bootstrapAttempt <= 3; bootstrapAttempt += 1) { try { @@ -33,40 +34,74 @@ export async function createTaskThroughUi(input: { ); } await newTask.click(); - await input.page.getByPlaceholder("Task title").fill(input.title); + const dialog = input.page.getByRole("dialog"); + const titleInput = dialog.getByPlaceholder("Task title"); + const hasTitleInput = await titleInput.count() > 0; + if (hasTitleInput) await titleInput.fill(input.title); await input.page .getByRole("dialog") .getByRole("textbox", { name: "editable markdown", exact: true }) .fill(input.prompt); if (input.workMode !== "standard") { - await input.page - .getByRole("dialog") - .locator(`[data-issue-work-mode-chip="standard"]`) - .click(); - await input.page - .locator(`[data-issue-work-mode="${input.workMode}"]`) - .click(); + const legacyMode = dialog.locator('[data-issue-work-mode-chip="standard"]'); + if (await legacyMode.count()) { + await legacyMode.click(); + await input.page.locator(`[data-issue-work-mode="${input.workMode}"]`).click(); + } else { + await dialog.getByRole("button", { name: "Add to composer", exact: true }).click(); + await input.page.getByRole("menuitem", { name: input.workMode === "planning" ? /^Plan mode/ : /^Ask mode/ }).click(); + } } await input.page - .getByRole("button", { name: "Assignee", exact: true }) + .getByRole("button", { name: /^(?:Select )?Assignee$/i }) .click(); await input.page - .getByPlaceholder("Search assignees...") + .getByPlaceholder(/^Search assignees(?:…|\.\.\.)?$/i) .fill(input.agentName); await input.page.getByText(input.agentName, { exact: true }).last().click(); if (input.projectName) { - const dialog = input.page.getByRole("dialog"); // Selecting the assignee advances focus to this selector and opens it. // Focus is idempotent here; clicking would toggle an already-open popover // closed before the search field can be filled. - await dialog.getByRole("button", { name: "Project", exact: true }).focus(); - await dialog.getByPlaceholder("Search projects...").fill(input.projectName); - await dialog.getByText(input.projectName, { exact: true }).last().click(); + const projectTrigger = dialog.getByRole("button", { name: "Project", exact: true }) + .or(dialog.getByRole("button", { name: input.projectName, exact: true })); + await projectTrigger.focus(); + const projectSearch = input.page.getByPlaceholder(/^Search projects(?:…|\.\.\.)?$/i); + await expect(projectSearch).toBeVisible({ timeout: 2_000 }).catch(async () => { + if (await projectTrigger.getAttribute("aria-expanded") !== "true") await projectTrigger.click(); + await expect(projectSearch).toBeVisible(); + }); + await projectSearch.fill(input.projectName); + await input.page.getByText(input.projectName, { exact: true }).last().click(); } const submittedAtMs = Date.now(); + if (input.attachments?.length) { + const chooser = input.page.waitForEvent("filechooser"); + const upload = dialog.getByRole("button", { name: "Upload", exact: true }); + if (await upload.count()) await upload.click(); + else { + await dialog.getByRole("button", { name: "Add to composer", exact: true }).click(); + await input.page.getByRole("menuitem", { name: /^Files and images/ }).click(); + } + await (await chooser).setFiles([...input.attachments]); + // Upload finishes as part of task creation; the dialog retains the selected files. + } + const created = input.page.waitForResponse(response => response.request().method() === "POST" && /^\/api\/companies\/[^/]+\/issues$/.test(new URL(response.url()).pathname)); await input.page - .getByRole("button", { name: "Create Task", exact: true }) + .getByRole("button", { name: /^Create task$/i }) .click(); + const response = await created; + if (!response.ok()) throw new Error("Task creation request failed"); + if (!hasTitleInput && input.title) { + // Prompt-only creation generates a title. Set only fixture naming metadata + // through the public API; execution and results remain entirely real. + const issue = await response.json(); + const renamed = await input.page.request.patch(new URL(`/api/issues/${issue.id}`, response.url()).href, { data: { title: input.title } }); + if (!renamed.ok()) throw new Error("Fixture task naming failed"); + } + // The create response precedes staged-file uploads. The production dialog + // closes after those uploads finish; navigating earlier can cancel them. + await expect(dialog).not.toBeVisible({ timeout: 30_000 }); return submittedAtMs; } diff --git a/ui/src/adapters/codex-local/config-fields.test.tsx b/ui/src/adapters/codex-local/config-fields.test.tsx index 1272376e2e2..b43781732f7 100644 --- a/ui/src/adapters/codex-local/config-fields.test.tsx +++ b/ui/src/adapters/codex-local/config-fields.test.tsx @@ -1,12 +1,32 @@ -import { renderToStaticMarkup } from "react-dom/server"; -import { describe, expect, it } from "vitest"; +// @vitest-environment jsdom +import { createRoot } from "react-dom/client"; +import { act } from "react"; +import type { ReactNode } from "react"; + +import { beforeAll, describe, expect, it } from "vitest"; import { TooltipProvider } from "@/components/ui/tooltip"; import { CodexLocalConfigFields } from "./config-fields"; -function renderRunner(config: Record): string { - return renderToStaticMarkup( +beforeAll(() => { Object.assign(globalThis, { IS_REACT_ACT_ENVIRONMENT: true }); }); + +async function renderMarkup(node: ReactNode, expand?: string): Promise { + const container = document.createElement("div"); + document.body.appendChild(container); + const root = createRoot(container); + await act(async () => root.render(node)); + if (expand) await act(async () => { + container.querySelector(`[aria-label="${expand}"]`)?.dispatchEvent(new KeyboardEvent("keydown", { key: "Enter", bubbles: true })); + }); + const html = document.body.innerHTML; + await act(async () => root.unmount()); + container.remove(); + return html; +} + +async function renderRunner(config: Record, expand?: string): Promise { + return renderMarkup( ): string { hideInstructionsFile /> , + expand, ); } @@ -30,19 +51,19 @@ describe("Paperclip Runner Codex configuration", () => { ["approve-paperclip", "Automatic Paperclip actions"], ["approve-reads", "Allow Paperclip reads"], ["deny-all", "Deny all"], - ])("displays Grok's default or saved permission mode %s", (acpxPermissionMode, label) => { - const html = renderRunner({ provider: "acpx", acpxAgent: "grok", acpxPermissionMode }); - expect(html).toContain(''); + ])("displays Grok's default or saved permission mode %s", async (acpxPermissionMode, label) => { + const html = await renderRunner({ provider: "acpx", acpxAgent: "grok", acpxPermissionMode }); + expect(html).toContain('Grok Build'); expect(html).toContain('aria-label="Permission mode"'); expect(html).toContain(label); }); - it("exposes all qualified provider choices", () => { - const html = renderRunner({ provider: "codex" }); + it("exposes all qualified provider choices", async () => { + const html = await renderRunner({ provider: "codex" }, "Harness"); - expect(html).toContain(''); + expect(html).toContain('aria-label="Harness"'); expect(html).toContain("OpenCode 1.18.34"); - expect(html).toContain(''); + expect(html).toContain('ACP agents'); expect(html).not.toContain("Permission mode"); expect(html).not.toContain("Ask when requested"); expect(html).not.toContain("Ask for untrusted operations"); @@ -51,14 +72,14 @@ describe("Paperclip Runner Codex configuration", () => { expect(html).not.toContain("Bypass sandbox"); }); - it("renders OpenCode's bounded permission modes", () => { - const html = renderRunner({ + it("renders OpenCode's bounded permission modes", async () => { + const html = await renderRunner({ provider: "opencode", opencodePermissionMode: "allow", }); expect(html).toContain( - '', + 'OpenCode 1.18.34', ); expect(html).toContain("Full auto (allow)"); expect(html).toContain('aria-label="Permission mode"'); @@ -66,27 +87,28 @@ describe("Paperclip Runner Codex configuration", () => { expect(html).not.toContain("Ask for untrusted operations"); }); - it("offers qualified Claude and keeps candidate ACP agents visibly disabled", () => { - const html = renderRunner({ + it("offers qualified Claude and keeps candidate ACP agents visibly disabled", async () => { + const html = await renderRunner({ provider: "acpx", acpxAgent: "claude", acpxPermissionMode: "approve-reads", - }); + }, "ACP agent"); - expect(html).toContain(''); + expect(html).toContain('ACP agents'); expect(html).toContain("ACP agent"); - expect(html).toContain(''); - expect(html).toContain(''); - expect(html).toContain(''); - expect(html).toContain(''); + expect(html).toContain('aria-label="ACP agent"'); + expect(html.match(/role="option"[^>]*data-disabled=""/g)).toHaveLength(3); + expect(html).toContain('Cursor — qualification pending'); + expect(html).toContain('GitHub Copilot — qualification pending'); + expect(html).toContain('Pi — qualification pending'); expect(html).not.toContain("Codex via ACPX"); expect(html).not.toContain("ACPX Codex"); expect(html).not.toContain("Pi via ACPX"); expect(html).toContain("Allow Paperclip reads"); }); - it("falls back to the fail-closed Codex permission mode", () => { - const html = renderRunner({ codexPermissionMode: "unrestricted" }); + it("falls back to the fail-closed Codex permission mode", async () => { + const html = await renderRunner({ codexPermissionMode: "unrestricted" }); expect(html).toContain("Unsupported saved mode — select a qualified mode"); expect(html).toContain("cannot start or recover a Paperclip Runner run"); @@ -94,12 +116,12 @@ describe("Paperclip Runner Codex configuration", () => { expect(html).not.toContain("Full auto (never ask)"); }); - it("shows a bounded idle timeout only for warm sessions", () => { - const warmHtml = renderRunner({ + it("shows a bounded idle timeout only for warm sessions", async () => { + const warmHtml = await renderRunner({ lifecycleMode: "warm", idleTimeoutMs: 45_000, }); - const turnHtml = renderRunner({ + const turnHtml = await renderRunner({ lifecycleMode: "per_turn", idleTimeoutMs: 45_000, }); diff --git a/ui/src/adapters/codex-local/config-fields.tsx b/ui/src/adapters/codex-local/config-fields.tsx index ce36b43769d..c16848c39c9 100644 --- a/ui/src/adapters/codex-local/config-fields.tsx +++ b/ui/src/adapters/codex-local/config-fields.tsx @@ -1,4 +1,5 @@ import { Select, SelectContent, SelectItem, SelectTrigger, SelectValue } from "../../components/ui/select"; +import { AdapterMark } from "../../components/AdapterMark"; import { configFieldsForSection } from "../config-sections"; import type { AdapterConfigFieldsProps } from "../types"; import { @@ -36,6 +37,14 @@ const defaultOpenCodeRunnerModel = "openrouter/deepseek/deepseek-v4-flash-0731"; const defaultAcpxClaudeModel = "claude-sonnet-5"; const defaultClaudeManagedModel = "claude-sonnet-5"; const defaultAwsAgentCoreModel = "global.anthropic.claude-sonnet-4-6"; +const runnerHarnessOptions = [ + { value: "codex", label: "Codex", adapter: "codex_local" }, + { value: "opencode", label: "OpenCode 1.18.34", adapter: "opencode_local" }, + { value: "claude_managed", label: "Claude Managed", adapter: "claude_local" }, + { value: "aws_agentcore", label: "AWS AgentCore", adapter: "aws_agentcore" }, + { value: "acpx", label: "ACP agents", adapter: "acpx_local" }, + { value: "grok", label: "Grok Build", adapter: "grok_local" }, +]; export function CodexLocalConfigFields({ section, @@ -191,16 +200,15 @@ export function CodexLocalConfigFields({ )} {runnerManaged && ( - + + + {runnerHarnessOptions.map((option) => ( + + + {option.label} + + ))} + + )} {runnerManaged && runnerProvider === "acpx" && runnerSchemaValue("acpxAgent", "claude") !== "grok" && ( - + + + {PAPERCLIP_RUNNER_ACPX_PROFILES.map(profile => ( + + + {profile.label}{profile.qualified ? "" : " — qualification pending"} + + ))} + + )} {runnerManaged && runnerProvider === "claude_managed" && ( diff --git a/ui/src/adapters/config-sections.test.tsx b/ui/src/adapters/config-sections.test.tsx index 5903e75e40d..8073eb5935a 100644 --- a/ui/src/adapters/config-sections.test.tsx +++ b/ui/src/adapters/config-sections.test.tsx @@ -1,6 +1,10 @@ -import { renderToStaticMarkup } from "react-dom/server"; +// @vitest-environment jsdom +import { createRoot } from "react-dom/client"; +import { act } from "react"; +import type { ReactNode } from "react"; + import type { ComponentType } from "react"; -import { describe, expect, it } from "vitest"; +import { beforeAll, describe, expect, it } from "vitest"; import { TooltipProvider } from "../components/ui/tooltip"; import type { AdapterConfigFieldsProps, AdapterConfigSection } from "./types"; import { CodexLocalConfigFields } from "./codex-local/config-fields"; @@ -10,13 +14,29 @@ import { ProcessConfigFields } from "./process/config-fields"; import { OpenClawGatewayConfigFields } from "./openclaw-gateway/config-fields"; import { HermesGatewayConfigFields } from "./hermes-gateway/config-fields"; -function renderSection( +beforeAll(() => { Object.assign(globalThis, { IS_REACT_ACT_ENVIRONMENT: true }); }); + +async function renderMarkup(node: ReactNode, expand?: string): Promise { + const container = document.createElement("div"); + document.body.appendChild(container); + const root = createRoot(container); + await act(async () => root.render(node)); + if (expand) await act(async () => { + container.querySelector(`[aria-label="${expand}"]`)?.dispatchEvent(new KeyboardEvent("keydown", { key: "Enter", bubbles: true })); + }); + const html = container.innerHTML; + await act(async () => root.unmount()); + container.remove(); + return html; +} + +async function renderSection( Component: ComponentType, adapterType: string, section: AdapterConfigSection, config: Record = {}, ) { - return renderToStaticMarkup( + return renderMarkup( { - it("separates provider selection from lifecycle and hides fixed Codex permissions", () => { + it("separates provider selection from lifecycle and hides fixed Codex permissions", async () => { const config = { provider: "codex", lifecycleMode: "warm", idleTimeoutMs: 45000, }; - const adapter = renderSection( + const adapter = await renderSection( CodexLocalConfigFields, "paperclip_runner", "adapter", config, ); - const configuration = renderSection( + const configuration = await renderSection( CodexLocalConfigFields, "paperclip_runner", "configuration", config, ); - const policy = renderSection( + const policy = await renderSection( CodexLocalConfigFields, "paperclip_runner", "runPolicy", config, ); - expect(adapter).toContain(''); + expect(adapter).toContain('aria-label="Harness"'); expect(adapter).not.toContain("Runner lifecycle"); expect(configuration).not.toContain("Permission mode"); expect(configuration).not.toContain("Runner lifecycle"); @@ -69,13 +89,12 @@ describe("adapter configuration sections", () => { expect(policy).not.toContain("ACP agents"); }); - it("keeps ACP agent admission choices in the adapter section", () => { + it("keeps ACP agent admission choices in the adapter section", async () => { const config = { provider: "acpx", acpxAgent: "claude" }; - const adapter = renderSection(CodexLocalConfigFields, "paperclip_runner", "adapter", config); - const policy = renderSection(CodexLocalConfigFields, "paperclip_runner", "runPolicy", config); + const adapter = await renderSection(CodexLocalConfigFields, "paperclip_runner", "adapter", config); + const policy = await renderSection(CodexLocalConfigFields, "paperclip_runner", "runPolicy", config); - expect(adapter).toContain(''); - expect(adapter).toContain(''); + expect(adapter).toContain('aria-label="ACP agent"'); expect(adapter).not.toContain("Runner lifecycle"); expect(policy).toContain("Runner lifecycle"); expect(policy).not.toContain("ACP agent"); @@ -88,31 +107,31 @@ describe("adapter configuration sections", () => { ["gemini_local", GeminiLocalConfigFields], ] as const)( "separates ACP commands and lifecycle for %s", - (type, Component) => { + async (type, Component) => { const config = { engine: "acp", agentCommand: "saved-command", warmHandleIdleMs: 1234, }; - expect(renderSection(Component, type, "advanced", config)).toContain( + expect(await renderSection(Component, type, "advanced", config)).toContain( 'value="saved-command"', ); expect( - renderSection(Component, type, "configuration", config), + await renderSection(Component, type, "configuration", config), ).not.toContain("ACP server command"); - const policy = renderSection(Component, type, "runPolicy", config); + const policy = await renderSection(Component, type, "runPolicy", config); expect(policy).toContain("ACP session mode"); expect(policy).toContain('value="1234"'); expect(policy).not.toContain("ACP server command"); }, ); - it("keeps process command and arguments under Advanced with saved values", () => { + it("keeps process command and arguments under Advanced with saved values", async () => { const config = { command: "node", args: ["worker.js", "--quiet"] }; expect( - renderSection(ProcessConfigFields, "process", "configuration", config), + await renderSection(ProcessConfigFields, "process", "configuration", config), ).toBe(""); - const advanced = renderSection( + const advanced = await renderSection( ProcessConfigFields, "process", "advanced", @@ -127,12 +146,12 @@ describe("adapter configuration sections", () => { ["hermes_gateway", HermesGatewayConfigFields], ] as const)( "moves %s timeouts without changing their values", - (type, Component) => { + async (type, Component) => { const config = { timeoutSec: 37 }; expect( - renderSection(Component, type, "configuration", config), + await renderSection(Component, type, "configuration", config), ).not.toContain('value="37"'); - expect(renderSection(Component, type, "runPolicy", config)).toContain( + expect(await renderSection(Component, type, "runPolicy", config)).toContain( 'value="37"', ); }, diff --git a/ui/src/components/AdapterMark.tsx b/ui/src/components/AdapterMark.tsx new file mode 100644 index 00000000000..8d293cbbacc --- /dev/null +++ b/ui/src/components/AdapterMark.tsx @@ -0,0 +1,59 @@ +import { getAdapterDisplay } from "@/adapters/adapter-display-registry"; +import { cn } from "@/lib/utils"; + +const brandMarks: Record = { + claude_local: { src: "/brands/claude-color.svg" }, + codex_local: { src: "/brands/codex-color.svg" }, + gemini_local: { src: "/brands/adapters/gemini-color.svg" }, + kimi_local: { + src: "/brands/adapters/kimi-color-light.svg", + dark: "/brands/adapters/kimi-color.svg", + }, + ...Object.fromEntries( + [ + ["cursor", "cursor"], + ["cursor_cloud", "cursor"], + ["grok_local", "grok"], + ["hermes_local", "hermesagent"], + ["hermes_gateway", "hermesagent"], + ["pi_local", "pi"], + ].map(([type, icon]) => [ + type, + { + src: `/brands/adapters/${icon}.svg`, + dark: `/brands/adapters/${icon}-dark.svg`, + }, + ]), + ), +}; +export function AdapterMark({ + type, + className = "size-6", +}: { + type: string; + className?: string; +}) { + const Icon = getAdapterDisplay(type).icon; + const mark = brandMarks[type]; + if (!mark) return ; + return ( + <> + + {mark.dark && ( + + )} + + ); +} diff --git a/ui/src/components/AgentConfigForm.render.test.tsx b/ui/src/components/AgentConfigForm.render.test.tsx index a95b011a1b3..e649c736a91 100644 --- a/ui/src/components/AgentConfigForm.render.test.tsx +++ b/ui/src/components/AgentConfigForm.render.test.tsx @@ -13,6 +13,10 @@ import { AgentConfigForm, AdapterLoginPanel, subtractPersistedOverlay, type Adap import { defaultCreateValues } from "./agent-config-defaults"; import { buildNewAgentHirePayload } from "../lib/new-agent-hire-payload"; import { ApiError } from "../api/client"; +import { aiConnectionsApi } from "../api/ai-connections"; +import { CodexLocalConfigFields } from "../adapters/codex-local/config-fields"; +import type { AdapterConfigFieldsProps } from "../adapters/types"; +import { DEFAULT_CODEX_LOCAL_MODEL } from "@paperclipai/adapter-codex-local"; const mockAgentsApi = vi.hoisted(() => ({ adapterModels: vi.fn(), @@ -103,12 +107,10 @@ vi.mock("../adapters", () => ({ // The stand-in also records the two gates the form resolves for every // adapter, so a test can assert the plumbing without rendering a real // adapter's fields. - ConfigFields: ({ adapterType, hideInstructionsFile, managedSandboxOnly }: { - adapterType: string; - hideInstructionsFile?: boolean; - managedSandboxOnly?: boolean; - }) => - adapterType === "hermes_gateway" + ConfigFields: (props: AdapterConfigFieldsProps) => { + if (type === "paperclip_runner") return ; + const { adapterType, hideInstructionsFile, managedSandboxOnly } = props; + return adapterType === "hermes_gateway" ?
Hermes Gateway fields
: (
({ data-hide-instructions-file={String(hideInstructionsFile === true)} data-managed-sandbox-only={String(managedSandboxOnly === true)} /> - ), + ); + }, buildAdapterConfig: (values: { model?: string }) => ({ model: values.model || undefined, }), @@ -784,6 +787,44 @@ describe("AgentConfigForm environment selector", () => { expect(result.onSave.mock.calls[0][0].adapterConfig.effort).toBeUndefined(); }); + it.each([ + ["Codex", "codex", undefined, DEFAULT_CODEX_LOCAL_MODEL], + ["ACP agents", "acpx", "claude", "claude-sonnet-5"], + ["Claude Managed", "claude_managed", undefined, "claude-sonnet-5"], + ])("saves the %s harness default without the previous OpenCode model prefix", async (label, provider, acpxAgent, model) => { + const accountList = vi.spyOn(aiConnectionsApi, "list").mockResolvedValue({ + currentUserId: "you", + canManageConnections: true, + connections: [], + }); + try { + const result = await renderForm([], { + adapterType: "paperclip_runner", + adapterConfig: { provider: "opencode", model: "openrouter/anthropic/claude-sonnet-4.6" }, + runtimeConfig: { aiConnection: { mode: "responsible_user", provider: "openrouter", method: "api_key" } }, + }); + roots.push(result.root); + await act(async () => { + result.container.querySelector('[aria-label="Harness"]')! + .dispatchEvent(new KeyboardEvent("keydown", { key: "Enter", bubbles: true })); + }); + await flushReact(); + const option = [...document.querySelectorAll('[role="option"]')] + .find(element => element.textContent === label)!; + expect(option).toBeTruthy(); + await act(async () => { + option.dispatchEvent(new KeyboardEvent("keydown", { key: "Enter", bubbles: true })); + }); + await flushReact(); + await clickByText(result.container, "Save"); + expect(result.onSave).toHaveBeenCalledWith(expect.objectContaining({ + adapterConfig: expect.objectContaining({ provider, model, ...(acpxAgent ? { acpxAgent } : {}) }), + })); + } finally { + accountList.mockRestore(); + } + }); + it("saves Grok 4.7 reasoning effort using the runtime key", async () => { const result = await renderForm([], { adapterType: "grok_local", adapterConfig: { model: "grok-4.7", reasoningEffort: "high" } }); roots.push(result.root); diff --git a/ui/src/components/AgentConfigForm.tsx b/ui/src/components/AgentConfigForm.tsx index 6f655057762..533985df6dd 100644 --- a/ui/src/components/AgentConfigForm.tsx +++ b/ui/src/components/AgentConfigForm.tsx @@ -1,3 +1,5 @@ +import { useConnectionModels } from "./ai-connections/useConnectionModels"; +import { aiRoutingHarness } from "@paperclipai/shared"; import { AiConnectionField } from "./ai-connections/AiConnectionField"; import { aiConnectionBindingSchema, aiRuntimeConnectionBindingSchema } from "@paperclipai/shared"; import { testAgentSetup } from "@/lib/test-agent-setup"; @@ -42,6 +44,9 @@ import { PopoverTrigger, } from "@/components/ui/popover"; import { Button } from "@/components/ui/button"; +import { Input } from "@/components/ui/input"; +import { NativeSelect, selectTriggerClassName } from "@/components/ui/select"; +import { AdapterMark } from "./AdapterMark"; import { FolderOpen, Heart, ChevronDown, X, Copy, Check, ExternalLink, Loader2, TriangleAlert, Bug } from "lucide-react"; import { asBoolean, asFiniteNumber, asObject, cn } from "../lib/utils"; import { copyTextToClipboard } from "../lib/clipboard"; @@ -76,7 +81,6 @@ import { getUIAdapter } from "../adapters"; import { ClaudeLocalAdvancedFields } from "../adapters/claude-local/config-fields"; import { MarkdownEditor } from "./MarkdownEditor"; import { ChoosePathButton } from "./PathInstructionsModal"; -import { OpenCodeLogoIcon } from "./OpenCodeLogoIcon"; import { ReportsToPicker } from "./ReportsToPicker"; import { EnvironmentVariablesEditor, @@ -894,6 +898,7 @@ export function AgentConfigForm(props: AgentConfigFormProps) { const modelProvider = adapterType === "opencode_local" && aiConnectionBindingSchema.safeParse( (overlay.runtime.runtimeConfig as Record | undefined)?.aiConnection ?? runtimeConfig.aiConnection, ).data?.provider === "openrouter" ? "openrouter" : runnerProvider; + const connectionModels = useConnectionModels(selectedCompanyId, isCreate ? undefined : aiConnectionBindingSchema.safeParse((overlay.runtime.runtimeConfig as Record | undefined)?.aiConnection ?? runtimeConfig.aiConnection).data, aiRoutingHarness(adapterType, runnerProvider, eff("adapterConfig", "acpxAgent", config.acpxAgent))); // Fetch adapter models for the effective provider, including unsaved changes. const modelQueryKey = selectedCompanyId ? queryKeys.agents.adapterModels(selectedCompanyId, adapterType, currentDefaultEnvironmentId || null, modelProvider) @@ -901,17 +906,19 @@ export function AgentConfigForm(props: AgentConfigFormProps) { const { data: fetchedModels, error: fetchedModelsError, + isLoading: fetchingModels, } = useQuery({ queryKey: modelQueryKey, queryFn: () => agentsApi.adapterModels(selectedCompanyId!, adapterType, { environmentId: currentDefaultEnvironmentId || null, provider: modelProvider, }), - enabled: Boolean(selectedCompanyId), + enabled: Boolean(selectedCompanyId) && !connectionModels, }); const [refreshModelsError, setRefreshModelsError] = useState(null); const [refreshingModels, setRefreshingModels] = useState(false); - const models = fetchedModels ?? externalModels ?? []; + const models = connectionModels?.models ?? fetchedModels ?? externalModels ?? []; + const modelError = connectionModels ? connectionModels.error : fetchedModelsError; const adapterCommandField = "command"; const { data: detectedModelData, @@ -946,7 +953,10 @@ export function AgentConfigForm(props: AgentConfigFormProps) { set: isCreate ? (patch: Partial) => props.onChange(patch) : null, config, eff: eff as (group: "adapterConfig", field: string, original: T) => T, - mark: mark as (group: "adapterConfig", field: string, value: unknown) => void, + // Harness transitions supply the new harness's default model. Resolve + // user-selected model IDs in ModelDropdown, not through the previous + // render's harness when adapter fields change several values together. + mark, models, // Resolve the effective instructions-file gate once. The instructions file // is an absolute host path, so the managed-sandbox-only policy hides it for @@ -1658,7 +1668,7 @@ export function AgentConfigForm(props: AgentConfigFormProps) { )} - {!isCreate && selectedCompanyId && | undefined)?.aiConnection ?? runtimeConfig.aiConnection).data} model={String(eff("adapterConfig", "model", config.model) ?? "")} environmentId={currentDefaultEnvironmentId || undefined} legacy onChange={binding => mark("runtime", "runtimeConfig", { ...runtimeConfig, aiConnection: binding })} />} @@ -1719,6 +1729,7 @@ export function AgentConfigForm(props: AgentConfigFormProps) { {isLocal && (<> { const supportedEfforts = setupEfforts(adapterType, v); @@ -1732,7 +1743,7 @@ export function AgentConfigForm(props: AgentConfigFormProps) { }); return; } - mark("adapterConfig", "model", v || undefined); + mark("adapterConfig", "model", connectionModels?.resolveModel(v) || v || undefined); if (clearUnsupportedEffort) { mark("adapterConfig", thinkingEffortKey, undefined); mark("adapterConfig", "reasoningEffort", undefined); @@ -1743,35 +1754,36 @@ export function AgentConfigForm(props: AgentConfigFormProps) { defaultLabel={adapterType === "claude_local" ? `Default (${DEFAULT_CLAUDE_LOCAL_MODEL})` : undefined} allowDefault={adapterType !== "opencode_local" && adapterType !== "pi_local" && adapterType !== "paperclip_runner"} required={adapterType === "opencode_local" || adapterType === "pi_local"} - groupByProvider={adapterType === "opencode_local" || adapterType === "pi_local"} - preserveOrder={adapterCuratesModelOrder(adapterType)} + groupByProvider={!connectionModels && (adapterType === "opencode_local" || adapterType === "pi_local")} + preserveOrder={Boolean(connectionModels) || adapterCuratesModelOrder(adapterType)} creatable - detectedModel={detectedModel} + detectedModel={connectionModels ? undefined : detectedModel} detectedModelCandidates={[]} - onDetectModel={adapterType === "opencode_local" || adapterType === "paperclip_runner" + onDetectModel={connectionModels || adapterType === "opencode_local" || adapterType === "paperclip_runner" ? undefined : async () => { const result = await refetchDetectedModel(); return result.data?.model ?? null; }} onRefreshModels={ - supportsAdapterModelRefresh(adapterType) + connectionModels ? connectionModels.refreshModels : supportsAdapterModelRefresh(adapterType) ? handleRefreshModels : undefined } - refreshingModels={refreshingModels} + refreshingModels={connectionModels?.refreshing ?? refreshingModels} detectModelLabel="Detect model" emptyDetectHint="No model detected. Select or enter one manually." /> - {(refreshModelsError || fetchedModelsError) && ( + {(refreshModelsError || modelError) && (

{refreshModelsError - ?? (fetchedModelsError instanceof Error - ? fetchedModelsError.message + ?? (modelError instanceof Error + ? modelError.message : "Failed to load adapter models.")}

)} {adapterType === "opencode_local" + && !connectionModels && currentDefaultEnvironment && currentDefaultEnvironment.driver !== "local" && (

@@ -3666,19 +3678,26 @@ export function AdapterTypeDropdown({ return ( - {adapterList.map((item) => ( } + + ); + } + return ( - @@ -4024,7 +4095,7 @@ export function ModelDropdown({ {filteredModels.length === 0 && !canCreateManualModel && promotedModelIds.size === 0 && (

- {onDetectModel + {loadingModels ? "Loading models…" : onDetectModel ? (emptyDetectHint ?? "No model detected yet. Enter a provider/model manually.") : "No models found."}

diff --git a/ui/src/components/ai-connections/AiConnectionAccountControls.tsx b/ui/src/components/ai-connections/AiConnectionAccountControls.tsx index 38605dce7db..b05f77f230d 100644 --- a/ui/src/components/ai-connections/AiConnectionAccountControls.tsx +++ b/ui/src/components/ai-connections/AiConnectionAccountControls.tsx @@ -27,7 +27,7 @@ export function AiConnectionAccountControls({ const activeDefault = account.isDefault && available; return (
- {ownPersonal && ( + {ownPersonal && !account.routing && (
- {aiMethodLabel(metadata.provider, metadata.method)} ·{" "} + {metadata.routing ? metadata.routing.kind === "bedrock" ? `Bedrock · ${metadata.routing.region}` : metadata.routing.kind === "openrouter" ? "OpenRouter" : metadata.routing.baseUrl : aiMethodLabel(metadata.provider, metadata.method)} ·{" "} {connection.credentialPolicy === "per_user" ? "Personal" : "Company shared"} @@ -90,6 +85,7 @@ export function ManagedAiConnectionDetails({ ); return (
+ {account.routing &&
Provider settings
Destination
{account.routing.baseUrl ?? (account.routing.kind === "bedrock" ? account.routing.region : "OpenRouter")}
API format
{account.routing.kind === "openrouter" ? "Selected by harness" : account.routing.protocol}
Authentication
{account.routing.auth}
Models
{account.routing.models.map(m => m.label ?? m.id).join(", ") || "Enter a model ID on the agent"}
} ({ aiConnectionsApi: { list: vi.fn() } })); +vi.mock("@/api/agents", () => ({ agentsApi: { adapterModels: vi.fn() } })); + +const binding: AiConnectionBinding = { provider: "openrouter", method: "api_key", mode: "shared", connectionId: "router", grantId: "grant" }; +const account: AiManagedConnectionSummary = { + id: "router", grantId: "grant", companyId: "company", provider: "openrouter", method: "api_key", + name: "Company OpenRouter", ownership: "shared", isDefault: false, status: "connected", + routing: { kind: "openrouter", protocol: "chat", auth: "bearer", models: [] }, +}; +const catalog = [ + { id: "openrouter/z-ai/glm-5", label: "Z.AI GLM" }, + { id: "openrouter/anthropic/claude-sonnet-4.5", label: "Claude Sonnet" }, + { id: "openrouter/openrouter/auto", label: "Auto Router" }, +]; +let root: Root; +let client: QueryClient; +let result: ReturnType; +let render: (value: AiConnectionBinding | undefined) => Promise; + +beforeEach(() => { + vi.mocked(agentsApi.adapterModels).mockResolvedValue(catalog); + vi.mocked(aiConnectionsApi.list).mockResolvedValue({ currentUserId: "you", connections: [account], canManageConnections: true }); +}); +afterEach(async () => { + await act(async () => root?.unmount()); + client?.clear(); + vi.resetAllMocks(); +}); + +async function mount(accounts: AiManagedConnectionSummary[] = [account], harness = "opencode_local", value: AiConnectionBinding | undefined = binding) { + client = new QueryClient({ defaultOptions: { queries: { retry: false, staleTime: Infinity } } }); + client.setQueryData(["ai-connections", "company"], { currentUserId: "you", connections: accounts }); + root = createRoot(document.createElement("div")); + function Probe({ value }: { value: AiConnectionBinding | undefined }) { + result = useConnectionModels("company", value, harness); + return null; + } + render = async next => { await act(async () => root.render()); }; + await render(value); +} + +it.each(["opencode_local", "codex_local", "claude_local", "hermes_local"])("automatically discovers an empty OpenRouter connection's models for %s", async harness => { + await mount([account], harness); + await vi.waitFor(() => expect(result?.models).toHaveLength(3)); + expect(agentsApi.adapterModels).toHaveBeenCalledWith("company", "opencode_local", { provider: "openrouter" }); + expect(result?.models.map(m => m.id)).toEqual(catalog.map(m => harness === "opencode_local" ? m.id : m.id.slice("openrouter/".length))); + expect(result?.models.map(m => m.label)).toEqual(catalog.map(m => m.label)); + expect(result?.resolveModel("anthropic/custom-model")).toBe(harness === "opencode_local" ? "openrouter/anthropic/custom-model" : "anthropic/custom-model"); +}); + +it("also discovers models for legacy OpenRouter connections without routing metadata", async () => { + await mount([{ ...account, routing: undefined }]); + await vi.waitFor(() => expect(result?.models).toEqual(catalog)); +}); + +it("preserves an explicit connection model list instead of replacing it with the public catalog", async () => { + await mount([{ ...account, routing: { ...account.routing!, models: [{ id: "company/model", label: "Approved model" }] } }]); + expect(result?.models).toEqual([{ id: "openrouter/company/model", label: "Approved model" }]); + expect(agentsApi.adapterModels).not.toHaveBeenCalled(); +}); + +it("does not reuse public OpenRouter models after switching to a custom gateway", async () => { + const gateway: AiManagedConnectionSummary = { ...account, id: "gateway", grantId: "gateway-grant", provider: "openai", routing: { kind: "gateway", protocol: "chat", auth: "bearer", baseUrl: "https://example.com/v1", models: [] } }; + await mount([account, gateway]); + await vi.waitFor(() => expect(result?.models).toEqual(catalog)); + await render({ provider: "openai", method: "api_key", mode: "shared", connectionId: gateway.id, grantId: gateway.grantId }); + expect(result?.models).toEqual([]); + expect(result?.resolveModel("private-model")).toBe("paperclip/private-model"); + expect(result?.refreshModels).toBeUndefined(); +}); + +it("leaves native model discovery alone", async () => { + await mount([], "codex_local", { provider: "openai", method: "subscription", mode: "responsible_user" }); + expect(result).toBeUndefined(); + expect(agentsApi.adapterModels).not.toHaveBeenCalled(); +}); + +it("only uses the current person's default connection models", async () => { + await mount([ + { ...account, ownership: "personal", ownerUserId: "someone-else", isDefault: true, routing: { ...account.routing!, models: [{ id: "other/private-model" }] } }, + { ...account, ownership: "personal", ownerUserId: "you", isDefault: true }, + ], "opencode_local", { provider: "openrouter", method: "api_key", mode: "responsible_user" }); + await vi.waitFor(() => expect(result?.models).toEqual(catalog)); +}); + +it("exposes loading and allows retry after a catalog failure", async () => { + let reject!: (error: Error) => void; + vi.mocked(agentsApi.adapterModels).mockReturnValueOnce(new Promise((_, rejectPromise) => { reject = rejectPromise; })); + await mount(); + expect(result?.isLoading).toBe(true); + await act(async () => reject(new Error("Catalog unavailable"))); + await vi.waitFor(() => expect(result?.error?.message).toBe("Catalog unavailable")); + expect(result?.isLoading).toBe(false); + expect(result?.resolveModel("manual/model")).toBe("openrouter/manual/model"); + await act(async () => { await result?.refreshModels?.(); }); + await vi.waitFor(() => expect(result?.models).toEqual(catalog)); + expect(result?.error).toBeNull(); +}); diff --git a/ui/src/components/ai-connections/useConnectionModels.ts b/ui/src/components/ai-connections/useConnectionModels.ts new file mode 100644 index 00000000000..1173a8e8994 --- /dev/null +++ b/ui/src/components/ai-connections/useConnectionModels.ts @@ -0,0 +1,55 @@ +import { useQuery } from "@tanstack/react-query"; +import { aiRoutingModel, type AiConnectionBinding, type AiProviderRouting } from "@paperclipai/shared"; +import { aiConnectionsApi } from "@/api/ai-connections"; +import { agentsApi } from "@/api/agents"; +import { queryKeys } from "@/lib/queryKeys"; + +export function useConnectionModels( + companyId: string | null | undefined, + binding: AiConnectionBinding | undefined, + harness: string, +) { + const accounts = useQuery({ + queryKey: ["ai-connections", companyId], + queryFn: () => aiConnectionsApi.list(companyId!), + enabled: Boolean(companyId && binding), + }); + const connection = accounts.data?.connections.find((c) => + binding?.mode === "responsible_user" + ? c.provider === binding.provider && c.isDefault && c.ownerUserId === accounts.data?.currentUserId && c.ownership === "personal" + : c.id === binding?.connectionId && c.grantId === binding?.grantId, + ); + const routing = connection?.routing; + const openRouter = routing?.kind === "openrouter" || (!routing && binding?.provider === "openrouter"); + const discover = openRouter && !routing?.models.length; + // Reuse the existing public catalog and its query cache. It returns OpenCode IDs; + // strip only that transport prefix for other harnesses (openrouter/auto is + // itself a valid upstream model ID). + const catalog = useQuery({ + queryKey: queryKeys.agents.adapterModels(companyId ?? "none", "opencode_local", null, "openrouter"), + queryFn: () => agentsApi.adapterModels(companyId!, "opencode_local", { provider: "openrouter" }), + enabled: Boolean(companyId && discover), + staleTime: 60_000, + retry: false, + }); + const effectiveRouting: AiProviderRouting | undefined = routing ?? (openRouter + ? { kind: "openrouter", protocol: "chat", auth: "bearer", models: [] } + : undefined); + const modelOptions = discover + ? (catalog.data ?? []).map((m) => ({ ...m, id: harness === "opencode_local" ? m.id : m.id.replace(/^openrouter\//, "") })) + : routing?.models ?? []; + return effectiveRouting + ? { + models: modelOptions.map((m) => ({ + id: aiRoutingModel(effectiveRouting, harness, m.id), + label: m.label ?? m.id, + })), + isLoading: discover && catalog.isLoading, + error: discover ? catalog.error : null, + refreshing: discover && catalog.isFetching, + refreshModels: discover ? async () => { await catalog.refetch(); } : undefined, + resolveModel: (model: string) => + aiRoutingModel(effectiveRouting, harness, model), + } + : undefined; +} diff --git a/ui/src/components/ai-connections/useLocalAiLogin.ts b/ui/src/components/ai-connections/useLocalAiLogin.ts index 8e6a34ecf19..b0de9683f5d 100644 --- a/ui/src/components/ai-connections/useLocalAiLogin.ts +++ b/ui/src/components/ai-connections/useLocalAiLogin.ts @@ -3,7 +3,7 @@ import type { AiConnectionLoginIntent, LocalAiLoginAttempt, LocalAiLoginStatus } import { aiConnectionsApi } from "@/api/ai-connections"; /** Every authentication host uses the same local credential check and login lifecycle. */ -export function useLocalAiLogin(companyId: string | null, intent: AiConnectionLoginIntent, enabled: boolean, _legacyOptions?: { allowHostClaude?: boolean }) { +export function useLocalAiLogin(companyId: string | null, intent: AiConnectionLoginIntent, enabled: boolean) { const isolated = true; const active = Boolean(companyId && enabled); const [attempt, setAttempt] = useState(null); diff --git a/ui/src/components/new-agent/AgentBasicsDialog.tsx b/ui/src/components/new-agent/AgentBasicsDialog.tsx index d778c2bb146..f7845a6c460 100644 --- a/ui/src/components/new-agent/AgentBasicsDialog.tsx +++ b/ui/src/components/new-agent/AgentBasicsDialog.tsx @@ -1,5 +1,6 @@ import { useCompany } from "@/context/CompanyContext"; import { useAgentAppearanceDraft } from "@/hooks/useAgentAppearanceDraft"; +import { AdapterMark } from "../AdapterMark"; import { AgentCharacter } from "../AgentCharacter"; import { useId, useState } from "react"; import { useQuery } from "@tanstack/react-query"; @@ -25,62 +26,8 @@ export type AgentBasics = { adapterType: string; runnerProvider: string; }; -const brandMarks: Record = { - claude_local: { src: "/brands/claude-color.svg" }, - codex_local: { src: "/brands/codex-color.svg" }, - gemini_local: { src: "/brands/adapters/gemini-color.svg" }, - kimi_local: { - src: "/brands/adapters/kimi-color-light.svg", - dark: "/brands/adapters/kimi-color.svg", - }, - ...Object.fromEntries( - [ - ["cursor", "cursor"], - ["cursor_cloud", "cursor"], - ["grok_local", "grok"], - ["hermes_local", "hermesagent"], - ["hermes_gateway", "hermesagent"], - ["pi_local", "pi"], - ].map(([type, icon]) => [ - type, - { - src: `/brands/adapters/${icon}.svg`, - dark: `/brands/adapters/${icon}-dark.svg`, - }, - ]), - ), -}; -export function AdapterMark({ - type, - className = "size-6", -}: { - type: string; - className?: string; -}) { - const Icon = getAdapterDisplay(type).icon; - const mark = brandMarks[type]; - if (!mark) return ; - return ( - <> - - {mark.dark && ( - - )} - - ); -} +export { AdapterMark } from "../AdapterMark"; + function AgentBasicsCharacter() { const { selectedCompanyId } = useCompany(); const { appearance } = useAgentAppearanceDraft(`${selectedCompanyId}:new-agent`); diff --git a/ui/src/components/new-agent/AgentProviderConnection.test.tsx b/ui/src/components/new-agent/AgentProviderConnection.test.tsx index a9116af73b1..0c0510428fd 100644 --- a/ui/src/components/new-agent/AgentProviderConnection.test.tsx +++ b/ui/src/components/new-agent/AgentProviderConnection.test.tsx @@ -1,6 +1,5 @@ // @vitest-environment jsdom import { flushSync } from "react-dom"; -import type { LocalAiLoginStatus } from "@paperclipai/shared"; import { createRoot, type Root } from "react-dom/client"; import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; import { afterEach, describe, expect, it, vi } from "vitest"; @@ -18,8 +17,8 @@ const managedApi = vi.hoisted(() => ({ list: vi.fn(async () => ({ currentUserId: "user-1", connections: [] })), loginResult: vi.fn(async () => ({ connectionId: "login-account", grantId: "login-grant" })), connectLocal: vi.fn(async () => ({ connectionId: "local-account", grantId: "local-grant" })), - startLocalLogin: vi.fn(async () => ({ sessionId: "local-attempt", command: "CODEX_HOME='/fixture/isolated-login' codex login", expiresAt: "2026-09-11T20:00:00Z" })), - checkLocalLogin: vi.fn(async (): Promise => ({ status: "sign_in_required" })), + startLocalLogin: vi.fn(async () => ({ sessionId: "local-attempt", expiresAt: "2099-01-01T00:00:00Z" })), + checkLocalLogin: vi.fn(async () => ({ status: "sign_in_required" as "ready" | "sign_in_required" })), cancelLocalLogin: vi.fn(async () => ({})), create: vi.fn(async () => ({ connectionId: "managed-connection", grantId: "managed-grant" })), })); @@ -56,6 +55,7 @@ async function mount( localEnvironment = false, deploymentMode: "local_trusted" | "authenticated" = "local_trusted", localAiLoginSupported = true, + advancedConnection?: Parameters[0]["advancedConnection"], ) { const key = adapterType === "claude_local" ? "ANTHROPIC_API_KEY" : "OPENAI_API_KEY"; @@ -128,18 +128,19 @@ async function mount( testConnection={test} onConnected={connected} managedAccount={managedAccount} + advancedConnection={advancedConnection} /> , ), ); await vi.waitFor(() => expect(mocks.personal).toHaveBeenCalled()); await vi.waitFor(() => expect(client.isFetching()).toBe(0)); - if (savedApiKeys && !managedAccount) await vi.waitFor(() => expect(host.textContent).toContain("2 saved API keys")); + if (savedApiKeys && !managedAccount && !advancedConnection) await vi.waitFor(() => expect(host.textContent).toContain("2 saved API keys")); return { test, connected, key }; } function click(text: string) { const button = [...host.querySelectorAll("button")].find((b) => - b.textContent?.includes(text), + b.getAttribute("aria-label") ? b.getAttribute("aria-label") === text : b.textContent?.includes(text), )!; expect(button).toBeTruthy(); flushSync(() => button.click()); @@ -150,6 +151,74 @@ function openProvider() { ); } describe("AgentProviderConnection reuse", () => { + it("keeps all three connection tiles visible while switching forms", async () => { + await mount("codex_local", false, true, true, true, false, undefined, false, "local_trusted", true, { + content:
Advanced connection picker
, + }); + const modeLabels = () => Array.from(host.querySelectorAll('[role="radio"]')).map(button => button.textContent); + expect(modeLabels()).toEqual(["OpenAISubscription", "OpenAIAPI key", "AdvancedCustom Gateway"]); + expect(host.querySelector('[aria-label="Saved subscription"]')).not.toBeNull(); + expect(host.textContent).not.toContain("Advanced connection picker"); + expect(host.textContent).not.toContain("instead"); + click("OpenAIAPI key"); + expect(host.querySelector('[aria-label="Saved API key"]')).not.toBeNull(); + expect(host.querySelector('[role="radio"][aria-checked="true"]')?.textContent).toBe("OpenAIAPI key"); + expect(host.querySelector('[aria-label="Saved subscription"]')).toBeNull(); + expect(modeLabels()).toEqual(["OpenAISubscription", "OpenAIAPI key", "AdvancedCustom Gateway"]); + click("Advanced"); + expect(host.textContent).toContain("Advanced connection picker"); + expect(host.querySelector('[role="radio"][aria-checked="true"]')?.textContent).toBe("AdvancedCustom Gateway"); + expect(host.querySelector("select")).toBeNull(); + expect(modeLabels()).toEqual(["OpenAISubscription", "OpenAIAPI key", "AdvancedCustom Gateway"]); + const useConnection = Array.from(host.querySelectorAll("button")).find(button => button.textContent?.includes("Use connection"))!; + expect(useConnection.disabled).toBe(true); + click("OpenAISubscription"); + expect(host.querySelector('[aria-label="Saved subscription"]')).not.toBeNull(); + expect(host.textContent).not.toContain("Advanced connection picker"); + }); + + it("continues with the selected advanced connection", async () => { + const binding = { provider: "openrouter" as const, method: "api_key" as const, mode: "shared" as const, connectionId: "router", grantId: "router-grant" }; + const { connected, test } = await mount("codex_local", false, true, true, true, false, undefined, false, "local_trusted", true, { + content:
Advanced connection picker
, value: binding, + }); + click("Advanced"); + click("Use connection"); + expect(connected).toHaveBeenCalledExactlyOnceWith({ env: {}, aiConnection: binding }); + expect(test).not.toHaveBeenCalled(); + }); + + it("preserves the API key when the selected tile is clicked again", async () => { + await mount("codex_local", false, true, false, false, false, undefined, false, "local_trusted", true, { + content:
Advanced connection picker
, + }); + click("OpenAIAPI key"); + const input = host.querySelector('input[type="password"]')!; + flushSync(() => { + Object.getOwnPropertyDescriptor(HTMLInputElement.prototype, "value")!.set!.call(input, "fixture-key"); + input.dispatchEvent(new Event("input", { bubbles: true })); + }); + click("OpenAIAPI key"); + expect(input.value).toBe("fixture-key"); + const apiTile = host.querySelector('[role="radio"][aria-checked="true"]')!; + flushSync(() => apiTile.dispatchEvent(new KeyboardEvent("keydown", { key: "ArrowRight", bubbles: true }))); + expect(host.querySelector('[role="radio"][aria-checked="true"]')?.textContent).toBe("AdvancedCustom Gateway"); + expect(host.querySelectorAll('[role="radio"]')).toHaveLength(3); + }); + + it("uses the saved subscription after switching back from an advanced connection", async () => { + const { connected } = await mount("codex_local", false, true, true, true, false, undefined, false, "local_trusted", true, { + content:
Advanced connection picker
, + value: { provider: "openrouter", method: "api_key", mode: "shared", connectionId: "router", grantId: "router-grant" }, + }); + click("Advanced"); + click("OpenAISubscription"); + click("Use saved subscription"); + await vi.waitFor(() => expect(connected).toHaveBeenCalled()); + expect(connected.mock.calls[0][0]).toMatchObject({ env: { CODEX_HOME: { type: "secret_ref", secretId: "codex-home" } } }); + expect(connected.mock.calls[0][0].aiConnection).toBeUndefined(); + }); + it.each([ ["anthropic", "claude_local"], ["openai", "codex_local"], ["xai", "grok_local"], ] as const)("lets %s recovery switch methods without overwriting the original account", async (provider, adapterType) => { @@ -273,11 +342,14 @@ describe("AgentProviderConnection reuse", () => { expect(mocks.loginPanel).not.toHaveBeenCalled(); }); it("leaves a completed local account saved when its host is cancelled", async () => { + managedApi.checkLocalLogin.mockResolvedValue({ status: "ready" }); let finish!: (result: { connectionId: string; grantId: string }) => void; managedApi.connectLocal.mockImplementationOnce(() => new Promise(resolve => { finish = resolve; })); const onComplete = vi.fn(); await mount("claude_local", false, false, false, false, false, { intent: { provider: "anthropic", method: "subscription", name: "My account", ownership: "personal", agentIds: [], allAgents: false }, onComplete }, true); - openProvider(); click("Connect"); flushSync(() => root.unmount()); + openProvider(); + await vi.waitFor(() => expect(host.textContent).toContain("is signed in")); + click("Connect"); flushSync(() => root.unmount()); finish({ connectionId: "saved", grantId: "saved-grant" }); await new Promise(resolve => setTimeout(resolve, 0)); expect(onComplete).not.toHaveBeenCalled(); diff --git a/ui/src/components/new-agent/AgentProviderConnection.tsx b/ui/src/components/new-agent/AgentProviderConnection.tsx index 9cd1c12b7ed..71a35f1f766 100644 --- a/ui/src/components/new-agent/AgentProviderConnection.tsx +++ b/ui/src/components/new-agent/AgentProviderConnection.tsx @@ -2,7 +2,8 @@ import { healthApi } from "@/api/health"; import { aiConnectionsApi } from "@/api/ai-connections"; import { useLocalAiLogin } from "../ai-connections/useLocalAiLogin"; import type { AiConnectionBinding, AiConnectionLoginIntent } from "@paperclipai/shared"; -import { useEffect, useRef, useState } from "react"; +import { useEffect, useRef, useState, type ReactNode } from "react"; +import { Cable } from "lucide-react"; import { useQuery } from "@tanstack/react-query"; import { motion } from "motion/react"; import { @@ -18,7 +19,7 @@ import { OnboardingCardField, OnboardingLoginCard, } from "../AdapterLoginChrome"; -import { ModelSourceTiles } from "../onboarding/ModelSourceTiles"; +import { ModelSourceTiles, type ModelConnectionMode } from "../onboarding/ModelSourceTiles"; import { CredentialModeLink } from "../onboarding/CredentialModeLink"; import { FooterNav } from "../onboarding/FooterNav"; import { MAKE_ROOM, CARD_ENTER } from "../onboarding/onboarding-motion"; @@ -40,10 +41,12 @@ export function AgentProviderConnection({ canLogin, localEnvironment = false, onConnected, + onDraftChanged, onBack, testConnection, testError, managedAccount, + advancedConnection, }: { companyId: string; adapterType: "claude_local" | "codex_local" | "grok_local"; @@ -51,9 +54,11 @@ export function AgentProviderConnection({ canLogin: boolean; localEnvironment?: boolean; onConnected: (connection: ProviderConnection) => void; + onDraftChanged?: () => void; onBack: () => void; testConnection: (connection: ProviderConnection) => Promise; testError?: string | null; + advancedConnection?: { content: ReactNode; value?: AiConnectionBinding }; /** Connections supplies its access intent; presentation and login controllers stay shared. */ managedAccount?: { intent: AiConnectionLoginIntent; @@ -81,7 +86,8 @@ export function AgentProviderConnection({ setLoginPhase("preparing"); }; const [methodChoice, setMethod] = useState<"subscription" | "api" | null>(managedAccount?.initialMethod === "api_key" ? "api" : managedAccount ? "subscription" : null); - const [opened, setOpened] = useState(false); + const [advanced, setAdvanced] = useState(false); + const [opened, setOpened] = useState(Boolean(advancedConnection)); const [authorizationUrl, setAuthorizationUrl] = useState(null); const [loginPhase, setLoginPhase] = useState<"preparing" | "ready" | "waiting" | "connecting">("preparing"); const phaseBeforeSubmit = useRef<"ready" | "waiting">("ready"); @@ -136,8 +142,8 @@ export function AgentProviderConnection({ const localLogin = useLocalAiLogin(companyId, managedIntent ?? { provider: aiProvider, method: "subscription", name: `My ${provider} subscription`, ownership: "personal", agentIds: [], allAgents: true, - }, canUseLocalLogin && method === "subscription" && !savedSubscription && !storedLogin.data, - { allowHostClaude: health.data?.deploymentMode === "local_trusted" }); + }, !advanced && canUseLocalLogin && method === "subscription" && !savedSubscription && !storedLogin.data, + ); const auth = useQuery({ queryKey: queryKeys.agents.authSignal( companyId, @@ -151,7 +157,7 @@ export function AgentProviderConnection({ environmentId ?? undefined, ), retry: false, - enabled: !managedAccount, + enabled: !managedAccount && !advanced, }); async function connect() { if (busy || managedAccount?.disabled) return; @@ -229,38 +235,58 @@ export function AgentProviderConnection({ !savedKeys.loading && !storedLogin.data && (Boolean(managedAccount) || auth.data?.status !== "present" || subscriptionId === ""); + const switchMode = (next: ModelConnectionMode) => { + if (advancedConnection && next === (advanced ? "advanced" : method)) return; + onDraftChanged?.(); + cancel(); + setAdvanced(next === "advanced"); + setOpened(advancedConnection ? true : opened); + savedManagedAccount.current = null; + if (next !== "advanced") setMethod(next); + setApiKey(""); + setStoredConnection(null); + setError(null); + }; return (
, credentialMode: "subscription" }, + { id: "api", label: provider, icon: , credentialMode: "api" }, + { id: "advanced", label: "Advanced", icon: , credentialMode: "advanced" }, + ] : [ { id: adapterType, label: provider, icon: , }, ]} - mode={method} - selectedId={opened ? adapterType : null} - collapsed={opened} - onSelect={() => { if (!managedAccount?.disabled) setOpened(true); }} + mode={advanced ? "advanced" : method} + selectedId={advancedConnection ? (advanced ? "advanced" : method) : opened ? adapterType : null} + collapsed={!advancedConnection && opened} + onSelect={id => { + if (managedAccount?.disabled) return; + if (advancedConnection) switchMode(id as ModelConnectionMode); + else setOpened(true); + }} /> - {(!opened || managedAccount) && !managedAccount?.fixedMethod && ( + {!advancedConnection && (!opened || managedAccount) && !managedAccount?.fixedMethod && (
- { - cancel(); - setOpened(opened); - savedManagedAccount.current = null; - setMethod(next); - setApiKey(""); - setStoredConnection(null); - setError(null); - }} - /> +
)} + {advanced && advancedConnection ? <> +
{advancedConnection.content}
+ { + if (advancedConnection.value) onConnected({ env: {}, aiConnection: advancedConnection.value }); + }} + /> + : <> {!opened && savedKeys.options.length > 0 && (

{savedKeys.options.length} saved API{" "} @@ -269,15 +295,17 @@ export function AgentProviderConnection({ )} {method === "subscription" && savedKeys.subscriptions.length > 0 && ( - +

+ { onDraftChanged?.(); setSubscriptionId(id); }} + loading={false} + error={false} + kind="subscription" + disabled={busy} + /> +
)} { + onDraftChanged?.(); setSelectedKeyId(id); setApiKey(""); setStoredConnection(null); @@ -318,6 +347,7 @@ export function AgentProviderConnection({ : "Enter API key here" } onChange={(value) => { + onDraftChanged?.(); setSelectedKeyId(""); setApiKey(value); setStoredConnection(null); @@ -395,7 +425,7 @@ export function AgentProviderConnection({ )} { - if (opened) cancel(); + if (opened && !advancedConnection) cancel(); else onBack(); }} primaryLabel={ @@ -416,6 +446,7 @@ export function AgentProviderConnection({ managedAccount?.disabled || (Boolean(managedAccount) && method === "subscription" && !canLogin && !canUseLocalLogin) || (localEnvironment && health.isPending) || localLogin.preparing || Boolean(localLogin.error) || + (method === "subscription" && canUseLocalLogin && !savedSubscription && !storedLogin.data && localLogin.status !== "ready") || (!managedAccount && auth.isPending) || savedKeys.loading || (adapterType === "claude_local" && storedLogin.isPending) || @@ -438,6 +469,7 @@ export function AgentProviderConnection({ } else void connect(); }} /> + }
); } diff --git a/ui/src/components/new-agent/NewAgentSetup.tsx b/ui/src/components/new-agent/NewAgentSetup.tsx index 33690f6e403..d6b38ffc481 100644 --- a/ui/src/components/new-agent/NewAgentSetup.tsx +++ b/ui/src/components/new-agent/NewAgentSetup.tsx @@ -1,3 +1,4 @@ +import { useConnectionModels } from "../ai-connections/useConnectionModels"; import { AgentCharacter } from "../AgentCharacter"; import { useAgentAppearanceDraft } from "../../hooks/useAgentAppearanceDraft"; import { AiConnectionField, aiProviderForAdapter } from "../ai-connections/AiConnectionField"; @@ -53,6 +54,7 @@ import { Field } from "../agent-config-primitives"; import { SecretPicker } from "../environment-variables-editor/SecretPicker"; import { Button } from "../ui/button"; import { Input } from "../ui/input"; +import { NativeSelect, nativeSelectClassName } from "../ui/select"; import { OnboardingCard, OnboardingHeading, @@ -66,8 +68,7 @@ import { } from "./AgentProviderConnection"; import { adapterCuratesModelOrder } from "../../lib/model-utils"; -const controlClass = - "w-full rounded-md border border-border bg-background px-2.5 py-1.5 text-sm leading-5 outline-none focus-visible:ring-2 focus-visible:ring-ring"; +const controlClass = nativeSelectClassName; const blocking = (result: AdapterEnvironmentTestResult) => result.status === "fail" || result.checks.some((check) => check.code === ADAPTER_AUTH_MISSING_CHECK_CODE); @@ -155,6 +156,7 @@ function Setup({ ); const [connection, setConnection] = useState(null); const aiBinding = runtimeAiBinding ?? connection?.aiConnection; + const connectionModels = useConnectionModels(companyId, aiBinding, brandType); const [repository, setRepository] = useState(""); const [branch, setBranch] = useState(""); const [createdInSession, setCreated] = useState(null); @@ -187,6 +189,12 @@ function Setup({ setTestState("idle"); setError(null); }; + const editConnection = () => { + setConnection(null); + setRuntimeAiBinding(undefined); + resetTest(); + setScreen("connect"); + }; const adapters = useQuery({ queryKey: queryKeys.adapters.all, queryFn: adaptersApi.list, @@ -218,7 +226,7 @@ function Setup({ const models = useQuery({ queryKey: queryKeys.agents.adapterModels(companyId, brandType, null, aiBinding?.provider), queryFn: () => agentsApi.adapterModels(companyId, brandType, { provider: aiBinding?.provider }), - enabled: Boolean(brandType) && showModel, + enabled: Boolean(brandType) && showModel && !connectionModels, retry: false, }); const companySecrets = useQuery({ @@ -682,9 +690,9 @@ function Setup({ type="button" aria-current={step === screen ? "step" : undefined} disabled={ - busy || Boolean(created) || index > steps.indexOf(screen) + busy || Boolean(created) || step === "saved" } - onClick={() => setScreen(step)} + onClick={() => step === "connect" && screen !== "connect" ? editConnection() : setScreen(step)} className={cn( "flex w-full items-center gap-3 rounded-md px-3 py-3 text-left text-sm", step === screen @@ -716,38 +724,42 @@ function Setup({
-
- - - -
- { + if (binding.mode === "router") return; + setConnection(null); + setRuntimeAiBinding(binding); + resetTest(); + }} + /> + ), + }} onBack={() => navigate("/agents/all")} + onDraftChanged={() => { + setConnection(null); + setRuntimeAiBinding(undefined); + resetTest(); + }} testConnection={runTest} testError={ error ?? @@ -764,10 +776,11 @@ function Setup({ } onConnected={(next) => { setConnection(next); + setRuntimeAiBinding(undefined); resetTest(); setScreen("runtime"); }} - /> + /> :

Loading connection settings…

} ) : screen === "saved" && created ? (
@@ -833,32 +846,24 @@ function Setup({
-

Runtime

- {aiProviderForAdapter(brandType) && ( - connection && !aiBinding ? ( -
-

- Using the connection selected in the Connect step. -

- -
- ) : ( - { binding.mode !== "router" && setRuntimeAiBinding(binding); resetTest(); }} /> - ) + {!connectionAdapter && aiProviderForAdapter(brandType) && ( + { binding.mode !== "router" && setRuntimeAiBinding(binding); resetTest(); }} /> )} - {models.error &&

Could not load models. Retry or enter a model ID manually.

} + {(connectionModels ? connectionModels.error : models.error) &&

Could not load models. Retry or enter a model ID manually.

} {((showModel && !usingKimiApi) || efforts.length > 0) && (
{showModel && !usingKimiApi && ( { - setModel(value); + setModel(connectionModels?.resolveModel(value) ?? value); if ( effort && !setupEfforts(adapterType, value).includes( @@ -883,15 +888,14 @@ function Setup({ allowDefault={!multiProvider} required={multiProvider} creatable - groupByProvider={multiProvider} - preserveOrder={adapterCuratesModelOrder(brandType)} + groupByProvider={multiProvider && !connectionModels} + preserveOrder={Boolean(connectionModels) || adapterCuratesModelOrder(brandType)} /> )} {efforts.length > 0 && ( - + )}
)} - {SETUP_LOGIN_HINTS[adapterType] && ( + {!aiBinding && SETUP_LOGIN_HINTS[adapterType] && (

{SETUP_LOGIN_HINTS[adapterType]}

)} - {showModel && models.error && ( -

- Couldn’t load models. You can enter a model ID - manually. -

- )} {hasCredentialField && !aiBinding && (
{chooseProvider && ( @@ -1156,7 +1154,7 @@ function Setup({ state={testState} result={result} error={error} - disabled={!ready || busy} + disabled={!ready || busy || Boolean(connectionAdapter && !connection)} onTest={() => void runTest()} /> {error && testState !== "fail" && ( @@ -1170,7 +1168,7 @@ function Setup({ type="button" variant="ghost" disabled={busy} - onClick={() => setScreen("connect")} + onClick={editConnection} > Connection @@ -1178,7 +1176,14 @@ function Setup({ ) : ( )} - : + }
)} diff --git a/ui/src/components/onboarding/CredentialModeLink.tsx b/ui/src/components/onboarding/CredentialModeLink.tsx index f8e1111c583..f700dd88963 100644 --- a/ui/src/components/onboarding/CredentialModeLink.tsx +++ b/ui/src/components/onboarding/CredentialModeLink.tsx @@ -18,9 +18,10 @@ import { LINK_LABEL_FADE_IN, LINK_LABEL_FADE_OUT } from "./onboarding-motion"; */ const LINK_LABEL: Record = { - subscription: "Use API key instead", - api: "Use subscription instead", + subscription: "Use subscription instead", + api: "Use API key instead", }; +const NATIVE_MODES: CredentialMode[] = ["subscription", "api"]; const OTHER_MODE: Record = { subscription: "api", @@ -33,11 +34,20 @@ export function CredentialModeLink({ }: { mode: CredentialMode; onChange: (next: CredentialMode) => void; +}) { + return onChange(OTHER_MODE[mode])} modes={NATIVE_MODES} />; +} + +function ModeLink({ destination, onClick, modes }: { + destination: CredentialMode; + onClick: () => void; + modes: CredentialMode[]; }) { return ( diff --git a/ui/src/components/onboarding/ModelSourceTiles.tsx b/ui/src/components/onboarding/ModelSourceTiles.tsx index 3f33dce8497..ecddf8527ac 100644 --- a/ui/src/components/onboarding/ModelSourceTiles.tsx +++ b/ui/src/components/onboarding/ModelSourceTiles.tsx @@ -21,17 +21,21 @@ import { /** How a source gets authenticated. Every tile is in the same mode at once. */ export type CredentialMode = "subscription" | "api"; +export type ModelConnectionMode = CredentialMode | "advanced"; export type ModelSource = { id: string; label: string; /** The brand mark, rendered into a 30px square. */ icon: ReactNode; + /** Override the row's credential mode; null leaves the tag slot empty. */ + credentialMode?: ModelConnectionMode | null; }; -const CREDENTIAL_TAG_LABEL: Record = { +const CREDENTIAL_TAG_LABEL: Record = { subscription: "Subscription", - api: "API", + api: "API key", + advanced: "Custom Gateway", }; /** @@ -43,7 +47,7 @@ const CREDENTIAL_TAG_LABEL: Record = { * what makes the outgoing label fall out of frame rather than slide past the * tile's padding and over the row below. */ -export function CredentialTag({ mode }: { mode: CredentialMode }) { +export function CredentialTag({ mode }: { mode: ModelConnectionMode }) { return ( @@ -70,7 +74,7 @@ function ModelSourceTile({ settling, }: { source: ModelSource; - mode: CredentialMode; + mode: ModelConnectionMode | null; selected: boolean; onSelect: () => void; buttonRef: (node: HTMLButtonElement | null) => void; @@ -123,7 +127,7 @@ function ModelSourceTile({ {source.label} - + {mode ? : } ); } @@ -138,7 +142,7 @@ export function ModelSourceTiles({ settling = false, }: { sources: ModelSource[]; - mode: CredentialMode; + mode: ModelConnectionMode; /** `null` before anything has been picked — the step opens this way. */ selectedId: string | null; onSelect: (id: string) => void; @@ -220,7 +224,7 @@ export function ModelSourceTiles({ > onSelect(source.id)} settling={settling} diff --git a/ui/src/pages/NewAgent.test.tsx b/ui/src/pages/NewAgent.test.tsx index 32b4906036f..04634cdb9ad 100644 --- a/ui/src/pages/NewAgent.test.tsx +++ b/ui/src/pages/NewAgent.test.tsx @@ -40,7 +40,7 @@ const state = vi.hoisted(() => ({ openNewIssue: vi.fn(), })); const managedApi = vi.hoisted(() => ({ - list: vi.fn(async () => ({ currentUserId: "user-1", connections: [] })), + list: vi.fn(async () => ({ currentUserId: "user-1", canManageConnections: true, connections: [] })), create: vi.fn(async () => ({ connectionId: "managed-connection", grantId: "managed-grant" })), setDefault: vi.fn(async () => ({})), })); @@ -157,6 +157,7 @@ const pass = { }; beforeEach(() => { vi.clearAllMocks(); + managedApi.list.mockResolvedValue({ currentUserId: "user-1", canManageConnections: true, connections: [] }); container = document.createElement("div"); document.body.append(container); root = createRoot(container); @@ -199,7 +200,49 @@ afterEach(async () => { container.remove(); }); describe("New agent setup", () => { - it.each([false, true])("selects the Grok sandbox before connecting (managed-only=%s)", async (managedOnly) => { + it.each(["claude_local", "codex_local"])("can choose a sign-in environment in Configure before connecting %s", async (adapterType) => { + settings.get.mockResolvedValue({ defaultEnvironmentId: "no-login" }); + envApi.list.mockResolvedValue([ + { id: "no-login", name: "No browser sign-in", status: "active", driver: "sandbox", config: { provider: "other" } }, + { id: "login-env", name: "Browser sign-in", status: "active", driver: "sandbox", config: { provider: "daytona" } }, + ]); + envApi.capabilities.mockResolvedValue({ sandboxProviders: { daytona: { supportsLoginPty: true } } }); + api.getAdapterAuthSignal.mockResolvedValue({ status: "missing" }); + await render(adapterType); + expect(container.textContent).toContain("does not support browser sign-in"); + expect(container.querySelector('select[aria-label="Environment"]')).toBeNull(); + await click("2Configure"); + expect(container.querySelector('button[type="submit"]')).toBeNull(); + expect([...container.querySelectorAll("button")].find(button => button.textContent?.trim() === "Run test")?.disabled).toBe(true); + await click("Connect model"); + expect(api.testEnvironment).not.toHaveBeenCalled(); + expect(api.hire).not.toHaveBeenCalled(); + await click("2Configure"); + const select = container.querySelector('select[aria-label="Environment"]') as HTMLSelectElement; + await act(async () => { + select.value = "login-env"; + select.dispatchEvent(new Event("change", { bubbles: true })); + }); + await settle(); + expect(container.querySelector('select[aria-label="Environment"]')).toBeNull(); + await click("Complete subscription login"); + await click("Finish setup"); + expect(api.hire.mock.calls[0][1]).toMatchObject({ defaultEnvironmentId: "login-env", runtimeConfig: { aiConnection: { method: "subscription" } } }); + }); + it("waits for the resolved environment before allowing a credential-mode selection", async () => { + let resolveEnvironments!: (value: object[]) => void; + envApi.list.mockImplementationOnce(() => new Promise(resolve => { resolveEnvironments = resolve; })); + await render("codex_local"); + expect(container.textContent).toContain("Loading connection settings"); + expect(container.querySelector('[role="radiogroup"][aria-label="Connection type"]')).toBeNull(); + await act(async () => resolveEnvironments([{ id: "local-1", name: "Local", driver: "local", config: {} }])); + await settle(); + await click("OpenAIAPI key"); + expect(container.querySelector('[aria-label="API key"]')).not.toBeNull(); + await settle(); + expect(container.querySelector('[aria-label="API key"]')).not.toBeNull(); + }); + it.each([false, true])("selects the Grok environment in Configure (managed-only=%s)", async (managedOnly) => { settings.getExperimental.mockResolvedValue({ enableNativeRunner: true, enableManagedSandboxOnly: managedOnly }); envApi.list.mockResolvedValue([ { id: "local-1", name: "Local", status: "active", driver: "local", config: {} }, @@ -210,6 +253,9 @@ describe("New agent setup", () => { secret: { companyId: "company-1", status: "active" }, }]); await render("paperclip_runner", "grok"); + expect(container.querySelector('select[aria-label="Environment"]')).toBeNull(); + await click("GrokAPI key"); + await click("Use saved API key"); const select = container.querySelector('select[aria-label="Environment"]') as HTMLSelectElement; expect(select.disabled).toBe(false); expect([...select.options].some((option) => option.value === "local-1")).toBe(!managedOnly); @@ -218,8 +264,10 @@ describe("New agent setup", () => { select.dispatchEvent(new Event("change", { bubbles: true })); }); await settle(); - await click("GrokAPI"); - await click("Use saved API key"); + if (container.querySelector('[role="radiogroup"][aria-label="Connection type"]')) { + await click("GrokAPI key"); + await click("Use saved API key"); + } expect(api.testEnvironment).toHaveBeenCalledWith("company-1", "paperclip_runner", expect.objectContaining({ environmentId: "grok-sandbox" })); await click("Finish setup"); expect(api.hire.mock.calls[0][1].defaultEnvironmentId).toBe("grok-sandbox"); @@ -259,13 +307,13 @@ describe("New agent setup", () => { api.getAdapterAuthSignal.mockResolvedValue({ status: "missing" }); api.testEnvironment.mockResolvedValue({ ...pass, adapterType }); await render(adapterType, "grok"); - expect(container.textContent).toContain("Connect Atlas to Grok"); + expect(container.textContent).toContain("Connect a model"); + expect(container.textContent).not.toContain("Connect Atlas to Grok"); if (method === "subscription") { await click("GrokSubscription"); await click("Complete subscription login"); } else { - await click("Use API key insteadUse subscription insteadUse API key instead"); - await click("GrokAPI"); + await click("GrokAPI key"); await fill("API key", "example-test-secret"); await click("Connect"); expect(managedApi.create).toHaveBeenCalledWith("company-1", expect.objectContaining({ @@ -375,7 +423,8 @@ describe("New agent setup", () => { }); it("uses the shared Grok connection flow and hides ignored Kimi and OpenCode effort controls", async () => { await render("grok_local"); - expect(container.textContent).toContain("Connect Atlas to Grok"); + expect(container.textContent).toContain("Connect a model"); + expect(container.textContent).not.toContain("Connect Atlas to Grok"); await render("opencode_local"); expect(container.querySelector('[aria-label="Thinking effort"]')).toBeNull(); }); @@ -462,14 +511,15 @@ describe("New agent setup", () => { ["paperclip_runner", "codex", "OpenAI", "OPENAI_API_KEY"], ])("stores %s %s as a reusable connection before hiring", async (adapter, runner, provider, key) => { await render(adapter, runner); - await click("Use API key insteadUse subscription insteadUse API key instead"); - await click(provider + "API"); + await click(provider + "API key"); await fill("API key", "connection-key"); await click("Connect"); const binding = { provider: key === "ANTHROPIC_API_KEY" ? "anthropic" : "openai", method: "api_key", mode: "responsible_user" }; expect(managedApi.create).toHaveBeenCalledWith("company-1", expect.objectContaining({ apiKey: "connection-key", provider: binding.provider })); expect(api.testEnvironment.mock.calls[0][2].testCredentials).toEqual({}); expect(api.testEnvironment.mock.calls[0][2].aiConnection).toEqual(binding); + expect(container.textContent).toContain("Configure your agent"); + expect(container.querySelector('[role="combobox"][aria-label="Connection"]')).toBeNull(); expect(secrets.createUserSecretDefinition).not.toHaveBeenCalled(); await click("Finish setup"); expect(api.hire.mock.calls[0][1].runtimeConfig.aiConnection).toEqual(binding); @@ -487,7 +537,7 @@ describe("New agent setup", () => { secret: { companyId: "company-1", status: "active" }, }]); await render(adapter, runner); - await click(provider + "API"); + await click(provider + "API key"); expect((container.querySelector("select[aria-label='Saved API key']") as HTMLSelectElement).value).toBe("user:existing-key"); await click("Use saved API key"); const binding = { type: "user_secret_ref", key, version: "latest" }; @@ -499,6 +549,71 @@ describe("New agent setup", () => { expect(secrets.createMyUserSecret).not.toHaveBeenCalled(); expect(secrets.rotateMyUserSecret).not.toHaveBeenCalled(); }); + it.each(["codex_local", "paperclip_runner"])("keeps a saved connection chosen in Connect when hiring %s", async (adapter) => { + await render(adapter, "codex"); + managedApi.list.mockResolvedValue({ + currentUserId: "user-1", + connections: [{ + id: "00000000-0000-4000-8000-000000000021", grantId: "00000000-0000-4000-8000-000000000022", companyId: "company-1", + provider: "openrouter", method: "api_key", name: "Company OpenRouter", + ownership: "shared", isDefault: false, status: "connected", + routing: { kind: "openrouter", protocol: "responses", auth: "bearer", models: [] }, + }], + } as any); + await click("AdvancedCustom Gateway"); + const picker = container.querySelector('[role="combobox"][aria-label="Connection"]')!; + await act(async () => picker.dispatchEvent(new KeyboardEvent("keydown", { key: "ArrowDown", bubbles: true }))); + await settle(); + const option = [...document.querySelectorAll('[role="option"]')].find(node => node.textContent?.includes("Company OpenRouter")) as HTMLElement; + expect(option).toBeTruthy(); + await act(async () => option.click()); + await settle(); + await click("Use connection"); + expect(container.textContent).toContain("Configure your agent"); + expect(container.querySelector('[role="combobox"][aria-label="Connection"]')).toBeNull(); + await click("Run test"); + const binding = { provider: "openrouter", method: "api_key", mode: "shared", connectionId: "00000000-0000-4000-8000-000000000021", grantId: "00000000-0000-4000-8000-000000000022" }; + expect(api.testEnvironment.mock.calls[0][2].aiConnection).toEqual(binding); + await click("Finish setup"); + expect(api.hire.mock.calls[0][1].runtimeConfig.aiConnection).toEqual(binding); + }); + it("requires the current connection choice to connect before Configure can save it", async () => { + await render("codex_local"); + managedApi.list.mockResolvedValue({ currentUserId: "user-1", connections: [{ + id: "00000000-0000-4000-8000-000000000021", grantId: "00000000-0000-4000-8000-000000000022", companyId: "company-1", + provider: "openrouter", method: "api_key", name: "Company OpenRouter", ownership: "shared", isDefault: false, status: "connected", + routing: { kind: "openrouter", protocol: "responses", auth: "bearer", models: [] }, + }] } as any); + await click("AdvancedCustom Gateway"); + const picker = container.querySelector('[role="combobox"][aria-label="Connection"]')!; + await act(async () => picker.dispatchEvent(new KeyboardEvent("keydown", { key: "ArrowDown", bubbles: true }))); + await settle(); + const option = [...document.querySelectorAll('[role="option"]')].find(node => node.textContent?.includes("Company OpenRouter")) as HTMLElement; + await act(async () => option.click()); + await settle(); + await click("2Configure"); + expect(container.textContent).not.toContain("Finish setup"); + expect(api.hire).not.toHaveBeenCalled(); + await click("Connect model"); + await click("OpenAIAPI key"); + await fill("API key", "new-api-credential"); + await click("2Configure"); + expect(container.textContent).not.toContain("Finish setup"); + expect(api.testEnvironment).not.toHaveBeenCalled(); + await click("Connect model"); + await click("OpenAIAPI key"); + await fill("API key", "new-api-credential"); + await click("Connect"); + await click("Connection"); + await click("2Configure"); + expect(container.textContent).not.toContain("Finish setup"); + await click("Connect model"); + await click("OpenAIAPI key"); + await fill("API key", "new-api-credential"); + await click("Connect"); + await click("Finish setup"); + expect(api.hire.mock.calls[0][1].runtimeConfig.aiConnection).toEqual({ provider: "openai", method: "api_key", mode: "responsible_user" }); + }); it.each(["pi_local"])( "persists %s OpenRouter credentials only as a secret reference", async (adapter) => { @@ -556,10 +671,13 @@ describe("New agent setup", () => { await settle(); expect(document.querySelector('[role="dialog"]')).toBeNull(); expect(managedApi.setDefault).not.toHaveBeenCalled(); + expect(api.hire).not.toHaveBeenCalled(); + expect(api.testEnvironment).not.toHaveBeenCalled(); expect(managedApi.create).toHaveBeenCalledWith("company-1", expect.objectContaining({ provider: "openrouter", method: "api_key", apiKey: "example-test-secret", })); - const binding = { provider: "openrouter", method: "api_key", mode: "delegated", connectionId: "managed-connection", grantId: "managed-grant" }; + const binding = { provider: "openrouter", method: "api_key", mode: "shared", connectionId: "managed-connection", grantId: "managed-grant" }; + await click("Run test"); expect(api.testEnvironment.mock.calls[0][2]).toEqual(expect.objectContaining({ aiConnection: binding, testCredentials: {}, @@ -656,7 +774,7 @@ describe("New agent setup", () => { { type: "paperclip_runner", loaded: true, disabled: true }, ]; await render("paperclip_runner"); - await connect("OpenAI"); + expect(container.textContent).toContain("This adapter is unavailable"); expect(api.testEnvironment).not.toHaveBeenCalled(); expect(api.hire).not.toHaveBeenCalled(); }); diff --git a/ui/src/pages/apps/Browse.test.tsx b/ui/src/pages/apps/Browse.test.tsx index edf7af00674..0961c1d20b1 100644 --- a/ui/src/pages/apps/Browse.test.tsx +++ b/ui/src/pages/apps/Browse.test.tsx @@ -251,6 +251,29 @@ describe("Connectors landing page", () => { return client; } + it("shows provider catalog rows and splits legacy gateway accounts by API format", async () => { + const slugs = ["notion", "openrouter", "bedrock", "google", "responses-api", "messages-api", "chat-completions-api", "local"]; + listGalleryMock.mockResolvedValue({ apps: slugs.map(slug => getAppStoreDefinition(slug)) }); + listApplicationsMock.mockResolvedValue({ applications: [application({ id: "gateway-app", name: "Model gateway", applicationKey: "app-gallery:gateway", metadata: { sourceTemplateKey: "gateway" } })] }); + listConnectionsMock.mockResolvedValue({ connections: ["responses", "messages"].map(protocol => connection({ + id: protocol, name: `${protocol} account`, applicationId: "gateway-app", connectionPurpose: "ai", + config: { sourceTemplateKey: "gateway", ai: { provider: protocol === "messages" ? "anthropic" : "openai", method: "api_key", routing: { kind: "gateway", protocol, auth: "bearer", baseUrl: "https://models.example.com/v1", models: [] } } }, + })) }); + await renderBrowse(); + expect(container.textContent).not.toContain("Connect a model provider"); + expect(container.textContent).not.toContain("Model gateway"); + for (const slug of slugs) expect(container.querySelector(`[data-app-slug="${slug}"]`)).not.toBeNull(); + const responses = container.querySelector('[data-app-slug="responses-api"]')!; + const messages = container.querySelector('[data-app-slug="messages-api"]')!; + expect(responses.textContent).toContain("responses account"); + expect(responses.textContent).not.toContain("messages account"); + expect(messages.textContent).toContain("messages account"); + const bedrock = container.querySelector('[data-app-slug="bedrock"]')!; + const connect = Array.from(bedrock.querySelectorAll('button')).find(button => button.textContent?.includes("Connect"))!; + await act(() => connect.click()); + expect(navigateMock).toHaveBeenCalledWith("/apps/connect?source=bedrock"); + }); + it("offers assistant setup from Connections without choosing an agent", async () => { await renderBrowse(); const button = container.querySelector('[aria-label="Set up Assistant Connection (MCP)"]'); diff --git a/ui/src/pages/apps/Browse.tsx b/ui/src/pages/apps/Browse.tsx index f79226594fd..987dadaa8d8 100644 --- a/ui/src/pages/apps/Browse.tsx +++ b/ui/src/pages/apps/Browse.tsx @@ -292,11 +292,11 @@ function connectorAction( }; } if (chatHref) return { label: "Connect", href: chatHref }; - // PAP-659 C4: the card's verb comes from the same four-state resolver the - // connect screen uses, so "Connect" never turns out to mean "paste a key". + // AI accounts share the Connect action across subscription and key methods. + // Tool connectors retain their capability-specific setup verbs. if (row.entry) { return { - label: connectionSetupVerbForApp(row.entry), + label: row.entry.methods?.some(method => method.purpose === "ai") ? "Connect" : connectionSetupVerbForApp(row.entry), href: connectHrefFor(row.entry), }; } @@ -601,7 +601,8 @@ export function Browse({ renderAccountDetails = (connection) => connection.conne const applicationSlug = appApplicationSourceSlug(application); const savedAppConnections = connectionsByApplicationId.get(application.id) ?? []; - const appConnections = savedAppConnections.filter( + if (applicationSlug === "gateway" && savedAppConnections.length === 0) continue; + let appConnections = savedAppConnections.filter( (connection) => !GOOGLE_CONNECTOR_SLUGS.has(appConnectionSourceSlug(connection) ?? ""), ); // Hide source-only Google rows, but keep independently identified connectors. @@ -610,6 +611,19 @@ export function Browse({ renderAccountDetails = (connection) => connection.conne savedAppConnections.length > 0 && appConnections.length === 0 ) continue; + // One legacy gateway application may contain several API formats. Group + // each saved AI account by its actual routing, not the old application slug. + const ungroupedCount = appConnections.length; + appConnections = appConnections.filter((connection) => { + if (connection.connectionPurpose !== "ai") return true; + const slug = appConnectionSourceSlug(connection); + const row = slug ? rowsBySlug.get(slug) : undefined; + if (!row) return true; + if (!row.applications.some((item) => item.id === application.id)) row.applications.push(application); + row.connections.push(connection); + return false; + }); + if (ungroupedCount > 0 && appConnections.length === 0) continue; const configuredConnectionSlug = appConnections .map( (connection) => diff --git a/ui/src/pages/apps/app-connect-policy.test.ts b/ui/src/pages/apps/app-connect-policy.test.ts index 481ce0e4c6b..9eb3c919f6b 100644 --- a/ui/src/pages/apps/app-connect-policy.test.ts +++ b/ui/src/pages/apps/app-connect-policy.test.ts @@ -32,6 +32,10 @@ describe("app connect policy", () => { expect(canEnterAppsConnect(new URLSearchParams("source=context7"))).toBe(false); expect(canEnterAppsConnect(new URLSearchParams("source=zapier"))).toBe(true); expect(canEnterAppsConnect(new URLSearchParams("source=unknown"))).toBe(false); + expect(canEnterAppsConnect(new URLSearchParams("source=model-provider"))).toBe(false); + for (const source of ["responses-api", "messages-api", "chat-completions-api", "bedrock", "local", "google", "openrouter"]) { + expect(canEnterAppsConnect(new URLSearchParams({ source })), source).toBe(true); + } expect(canEnterAppsConnect(new URLSearchParams("byo=1"))).toBe(false); expect(canEnterAppsConnect(new URLSearchParams("byo=1&source=zapier"))).toBe(false); }); diff --git a/ui/src/pages/apps/app-connect-policy.ts b/ui/src/pages/apps/app-connect-policy.ts index b54f8c9e7c1..fd5384a1cf3 100644 --- a/ui/src/pages/apps/app-connect-policy.ts +++ b/ui/src/pages/apps/app-connect-policy.ts @@ -67,6 +67,7 @@ export function canEnterAppsConnect( ); } const source = searchParams.get("source") ?? ""; + if (["google-ai", "gateway"].includes(source)) return true; // Router connectors are registered by installed plugins, not the static // catalog. AppsConnect resolves this reserved slug against the host gallery // before rendering the pool wizard; unknown/uninstalled routers show its diff --git a/ui/src/pages/apps/app-definition-display.ts b/ui/src/pages/apps/app-definition-display.ts index 2c86abbe55a..b7772797781 100644 --- a/ui/src/pages/apps/app-definition-display.ts +++ b/ui/src/pages/apps/app-definition-display.ts @@ -1,5 +1,5 @@ +import { aiConnectionCatalogSlug, aiConnectionMetadataSchema, type AppDefinition, type ToolApplication, type ToolConnection } from "@paperclipai/shared"; import { aiConnectionRouterSlug, aiConnectionRouterPluginKey } from "@paperclipai/shared"; -import type { AppDefinition, ToolApplication, ToolConnection } from "@paperclipai/shared"; export type AppGalleryDisplayEntry = AppDefinition & { key?: string; @@ -50,6 +50,10 @@ export function appApplicationSourceSlug(application: ToolApplication | null | u export function appConnectionSourceSlug(connection: ToolConnection | null | undefined): string | null { if (!connection) return null; + if (connection.connectionPurpose === "ai") { + const metadata = aiConnectionMetadataSchema.safeParse(connection.config?.ai); + if (metadata.success) return aiConnectionCatalogSlug(metadata.data.provider, metadata.data.routing); + } const router = aiConnectionRouterPluginKey(connection); if (router) return aiConnectionRouterSlug(router); const source = connection.config?.sourceTemplateKey ?? connection.transportConfig?.sourceTemplateKey; diff --git a/ui/storybook/prototypes/agent-settings/AgentSettingsPreview.tsx b/ui/storybook/prototypes/agent-settings/AgentSettingsPreview.tsx index d8065003655..bca3a2a3ec3 100644 --- a/ui/storybook/prototypes/agent-settings/AgentSettingsPreview.tsx +++ b/ui/storybook/prototypes/agent-settings/AgentSettingsPreview.tsx @@ -4,6 +4,7 @@ import { useRef, useState, type ComponentProps, + type ReactNode, } from "react"; import { Route, Routes, useLocation, useNavigate } from "react-router-dom"; import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query"; @@ -67,11 +68,14 @@ export function AgentSettingsPreview({ adapterType = "claude_local", testOutcome = "pass", saveFails = false, + runtimeContent, }: { initialTab?: AgentLocalDetailView; adapterType?: string; testOutcome?: TestOutcome; saveFails?: boolean; + /** Storybook-only extension point for reviewing proposed runtime controls in the existing page. */ + runtimeContent?: ReactNode; }) { const [fixtures] = useState(() => createSettingsFixtures(adapterType, testOutcome, saveFails), @@ -115,7 +119,7 @@ export function AgentSettingsPreview({ } + element={} />
{ if ( dirty && @@ -267,6 +271,16 @@ function SettingsPage() {
+
+ + {agent.name} · Agent navigation + + +
)} - {(view === "runtime" || view === "secrets") && ( -
-
- -
- {view === "runtime" && ( - <> - test.current?.()} - /> - {feedback.login && ( - test.current?.()} - onApplyStored={() => test.current?.()} + {view === "runtime" && runtimeContent + ? runtimeContent + : (view === "runtime" || view === "secrets") && ( +
+
+ +
+ {view === "runtime" && ( + <> + test.current?.()} + /> + {feedback.login && ( + test.current?.()} + onApplyStored={() => test.current?.()} + /> + )} + )} - +
)} -
- )} {view === "permissions" && ( -

- {saving - ? "Saving changes…" - : dirty - ? "You have unsaved changes." - : saved - ? "Changes saved." - : ""} -

-
- - -
- - )} + ) && + !(view === "runtime" && runtimeContent) && ( +
+

+ {saving + ? "Saving changes…" + : dirty + ? "You have unsaved changes." + : saved + ? "Changes saved." + : ""} +

+
+ + +
+
+ )}
diff --git a/ui/storybook/prototypes/provider-routing/AgentSetup.tsx b/ui/storybook/prototypes/provider-routing/AgentSetup.tsx new file mode 100644 index 00000000000..18639ba8fef --- /dev/null +++ b/ui/storybook/prototypes/provider-routing/AgentSetup.tsx @@ -0,0 +1,573 @@ +import { useEffect, useRef, useState } from "react"; +import { Button } from "@/components/ui/button"; +import { RuntimeTestCard, type TestState } from "@/components/RuntimeTestCard"; +import { ModelSourceTiles } from "@/components/onboarding/ModelSourceTiles"; +import { CredentialModeLink } from "@/components/onboarding/CredentialModeLink"; +import { AdapterMark } from "@/components/AdapterMark"; +import { + AdapterTypeDropdown, + ModelDropdown, +} from "@/components/AgentConfigForm"; +import { Field } from "@/components/agent-config-primitives"; +import { ConnectionChoiceList } from "@/features/connections/ConnectionChoiceList"; +import { AiConnectionLegacyNotice } from "@/components/ai-connections/AiConnectionManagement"; +import { AgentSettingsPreview } from "../agent-settings/AgentSettingsPreview"; +import { ProviderApiKeyCard } from "@/components/AdapterLoginChrome"; +import { ConnectionSetup } from "./ConnectionSetup"; +import { ConnectionPicker } from "./ConnectionPicker"; +import { + AdvancedOptions, + Choice, + Footer, + Notice, + Surface, + TextField, +} from "./shared"; +import { + compatibility, + connections, + defaultConnection, + harnesses, + modelsFor, + type Connection, + type Harness, +} from "./model"; + +export type AgentProps = { + harness?: Harness; + runner?: "new" | "legacy"; + initialConnectionId?: string; + addedConnection?: Connection; + initialModel?: string; + initialTest?: TestState; + scenario?: + | "normal" + | "empty" + | "loading" + | "catalog-error" + | "denied" + | "missing-personal" + | "legacy" + | "readonly"; + testFailure?: "tools" | "network"; + newAgent?: boolean; + initialAdvanced?: boolean; +}; + +const adapterTypes: Record = { + codex: "codex_local", + claude: "claude_local", + opencode: "opencode_local", + grok: "grok_local", + pi: "pi_local", + gemini: "gemini_local", + kimi: "kimi_local", + hermes: "hermes_local", + cursor: "cursor", + copilot: "copilot_local", +}; +const excludedAdapters = new Set([ + "process", + "http", + "openclaw_gateway", + "hermes_gateway", + "claude_managed", + "aws_agentcore", + "acpx_local", + "paperclip_runner", + "cursor_cloud", +]); + +export function AgentSetup(props: AgentProps) { + const content = ; + return props.newAgent ? ( + content + ) : ( + + ); +} + +function AgentRoutingFields({ + harness: initialHarness = "codex", + runner = "new", + initialConnectionId, + addedConnection, + initialModel, + initialTest = "idle", + scenario = "normal", + testFailure, + newAgent, + initialAdvanced = false, +}: AgentProps) { + const startingConnection = + addedConnection ?? + connections.find((entry) => entry.id === initialConnectionId) ?? + defaultConnection(initialHarness); + const [harness, setHarness] = useState(initialHarness); + const [rows, setRows] = useState( + addedConnection + ? [ + ...connections.filter((entry) => entry.id !== addedConnection.id), + addedConnection, + ] + : scenario === "empty" + ? [] + : connections, + ); + const [connectionId, setConnectionId] = useState(startingConnection.id); + const [model, setModel] = useState( + initialModel ?? modelsFor(startingConnection, harness)[0], + ); + const [manual, setManual] = useState(false); + const [modelOpen, setModelOpen] = useState(false); + const [advancedOpen, setAdvancedOpen] = useState(initialAdvanced); + const [effort, setEffort] = useState("default"); + const [environment, setEnvironment] = useState("cloud"); + const [state, setState] = useState(initialTest); + const [catalogError, setCatalogError] = useState( + scenario === "catalog-error", + ); + const [creating, setCreating] = useState(false); + const [saved, setSaved] = useState(false); + const [discarded, setDiscarded] = useState(false); + const timer = useRef | undefined>(undefined); + useEffect(() => () => clearTimeout(timer.current), []); + const selected = rows.find((entry) => entry.id === connectionId); + const availableModels = selected ? modelsFor(selected, harness) : []; + const unknownModel = !manual && !availableModels.includes(model); + const reason = selected ? compatibility(selected, harness) : undefined; + const denied = scenario === "denied" || scenario === "missing-personal"; + const readOnly = scenario === "readonly"; + const nativePending = + runner === "new" && + !harnesses.find((entry) => entry.value === harness)!.native; + const invalid = + !selected || + !!reason || + unknownModel || + !model.trim() || + denied || + nativePending || + readOnly; + const change = () => { + clearTimeout(timer.current); + setState("idle"); + setSaved(false); + setDiscarded(false); + }; + const error = + testFailure === "network" + ? "This endpoint is unreachable from the selected environment. Check network access or choose another environment." + : "The model responded, but the tool-call test failed. Choose a model with tool support or check the gateway’s API format."; + const test = () => { + setState("running"); + timer.current = setTimeout( + () => setState(testFailure ? "fail" : "pass"), + 400, + ); + }; + + if (creating) + return ( + setCreating(false)} + onComplete={(connection) => { + setRows((current) => [ + ...current.filter((entry) => entry.id !== connection.id), + connection, + ]); + setConnectionId(connection.id); + setCreating(false); + change(); + }} + /> + ); + return ( + +
+ + { + const next = (Object.keys(adapterTypes) as Harness[]).find( + (key) => adapterTypes[key] === value, + ); + if (next) { + setHarness(next); + change(); + } + }} + /> + +
+ {nativePending && ( + + This harness is not yet available through Paperclip Runner. Select a + supported harness or configure its legacy adapter. + + )} + {scenario === "legacy" && ( + setCreating(true)} + /> + )} + {scenario === "loading" ? ( +

+ Loading your connections… +

+ ) : selected ? ( + <> + !compatibility(entry, harness))} + disabled={readOnly} + onSelect={(id) => { + if (id === connectionId) return; + setConnectionId(id); + setManual(false); + change(); + }} + onConnect={() => setCreating(true)} + /> + {reason && ( + {reason} Choose a compatible connection. + )} + {denied && ( + + {scenario === "missing-personal" + ? "Sam has no credential for Engineering gateway. Sam needs to connect to this destination, or choose an authorized company credential." + : "You don’t have access to this company credential. Ask its owner for access or choose another connection."} + + )} + + ) : ( + + No compatible connections yet. Connect a provider to choose a model + for Nova. + + )} + {!selected && ( + + )} + {selected && scenario !== "loading" && ( + <> + {catalogError ? ( + +

Couldn’t load models from this connection.

+ + + ) : ( +
+ ({ id, label: id }))} + value={model} + open={modelOpen} + onOpenChange={setModelOpen} + allowDefault={false} + required + groupByProvider={selected.provider === "openrouter"} + creatable={advancedOpen} + onChange={(value) => { + setManual(!availableModels.includes(value)); + setModel(value); + change(); + }} + /> +
+ )} + {catalogError && advancedOpen && ( + { + setManual(true); + setModel(event.target.value); + change(); + }} + hint="Use the exact ID accepted by this connection. Custom model metadata is managed on the connection." + /> + )} + {unknownModel && !catalogError && ( + + Your previous model is unavailable through {selected.name}. Choose + a model to continue. Your harness is still{" "} + {harnesses.find((entry) => entry.value === harness)!.label}. + + )} + { + setEffort(value); + change(); + }} + options={[ + { value: "default", label: "Model default" }, + { value: "low", label: "Low" }, + { value: "high", label: "High" }, + ]} + /> + { + setEnvironment(value); + change(); + }} + options={[ + { value: "cloud", label: "Company AWS workspace" }, + { value: "local", label: "Local machine" }, + ]} + /> + + +

+ To use a custom model ID or gateway alias, enter it in the Model + picker above. +

+ + + + )} + {saved && ( + + {scenario === "legacy" + ? "Managed connection adopted." + : "Nova’s configuration saved."}{" "} + {selected?.name} · {model} + + )} + {discarded && Changes discarded.} +