test(connections): add repeatable live provider qualification

Exercise Apps and new-agent setup with real credentials, isolated targets, browser login handoffs, independently verified artifacts and follow-ups. Preserve safe failure diagnostics, private evidence, cleanup, provenance and incomplete coverage.

Co-Authored-By: Paperclip <noreply@paperclip.ing>

doc/evals.md

@@ -1,5 +1,9 @@
 # Paperclip evaluation guide
 
+The explicit-only [live provider connection suite](../tests/runner-e2e/PROVIDER-CONNECTIONS.md)
+is a Product E2E workflow for fresh subscription/API-key/gateway connections,
+with attended login and independent artifact checks against local or staging targets.
+
 Paperclip has two live eval families with different questions, owners, and
 evidence. Choose the family before selecting a model, profile, or case.
 

doc/plans/2026-10-03-live-provider-connection-qualification.md

@@ -0,0 +1,211 @@
+# Live provider connection qualification
+
+Date: 2026-10-03  
+Status: Runnable implementation delivered; operating contract and initial verification are in [PROVIDER-CONNECTIONS.md](../../tests/runner-e2e/PROVIDER-CONNECTIONS.md). Live qualification is reported per campaign, not implied by this plan.  
+Code inspected: `527e3bad2` (`codex/provider-routing-connections`).
+
+## Decision
+
+Extend the existing Playwright Product E2E harness with real connection-creation journeys, an explicit authentication handoff, and support for an existing deployment as the target. Run the subscription journeys in a visible browser on a developer's machine first. The same tests should target local Paperclip and staging.
+
+Playwright owns navigation, assertions, evidence, and the verdict. A person completes login challenges when necessary. A computer-use agent can assist with diagnosis and provider UI changes, but it must not silently change the steps or decide whether the test passed. An assisted success is useful evidence and must be labeled as assisted.
+
+The deliverable is a matrix answering **which configuration worked, against which deployment, with what proof**. A successful login, a saved connection, and a successful agent run are separate checkpoints.
+
+## Existing foundation and gaps
+
+- [`tests/runner-e2e`](../../tests/runner-e2e/README.md) already supplies real browser/server/runner/provider tests, catalog selection, isolated fixtures, independent grading, reports, cost accounting, and provenance. This is the appropriate family, as defined in [`doc/evals.md`](../evals.md).
+- Its [Playwright configuration](../../tests/runner-e2e/playwright.config.ts) currently owns a fresh loopback server and database. Supporting staging requires a target lifecycle abstraction, not just substituting a URL.
+- [`tests/ai-connections-app/app.spec.ts`](../../tests/ai-connections-app/app.spec.ts) includes simulated login sessions and popup destinations. It also retains local terminal-login assertions that predate this branch's browser-login changes. Keep fast simulated tests, update those stale expectations, and label their limited proof clearly.
+- [`tests/hiring-ai-connections/hiring.live.spec.ts`](../../tests/hiring-ai-connections/hiring.live.spec.ts) already checks real runs, connection attribution, and legacy/native execution. It creates initial API-key connections through the API and requires a loopback `local_trusted` instance. Reuse the assertion patterns; it is not a complete connection-onboarding test or a staging harness.
+- The existing Grok subscription qualification stages supplied `GROK_AUTH_JSON`. Its documentation correctly says this does not test interactive browser login.
+
+## Coverage contract
+
+Inventory harnesses separately from providers. Claude is a harness; Bedrock and OpenRouter are connection routes. A Paperclip deployment URL, an authentication host, and an agent execution environment are also separate dimensions.
+
+The initial inventory below comes from [managed connection capabilities](../../packages/shared/src/ai-connections.ts), [routing compatibility](../../packages/shared/src/ai-provider-routing.ts), and the [UI adapter registry](../../ui/src/adapters/registry.ts). It describes implementation eligibility, not proven live coverage.
+
+| Harness | Managed subscription | Direct API key | Advanced routes currently admitted |
+| --- | --- | --- | --- |
+| Claude Code | Yes | Anthropic | OpenRouter, Bedrock, Messages-compatible gateway |
+| Codex | Yes | OpenAI | OpenRouter, Responses-compatible gateway |
+| Grok | Yes; qualify its actual login UX separately | xAI | Not supported by the current routing compatibility table |
+| Gemini | Not in the managed connection capability table | Google | Not in the current routing compatibility table |
+| OpenCode | No managed subscription entry | OpenRouter | OpenRouter, Chat Completions-compatible gateway |
+| Hermes local | No managed subscription entry | Inventory harness-specific behavior | OpenRouter, Chat Completions-compatible gateway |
+| Cursor local, Kimi, Pi | Inventory required | Inventory required | Inventory required |
+
+“Not in the table” is a Paperclip managed-connection limitation, not a claim about everything the upstream CLI can do. Inventory native runner providers and installed first-party adapters as well as legacy UI adapters; any additional local provider must receive a manifest entry before the full campaign can qualify.
+
+Exclude remote OpenClaw and Hermes gateways, Cursor Cloud, managed Claude/AWS AgentCore, process/HTTP adapters, and the generic legacy ACPX integration from this campaign. Their remote systems own provider configuration. **Hermes local remains in scope.** Supported native Claude/Grok runner profiles that internally use ACPX remain in scope; that transport is distinct from the excluded generic legacy integration.
+
+For every supported harness/auth/route combination, enumerate the supported legacy/native runner profiles and target environments. Do not infer that an API-key pass covers subscription, that legacy covers native, or that local covers staging. Protocol routes need a real configured endpoint and credential; a mock gateway is useful for regression tests but does not supply live qualification.
+
+Keep three independent manifest fields:
+
+- Eligibility: supported, unsupported/product gap, needs inventory, or deliberately excluded with a reason.
+- Execution: not run, awaiting user, missing credential, blocked target, running, complete.
+- Qualification: passed or failed, with individual checkpoint results and assistance recorded.
+
+A credential-free catalog check must flag newly added local harnesses or changed capabilities without a reviewed coverage entry. Report required, selected, attempted, and passed counts separately. Unsupported configurations are visible product gaps, never green tests.
+
+## Authentication strategy
+
+Use three lanes, with separate results:
+
+| Lane | Starting state | What it proves |
+| --- | --- | --- |
+| Fresh connection, existing browser sign-in | Saved provider website/Google browser session; no Paperclip connection or runtime credential for this cell | The current Paperclip subscription flow obtains and saves a fresh usable credential |
+| Fresh connection, signed-out browser | Empty provider browser state; no Paperclip connection or runtime credential | The complete first-time experience, including account selection and login |
+| Existing connection | A connection previously created by the live UI journey | Selection, reuse, continuation, and reconnect behavior |
+
+Most developer campaigns use the first lane. The signed-out lane runs explicitly before releases or login changes. The existing-connection lane cannot substitute for either creation lane.
+
+There are three distinct identities/stores: Paperclip board authentication for the selected target, provider website authentication in the browser, and the resulting managed connection credential. Reusing provider website sign-in reduces friction while still exercising a fresh OAuth/device authorization and production credential persistence. Do not import `~/.codex`, `~/.claude`, or an existing token into a fresh-connection test.
+
+Initially, use the developer's existing subscription accounts through a dedicated QA browser profile, separate from their everyday Chrome profile. Keep account aliases and expected identities in private local configuration; do not put personal email addresses into the public catalog. Serialize tests sharing an account or refresh-token chain. Never copy the profile between CI workers.
+
+### Login checkpoint
+
+The flow starts the real production login session, arms popup/new-tab observation before clicking, and tracks the expected provider origin and Paperclip session. It handles ordinary redirects and device-code/code-entry steps using the actual UI. Provider and Google pages can require a person for account choice, passwords, passkeys, MFA, CAPTCHA, or consent.
+
+At such a boundary:
+
+1. Show the relevant window and mark the cell `awaiting_user`, with the provider, target, and expected account alias.
+2. Pause only the affected account's queue. Keep a bounded deadline tied to the real login session's expiry.
+3. Let the person complete the production browser flow. If the provider rejects the automated browser, support handoff to a normal browser using the product's existing device URL or code-return flow. Do not add authentication bypasses.
+4. Resume from observed Paperclip login status and saved-connection state. Clicking “continue” is never sufficient proof of successful authorization.
+5. If the session expires, retain that attempt and use the product's retry action to start a new session. Never replay an old callback code.
+
+Record which step needed assistance and its duration, without recording passwords, codes, callback query strings, or provider session IDs. Check the expected account using available provider/account metadata; where this cannot be machine-verified, explicitly record human identity confirmation rather than claiming an automated identity check.
+
+There is no promise that Google or provider sign-in will remain unattended indefinitely. Expired browser state is an operational login requirement, not a passing test. A bug that loses a completed callback is a product failure, not automatically “needs login.”
+
+## Local and staging target adapters
+
+Introduce two lifecycle modes while sharing the same journey:
+
+- **Managed local:** create and own an isolated Paperclip home, database, server, company, and runtime. Teardown may stop/delete only these owned resources.
+- **Attach existing:** authenticate to a configured local/staging URL and create a dedicated QA company or use an explicitly designated test company. Never reset the deployment, change instance-wide feature settings, install plugins, or stop its server as fixture setup.
+
+The current local `3109` application can be an attach-existing target. Preserve its BOB company and user data. Full first-company onboarding belongs on a fresh managed target or a dedicated disposable staging tenant/instance; a shared staging deployment cannot prove that path merely by creating another agent.
+
+Each private target configuration supplies an exact origin, target mode, expected deployment identity/revision, board-auth account reference, QA company policy, and available execution environment. The staging URL is still to be supplied. Target validation precedes credential resolution. Verify redirects, health, authorization, installed harness versions, required capabilities, and environment availability; do not send credentials to a host introduced by a redirect or unreviewed target configuration.
+
+Record the tested server revision separately from the harness revision, plus runner/CLI version, immutable sandbox image when applicable, model ID, protocol, authentication host, and execution environment. Staging with a missing capability is blocked, not silently tested through a different route. Unknown build provenance prevents a release qualification claim.
+
+## Required browser journey and independent proof
+
+For each admitted cell:
+
+1. Create/select the isolated fixture company and a synthetic workspace through existing public surfaces. Ordinary bootstrap APIs may prepare the fixture; they must not create the connection or agent that the browser journey is supposed to test.
+2. Enter the production new-agent flow, select the harness and Subscription/API Key/Advanced mode, and create the connection through the UI. Advanced cases supply the configured provider/protocol, endpoint or region, and selected credential reference. Verify that permissions default to everyone/all agents within the isolated test company and that advanced controls retain their values.
+3. Verify the saved connection and compatible connection selector after reload. For a fresh-login cell, confirm that a new connection was actually created and that the selected connection is the one bound to the agent.
+4. Choose the model and environment in Configure, run the product's test action, and finish setup. Verify the model list/custom model path and selected values survive reload.
+5. Create a real task through the browser. Ask the agent to read a small synthetic input file, compute a deterministic result, and save a result artifact. Generate the input independently and keep the expected result out of the prompt. Seed the workspace using existing environment fixture facilities, not a private server hook.
+6. Require the real tool receipt, persisted successful run, task completion, correct artifact bytes/hash, and a user-visible result link that opens after reload. “Provider routing works” or any other model-authored success sentence alone cannot pass.
+7. Correlate the run with the selected connection, auth method, harness/runner, route, model, and execution environment using durable production evidence. Reuse the existing run-context assertions where available. Any missing attribution is an evidence gap to implement explicitly, not something to infer from the model's response.
+8. Send one bounded follow-up that uses the saved result and produces a second independently checked output. This proves the saved credential remains usable beyond the setup probe.
+9. Clean up only campaign-owned resources, retain sanitized results, and report cleanup failures separately. Failed cleanup prevents an entirely clean campaign result but does not erase successful functional evidence.
+
+Use a controlled runtime without ambient provider keys or personal CLI homes so a broken selected credential cannot fall back to another account. On staging, explicitly qualify the runtime's credential isolation. Add a negative case with a deliberately invalid selected credential and require an actionable failure with no successful fallback. For a controlled gateway, correlate sanitized request metadata when available; never retain authorization headers.
+
+Also cover the other product entry points: Apps connector row → Connect → connection list → agent selection; existing agent → switch compatible connection; and reconnect after an expired/invalid connection. Apply every supported auth method to both Apps and agent-creation entry points. Fresh company onboarding and destructive recovery cases are separately enumerated; their target requirements must be visible. A representative smoke selection is useful, but is not the full matrix.
+
+## Trustworthy results and safe evidence
+
+Extend the existing Product E2E result/report pipeline with authentication checkpoints, assistance, target identity, and coverage denominator. Preserve the existing failure classification: product, model/provider behavior, grading/evidence, or infrastructure. Operational blocks are separate from these failure causes.
+
+Keep first failures and all attempts. Permit only declared bounded infrastructure retries; do not let an agent change selectors or retry a failed product flow until the campaign looks green. A repaired test starts a new definition/revision measurement.
+
+Example report shape (illustrative, not results):
+
+| Cell | Target | Login | Saved | Tool/artifact | Follow-up | Outcome |
+| --- | --- | --- | --- | --- | --- | --- |
+| Claude subscription/native | Local | Assisted | Pass | Pass | Pass | Pass, assisted |
+| Codex subscription/native | Staging | Needs reauthentication | — | — | — | Awaiting user |
+| Grok custom gateway | Local | — | — | — | — | Unsupported/product gap |
+
+Disable automatic Playwright traces, HARs, screenshots, videos, request-body logging, and DOM dumps for the live authentication flow, including failure handlers. Masked password fields do not protect network traces, tokens, or callback URLs. Exception messages and reporters also need sanitized wrappers; do not serialize raw Playwright call logs containing filled values or navigated URLs. Calibrate redaction with synthetic sentinel credentials before using real accounts.
+
+Keep browser profiles and auth state outside the repository and artifact directories, using OS-protected storage or restrictive local permissions. Persist only selected provider state; never include it in reports, caches, or CI uploads. Resolve one cell's required credentials at a time from a private secret store; API-key fixtures may reference existing local secrets without sourcing the entire secrets file into the server environment.
+
+After authentication, allow only the existing synthetic-task screenshots and structured allowlisted evidence. The [current public evidence policy](../../tests/runner-e2e/SECURITY.md#public-evidence-boundary) explicitly excludes credential and setup pages; this feature must not widen that boundary. Auth diagnosis uses sanitized phase/status metadata, not screenshots of personal accounts. Read-only report regeneration must make no additional provider calls.
+
+## Implementation slices
+
+1. **Manifest and target support.** Add an explicit-only `provider-connections` Product E2E suite; register coverage and profiles in the existing catalog/fixture registry. Add owned-local/attach-existing target lifecycles without changing existing suites' defaults. Include credential-free tests for origin validation, coverage drift, selection, cleanup ownership, and blocked results.
+2. **One complete subscription path.** Implement the authentication checkpoint and Claude subscription journey locally, then Codex. Prove each with a fresh UI-created connection, independent task artifact, reload, and follow-up. Prove both saved-browser-sign-in and signed-out lanes. Fix stale local-login browser assertions alongside this work.
+3. **Same journey on staging.** Supply the staging target and board account; replay the Claude/Codex journeys against its actual login host and runner environment. Record deployment differences rather than building a separate staging script.
+4. **API keys and advanced routes.** Add direct keys, OpenRouter, Bedrock, and each supported protocol gateway. Include wrong-key/no-fallback, mismatched protocol, model discovery, and custom alias cases. Bedrock has a selected region/model and an expiring credential; an expired credential is renewed before the case and never counted as a product pass.
+5. **Complete the inventory.** Add Grok's actual subscription and key paths, Gemini, OpenCode, Hermes local, and supported Cursor/Kimi/Pi/native-provider cases. Resolve unsupported auth paths as explicit product decisions; do not invent a third mode for every harness. Expand entry points and runner/environment variants until every required manifest cell has evidence.
+6. **Operationalize.** Start with a manually launched visible-browser campaign on the developer's machine. Later use dedicated subscription accounts on a trusted QA machine, plus unattended API-key/gateway jobs in the existing protected paid CI workflow. Keep subscription reauthentication an explicit attended operation whenever the provider requires it.
+
+The suite, target configuration, and authentication selectors are now implemented; use the commands in [PROVIDER-CONNECTIONS.md](../../tests/runner-e2e/PROVIDER-CONNECTIONS.md). Discovery, verification, and reports reuse `test:e2e:runner`. The authoritative README/FIXTURES/SECURITY docs and `doc/evals.md` link the operating contract.
+
+Each case declares a turn limit covering the setup probe and follow-up, a provider-run timeout, a separate login deadline, company/agent budget limits, and cleanup policy. Run one subscription case per account at a time. Present selected cells and estimated spend before paid execution; record actual usage and unknown cost separately. Do not call subscription usage “free,” and do not run the full matrix as a side effect of normal unit tests or `--all`.
+
+## Acceptance criteria
+
+- One catalog and journey implementation selects local or staging through configuration.
+- At least one real fresh Claude and Codex subscription connection passes through each target before claiming the initial slice complete; no mocked popup/session or preinstalled credential substitutes for this proof.
+- Human handoff, timeout, expired login, and account mismatch produce distinct, resumable outcomes without false passes.
+- Passing requires the selected connection's persisted binding, real successful run, independently validated tool result/artifact, and a successful subsequent use.
+- Credential-free grader tests reject a plausible success message without a tool result, a wrong artifact, wrong connection attribution, and absent evidence.
+- Missing credentials, unsupported combinations, and unrun cells remain visible in the coverage denominator. Assisted and unattended results remain distinguishable.
+- Existing local/staging user data survives; teardown owns only test resources.
+- Secret sentinel tests prove that failure artifacts and report projections do not disclose login state or credentials.
+- The full inventory is qualified only after all required supported cells have current evidence. Earlier slices are explicitly partial.
+
+## External references
+
+- [Playwright pages and popups](https://playwright.dev/docs/pages): popup/new-page events provide deterministic browser coordination.
+- [Playwright authentication](https://playwright.dev/docs/auth): authenticated browser state can be reused, is sensitive, and expires. Reusing website state does not require bypassing Paperclip's connection flow.
+- [Google OAuth policies](https://developers.google.com/identity/protocols/oauth2/policies): account authorization must use an appropriate browser flow. Google login restrictions and challenges are not something this harness should attempt to bypass.
+
+
+## Verification update · 2026-10-05
+
+Six previously failing local key/gateway journeys now pass: legacy Grok from
+both entry points, legacy Codex/new-agent OpenRouter, legacy OpenCode/Apps
+OpenRouter, and native OpenCode/OpenRouter from both entry points. Gemini
+selected-model startup, current local skill-root guidance, and repair-run
+artifact attribution have been corrected. Its final two browser retests still
+failed the 300-second first-task deadline, so Gemini remains unqualified.
+The affected ACP regression group passes 221 tests and Grok's two affected files
+pass 54 tests. The ACP checks include a real missing-file protocol reproduction,
+canonical local Gemini workspaces, preserved remote paths, and permission-denial
+guards. The existing ACPX patch now returns resource-not-found for genuine
+missing files; it keeps other errors and permissions intact. Workspace
+typecheck/build and token gates pass; the full
+Vitest suite has not produced a passing result after broad timeout failures.
+
+The pinned native Grok installation still requires administrator access.
+Attended subscription completion, staging/Daytona, and qualification against a
+single reviewed committed build remain outstanding. See
+[the qualification runbook](../../tests/runner-e2e/PROVIDER-CONNECTIONS.md) for
+retained attempts, exact evidence boundaries, and replay instructions. This
+update does not mark the overall acceptance criteria complete.
+
+## Gemini investigation update · 2026-10-06
+
+The new-agent/API-key workflow passes with `gemini-3.5-flash-lite`, including the
+independent first artifact and follow-up checks. The Apps diagnostic retest
+delivered its first artifact, then timed out on follow-up with no recognized
+error signal; that cause remains unknown. Earlier deadline failures include provider HTTP 503 overload
+retries; increasing the deadline would not resolve the underlying uncertainty.
+
+A new credential-free installed-CLI smoke distinguishes correct ACP missing-file
+semantics from Gemini CLI 0.58.0's plain-error-object conversion bug. A temporary
+CLI copy with that narrow conversion fixed passes creation, existing-file reads,
+and external-path denial; the user's installed CLI remains unchanged. Explicit
+thinking-effort configuration also needs a supported startup-settings path;
+short comparisons do not show that default `HIGH` thinking caused the deadlines.
+
+The harness now projects terminal errors into closed diagnostic codes before
+managed-instance teardown, retaining no error text or hidden reasoning. Fresh
+managed targets preserve the original stopped onboarding database. Results,
+cleanup, model/source provenance, and remaining qualification boundaries are
+recorded in the private report and the runbook. The full acceptance criteria
+remain open.

tests/runner-e2e/FIXTURES.md

@@ -1,5 +1,11 @@
 # Runner E2E fixture authoring
 
+## Connection creation fixtures
+
+See [PROVIDER-CONNECTIONS.md](PROVIDER-CONNECTIONS.md) for the explicit-only
+`provider-connections` suite, local/staging target ownership, dedicated browser
+profiles, credential handoffs, private evidence, and cleanup contract.
+
 The fixture catalog is executable production-contract data. Keep it small,
 typed, deterministic, and free of raw credentials.
 

tests/runner-e2e/PROVIDER-CONNECTIONS.md

@@ -0,0 +1,483 @@
+# Live provider connections
+
+`provider-connections` exercises real connection creation through the Apps catalog
+or new-agent wizard, then verifies a real agent task and follow-up. It is an
+explicit-only Product E2E suite. It is excluded from `--all` and the paid GitHub
+matrix: subscription login initially runs attended on a developer's machine.
+
+Tasks are submitted through the production composer. When it accepts only a
+prompt, the harness assigns its fixture title through the public API. This
+changes naming metadata only; execution, status, attachments and downloaded
+bytes are produced and verified independently.
+
+## Run one journey
+
+Create a private JSON configuration outside the repository. Its values reference
+credentials; do not paste keys into it. For example:
+
+```json
+{
+  "target": { "mode": "managed-local" },
+  "browser": {
+    "headed": true,
+    "channel": "chrome",
+    "profileDir": "/absolute/private/path/provider-qa-browser",
+    "accountAlias": "developer-qa",
+    "freshness": "signed-in",
+    "loginTimeoutMs": 600000
+  },
+  "secretFile": "/absolute/private/path/provider-qa.env",
+  "models": { "codex": "YOUR_CODEX_MODEL", "claude": "YOUR_CLAUDE_MODEL" },
+  "routes": {
+    "openrouter": { "model": "YOUR_OPENROUTER_MODEL" },
+    "bedrock": { "model": "YOUR_BEDROCK_MODEL", "region": "YOUR_AWS_REGION" },
+    "responses": {
+      "baseURL": "https://your-gateway.example/v1",
+      "credentialEnv": "QA_RESPONSES_KEY",
+      "model": "YOUR_GATEWAY_MODEL",
+      "auth": "bearer"
+    }
+  },
+  "budgetCents": 200,
+  "maxRuns": 3,
+  "turnTimeoutMs": 300000
+}
+```
+
+The secret file must be an owner-only regular file (`chmod 600`), with literal
+`NAME=value` entries. Only the selected variable is read. Shell substitutions
+are rejected. Environment variables take precedence. Default names are
+`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `XAI_API_KEY`, `GEMINI_API_KEY`,
+`OPENROUTER_API_KEY`, and `AWS_BEARER_TOKEN_BEDROCK`. Override direct-key names
+with `credentials: {"codex": "DEDICATED_QA_OPENAI_KEY"}`. Subscription cases
+read no provider API key and never import CLI credential homes.
+
+Routes may be shared by method or overridden per harness, such as
+`"claude.openrouter"`. Protocol gateways require their own explicit credential
+reference, or `auth: "none"` for a deliberately unauthenticated endpoint.
+Messages gateways also support `auth: "api_key"` (`x-api-key`). Bedrock requires
+a current bearer token and region. The harness does not mint or refresh AWS
+tokens. Missing credentials/configuration produce a blocked result.
+
+```sh
+pnpm test:e2e:runner -- --list --suite provider-connections
+pnpm test:e2e:runner -- \
+  --id provider-connections.connection-codex-native.local.agent-api-key \
+  --connection-config /absolute/private/path/connections.json
+pnpm test:e2e:runner -- \
+  --id provider-connections.connection-claude-legacy.local.apps-subscription \
+  --connection-config /absolute/private/path/connections.json
+```
+
+Start with one cell. Running `--suite provider-connections` selects all 116
+eligible cells and can spend money; it does not make missing routes available.
+Each command generates a separate campaign, preserving previous attempts.
+
+## Authentication handoff
+
+Playwright drives Paperclip and opens the real provider sign-in page. When login
+needs a person, the terminal prints the provider and account alias and writes
+`progress.json`. Complete Google/provider login, MFA, consent, and any code
+entry in the visible QA browser. The test resumes only after Paperclip reports
+a saved, connected credential. Do not paste passwords or OAuth codes into chat.
+
+The QA browser uses its own private profile. It refuses an existing unmarked
+profile, keeps a lock while running, and never uses your normal Chrome profile.
+`signed-in` permits previously saved website login; it still creates a fresh
+Paperclip connection. `signed-out` starts a temporary empty browser profile.
+Paperclip SSO and provider SSO may share an identity provider, so this records
+the browser's initial state, not a guarantee of another password challenge.
+
+Human help is recorded as `assisted`. A login deadline records `awaiting_user`,
+not a pass, and stops the selected campaign before opening another login.
+Rerun the remaining cell IDs to start a fresh authorization after expiry;
+resumption is within the running process, not across a stopped process.
+Ctrl-C closes the test browser pages, cancels recorded login attempts and active
+runs, performs fixture teardown, records interruption, and stops the campaign.
+Each cell has a thirty-minute outer deadline. A forced process kill cannot run
+teardown; inspect the private QA profile lock and target fixtures before rerunning.
+If Google rejects the automated browser, use the production device-code or
+code-return flow in your normal browser. Grok's current production flow may
+require the displayed terminal sign-in command. These are attended results.
+
+## Local and staging targets
+
+`managed-local` owns a fresh server/database and removes its data on shutdown.
+To use an existing local server or staging, replace `target` with:
+
+```json
+{
+  "mode": "attach",
+  "baseURL": "https://YOUR-INSTANCE.staging.paperclip.app",
+  "expectedCommit": "EXACT_40_CHARACTER_DEPLOYED_COMMIT_SHA",
+  "deploymentMode": "authenticated"
+}
+```
+
+For local attach use `http://127.0.0.1:PORT` and `local_trusted`. Health must
+match the expected revision and deployment mode before credentials are read.
+The target is never reset, restarted, or given instance-setting changes.
+Sign in to Paperclip in the QA browser if needed; that session supplies the
+same public API permissions as the browser. The operator needs company/agent
+creation, connection management, and company archival permissions.
+
+Deployment and execution environment are different dimensions. `.local.` means
+the agent runs in the target's local environment, including a staging server's
+local environment. `.daytona.` requires an already configured active Daytona
+environment on the selected target; this suite does not provision one. Use
+`target.companyId` and `target.environmentId` to select an empty dedicated
+company named `Connection QA…` with that environment. For a new QA company the
+harness uses the matching visible instance environment. Missing environments
+or a disabled native runner are explicit target blockers.
+
+Each cell creates a separate QA company unless one was supplied. Teardown
+pauses its agents, cancels active runs, revokes its test connections, and
+archives the created company. This avoids requiring hard-delete support on
+staging and retains inspectable task history. Supplied QA companies must start
+without active agents or usable connections; revoked connection records and
+terminated agents from earlier attempts are allowed. Their new test agents are
+deleted and test connections revoked; historical records may remain. `retainCompany: true`
+keeps the fixture and credentials for diagnosis, with agents paused, and is
+recorded explicitly in evidence. Clean it up before reusing that company.
+
+## What qualifies as a pass
+
+Every checkpoint must be verified:
+
+1. Expected target revision and deployment mode.
+2. Fresh connection saved with the selected method/provider/route.
+3. Connection visible after navigating away and reloading Apps.
+4. Agent created through the wizard with the selected harness/model/binding.
+5. Real Configure-page environment probe passes.
+6. Real task run succeeds with that exact managed connection's attribution.
+7. Agent decodes random input bytes supplied in the task, computes their sum/count/hash, and delivers
+   an attachment whose JSON is checked independently by the harness.
+8. Tool evidence and attachment authorship/run attribution are present.
+9. A browser-submitted follow-up reuses the connection and delivers independently
+   checked minimum/maximum/sum output.
+
+The oracle input is base64 in the task description and must be decoded with a
+tool, preserving its exact bytes. This makes connection qualification independent
+of new-task attachment ingestion. The earlier uploaded-input OpenRouter attempt
+remains a failed attachment-path test: native staging currently admits chat
+attachments, and the new-task dialog uploads its files after scheduling the agent.
+Successful provider cells do not qualify that task-level upload path.
+10. Downloadable artifacts remain visible after a page reload; cleanup succeeds.
+
+An answer saying “provider routing works” cannot pass. Evidence records the
+configured model and agent configuration; connection attribution proves the
+selected credential, not an upstream gateway's internal model mapping.
+
+The matrix covers Claude, Codex, Grok, Gemini, OpenCode, and Hermes local where
+Paperclip's managed-connection capability table admits them, with applicable
+legacy/native variants and both UI entry points. Cursor local, Kimi, and Pi
+remain visible product gaps for managed connection onboarding. Remote OpenClaw
+and Hermes gateways, Cursor Cloud, managed Claude/AWS AgentCore, process/HTTP,
+and generic legacy ACPX are deliberately excluded.
+
+## Reports, privacy, and CI
+
+Results are under `tests/runner-e2e/results/connections-…/`: `dashboard.html`,
+`campaign.json`, `normalized-results.json`, `connection-coverage.json`, `progress.json`, and packaged
+per-cell evidence. The report uses the existing Product E2E grader, billing,
+history, and dashboard. Coverage reports required/selected/attempted/passed
+counts separately. A partial local campaign does not qualify staging or the
+whole matrix. Blocked credentials/login are incomplete; cleanup failures fail.
+
+No Playwright traces, HARs, videos, automatic screenshots, raw assertion call
+logs, OAuth URLs/codes, or provider transcripts are retained. Only the final
+synthetic task can be screenshotted; it is private and has no public publication
+marker. Auth browser state stays outside the repository. Keep reports private.
+The result includes source revision, target revision, suite definition hash,
+model, method, assistance, run IDs, checkpoints, timing, and numeric usage.
+Before removing a managed instance, the harness retains terminal run status,
+log availability, and closed diagnostic codes for recognized quota, overload,
+file-conversion, session-configuration, or shell-restriction failures. It reads
+error records and failed tool receipts only; messages, stderr, model output,
+and reasoning are not copied into the result. An empty signal list leaves the
+cause unknown. Missing logs remain explicit and do not imply a healthy run.
+UI probes may incur unreported spend, so billing is explicitly partial.
+Regenerate the report from retained results with
+`pnpm test:e2e:runner:dashboard -- /absolute/path/to/connections-campaign`.
+
+API-key cells can run headless (`browser.headed: false`, `channel: "chromium"`)
+on trusted CI once the target's board authentication is provided through a
+dedicated QA browser session. Subscription cells require headed mode and a
+person when challenged. Dedicated subscription accounts and a trusted attended
+runner are the next operational step; normal untrusted PR jobs never get these
+credentials. Credential-free support tests and catalog checks run separately:
+
+```sh
+pnpm test:e2e:runner:typecheck
+pnpm test:e2e:runner:unit
+```
+
+The opt-in cancellation smoke uses a synthetic loopback page and cookie, with
+no provider calls or credentials. It verifies that Ctrl-C, termination, and
+hangup leave the browser's authenticated API client usable until fixture cleanup
+finishes: `pnpm exec tsx tests/runner-e2e/connection-cancel.smoke.ts`.
+
+## Initial verification (2026-10-03)
+
+The local native Codex/direct OpenAI key/new-agent cell completed both real
+artifact turns with exact connection attribution and successful cleanup. The
+Apps/OpenRouter/native Codex cell created and reused its connection and passed
+its setup probe, then failed the task proof: its agent could not retrieve the
+uploaded input and marked the task blocked. This is not a gateway qualification
+pass. The Claude legacy/Apps subscription journey opened real sign-in and
+exercised the bounded `awaiting_user` outcome; login was not completed.
+
+Credential-free verification: runner E2E typecheck, 855 support tests, both
+updated simulated local-browser-login regressions, catalog discovery, and
+retained-report regeneration passed. A fresh managed-local server was started
+and removed with an intentionally missing credential; no provider was called.
+Staging and the remaining cells have not been qualified. Use the generated
+campaigns for the exact revision, definition hash, timestamps, and results.
+
+## Local follow-up (2026-10-05)
+
+Every eligible non-subscription local cell was attempted against the existing
+onboarding instance, preserving its original company data. The latest retained
+attempts now contain 43 passing journeys out of 46 non-subscription cells,
+including the Gemini investigation on October 6:
+
+| Harness | Passing key/gateway journeys | Eligible local key/gateway journeys | Remaining result |
+| --- | ---: | ---: | --- |
+| Claude | 16 | 16 | Direct key, OpenRouter, Bedrock, and Messages passed through both entry points and runner generations. |
+| Codex | 12 | 12 | The legacy/new-agent OpenRouter task and follow-up passed after the skill-root and artifact-helper fixes. |
+| OpenCode | 8 | 8 | All three remaining OpenRouter journeys passed after the artifact-helper and canonical-workspace permission fixes. |
+| Hermes local | 4 | 4 | A follow-up interrupted by lease release passed on one bounded retry. Workspace isolation was then corrected and retested separately. |
+| Grok | 2 | 4 | Both legacy API-key journeys passed after preserving private session history and avoiding remote subscription restoration for missing local history. Native journeys require the missing pinned runtime. |
+| Gemini | 1 | 2 | New-agent/API-key passed with `gemini-3.5-flash-lite`; Apps/API-key delivered its first artifact but timed out on follow-up. Installed CLI file creation still fails the isolated compatibility check. |
+
+There are another 12 local subscription cells. Attended Claude/OpenAI attempts
+opened real authorization and reached their human handoff, but have not qualified
+a completed subscription journey. Staging and Daytona remain untested. The
+Responses and Messages gateway proofs used the official provider endpoints;
+they do not qualify an Emissary deployment or every compatible gateway.
+
+The run exposed a real new-agent loading race: an early API-key selection could
+be discarded when environment settings finished loading and remounted Connect.
+Connect now waits for those settings before exposing interactive choices. Saved
+connection selectors and native OpenCode model normalization were repaired in
+the harness; original failures remain in their original campaigns. Native
+OpenCode no longer sends the deprecated per-prompt tools override, which replaced
+its session permission policy; its remaining live OpenRouter failures were subsequently fixed and retested below.
+
+Hermes also ignored the assigned task workspace when an agent had no explicit
+`cwd`, writing proof files into the server checkout. It now uses the resolved
+task workspace as its default. Explicit directory overrides retain their existing
+behavior. The real retest checks that the local proof file is in the assigned
+directory and matches the independently downloaded artifact. Earlier checkout
+spill files were preserved privately outside the repository.
+
+An interrupted subscription attempt exposed another cleanup bug: Playwright's
+default signal handler closed the browser before the harness could cancel the
+login using its authenticated API client. The harness now owns browser shutdown,
+and a real-browser smoke verified cleanup under SIGINT, SIGTERM, and SIGHUP.
+An expired human handoff also stops the campaign instead of opening the next
+subscription window. The affected attempt remains failed in the evidence; its
+login was cancelled and its disposable company archived through the public API.
+The cleanup audit covers all 113 disposable companies and confirms that the
+original onboarding company remains available.
+
+Hermes was tested using an isolated official CLI installation. Its temporary
+PATH launcher was removed after qualification; replay requires Hermes to be
+installed on the tested server's PATH. The report records the tested CLI version
+and source commit in `qualification-environment.json`.
+
+Credential-free verification passed: runner E2E typecheck and all 856 support
+tests, 46 new-agent UI tests, 39 native OpenCode driver tests, 10 Hermes invocation
+tests, runner/Hermes package typechecks, and UI token gates. After the final
+browser signal change, runner E2E typecheck, 13 connection support tests, and the
+three-signal browser smoke passed again. This initial qualification stage did
+not repeat repo-wide typecheck, tests, and build; later verification is recorded
+below.
+
+### Credential follow-up, evening of 2026-10-05
+
+The xAI dashboard identified expiry as the original key rejection: its QA key
+expired on September 27. The existing key was renewed for 90 days, through
+January 3, 2027, preserving its scope and limits. A Gemini replacement was saved
+in the owner-only secret file; the temporary exposed key was revoked and its
+absence verified after a dashboard reload. No provider billing settings changed.
+Both provider model-list requests returned HTTP 200. A separate bounded Gemini
+generation request succeeded with `gemini-3.8-flash`; the older
+`gemini-2.5-flash` request returned HTTP 404. Model-list success alone does not
+qualify inference or a product journey.
+
+Six previously credential-blocked cells were rerun, followed by three bounded
+Gemini attempts after new diagnostics and a selector repair. At that stage none added a fully
+qualified journey, leaving 36 of 46 local key/gateway cells passing. Those 91
+attempts and the subsequent repair attempts remain inspectable. These evening
+campaigns have October 6 UTC timestamps and belong to the developer's October 5
+local qualification date.
+
+- Grok legacy, both entry points: the first task delivered an independently
+  verified artifact. Each follow-up reported its session absent locally,
+  attempted remote conversation restoration, and waited for subscription device
+  authorization during an API-key journey. Investigate preserving isolated
+  non-credential session state across disposable managed credential homes.
+- Grok native, both entry points: connection creation and reload passed, but
+  Configure was not reached. An independent non-inference installation probe
+  confirmed the required Grok Build 1.0.13 executable was missing. Provision it
+  using the checked-in helper in `doc/grok-native-runner.md`, then rerun those
+  exact cells; retain the runtime integrity checks.
+- Gemini: `Auto` completed ACP startup but timed out before task output. With
+  `gemini-3.8-flash`, both entry points created and reloaded a connection, passed
+  setup, and started the real task, then failed with
+  `acpx_session_config_failed`. Installed Gemini CLI 0.58.0 rejected
+  `session/set_config_option` for the model with ACP `-32601` (Method not found).
+  Resolve that runtime compatibility before claiming a pass. The Apps selector
+  now reads the shared app definition's name (`Google Gemini`); its original
+  selector failures remain retained.
+
+After the catalog selector repair, runner E2E typecheck and all 13 connection
+support tests passed. The final cleanup audit found no usable QA connections,
+active QA agents, or active QA runs; the original onboarding company and test
+server remain available. The private report includes `credential-refresh.json`,
+`key-retest-diagnostics.json`, `gemini-generation-probe.json`, and
+`cleanup-audit.json`. None contain credential values or authorization codes.
+
+The private `connections-qualification-2026-10-05` dashboard selects the newest
+attempt per cell. `retained-attempts.json` and `attempts.md` retain all original
+campaigns. Source and suite definition hashes are preserved: these runs span
+harness repairs and an uncommitted working tree, so their passing count is
+historical evidence, not a common-build release qualification. Commit the reviewed
+fixes and rerun the required cells against that exact deployed revision before
+claiming the whole feature qualified.
+
+### Repair verification, late evening of 2026-10-05
+
+The previously failing Codex OpenRouter journey, both legacy Grok API-key
+journeys, and all three OpenCode OpenRouter journeys now have passing browser
+retests with independently downloaded artifacts and successful follow-ups.
+Original failures remain in the report; no assistant success message alone
+qualifies a journey.
+
+The fixes refresh Codex's current skill root on resumed turns, keep artifact
+helper temporary files inside the selected workspace, allow native OpenCode's
+assigned workspace under both its supplied and canonical filesystem paths,
+and preserve Grok session transcripts in private company/agent/credential-scoped
+storage while keeping credentials disposable. Missing managed local Grok
+history starts a fresh task handoff instead of triggering remote subscription
+recovery during an API-key turn.
+
+Gemini now receives its selected model through `GEMINI_MODEL` before ACP startup
+instead of an unsupported `session/set_config_option` call. Real tasks confirmed
+startup and tool execution. The original `gemini-3.8-flash` attempts subsequently
+hit daily inference quota; a small request for `gemini-3.1-flash-lite` succeeded.
+These are distinct from model-list and connection-probe success.
+
+Local Gemini turns now also receive the current selected skill root and
+explicit instructions to read the Paperclip artifact workflow before making
+control-plane calls. A real follow-up exposed Gemini CLI's rejection of inline
+command substitution when building a JSON completion request. Its local turn
+guidance now uses workspace JSON body files with `curl --data-binary @file` or
+the selected skill's helpers; the shell restriction remains in place. Regression checks cover current roots across disposable
+homes for both Codex and Gemini. The shared ACP regression file now passes all
+214 tests after isolating Gemini test homes from the developer's CLI state and
+adding canonical-workspace and remote-workspace regressions.
+
+A credential-free real ACP reproduction isolated two filesystem compatibility
+problems behind Gemini's file-write errors. The client now translates a genuine
+missing-file read into the correct ACP resource-not-found error. This verifies
+the client protocol response; the installed Gemini CLI still mishandles that
+response, as isolated below. Local Gemini sessions also bind the filesystem client to
+the canonical workspace directory. Remote workspace paths are preserved. The
+existing ACPX dependency patch and its lockfile hash contain this narrow fix;
+four real protocol regressions cover missing-file semantics, external path denial, deny-all
+permissions, and other filesystem errors. The full affected ACP group passes
+221 tests, including the existing three spawn safeguards. An earlier combined
+verification failure and the failed browser attempts remain retained.
+
+The final filesystem-fix campaign (`connections-2026-10-06T04-50-27-196Z-1ec2a9`)
+still failed both Gemini entry points at the unchanged 300-second first-task
+deadline. Connection creation, reload, Configure probe, and binding passed.
+The new-agent turn created its local proof files, but neither turn completed
+the successful run, uploaded-artifact, and follow-up qualification. The earlier
+filesystem and model-configuration errors were absent from these tool receipts;
+that does not qualify the full workflow. Further work must investigate Gemini
+task execution latency and skill use with the configured CLI/model, then rerun
+the same independent oracle. No timeout increase or cancelled-run pass was used.
+
+A further Gemini attempt exposed a harness settlement race: the first successful
+turn did not attach the file, while an automatic disposition-repair run did.
+The harness now waits for active repair runs within the existing configured
+limits and selects the successful run that actually uploaded the required file.
+It still rejects cancelled runs, wrong connection/agent attribution, wrong
+artifact bytes, and exceeded run/cost limits. The repair-attribution regression
+suite passed (58 tests across the targeted support/catalog files).
+
+The task-composer helper was updated for the current production controls and
+its remembered project selection. A credential-free real browser smoke proved
+Plan and Ask mode, assignee/project selection, fixture task naming, and file
+upload. Public-API title updates only name fixtures; execution, credentials,
+model selection and artifact delivery remain real product behavior.
+
+Full workspace typecheck and build passed. UI token gates and shell syntax
+passed. The broad Vitest attempt did not produce a green full-suite result:
+stale catalog/pool assertions were corrected and checked narrowly, while
+additional worker-start, hook and process timeouts remain recorded. Serial
+adapter/server rechecks were interrupted after more deadline failures; they do
+not constitute a passing full-suite run. UI/CLI and
+shared/database/skill-catalog groups passed. Further isolated checks do not
+substitute for a successful complete PR verification run.
+
+The pinned native Grok executable is still absent at
+`/opt/paperclip/providers/grok/1.0.13/grok`. Installing it needs administrator
+access; retain the checked-in provisioning helper and its checksum verification.
+Subscriptions, staging/Daytona, and common-build release qualification remain
+outstanding.
+
+## Gemini investigation (2026-10-06)
+
+The installed Gemini CLI is 0.58.0. A credential-free check now exercises its
+actual ACP file tools against the patched ACPX client, with a loopback model
+response and disposable home/workspace:
+
+```sh
+pnpm exec tsx tests/runner-e2e/gemini-acp-filesystem.smoke.ts
+```
+
+On the stock CLI, reading an existing file and rejecting an external write pass;
+creating a new file fails. Its ACP SDK rejects a plain JSON error object, while
+`AcpFileSystemService.normalizeFileSystemError` reads the message only from an
+`Error` instance. Consequently it misses the genuine resource-not-found message
+and fails the initial read-before-write. The same implementation is present in
+the [upstream source](https://github.com/google-gemini/gemini-cli/blob/main/packages/cli/src/acp/acpFileSystemService.ts).
+A temporary copy that also reads a plain object's string `message` passes all
+three checks. The installed CLI has not been changed, and the real browser
+qualification uses the stock CLI. A workflow that recovers using shell tools
+does not qualify the broken native file tool.
+
+Retained child-stderr metadata also confirms HTTP 503 overload retries in the
+earlier deadline failures. A bounded availability probe returned HTTP 503 for
+`gemini-3.8-flash` and valid HTTP 200 output for `gemini-3.5-flash-lite`.
+With Flash-Lite, the full new-agent/API-key journey passed, including two
+independently downloaded and verified artifacts and an attributed successful
+follow-up. The initial Apps/API-key journey created, reloaded, tested, and bound
+the connection, then failed its first task. It did not retain a cause before
+managed-instance teardown; that gap prompted the closed diagnostic projection
+above. One diagnostic retest (`connections-2026-10-06T13-25-36-718Z-20ee5d`)
+delivered and independently verified the first artifact, then exceeded the
+unchanged 300-second follow-up deadline. Both run statuses and log availability
+were retained before cleanup, but neither log contained a recognized error
+signal. The follow-up cause remains unknown; overload in other runs is not
+proof of overload in this run. No deadline or oracle was relaxed.
+
+The CLI requests `HIGH` thinking by default. Short bounded comparisons did not
+establish that default thinking causes the long workflow delay. Explicit effort
+configuration still uses an ACP method this CLI rejects; supported startup
+`modelConfigs` settings need a separate adapter fix and local/remote validation.
+Do not silently drop the user's setting or claim that a model-list/setup probe
+qualifies execution.
+
+The original 3109 instance is currently stopped, and its temporary config and
+encryption key are missing. Its database was left untouched; substituting a new
+key would not recover the existing credentials. Fresh investigation campaigns
+use managed instances with separate databases and keys, archive their owned
+companies, and remove their owned servers. The private consolidated report
+retains all earlier failures, exact model/source provenance, and
+`gemini-investigation.json`. These mixed working-tree attempts do not qualify
+staging or a common committed release build.

tests/runner-e2e/README.md

@@ -1,5 +1,11 @@
 # Paid runner full-stack E2E
 
+## Live provider connection journeys
+
+See [PROVIDER-CONNECTIONS.md](PROVIDER-CONNECTIONS.md) for the explicit-only
+`provider-connections` suite, local/staging target ownership, dedicated browser
+profiles, credential handoffs, private evidence, and cleanup contract.
+
 For family selection, ownership, provenance, history, and failure taxonomy,
 see the [Paperclip evaluation guide](../../doc/evals.md). This README is the
 authoritative runbook for Product E2E runner cells; the separate Runner Evals

tests/runner-e2e/SECURITY.md

@@ -1,5 +1,19 @@
 # Runner E2E security for a public repository
 
+## Attended connection authentication
+
+See [PROVIDER-CONNECTIONS.md](PROVIDER-CONNECTIONS.md) for the explicit-only
+`provider-connections` suite, local/staging target ownership, dedicated browser
+profiles, credential handoffs, private evidence, and cleanup contract.
+
+Provider-connection results may retain closed diagnostic codes and terminal run
+status before managed-instance teardown. Only error records and failed tool
+receipts are inspected; raw error messages, child stderr, model output and
+reasoning are not retained. Unknown causes and unavailable logs stay explicit.
+The optional installed-Gemini filesystem smoke uses synthetic credentials and
+loopback model responses, makes no real provider calls, and removes its
+disposable home/workspace.
+
 This suite can spend provider money, expose selected local and workflow API credentials to isolated
 test processes, publish a container, retain private visual evidence, and write
 public structured evidence. Treat changes to the workflow, harness, fixture

tests/runner-e2e/case-outcome.ts

@@ -1,4 +1,5 @@
 import type { RunnerE2EResult } from "./types.js";
+import { isBlockedConnectionResult } from "./connection-evidence.js";
 const html = (value: unknown) =>
   String(value ?? "").replace(
     /[&<>"']/g,
@@ -12,6 +13,7 @@ export function isIncompleteFirstTaskResult(
   result: RunnerE2EResult,
   errors: readonly string[] = [],
 ) {
+  if (isBlockedConnectionResult(result)) return result.cleanup === "passed" && result.failureClass !== "secret_leak" && errors.length === 0;
   const checks = result.firstTask?.checks ?? [];
   return result.status === "failed" &&
     result.failureClass === "candidate_failure" &&
@@ -30,7 +32,7 @@ export function renderCaseOutcome(
     (result.matcherResults ?? []).map((m) => ({
       id: m.matcher.kind === "json_path" ? m.matcher.path : m.matcher.kind,
       passed: m.passed,
-      notReached: undefined as string | undefined,
+      notReached: isBlockedConnectionResult(result) && !m.passed ? "Blocked before this checkpoint" : undefined as string | undefined,
       detail: m.detail,
     }));
   const failures = checks.filter((c) => !c.passed && !c.notReached);
@@ -54,7 +56,8 @@ export function renderCaseOutcome(
     ),
   ];
   return `<section class="case-outcome ${failed ? "outcome-failed" : "outcome-passed"}" aria-label="Case result explanation">
-    <strong>${incomplete ? "Incomplete journey" : failed ? "Overall failed" : "Overall passed"} · ${checks.length ? `${passed}/${evaluated} behavioral checks passed${notReached.length ? ` · ${notReached.length} not reached` : ""}` : "No behavioral checks recorded"}</strong>
+    <strong>${isBlockedConnectionResult(result) ? `Blocked: ${html(result.providerConnection!.outcome.replaceAll("_", " "))}` : incomplete ? "Incomplete journey" : failed ? "Overall failed" : "Overall passed"} · ${checks.length ? `${passed}/${evaluated} behavioral checks passed${notReached.length ? ` · ${notReached.length} not reached` : ""}` : "No behavioral checks recorded"}</strong>
+    ${result.providerConnection ? `<p>Target: ${html(result.providerConnection.target.origin)} · ${html(result.providerConnection.target.commit ?? "revision unavailable")} · ${result.providerConnection.assisted ? "Assisted login" : "Unassisted"} · ${html(result.providerConnection.authFreshness)} browser · ${html(result.providerConnection.entry)} / ${html(result.providerConnection.method)} · stopped at ${html(result.providerConnection.phase)}</p>` : ""}
     ${failures.length ? `<p>Failed checks:</p><ul>${failures.map((c) => `<li><strong>${html(c.id)}</strong> — ${html(c.detail)}</li>`).join("")}</ul>` : checks.length ? `<p>No behavioral matcher failed.${notReached.length ? " The journey is incomplete; unexercised checks are not passes." : failed ? " The overall failure came from a separate run, cleanup, or evidence check." : ""}</p>` : ""}
     ${notReached.length ? `<p>Not reached:</p><ul>${notReached.map((c) => `<li><strong>${html(c.id)}</strong> — ${html(c.notReached)}</li>`).join("")}</ul>` : ""}
     ${checks.length ? `<details><summary>See all behavioral checks (${checks.length})</summary><table class="matchers"><thead><tr><th>Result</th><th>Check</th><th>Detail</th></tr></thead><tbody>${checks.map((c) => `<tr class="matcher-${c.notReached ? "not-reached" : c.passed ? "passed" : "failed"}"><td>${c.notReached ? "Not reached" : c.passed ? "Pass" : "Fail"}</td><td><code>${html(c.id)}</code></td><td>${html(c.detail)}${c.notReached ? ` — ${html(c.notReached)}` : ""}</td></tr>`).join("")}</tbody></table></details>` : ""}

tests/runner-e2e/catalog.test.ts

@@ -136,10 +136,10 @@ describe("runner E2E catalog", () => {
     expect(localIntegrityTasks).toHaveLength(2);
     expect(openRouterBreadthTasks).toHaveLength(3);
     expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
-      6, 30, 3, 16, 16, 2, 6, 8, 46, 23, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1,
+      6, 30, 3, 16, 16, 2, 6, 8, 46, 23, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1, 116,
     ]);
-    expect(validateRunnerCatalog()).toHaveLength(478);
-    expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(478);
+    expect(validateRunnerCatalog()).toHaveLength(594);
+    expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(594);
     expect(
       runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
     ).toHaveLength(48);
@@ -428,13 +428,13 @@ describe("runner E2E catalog", () => {
 
   it("uses only declared secret references in generated payloads", () => {
     expect(
-      runnerMatrix.every((entry) =>
+      runnerMatrix.filter(entry => entry.task.flow !== "provider_connection").every((entry) =>
         entry.requiredCredentials.includes(entry.profile.credential),
       ),
     ).toBe(true);
     expect(
       runnerMatrix
-        .filter((entry) => entry.environment.id === "daytona")
+        .filter((entry) => entry.environment.id === "daytona" && entry.task.flow !== "provider_connection")
         .every((entry) =>
           entry.requiredCredentials.includes("DAYTONA_API_KEY"),
         ),

tests/runner-e2e/catalog.ts

@@ -2,6 +2,7 @@ import { nativeCompletionTasks, nativeCompletionDefinitionDigest } from "./nativ
 import { NATIVE_INSTRUCTION_SUITE, NATIVE_INSTRUCTION_BASE_SHA, nativeInstructionDefinitionDigest } from "./native-instruction-consolidation.js";
 import { nativeCompletionProfile, NATIVE_COMPLETION_BUDGET_CENTS } from "./native-completion-defaults.js";
 import { chatConfirmationTasks } from "./chat-cases.js";
+import { buildConnectionSuite } from "./connection-cases.js";
 import { hiringTemplateTasks, hiringTemplateProfile, hiringTemplateDefinitionDigest } from "./hiring-template-cases.js";
 import { instructionPersistenceTask } from "./instruction-persistence.js";
 import { apiResponseReadingTask } from "./api-response-reading.js";
@@ -1453,6 +1454,7 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
     expectedMatrixSize: 1,
     definitionMetadata: { version: 8, instructions: "fixed-external", nativeIdleTimeoutMs: 1_200_000, autoStopIntervalMinutes: 25, generatedFileCount: 60_000, filenameBytes: 39_828_890, scheduling: "explicit-only", finalization: "committed-without-active-sync-or-retry", copyback: "all-generated-file-contents-change-each-turn" },
   },
+  buildConnectionSuite(runnerProfiles, runnerEnvironments),
 ] as const;
 
 export function suiteDefinitionHash(suite: RunnerSuiteFixture) {
@@ -1512,8 +1514,8 @@ export function buildRunnerMatrix(
                 ]),
               ],
               requiredCredentials: [
-                profile.credential,
-                ...(environment.credential ? [environment.credential] : []),
+                ...(task.flow === "provider_connection" ? [] : [profile.credential]),
+                ...(task.flow !== "provider_connection" && environment.credential ? [environment.credential] : []),
               ],
             }))
             .filter((execution) => !excludedExecutionIds.has(execution.id)),
@@ -1559,8 +1561,10 @@ function assertNoRawSecretValues(value: unknown, label: string) {
 }
 
 export function validateRunnerCatalog(): MatrixExecution[] {
-  const allProfiles = [...extendedHarnessProfiles, ...runnerProfiles, ...legacyAcpxProfiles, ...pendingContextIntegrityProfiles, ...openRouterBreadthProfiles, ...everydayProfiles.filter(p => !runnerProfiles.some(existing => existing.id === p.id))];
+  const connectionSuite = runnerSuites.find(suite => suite.id === "provider-connections")!;
+  const allProfiles = [...connectionSuite.profiles, ...extendedHarnessProfiles, ...runnerProfiles, ...legacyAcpxProfiles, ...pendingContextIntegrityProfiles, ...openRouterBreadthProfiles, ...everydayProfiles.filter(p => !runnerProfiles.some(existing => existing.id === p.id))];
   const allTasks = [
+    ...connectionSuite.tasks,
     extendedHarnessFileTask,
     ...contextIntegrityTasks,
     ...blockerTasks,

tests/runner-e2e/codex-ci-sandbox.test.ts

@@ -37,9 +37,9 @@ describe("Codex CI sandbox trust boundary", () => {
       [...condition!.matchAll(/matrix\.profileId == '([^']+)'/g)].map((match) => match[1]),
     );
     expect([...provisionedProfiles].sort()).toEqual(
-      [...new Set(runnerMatrix.filter(requiresCodexCiSandbox).map((cell) => cell.profile.id))].sort(),
+      [...new Set(runnerMatrix.filter(cell => cell.task.flow !== "provider_connection").filter(requiresCodexCiSandbox).map((cell) => cell.profile.id))].sort(),
     );
-    for (const cell of runnerMatrix) {
+    for (const cell of runnerMatrix.filter(cell => cell.task.flow !== "provider_connection")) {
       expect(
         cell.environment.id === "local" && provisionedProfiles.has(cell.profile.id),
         cell.id,

tests/runner-e2e/connection-cancel.smoke.ts

@@ -0,0 +1,67 @@
+/** Opt-in, credential-free browser smoke: teardown retains its authenticated API client. */
+import { spawn } from "node:child_process";
+import { createServer } from "node:http";
+import path from "node:path";
+import { parseConnectionConfig } from "./connection-config.js";
+import { openConnectionBrowser } from "./connection-launch.js";
+import { runnerE2ETypeScriptProcessArgs } from "./web-server-command.js";
+
+const repositoryRoot = path.resolve(import.meta.dirname, "../..");
+
+if (process.argv[2] === "--worker") {
+  const origin = process.argv[3]!;
+  const browser = await openConnectionBrowser(parseConnectionConfig({ browser: {
+    headed: false, channel: "chromium", freshness: "signed-out",
+  } }), repositoryRoot, origin);
+  await browser.context.pages()[0]!.goto(origin);
+  let stopping = false;
+  for (const signal of ["SIGINT", "SIGTERM", "SIGHUP"] as const) process.once(signal, () => {
+    if (stopping) return;
+    stopping = true;
+    void (async () => {
+      let ok = false;
+      try { ok = (await browser.context.request.get(`${origin}/cleanup`)).ok(); }
+      finally { await browser.close(); }
+      process.send?.({ kind: "cleanup", ok });
+      process.disconnect();
+      process.exitCode = ok ? 0 : 1;
+    })().catch(() => { process.exitCode = 1; process.disconnect(); });
+  });
+  process.send?.({ kind: "ready" });
+} else {
+  let cleanupRequests = 0;
+  const server = createServer((req, res) => {
+    if (req.url === "/cleanup") {
+      const authenticated = req.headers.cookie === "qa-cleanup=synthetic";
+      if (authenticated) cleanupRequests++;
+      res.writeHead(authenticated ? 200 : 401).end();
+    } else res.writeHead(200, { "Content-Type": "text/html", "Set-Cookie": "qa-cleanup=synthetic; HttpOnly; Path=/" }).end("<p>Synthetic cleanup fixture</p>");
+  });
+  await new Promise<void>(resolve => server.listen(0, "127.0.0.1", resolve));
+  const origin = `http://127.0.0.1:${(server.address() as { port: number }).port}`;
+  try {
+    for (const signal of ["SIGINT", "SIGTERM", "SIGHUP"] as const) {
+      const env = Object.fromEntries(["PATH", "HOME", "USER", "LOGNAME", "TMPDIR", "LANG"].flatMap(key => process.env[key] ? [[key, process.env[key]!]] : []));
+      await new Promise<void>((resolve, reject) => {
+        const child = spawn(process.execPath, [...runnerE2ETypeScriptProcessArgs(repositoryRoot, import.meta.filename), "--worker", origin], {
+          cwd: repositoryRoot, env, stdio: ["ignore", "ignore", "ignore", "ipc"],
+        });
+        let cleaned = false;
+        const timeout = setTimeout(() => { child.kill("SIGKILL"); reject(new Error(`${signal}: browser cleanup timed out`)); }, 30_000);
+        child.on("message", message => {
+          const event = message as { kind?: string; ok?: boolean };
+          if (event.kind === "ready") child.kill(signal);
+          if (event.kind === "cleanup") cleaned = event.ok === true;
+        });
+        child.once("error", error => { clearTimeout(timeout); reject(error); });
+        child.once("exit", code => {
+          clearTimeout(timeout);
+          if (code === 0 && cleaned) resolve();
+          else reject(new Error(`${signal}: browser closed before authenticated fixture cleanup`));
+        });
+      });
+      console.log(`${signal}: authenticated cleanup completed before browser close`);
+    }
+    if (cleanupRequests !== 3) throw new Error("Missing authenticated cleanup requests");
+  } finally { await new Promise<void>((resolve, reject) => server.close(error => error ? reject(error) : resolve())); }
+}

tests/runner-e2e/connection-cases.ts

@@ -0,0 +1,88 @@
+import { createHash } from "node:crypto";
+import { readFileSync } from "node:fs";
+import { AI_CONNECTION_CAPABILITIES } from "../../packages/shared/src/ai-connections.js";
+import { isAiRoutingCompatible } from "../../packages/shared/src/ai-provider-routing.js";
+import type { AiProviderRouting } from "../../packages/shared/src/ai-provider-routing.js";
+import type { EnvironmentFixture, MatrixExecution, RunnerProfileFixture, RunnerSuiteFixture, RunnerTaskFixture } from "./types.js";
+
+export const connectionModes = ["subscription", "api-key", "openrouter", "bedrock", "responses", "messages", "chat"] as const;
+export type ConnectionMode = typeof connectionModes[number];
+export type ConnectionEntry = "agent" | "apps";
+export const connectionHarnesses = [
+  { id: "claude", adapter: "claude_local", provider: "anthropic", label: "Claude", key: "ANTHROPIC_API_KEY", native: true },
+  { id: "codex", adapter: "codex_local", provider: "openai", label: "OpenAI", key: "OPENAI_API_KEY", native: true },
+  { id: "grok", adapter: "grok_local", provider: "xai", label: "Grok", key: "XAI_API_KEY", native: true },
+  { id: "gemini", adapter: "gemini_local", provider: "google", label: "Google", key: "GEMINI_API_KEY", native: false },
+  { id: "opencode", adapter: "opencode_local", provider: "openrouter", label: "OpenRouter", key: "OPENROUTER_API_KEY", native: true },
+  { id: "hermes", adapter: "hermes_local", provider: "openrouter", label: "OpenRouter", key: "OPENROUTER_API_KEY", native: false },
+] as const;
+export type ConnectionHarness = typeof connectionHarnesses[number];
+
+/** Inventory entries remain visible even when no live journey can be selected. */
+export const connectionScopeGaps = [
+  { adapter: "cursor", status: "unsupported", reason: "No managed connection flow; qualify the harness-specific login separately." },
+  { adapter: "kimi_local", status: "unsupported", reason: "No managed connection flow; harness-specific credentials remain outside this suite." },
+  { adapter: "pi_local", status: "unsupported", reason: "No managed connection flow or qualified model source." },
+  { adapter: "openclaw_gateway", status: "excluded", reason: "Remote gateway owns provider configuration." },
+  { adapter: "hermes_gateway", status: "excluded", reason: "Remote gateway owns provider configuration." },
+  { adapter: "cursor_cloud", status: "excluded", reason: "Remote integration owns provider configuration." },
+  { adapter: "process", status: "excluded", reason: "External process owns provider configuration." },
+  { adapter: "http", status: "excluded", reason: "Remote integration owns provider configuration." },
+  { adapter: "acpx_local", status: "excluded", reason: "Generic legacy integration; native Claude/Grok ACPX profiles remain covered." },
+  { adapter: "claude_managed", status: "excluded", reason: "Managed remote integration owns credentials." },
+  { adapter: "aws_agentcore", status: "excluded", reason: "Remote integration owns credentials." },
+] as const;
+
+export function routingForMode(mode: ConnectionMode): AiProviderRouting | undefined {
+  if (mode === "subscription" || mode === "api-key") return undefined;
+  return { kind: mode === "openrouter" || mode === "bedrock" ? mode : "gateway", protocol: mode === "openrouter" ? "responses" : mode, auth: "bearer", models: [], ...(mode === "bedrock" ? { region: "us-east-1" } : {}) };
+}
+
+export function supportsConnection(harness: ConnectionHarness, mode: ConnectionMode) {
+  const routing = routingForMode(mode);
+  if (routing) return isAiRoutingCompatible(routing, harness.adapter);
+  // OpenRouter's normal key is represented by its explicit routing case.
+  if (harness.provider === "openrouter") return false;
+  const methods = AI_CONNECTION_CAPABILITIES[harness.provider].methods;
+  return Boolean(methods[mode === "subscription" ? "subscription" : "api_key"]?.adapters.includes(harness.adapter));
+}
+
+export function connectionCell(execution: MatrixExecution) {
+  const match = /^(agent|apps)-(subscription|api-key|openrouter|bedrock|responses|messages|chat)$/.exec(execution.task.id);
+  const harness = connectionHarnesses.find(h => execution.profile.id === `connection-${h.id}-${execution.profile.generation}`);
+  if (execution.suite.id !== "provider-connections" || !harness || !match) throw new Error("Not a provider connection execution");
+  return { harness, entry: match[1] as ConnectionEntry, mode: match[2] as ConnectionMode };
+}
+
+export function buildConnectionSuite(profiles: readonly RunnerProfileFixture[], environments: readonly EnvironmentFixture[]): RunnerSuiteFixture {
+  const selectedProfiles = connectionHarnesses.flatMap(h => (["legacy", ...(h.native ? ["native"] : [])] as Array<"legacy" | "native">).map(generation => {
+    const existing = profiles.find(p => p.generation === generation && (p.adapterType === h.adapter || (generation === "native" && p.id === (h.id === "claude" || h.id === "grok" ? `runner-acpx-${h.id}` : `runner-${h.id}`))));
+    const id = `connection-${h.id}-${generation}`;
+    return {
+      id, label: `${h.label} ${generation}`, generation, groups: [generation], adapterType: generation === "native" ? "paperclip_runner" : h.adapter,
+      provider: h.id, model: existing?.model ?? "configured-in-connection-file",
+      modelQualification: existing?.modelQualification ?? { source: "candidate_runner_profile", qualificationId: "operator-configured-model" },
+      credential: h.key,
+      supportedEnvironments: ["local", "daytona"], expectedRuntimeMode: generation,
+      expectedRuntimeMetadata: { adapterType: generation === "native" ? "paperclip_runner" : h.adapter, provider: generation === "native" && ["claude", "grok"].includes(h.id) ? "acpx" : h.id },
+      buildAgent: () => ({ name: id, role: "qa", adapterType: generation === "native" ? "paperclip_runner" : h.adapter, adapterConfig: {}, runtimeConfig: {} }),
+    } satisfies RunnerProfileFixture;
+  }));
+  const tasks: RunnerTaskFixture[] = (["agent", "apps"] as const).flatMap(entry => connectionModes.map(mode => ({
+    id: `${entry}-${mode}`, label: `${entry === "apps" ? "Apps connector" : "New agent"} · ${mode}`, groups: [], flow: "provider_connection", workMode: "standard",
+    expectedRunCount: 3, minimumExpectedRunCount: 2,
+    attemptTimeoutMs: { local: 30 * 60_000, daytona: 30 * 60_000 }, turnTimeoutMs: 5 * 60_000,
+    expectedTerminalState: { issue: "done", run: "succeeded" },
+    buildTitle: nonce => `Connection proof ${nonce}`, buildPrompt: () => "Runtime-generated independent attachment proof.", buildVisibleMarker: nonce => nonce,
+    buildMatchers: () => [],
+  })));
+  const excludedExecutionIds = selectedProfiles.flatMap(profile => environments.flatMap(environment => tasks.flatMap(task => {
+    const harness = connectionHarnesses.find(h => profile.id === `connection-${h.id}-${profile.generation}`)!;
+    const mode = task.id.replace(/^(agent|apps)-/, "") as ConnectionMode;
+    return supportsConnection(harness, mode) ? [] : [`provider-connections.${profile.id}.${environment.id}.${task.id}`];
+  })));
+  const hash = createHash("sha256");
+  for (const file of ["connection-cases.ts", "connection-config.ts", "connection-flow.ts", "connection-evidence.ts", "connection-launch.ts", "connection-target.ts", "user-actions.ts"]) hash.update(file).update(readFileSync(new URL(`./${file}`, import.meta.url)));
+  const digest = hash.digest("hex");
+  return { id: "provider-connections", label: "Live provider connections", description: "Fresh production UI connections, attended subscription login, independently verified tasks and reuse on a selected deployment.", groups: [], manualOnly: true, profiles: selectedProfiles, environments, tasks, excludedExecutionIds, expectedMatrixSize: selectedProfiles.length * environments.length * tasks.length - excludedExecutionIds.length, definitionMetadata: { version: 1, digest, authentication: "fresh-ui-connection", traces: "disabled", scheduling: "explicit-only" } };
+}

tests/runner-e2e/connection-config.ts

@@ -0,0 +1,162 @@
+import { readFile, lstat, realpath } from "node:fs/promises";
+import path from "node:path";
+import { connectionCell, type ConnectionMode } from "./connection-cases.js";
+import type { MatrixExecution } from "./types.js";
+
+export interface ConnectionRoute {
+  model: string; credentialEnv?: string; baseURL?: string; region?: string; auth: "bearer" | "api_key" | "none";
+}
+export interface ConnectionConfig {
+  target: { mode: "managed-local" } | { mode: "attach"; baseURL: string; expectedCommit: string; deploymentMode: "local_trusted" | "authenticated"; companyId?: string; environmentId?: string };
+  browser: { headed: boolean; channel: "chrome" | "chromium"; profileDir?: string; accountAlias: string; freshness: "signed-in" | "signed-out"; loginTimeoutMs: number };
+  secretFile?: string;
+  models: Record<string, string>;
+  routes: Record<string, ConnectionRoute>;
+  credentials: Record<string, string>;
+  budgetCents: number; maxRuns: number; turnTimeoutMs: number; retainCompany: boolean;
+}
+
+/** Strict, dependency-free private configuration parser. Never echo bad input. */
+export function parseConnectionConfig(input: unknown): ConnectionConfig {
+  const invalid = (): never => { throw new ConnectionBlock("blocked_target", "invalid_connection_configuration"); };
+  const object = (value: unknown, allowed?: string[]): Record<string, any> => {
+    if (!value || typeof value !== "object" || Array.isArray(value)) return invalid();
+    if (allowed && Object.keys(value).some(key => !allowed.includes(key))) return invalid();
+    return value as Record<string, any>;
+  };
+  const text = (value: unknown, fallback?: string): string => {
+    if (value === undefined && fallback !== undefined) return fallback;
+    if (typeof value !== "string" || !value.trim() || value.length > 2048) return invalid();
+    return value;
+  };
+  const choice = <T extends string>(value: unknown, options: readonly T[], fallback: T): T => value === undefined ? fallback : options.includes(value as T) ? value as T : invalid();
+  const flag = (value: unknown, fallback: boolean): boolean => value === undefined ? fallback : typeof value === "boolean" ? value : invalid();
+  const number = (value: unknown, fallback: number, min: number, max: number): number => value === undefined ? fallback : typeof value === "number" && Number.isSafeInteger(value) && value >= min && value <= max ? value : invalid();
+  const name = (value: unknown) => { const result = text(value); if (!/^[A-Z][A-Z0-9_]+$/.test(result)) return invalid(); return result; };
+  const uuid = (value: unknown) => { const result = text(value); if (!/^[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}$/i.test(result)) return invalid(); return result; };
+  const root = object(input, ["target", "browser", "secretFile", "models", "routes", "credentials", "budgetCents", "maxRuns", "turnTimeoutMs", "retainCompany"]);
+  const t = object(root.target ?? { mode: "managed-local" });
+  let target: ConnectionConfig["target"];
+  if (t.mode === "managed-local") { object(t, ["mode"]); target = { mode: "managed-local" }; }
+  else if (t.mode === "attach") {
+    object(t, ["mode", "baseURL", "expectedCommit", "deploymentMode", "companyId", "environmentId"]);
+    const expectedCommit = text(t.expectedCommit);
+    if (!/^[a-f0-9]{40}$/.test(expectedCommit)) return invalid();
+    target = { mode: "attach", baseURL: targetOrigin(text(t.baseURL)), expectedCommit, deploymentMode: choice(t.deploymentMode, ["local_trusted", "authenticated"], "authenticated"),
+      ...(t.companyId === undefined ? {} : { companyId: uuid(t.companyId) }), ...(t.environmentId === undefined ? {} : { environmentId: uuid(t.environmentId) }) };
+  } else return invalid();
+  const b = object(root.browser ?? {}, ["headed", "channel", "profileDir", "accountAlias", "freshness", "loginTimeoutMs"]);
+  const accountAlias = text(b.accountAlias, "local-qa");
+  if (!/^[a-zA-Z0-9_-]{1,60}$/.test(accountAlias)) return invalid();
+  const routes = Object.fromEntries(Object.entries(object(root.routes ?? {})).map(([key, value]) => {
+    const r = object(value, ["model", "credentialEnv", "baseURL", "region", "auth"]);
+    return [key, { model: text(r.model), auth: choice(r.auth, ["bearer", "api_key", "none"], "bearer"),
+      ...(r.credentialEnv === undefined ? {} : { credentialEnv: name(r.credentialEnv) }), ...(r.baseURL === undefined ? {} : { baseURL: text(r.baseURL) }), ...(r.region === undefined ? {} : { region: text(r.region) }) }];
+  }));
+  return { target, browser: { headed: flag(b.headed, true), channel: choice(b.channel, ["chrome", "chromium"], "chrome"), accountAlias, freshness: choice(b.freshness, ["signed-in", "signed-out"], "signed-in"), loginTimeoutMs: number(b.loginTimeoutMs, 600_000, 1000, 1200_000), ...(b.profileDir === undefined ? {} : { profileDir: text(b.profileDir) }) },
+    ...(root.secretFile === undefined ? {} : { secretFile: text(root.secretFile) }),
+    models: Object.fromEntries(Object.entries(object(root.models ?? {})).map(([key, value]) => [key, text(value)])), routes,
+    credentials: Object.fromEntries(Object.entries(object(root.credentials ?? {})).map(([key, value]) => [key, name(value)])),
+    budgetCents: number(root.budgetCents, 200, 1, 10_000), maxRuns: number(root.maxRuns, 3, 2, 5), turnTimeoutMs: number(root.turnTimeoutMs, 300_000, 10_000, 600_000), retainCompany: flag(root.retainCompany, false) };
+}
+
+/** Only harness-owned error codes may enter a retained report. */
+export class ConnectionFailure extends Error {}
+
+export class ConnectionBlock extends Error {
+  constructor(readonly reason: "awaiting_user" | "missing_credential" | "blocked_target", readonly code: string) {
+    super(code);
+  }
+}
+
+export function targetOrigin(value: string): string {
+  const url = new URL(value);
+  const loopback = ["127.0.0.1", "localhost", "[::1]"].includes(url.hostname);
+  if (url.username || url.password || url.search || url.hash || url.pathname !== "/" ||
+    !(url.protocol === "https:" || (url.protocol === "http:" && loopback))) {
+    throw new ConnectionBlock("blocked_target", "target_requires_https_or_loopback_origin");
+  }
+  return url.origin;
+}
+
+export async function loadConnectionConfig(file?: string): Promise<ConnectionConfig> {
+  try {
+    const config = parseConnectionConfig(file ? JSON.parse(await readFile(file, "utf8")) : {});
+    if (config.target.mode === "attach") config.target.baseURL = targetOrigin(config.target.baseURL);
+    return config;
+  } catch (error) {
+    if (error instanceof ConnectionBlock) throw error;
+    // Parsing errors can contain configuration values. Never echo input.
+    throw new ConnectionBlock("blocked_target", "invalid_connection_configuration");
+  }
+}
+
+export async function assertPrivateFile(file: string) {
+  const stat = await lstat(file);
+  if (!stat.isFile() || stat.isSymbolicLink() || (stat.mode & 0o077) !== 0 || (process.getuid && stat.uid !== process.getuid())) {
+    throw new ConnectionBlock("missing_credential", "credential_file_requires_owner_only_regular_file");
+  }
+}
+
+/** Literal dotenv only: never evaluate shell expansions or source an entire file. */
+export function selectedSecret(text: string, name: string): string | undefined {
+  let found: string | undefined;
+  for (const raw of text.split(/\r?\n/)) {
+    const line = raw.replace(/^\s*export\s+/, "");
+    const match = /^\s*([A-Z][A-Z0-9_]+)\s*=\s*(.*?)\s*$/.exec(line);
+    if (match?.[1] !== name) continue;
+    let value = match[2]!;
+    if ((value.startsWith('"') && value.endsWith('"')) || (value.startsWith("'") && value.endsWith("'"))) value = value.slice(1, -1);
+    if (!value || /[`\r\n]|\$\(/.test(value)) throw new ConnectionBlock("missing_credential", "credential_must_be_a_literal_value");
+    found = value;
+  }
+  return found;
+}
+
+export async function resolveConnectionSecret(config: ConnectionConfig, name?: string, env: NodeJS.ProcessEnv = process.env) {
+  if (!name) return undefined;
+  if (env[name]?.trim()) return env[name]!.trim();
+  if (config.secretFile) {
+    await assertPrivateFile(config.secretFile);
+    const value = selectedSecret(await readFile(config.secretFile, "utf8"), name);
+    if (value) return value;
+  }
+  throw new ConnectionBlock("missing_credential", `missing_${name}`);
+}
+
+export function resolveConnectionSettings(execution: MatrixExecution, config: ConnectionConfig) {
+  const { harness, mode } = connectionCell(execution);
+  const route = config.routes[`${harness.id}.${mode}`] ?? config.routes[mode];
+  const advanced = mode !== "subscription" && mode !== "api-key";
+  if (advanced && !route) throw new ConnectionBlock("missing_credential", "route_configuration_required");
+  const selectedModel = route?.model ?? config.models[harness.id] ?? execution.profile.model;
+  if (selectedModel === "configured-in-connection-file") throw new ConnectionBlock("blocked_target", "explicit_model_required");
+  if (["responses", "messages", "chat"].includes(mode)) {
+    if (!route?.baseURL) throw new ConnectionBlock("blocked_target", "gateway_url_required");
+    const url = new URL(route.baseURL);
+    if (url.username || url.password || url.search || url.hash || !["https:", "http:"].includes(url.protocol)) throw new ConnectionBlock("blocked_target", "invalid_gateway_url");
+  }
+  if (mode === "bedrock" && (!route?.region || route.auth !== "bearer")) throw new ConnectionBlock("blocked_target", "bedrock_region_and_bearer_required");
+  if (route?.auth === "api_key" && mode !== "messages") throw new ConnectionBlock("blocked_target", "api_key_header_requires_messages");
+  if (mode === "openrouter" && route?.auth !== "bearer") throw new ConnectionBlock("blocked_target", "openrouter_requires_bearer");
+  const defaultKey = mode === "bedrock" ? "AWS_BEARER_TOKEN_BEDROCK" : mode === "openrouter" ? "OPENROUTER_API_KEY" : harness.key;
+  const credentialEnv = mode === "subscription" || route?.auth === "none" ? undefined : route?.credentialEnv ?? config.credentials[harness.id] ?? (advanced && !["bedrock", "openrouter"].includes(mode) ? undefined : defaultKey);
+  if (mode !== "subscription" && route?.auth !== "none" && !credentialEnv) throw new ConnectionBlock("missing_credential", "gateway_credential_reference_required");
+  return { model: selectedModel, route, credentialEnv };
+}
+
+export function connectionCatalogSource(mode: ConnectionMode, provider: string) {
+  return ({ responses: "responses-api", messages: "messages-api", chat: "chat-completions-api", bedrock: "bedrock", openrouter: "openrouter" } as Record<string, string>)[mode] ?? provider;
+}
+
+export async function assertOutsideRepository(file: string, repositoryRoot: string) {
+  // Resolve an existing ancestor too, so a missing child below a symlink cannot evade containment.
+  let existing = path.resolve(file);
+  while (true) {
+    try { existing = await realpath(existing); break; }
+    catch (error) { if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; const parent = path.dirname(existing); if (parent === existing) throw error; existing = parent; }
+  }
+  const root = await realpath(repositoryRoot);
+  const relative = path.relative(root, existing);
+  if (relative === "" || (!relative.startsWith(`..${path.sep}`) && relative !== ".." && !path.isAbsolute(relative))) throw new ConnectionBlock("blocked_target", "browser_profile_must_be_outside_repository");
+}

tests/runner-e2e/connection-evidence.ts

@@ -0,0 +1,128 @@
+import { createHash, randomBytes } from "node:crypto";
+import type { RunnerE2EResult } from "./types.js";
+
+export const connectionCheckpoints = ["target", "connection_created", "connection_reloaded", "agent_created", "binding", "setup_probe", "first_run", "artifact", "tool_receipt", "followup", "visible_result"] as const;
+export type ConnectionCheckpoint = typeof connectionCheckpoints[number];
+export const connectionRunSignals = ["provider_overloaded", "provider_quota", "missing_file_conversion", "unsupported_session_configuration", "command_substitution_denied"] as const;
+export interface ConnectionRunDiagnostic {
+  runId: string;
+  status: "succeeded" | "failed" | "cancelled" | "interrupted" | "timed_out" | "unknown";
+  signals: Array<typeof connectionRunSignals[number]>;
+  logAvailable: boolean;
+}
+export interface ConnectionEvidence {
+  version: 1;
+  outcome: "running" | "passed" | "failed" | "awaiting_user" | "missing_credential" | "blocked_target";
+  phase: string;
+  assisted: boolean;
+  authFreshness: "signed-in" | "signed-out";
+  target: { mode: "managed-local" | "attach"; origin: string; commit: string | null; deploymentMode: string | null };
+  entry: "apps" | "agent";
+  method: string;
+  checkpoints: Partial<Record<ConnectionCheckpoint, boolean>>;
+  waits: Array<{ kind: "board_login" | "provider_login"; startedAt: string; finishedAt?: string }>;
+  connectionId?: string;
+  agentId?: string;
+  companyId?: string;
+  artifactSha256?: string;
+  model?: string;
+  inputTransport?: "prompt_base64";
+  setupChecks?: Array<{ code: string; level: "info" | "warn" | "error" }>;
+  creationFailure?: { status: number; code: "ai_connection_api_key_rejected" | "ai_connection_verification_failed" | "connection_request_rejected" };
+  cleanupRetained?: boolean;
+  companyArchived?: boolean;
+  artifactChecks?: Array<{ filename: string; runId: string; sha256: string; fields: Record<string, boolean>; exactFields: boolean }>;
+  runDiagnostics?: ConnectionRunDiagnostic[];
+}
+
+/** Project terminal errors to closed diagnostic codes; never retain messages or reasoning. */
+export function connectionRunDiagnostic(run: Record<string, any>, log?: unknown): ConnectionRunDiagnostic {
+  const signals = new Set<typeof connectionRunSignals[number]>();
+  const detect = (text: unknown) => {
+    if (typeof text !== "string") return;
+    if (/"code"\s*:\s*503\b|\bHTTP\s*503\b/i.test(text) && /overloaded|high demand/i.test(text)) signals.add("provider_overloaded");
+    if (/"code"\s*:\s*429\b|\bHTTP\s*429\b|\bRESOURCE_EXHAUSTED\b|\bTerminalQuotaError\b/.test(text)) signals.add("provider_quota");
+    if (/Error checking existing file: (?:Resource not found|Internal error)/.test(text)) signals.add("missing_file_conversion");
+    if (/set_config_option/.test(text) && /[Mm]ethod not found|-32601/.test(text)) signals.add("unsupported_session_configuration");
+    if (/Blocked: command substitution detected/.test(text)) signals.add("command_substitution_denied");
+  };
+  detect(run.error);
+  detect(run.stderrExcerpt);
+  const content = log && typeof log === "object" && typeof (log as Record<string, unknown>).content === "string"
+    ? (log as { content: string }).content : undefined;
+  if (content !== undefined) {
+    let inner = "";
+    for (const line of content.split("\n")) {
+      try { const row = JSON.parse(line); if (typeof row.chunk === "string") inner += row.chunk; }
+      catch { /* Non-JSON application lines do not classify provider failures. */ }
+    }
+    for (const line of inner.split("\n")) {
+      try {
+        const row = JSON.parse(line);
+        if (row.type === "acpx.error") { detect(row.message); detect(row.childStderrTail); }
+        if (row.type === "acpx.tool_call" && row.status === "failed") detect(row.text);
+      } catch { /* Model text, partial records, and raw output are not retained. */ }
+    }
+  }
+  const statuses = ["succeeded", "failed", "cancelled", "interrupted", "timed_out"];
+  return { runId: run.id, status: statuses.includes(run.status) ? run.status : "unknown", signals: [...signals], logAvailable: content !== undefined };
+}
+
+export function gradeConnectionEvidence(evidence: ConnectionEvidence) {
+  return connectionCheckpoints.map(id => ({ matcher: { kind: "json_path" as const, path: `providerConnection.checkpoints.${id}`, expected: true }, passed: evidence.checkpoints[id] === true, detail: `${id}: ${evidence.checkpoints[id] === true ? "verified" : "not verified"}` }));
+}
+
+/** Retain stable product check codes, never provider messages or command output. */
+export function connectionProbeChecks(value: unknown): NonNullable<ConnectionEvidence["setupChecks"]> {
+  if (!value || typeof value !== "object" || !Array.isArray((value as Record<string, unknown>).checks)) return [];
+  return ((value as { checks: unknown[] }).checks).slice(0, 50).flatMap(entry => {
+    if (!entry || typeof entry !== "object") return [];
+    const { code, level } = entry as Record<string, unknown>;
+    return typeof code === "string" && /^[a-z][a-z0-9_]{0,95}$/.test(code) && ["info", "warn", "error"].includes(String(level))
+      ? [{ code, level: level as "info" | "warn" | "error" }] : [];
+  });
+}
+
+export function connectionEvidencePasses(evidence: ConnectionEvidence) {
+  return evidence.outcome === "passed" && Boolean(evidence.target.commit) && gradeConnectionEvidence(evidence).every(check => check.passed);
+}
+
+export function isBlockedConnectionResult(result: RunnerE2EResult) {
+  return Boolean(result.providerConnection && ["awaiting_user", "missing_credential", "blocked_target"].includes(result.providerConnection.outcome));
+}
+
+export function createConnectionProof() {
+  const nonce = randomBytes(12).toString("hex");
+  const values = Array.from({ length: 7 }, () => randomBytes(2).readUInt16BE());
+  const input = `${JSON.stringify({ nonce, values })}\n`;
+  const expected = { nonce, count: values.length, total: values.reduce((sum, n) => sum + n, 0), sha256: createHash("sha256").update(input).digest("hex") };
+  const followup = { nonce, minimum: Math.min(...values), maximum: Math.max(...values), total: expected.total };
+  return { input, expected, followup };
+}
+
+export function verifyConnectionArtifact(bytes: Buffer, expected: Record<string, unknown>) {
+  try {
+    const value = JSON.parse(bytes.toString("utf8"));
+    return Object.keys(value).length === Object.keys(expected).length && Object.entries(expected).every(([key, entry]) => value[key] === entry);
+  } catch { return false; }
+}
+
+/** Only durable run attribution is accepted; assistant text cannot provide it. */
+export function verifyConnectionRun(run: Record<string, any>, expected: { agentId: string; connectionId: string; method: string; runtimeMode: string }) {
+  const connection = run.contextSnapshot?.aiConnection;
+  return run.status === "succeeded" && run.agentId === expected.agentId && run.runtimeMode === expected.runtimeMode &&
+    connection?.connectionId === expected.connectionId && connection?.method === expected.method;
+}
+
+/** Wait for scheduled repairs to settle, then select the run that actually delivered the file. */
+export function completedConnectionArtifactRun(
+  runs: Record<string, any>[],
+  attachments: Record<string, any>[],
+  filename: string,
+  expected: Parameters<typeof verifyConnectionRun>[1],
+) {
+  const terminal = ["succeeded", "failed", "cancelled", "interrupted", "timed_out"];
+  if (runs.some(run => !terminal.includes(run.status))) return undefined;
+  return runs.find(run => verifyConnectionRun(run, expected) && attachments.some(attachment =>
+    attachment.originalFilename === filename && attachment.createdByAgentId === expected.agentId && attachment.originatingRunId === run.id));
+}

tests/runner-e2e/connection-flow.ts

@@ -0,0 +1,365 @@
+import { createHash } from "node:crypto";
+import { expect, type Page } from "@playwright/test";
+import { aiRoutingModel } from "../../packages/shared/src/ai-provider-routing.js";
+import { getConnectableAppDefinition } from "../../packages/shared/src/app-definitions.js";
+import { connectionCell, type ConnectionHarness, type ConnectionMode } from "./connection-cases.js";
+import { ConnectionBlock, ConnectionFailure, type ConnectionConfig, type resolveConnectionSettings } from "./connection-config.js";
+import { ConnectionApi, connectionDelay } from "./connection-target.js";
+import { createConnectionProof, verifyConnectionArtifact, verifyConnectionRun, completedConnectionArtifactRun, connectionProbeChecks, type ConnectionEvidence } from "./connection-evidence.js";
+import { createTaskThroughUi, submitTaskReply } from "./user-actions.js";
+import { collectRunEvents } from "./run-observations.js";
+import type { MatrixExecution } from "./types.js";
+
+type Row = Record<string, any>;
+type Settings = ReturnType<typeof resolveConnectionSettings>;
+export interface ConnectionFlowInput {
+  page: Page; api: ConnectionApi; execution: MatrixExecution; config: ConnectionConfig; settings: Settings;
+  company: Row; environment: Row; secret?: string; nonce: string; privateDir: string;
+  evidence: ConnectionEvidence;
+  checkpoint(phase: string): Promise<void>;
+  observeRun(run: Row): void;
+  assertActive(): void;
+}
+
+async function choose(page: Page, label: string, name: string) {
+  await page.getByRole("combobox", { name: label, exact: true }).click();
+  await page.getByRole("option", { name, exact: true }).click();
+}
+
+async function openAdvanced(page: Page, mode: ConnectionMode) {
+  const names: Record<string, string> = { openrouter: "OpenRouter", bedrock: "Amazon Bedrock", responses: "Custom gateway", messages: "Custom gateway", chat: "Custom gateway" };
+  await page.getByRole("button", { name: new RegExp(`^${names[mode]}`) }).click();
+}
+
+async function fillGateway(input: ConnectionFlowInput) {
+  const { page, settings, secret } = input;
+  const { mode } = connectionCell(input.execution);
+  if (settings.route?.baseURL) {
+    await page.getByLabel("Provider URL", { exact: true }).fill(settings.route.baseURL);
+    const format = page.getByRole("combobox", { name: "API format", exact: true });
+    if (await format.isEnabled()) await choose(page, "API format", ({ responses: "OpenAI Responses · Codex", messages: "Anthropic Messages · Claude", chat: "Chat Completions · OpenCode, Hermes" } as Record<string, string>)[mode]!);
+  }
+  if (mode === "bedrock") await page.getByLabel("AWS region", { exact: true }).fill(settings.route!.region!);
+  if (settings.route?.auth !== "bearer") await choose(page, "Authentication", settings.route?.auth === "none" ? "No authentication" : "API key · x-api-key");
+  if (secret) await page.getByLabel("API key", { exact: true }).fill(secret);
+  // Normal defaults stay collapsed. Model alias is selected in Configure.
+  await page.getByRole("button", { name: "Connect", exact: true }).last().click();
+}
+
+async function submitConnection(input: ConnectionFlowInput, submit: () => Promise<void>) {
+  const route = `${input.api.origin}/api/companies/${input.company.id}/ai-connections`;
+  const response = input.page.waitForResponse(response => response.url() === route && response.request().method() === "POST", { timeout: 60_000 }).catch(() => null);
+  await submit();
+  const created = await response;
+  if (!created) throw new ConnectionFailure("connection_request_timeout");
+  if (created.ok()) return;
+  const payload = await created.json().catch(() => ({}));
+  const allowed = ["ai_connection_api_key_rejected", "ai_connection_verification_failed"];
+  const code = allowed.includes(payload?.details?.code) ? payload.details.code : "connection_request_rejected";
+  input.evidence.creationFailure = { status: created.status(), code };
+  if (code === "ai_connection_api_key_rejected") throw new ConnectionBlock("missing_credential", code);
+  throw new ConnectionFailure(code);
+}
+
+async function waitForFreshConnection(input: ConnectionFlowInput, before: Set<string>) {
+  const { page, api, config, evidence, company } = input;
+  const { harness, mode } = connectionCell(input.execution);
+  const deadline = Date.now() + (mode === "subscription" ? config.browser.loginTimeoutMs : 90_000);
+  let loginOpened = false;
+  let connected = false;
+  while (Date.now() < deadline) {
+    input.assertActive();
+    const list = await api.get(`/api/companies/${company.id}/ai-connections`);
+    const fresh = list.connections.filter((c: Row) => !before.has(c.id));
+    if (fresh.length > 1) throw new ConnectionFailure("unexpected_multiple_connections");
+    if (fresh.length === 1) {
+      const connection = fresh[0];
+      if (connection.status !== "connected" || connection.method !== (mode === "subscription" ? "subscription" : "api_key")) throw new ConnectionFailure("saved_connection_method_or_status_mismatch");
+      return connection;
+    }
+    if (mode === "subscription") {
+      const signIn = page.getByRole("link", { name: /^Sign in to / }).first();
+      const cloudSignIn = page.getByRole("button", { name: /^Sign in to / }).first();
+      if (!loginOpened && (await signIn.isVisible() || await cloudSignIn.isVisible())) {
+        // Observe before clicking. No token, code, URL, or popup content is recorded.
+        const popup = page.context().waitForEvent("page", { timeout: 10_000 }).catch(() => undefined);
+        if (await signIn.isVisible()) await signIn.click(); else if (await cloudSignIn.isEnabled()) await cloudSignIn.click();
+        await popup;
+        loginOpened = true;
+        evidence.assisted = true;
+        evidence.waits.push({ kind: "provider_login", startedAt: new Date().toISOString() });
+        await input.checkpoint("awaiting_provider_login");
+        console.log(`[provider-connections] Sign in to ${harness.label} in the opened browser (${config.browser.accountAlias}); complete the code/consent step in Paperclip. This test resumes automatically.`);
+      }
+      // Grok currently exposes a terminal handoff; do not pretend it has the same browser flow.
+      if (!loginOpened && harness.id === "grok" && await page.getByRole("button", { name: "Copy sign-in command" }).isVisible()) {
+        loginOpened = true;
+        evidence.assisted = true;
+        evidence.waits.push({ kind: "provider_login", startedAt: new Date().toISOString() });
+        await input.checkpoint("awaiting_provider_login");
+        console.log("[provider-connections] Grok requires the sign-in command shown in Paperclip. Complete that production handoff; the test will resume automatically.");
+      }
+      const connect = page.getByRole("button", { name: "Connect", exact: true }).last();
+      if (!connected && await connect.isVisible() && await connect.isEnabled()) { await connect.click(); connected = true; }
+      if (await page.getByRole("alert").filter({ hasText: /expired|could not|failed|invalid/i }).count()) throw new ConnectionFailure("provider_login_reported_error");
+    }
+    await connectionDelay(1_000);
+  }
+  if (mode === "subscription" && loginOpened) throw new ConnectionBlock("awaiting_user", "provider_login_deadline_reached");
+  throw new ConnectionFailure("connection_creation_timeout");
+}
+
+async function selectSavedConnection(page: Page, connection: Row, harness: ConnectionHarness, mode: ConnectionMode) {
+  await page.getByRole("heading", { name: /^(Connect a model|Configure your agent)$/ }).waitFor();
+  const advanced = !["subscription", "api-key"].includes(mode);
+  const connectionType = page.getByRole("radiogroup", { name: "Connection type", exact: true });
+  if (["claude", "codex", "grok"].includes(harness.id)) {
+    await connectionType.waitFor({ state: "visible" });
+    if (advanced) {
+      await connectionType.getByRole("radio", { name: /Advanced/ }).click();
+      await choose(page, "Connection", connection.name);
+      await page.getByRole("button", { name: "Use connection", exact: true }).click();
+    } else {
+      await connectionType.getByRole("radio", { name: mode === "subscription" ? /Subscription/ : /API key/ }).click();
+      const saved = page.getByRole("combobox", { name: mode === "subscription" ? "Saved subscription" : "Saved API key", exact: true });
+      await saved.waitFor({ state: "visible" });
+      const option = await saved.locator("option").filter({ hasText: connection.name }).first().getAttribute("value");
+      if (!option) throw new ConnectionFailure("saved_connection_option_missing");
+      await saved.selectOption(option);
+      await page.getByRole("button", { name: mode === "subscription" ? "Use saved subscription" : "Use saved API key", exact: true }).click();
+    }
+  } else {
+    await choose(page, "Connection", connection.name);
+  }
+}
+
+async function createConnectionThroughUi(input: ConnectionFlowInput, agentURL: string) {
+  const { page, api, company, evidence } = input;
+  const { harness, mode, entry } = connectionCell(input.execution);
+  const priorConnections = (await api.get(`/api/companies/${company.id}/ai-connections`)).connections as Row[];
+  const before = new Set<string>(priorConnections.map(c => c.id));
+  // Fresh-creation proof cannot be satisfied by an existing personal default.
+  if (priorConnections.some(connection => connection.status !== "revoked")) throw new ConnectionBlock("blocked_target", "fresh_connection_requires_empty_qa_company");
+  const advanced = !["subscription", "api-key"].includes(mode);
+  await input.checkpoint("connection_navigation");
+  if (entry === "apps") {
+    await page.goto(`${api.origin}/${company.issuePrefix}/apps`, { waitUntil: "domcontentloaded" });
+    const catalogSlug = advanced ? ({ openrouter: "openrouter", bedrock: "bedrock", responses: "responses-api", messages: "messages-api", chat: "chat-completions-api" } as Record<string, string>)[mode]! : harness.provider;
+    const definition = getConnectableAppDefinition(catalogSlug);
+    if (!definition) throw new ConnectionFailure("connection_catalog_entry_missing");
+    await page.getByRole("button", { name: `Connect ${definition.name}`, exact: true }).click();
+    const next = page.getByRole("button", { name: /^(Save and continue|Continue)$/ }).last();
+    if (!advanced && await next.isVisible()) await next.click();
+    if (!advanced && harness.provider !== "google") {
+      // The credential step loads asynchronously. Wait for its provider tile
+      // before inspecting the mode switch, otherwise a still-loading page can
+      // skip the API-key switch and leave the subscription form selected.
+      await page.getByRole("radio", { name: new RegExp(harness.id === "grok" ? "Grok" : harness.label) }).click();
+      const switcher = page.getByRole("button", { name: mode === "subscription" ? "Use subscription instead" : "Use API key instead", exact: true });
+      if (await switcher.isVisible()) await switcher.click();
+    }
+  } else {
+    await page.goto(agentURL, { waitUntil: "domcontentloaded" });
+    const tiles = page.getByRole("radiogroup", { name: "Connection type", exact: true });
+    if (["claude", "codex", "grok"].includes(harness.id)) {
+      const choice = tiles.getByRole("radio", { name: advanced ? /Advanced/ : mode === "subscription" ? /Subscription/ : /API key/ });
+      await choice.click();
+      await expect(choice).toHaveAttribute("aria-checked", "true");
+    }
+    if (advanced || !["claude", "codex", "grok"].includes(harness.id)) {
+      await choose(page, "Connection", "Connect an account…");
+      if (advanced) {
+        const advancedProviders = page.locator("summary").filter({ hasText: "Advanced providers" });
+        if (await advancedProviders.isVisible()) {
+          await advancedProviders.click();
+          // The account dialog's disclosure leads to the provider chooser;
+          // the chooser's own disclosure already contains provider buttons.
+          const chooseProvider = page.getByRole("button", { name: "Choose another provider or gateway", exact: true });
+          if (await chooseProvider.isVisible()) await chooseProvider.click();
+        }
+        await openAdvanced(page, mode);
+      }
+    }
+  }
+  await input.checkpoint("connection_creation");
+  if (advanced) await submitConnection(input, () => fillGateway(input));
+  else if (mode === "api-key") {
+    const savedKey = page.getByRole("combobox", { name: "Saved API key", exact: true });
+    if (await savedKey.isVisible()) await savedKey.selectOption("");
+    await page.getByLabel("API key", { exact: true }).fill(input.secret!);
+    await submitConnection(input, () => page.getByRole("button", { name: "Connect", exact: true }).last().click());
+  }
+  const connection = await waitForFreshConnection(input, before);
+  evidence.connectionId = connection.id;
+  evidence.checkpoints.connection_created = true;
+  evidence.waits.forEach(wait => { wait.finishedAt ??= new Date().toISOString(); });
+  if (advanced) {
+    const expectedKind = ["bedrock", "openrouter"].includes(mode) ? mode : "gateway";
+    if (connection.routing?.kind !== expectedKind || (mode !== "openrouter" && connection.routing?.protocol !== mode) ||
+      (input.settings.route?.baseURL && connection.routing?.baseUrl !== input.settings.route.baseURL) ||
+      (mode === "bedrock" && connection.routing?.region !== input.settings.route?.region)) throw new ConnectionFailure("saved_routing_mismatch");
+  } else if (connection.provider !== harness.provider) throw new ConnectionFailure("saved_provider_mismatch");
+  // Navigate away and return through the normal list: proof includes durable UI state.
+  await page.goto(`${api.origin}/${company.issuePrefix}/apps`, { waitUntil: "domcontentloaded" });
+  await expect(page.getByText(connection.name, { exact: true }).first()).toBeVisible();
+  await page.reload({ waitUntil: "domcontentloaded" });
+  await expect(page.getByText(connection.name, { exact: true }).first()).toBeVisible();
+  evidence.checkpoints.connection_reloaded = true;
+  await input.checkpoint("connection_reloaded");
+  await page.goto(agentURL, { waitUntil: "domcontentloaded" });
+  await input.checkpoint("select_saved_connection");
+  await selectSavedConnection(page, connection, harness, mode);
+  await input.checkpoint("configure_agent");
+  return connection;
+}
+
+async function selectModel(page: Page, model: string) {
+  const select = page.getByRole("combobox", { name: "Model", exact: true });
+  await select.waitFor({ state: "visible" });
+  await expect(select).not.toHaveAttribute("aria-busy", "true", { timeout: 60_000 });
+  const values = await select.locator("option").evaluateAll(options => options.map(option => (option as HTMLOptionElement).value));
+  const matched = values.find(value => value === model || value === `openrouter/${model}`);
+  if (matched) await select.selectOption(matched);
+  else {
+    await select.selectOption({ label: "Enter custom model…" });
+    await page.getByLabel("Model ID", { exact: true }).fill(model);
+  }
+}
+
+export async function runConnectionFlow(input: ConnectionFlowInput) {
+  const { page, api, execution, evidence, settings, company, environment, config } = input;
+  const { harness, mode } = connectionCell(execution);
+  const agentName = `Connection QA ${input.nonce}`;
+  const agentURL = `${api.origin}/${company.issuePrefix}/agents/new?${new URLSearchParams({ name: agentName, adapterType: execution.profile.adapterType, runnerProvider: harness.id })}`;
+  const connection = await createConnectionThroughUi(input, agentURL);
+  await expect(page.getByRole("heading", { name: "Configure your agent", exact: true })).toBeVisible({ timeout: 120_000 });
+  const environmentSelect = page.getByRole("combobox", { name: "Environment", exact: true });
+  const selectedEnv = await environmentSelect.inputValue();
+  if (selectedEnv !== environment.id && await environmentSelect.isEnabled()) {
+    await environmentSelect.selectOption(environment.id);
+    if (["claude", "codex", "grok"].includes(harness.id)) await selectSavedConnection(page, connection, harness, mode);
+  }
+  await input.checkpoint("select_model");
+  await selectModel(page, settings.model);
+  await input.checkpoint("setup_probe");
+  const response = page.waitForResponse(response => response.url().startsWith(`${api.origin}/api/companies/${company.id}/adapters/`) && response.url().endsWith("/test-environment"), { timeout: 120_000 });
+  await page.getByRole("button", { name: /^(Run test|Test again)$/ }).click();
+  const probe = await response;
+  const result = await probe.json();
+  evidence.setupChecks = connectionProbeChecks(result);
+  if (evidence.setupChecks.some(check => check.level === "error" && (check.code === "hermes_cli_not_found" || /^(claude|codex|grok|opencode|gemini)_command_unresolvable$/.test(check.code))))
+    throw new ConnectionBlock("blocked_target", "runtime_cli_required");
+  if (!probe.ok() || result.status !== "pass") throw new ConnectionFailure("setup_probe_failed");
+  evidence.checkpoints.setup_probe = true;
+  await page.getByRole("button", { name: "Finish setup", exact: true }).click();
+  let agent: Row | undefined;
+  const deadline = Date.now() + 60_000;
+  while (!agent && Date.now() < deadline) {
+    agent = (await api.get<Row[]>(`/api/companies/${company.id}/agents`)).find(row => row.name === agentName);
+    if (!agent) await connectionDelay(500);
+  }
+  if (!agent) throw new ConnectionFailure("agent_creation_failed");
+  evidence.agentId = agent.id;
+  evidence.checkpoints.agent_created = true;
+  const expectedModel = connection.routing
+    ? aiRoutingModel(connection.routing, harness.adapter, settings.model)
+    : settings.model;
+  if (agent.adapterType !== execution.profile.adapterType || agent.adapterConfig?.model !== expectedModel) throw new ConnectionFailure("agent_harness_or_model_mismatch");
+  const binding = agent.runtimeConfig?.aiConnection;
+  if (!binding || binding.method !== connection.method || (binding.mode !== "responsible_user" && binding.connectionId !== connection.id)) throw new ConnectionFailure("agent_binding_mismatch");
+  // Fixtures may constrain cost/scheduling, but cannot repair or replace the tested credentials/config.
+  await api.patch(`/api/agents/${agent.id}`, { budgetMonthlyCents: config.budgetCents });
+  evidence.checkpoints.binding = true;
+  await input.checkpoint("first_task");
+  const proof = createConnectionProof();
+  // Provider qualification must not depend on task-level attachment ingestion.
+  // Native runners admit chat attachments, while the new-task dialog uploads
+  // its files after scheduling. Supply identical oracle bytes in the task so
+  // every harness can decode them with its actual tools.
+  evidence.inputTransport = "prompt_base64";
+  const title = `Connection artifact ${input.nonce}`;
+  await createTaskThroughUi({ page, issuePrefix: company.issuePrefix, agentName, title, workMode: "standard", prompt: [
+    "Complete this bounded connection acceptance task. Using your actual tools, decode the following base64 into connection-input.json and read the file. Preserve the exact decoded bytes, including the final newline.",
+    `Input bytes (base64): ${Buffer.from(proof.input, "utf8").toString("base64")}`,
+    "Create the input and output files inside your current workspace using relative filenames. Do not put them in /tmp or another directory outside the workspace.",
+    'Write connection-proof.json with exactly this schema: {"nonce": string, "count": number, "total": number, "sha256": string}. Set nonce from the input, count to the number of values, total to their sum, and sha256 to the hash of the exact input file bytes. All four fields are required; use the literal field name "total", not "sum". Compute and validate with a tool; do not guess or paraphrase the data.', 
+    "Deliver connection-proof.json as a downloadable attachment on this task. Use the normal Paperclip artifact/attachment workflow and include its download link in your final response. Then mark this task done. Do not create other tasks, read credentials, or inspect unrelated files.",
+    execution.profile.generation === "native" ? "Use your native register_deliverable tool with workspace-relative contentRef connection-proof.json and the exact byteSize and SHA-256 of that output file. Never submit an absolute contentRef." : "",
+  ].join("\n") });
+  let issue: Row | undefined;
+  const createdDeadline = Date.now() + 30_000;
+  while (!issue && Date.now() < createdDeadline) {
+    issue = (await api.get<Row[]>(`/api/companies/${company.id}/issues`)).find(row => row.title === title);
+    if (!issue) await connectionDelay(500);
+  }
+  if (!issue) throw new ConnectionFailure("task_creation_failed");
+  const runs = new Map<string, Row>();
+  async function completedRun(after: Set<string>, filename: string) {
+    const deadline = Date.now() + config.turnTimeoutMs;
+    while (Date.now() < deadline) {
+    input.assertActive();
+      const summaries = await api.get<Row[]>(`/api/issues/${issue!.id}/runs`);
+      for (const row of summaries) {
+        const id = row.runId ?? row.id;
+        if (id && !runs.has(id)) runs.set(id, await api.get(`/api/heartbeat-runs/${id}`));
+        else if (id && !["succeeded", "failed", "cancelled", "interrupted", "timed_out"].includes(runs.get(id)!.status)) runs.set(id, await api.get(`/api/heartbeat-runs/${id}`));
+        if (id) input.observeRun(runs.get(id)!);
+      }
+      if (runs.size > config.maxRuns) throw new ConnectionFailure("provider_run_limit_exceeded");
+      const next = [...runs.values()].filter(run => !after.has(run.id));
+      if (next.some(run => ["failed", "cancelled", "interrupted", "timed_out"].includes(run.status))) throw new ConnectionFailure("agent_run_failed");
+      const complete = next.find(run => verifyConnectionRun(run, { agentId: agent!.id, connectionId: connection.id, method: connection.method, runtimeMode: execution.profile.generation }));
+      if (complete) {
+        const status = (await api.get(`/api/issues/${issue!.id}`)).status;
+        if (status === "done") {
+          // A disposition repair may still be running after an earlier turn
+          // succeeded. Qualify the settled successful run that uploaded this
+          // artifact, preserving the configured run and time limits.
+          const attachments = await api.get<Row[]>(`/api/issues/${issue!.id}/attachments`);
+          const delivered = completedConnectionArtifactRun(next, attachments, filename,
+            { agentId: agent!.id, connectionId: connection.id, method: connection.method, runtimeMode: execution.profile.generation });
+          if (delivered) return delivered;
+          if (next.every(run => ["succeeded", "failed", "cancelled", "interrupted", "timed_out"].includes(run.status))) throw new ConnectionFailure("run_attributed_artifact_missing");
+        }
+        if (status === "blocked") throw new ConnectionFailure("task_blocked_after_provider_run");
+      }
+      if (next.some(run => run.status === "succeeded" && !verifyConnectionRun(run, { agentId: agent!.id, connectionId: connection.id, method: connection.method, runtimeMode: execution.profile.generation }))) throw new ConnectionFailure("run_attribution_mismatch");
+      await connectionDelay(1500);
+    }
+    throw new ConnectionFailure("agent_run_or_attribution_timeout");
+  }
+  async function artifact(filename: string, expected: Record<string, unknown>, run: Row) {
+    const attachments = await api.get<Row[]>(`/api/issues/${issue!.id}/attachments`);
+    const attachment = attachments.find(row => row.originalFilename === filename && row.createdByAgentId === agent!.id && row.originatingRunId === run.id);
+    if (!attachment) throw new ConnectionFailure("run_attributed_artifact_missing");
+    const bytes = await api.bytes(`/api/attachments/${attachment.id}/content`);
+    let actual: Record<string, unknown> = {};
+    try { actual = JSON.parse(bytes.toString("utf8")); } catch { /* The independent verifier fails below. */ }
+    evidence.artifactChecks ??= [];
+    evidence.artifactChecks.push({ filename, runId: run.id, sha256: createHash("sha256").update(bytes).digest("hex"),
+      fields: Object.fromEntries(Object.entries(expected).map(([key, value]) => [key, actual?.[key] === value])), exactFields: Boolean(actual && Object.keys(actual).length === Object.keys(expected).length) });
+    if (!verifyConnectionArtifact(bytes, expected)) throw new ConnectionFailure("independent_artifact_verification_failed");
+    await page.goto(`${api.origin}/${company.issuePrefix}/issues/${issue!.identifier}`, { waitUntil: "domcontentloaded" });
+    await page.reload({ waitUntil: "domcontentloaded" });
+    await expect(page.locator(`a[href*="/api/attachments/${attachment.id}/content"]`).first()).toBeVisible({ timeout: 30_000 });
+    evidence.artifactSha256 = createHash("sha256").update(bytes).digest("hex");
+  }
+  const first = await completedRun(new Set(), "connection-proof.json");
+  evidence.checkpoints.first_run = true;
+  await artifact("connection-proof.json", proof.expected, first);
+  evidence.checkpoints.artifact = true;
+  // Artifact upload provenance is a server-owned tool-side effect in both runner generations.
+  const events = await collectRunEvents<Row>((afterSeq, limit) => api.get(`/api/heartbeat-runs/${first.id}/events?afterSeq=${afterSeq}&limit=${limit}`));
+  const toolReceipt = events.some(event => event.eventType === "tool.execution.completed" || (event.eventType === "item.completed" && ["tool_result", "command_execution"].includes(event.payload?.prpEvent?.payload?.item?.type)));
+  // Legacy adapters may expose their tool transcript only in the run log; upload attribution is still a durable successful tool effect.
+  evidence.checkpoints.tool_receipt = execution.profile.generation === "legacy" || toolReceipt;
+  if (!evidence.checkpoints.tool_receipt) throw new ConnectionFailure("native_tool_receipt_missing");
+  await input.checkpoint("followup");
+  const previousRuns = new Set(runs.keys());
+  await submitTaskReply(page, "Using the original connection-input.json decoded from this task, deliver connection-followup.json with exactly nonce, minimum, maximum, and total of the original values. Use exactly this JSON schema: {\"nonce\": string, \"minimum\": number, \"maximum\": number, \"total\": number}. All four fields are required. Compute with a tool and validate the file against all four fields before attaching it and its download link. Then mark this task done. Do not create other tasks or read credentials.");
+  const second = await completedRun(previousRuns, "connection-followup.json");
+  await artifact("connection-followup.json", proof.followup, second);
+  evidence.checkpoints.followup = true;
+  evidence.checkpoints.visible_result = true;
+  return { issueId: issue.id as string, issueIdentifier: issue.identifier as string };
+}

tests/runner-e2e/connection-launch.ts

@@ -0,0 +1,300 @@
+import { chromium, type BrowserContext } from "@playwright/test";
+import { execFileSync } from "node:child_process";
+import { randomBytes, createHash } from "node:crypto";
+import { mkdir, mkdtemp, writeFile, rm, readdir, lstat, open } from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import { FixtureRegistry } from "./fixture-registry.js";
+import { buildRunnerCampaign } from "./history.js";
+import { renderRunnerE2EDashboard } from "./dashboard.js";
+import { buildRuntimeUsage, summarizeExecutionBilling } from "./billing.js";
+import { stageDashboardBrandAssets } from "./report-assets.js";
+import { classifyFailure } from "./failure-classifier.js";
+import { packageEvidence } from "./evidence.js";
+import { assertSecretFree, sanitizeJson } from "./redaction.js";
+import { connectionCell, connectionHarnesses, connectionModes, connectionScopeGaps, supportsConnection } from "./connection-cases.js";
+import { assertOutsideRepository, ConnectionBlock, ConnectionFailure, loadConnectionConfig, resolveConnectionSecret, resolveConnectionSettings, type ConnectionConfig } from "./connection-config.js";
+import { connectionEvidencePasses, connectionRunDiagnostic, gradeConnectionEvidence, type ConnectionEvidence } from "./connection-evidence.js";
+import { ConnectionApi, connectionDelay, startConnectionTarget, verifyConnectionTarget } from "./connection-target.js";
+import { runConnectionFlow } from "./connection-flow.js";
+import type { MatrixExecution, RunnerE2EResult } from "./types.js";
+
+type Row = Record<string, any>;
+
+export async function openConnectionBrowser(config: ConnectionConfig, repositoryRoot: string, origin: string) {
+  const temporary = config.browser.freshness === "signed-out" || !config.browser.profileDir;
+  const profile = temporary ? await mkdtemp(path.join(os.tmpdir(), "paperclip-qa-browser-")) : path.resolve(config.browser.profileDir!);
+  await assertOutsideRepository(profile, repositoryRoot);
+  await mkdir(profile, { recursive: true, mode: 0o700 });
+  const stat = await lstat(profile);
+  if (stat.isSymbolicLink() || !stat.isDirectory() || (stat.mode & 0o077) !== 0) throw new ConnectionBlock("blocked_target", "browser_profile_requires_private_directory");
+  const marker = path.join(profile, ".paperclip-provider-qa");
+  const entries = await readdir(profile);
+  if (entries.length && !entries.includes(path.basename(marker))) throw new ConnectionBlock("blocked_target", "use_a_dedicated_empty_qa_browser_profile");
+  if (!entries.length) await writeFile(marker, "Dedicated Paperclip provider QA browser. Contains private authentication state.\n", { mode: 0o600, flag: "wx" });
+  const lock = path.join(profile, ".paperclip-qa.lock");
+  try { await (await open(lock, "wx", 0o600)).close(); }
+  catch { throw new ConnectionBlock("blocked_target", "qa_browser_profile_in_use"); }
+  let context: BrowserContext;
+  try {
+    const browserEnv = Object.fromEntries(["PATH", "HOME", "USER", "LOGNAME", "TMPDIR", "DISPLAY", "WAYLAND_DISPLAY", "XAUTHORITY", "LANG"].flatMap(key => process.env[key] ? [[key, process.env[key]!]] : []));
+    context = await chromium.launchPersistentContext(profile, {
+      channel: config.browser.channel === "chrome" ? "chrome" : undefined,
+      headless: !config.browser.headed, baseURL: origin, viewport: { width: 1440, height: 1000 }, reducedMotion: "reduce",
+      env: browserEnv, acceptDownloads: true,
+      // Teardown needs this context's authenticated request client to cancel
+      // login sessions and revoke fixtures before the browser is closed.
+      handleSIGINT: false, handleSIGTERM: false, handleSIGHUP: false,
+      // Deliberately use the Playwright library without Test's automatic reporters,
+      // traces, HARs, videos, snapshots, assertion call logs, or failure screenshots.
+    });
+    context.setDefaultTimeout(30_000);
+    context.setDefaultNavigationTimeout(30_000);
+  } catch {
+    await rm(lock, { force: true });
+    if (temporary) await rm(profile, { recursive: true, force: true });
+    throw new ConnectionBlock("blocked_target", "qa_browser_launch_failed");
+  }
+  return { context, close: async () => { try { await context.close(); } finally { await rm(lock, { force: true }); if (temporary) await rm(profile, { recursive: true, force: true }); } } };
+}
+
+export function safeConnectionFailure(error: unknown, phase: string) {
+  if (error instanceof ConnectionBlock) return error.code;
+  // Arbitrary Playwright/provider messages can contain filled values and URLs.
+  const known = error instanceof ConnectionFailure;
+  const frame = error instanceof Error ? error.stack?.match(/connection-(?:flow|target|launch)\.ts:\d+:\d+/)?.[0] : undefined;
+  return known ? error.message : `${phase}_failed${frame ? ` (${frame})` : ""}`;
+}
+
+function sourceRevision(repositoryRoot: string) {
+  const git = (args: string[]) => execFileSync("git", args, { cwd: repositoryRoot, encoding: "utf8", stdio: ["ignore", "pipe", "ignore"] }).trim();
+  return { sha: git(["rev-parse", "HEAD"]), ref: `${git(["branch", "--show-current"])}${git(["status", "--porcelain"]) ? " (working tree)" : ""}`, workflowRunUrl: null };
+}
+
+async function safeWrite(file: string, value: unknown, secrets: readonly string[]) {
+  const text = `${JSON.stringify(sanitizeJson(value, secrets), null, 2)}\n`;
+  assertSecretFree(text, secrets, path.basename(file));
+  await writeFile(file, text, { mode: 0o600 });
+}
+
+async function connectionAttempt(execution: MatrixExecution, config: ConnectionConfig, origin: string, summaryDir: string, repositoryRoot: string): Promise<RunnerE2EResult> {
+  const startedAt = new Date().toISOString();
+  const nonce = randomBytes(6).toString("hex");
+  const privateDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-connection-proof-"));
+  const evidenceDir = path.join(summaryDir, execution.suite.id, execution.profile.id, execution.environment.id, execution.task.id, "attempt-1");
+  await mkdir(evidenceDir, { recursive: true, mode: 0o700 });
+  const { entry, mode } = connectionCell(execution);
+  const evidence: ConnectionEvidence = { version: 1, outcome: "running", phase: "target", assisted: false, authFreshness: config.browser.freshness,
+    target: { mode: config.target.mode, origin, commit: null, deploymentMode: null }, entry, method: mode, checkpoints: {}, waits: [] };
+  const secrets: string[] = [];
+  const checkpoint = async (phase: string) => { evidence.phase = phase; await safeWrite(path.join(summaryDir, "progress.json"), { executionId: execution.id, ...evidence }, secrets); };
+  const result: RunnerE2EResult = { schema: "paperclip.runner-e2e.result/v2", executionId: execution.id, suiteId: execution.suite.id, suiteDefinitionHash: execution.suiteDefinitionHash,
+    source: sourceRevision(repositoryRoot), attempt: 1, status: "failed", profileId: execution.profile.id, environmentId: execution.environment.id, caseId: execution.task.id,
+    provider: execution.profile.provider, model: execution.profile.model, runtimeMode: execution.profile.generation, startedAt, finishedAt: startedAt, durationMs: 0, cleanup: "not_started", providerConnection: evidence };
+  let browser: Awaited<ReturnType<typeof openConnectionBrowser>> | undefined;
+  let fixtures: Awaited<ReturnType<FixtureRegistry["setupAll"]>> | undefined;
+  const observedRuns = new Map<string, Row>();
+  let diagnosticApi: ConnectionApi | undefined;
+  const loginSessions = new Map<string, "DELETE" | "POST">();
+  const loginObservers: Promise<void>[] = [];
+  let stopped: "test_interrupted" | "cell_deadline_reached" | undefined;
+  const stopBrowserActions = () => { for (const page of browser?.context.pages() ?? []) void page.close().catch(() => {}); };
+  const interrupt = () => { stopped = "test_interrupted"; stopBrowserActions(); };
+  const assertActive = () => { if (stopped) throw new ConnectionBlock("blocked_target", stopped); };
+  process.once("SIGINT", interrupt);
+  process.once("SIGTERM", interrupt);
+  const deadlineTimer = setTimeout(() => { stopped = "cell_deadline_reached"; stopBrowserActions(); }, execution.task.attemptTimeoutMs?.[execution.environment.id] ?? 30 * 60_000);
+  try {
+    const identity = await verifyConnectionTarget(origin, config);
+    Object.assign(evidence.target, identity);
+    evidence.checkpoints.target = true;
+    assertActive();
+    const settings = resolveConnectionSettings(execution, config);
+    result.model = settings.model;
+    evidence.model = settings.model;
+    if (mode === "subscription" && !config.browser.headed) throw new ConnectionBlock("awaiting_user", "subscription_requires_headed_browser");
+    // Resolve only the selected credential, and only after validating the target.
+    const secret = await resolveConnectionSecret(config, settings.credentialEnv);
+    if (secret) secrets.push(secret);
+    browser = await openConnectionBrowser(config, repositoryRoot, origin);
+    const page = browser.context.pages()[0] ?? await browser.context.newPage();
+    const api = new ConnectionApi(browser.context.request, origin);
+    diagnosticApi = api;
+    browser.context.on("response", response => {
+      const url = new URL(response.url());
+      if (url.origin !== origin || response.request().method() !== "POST" || !response.ok()) return;
+      const local = /^\/api\/companies\/([a-f0-9-]+)\/ai-connections\/local\/attempts$/.test(url.pathname);
+      const cloud = /^\/api\/companies\/([a-f0-9-]+)\/(?:setup-token-login-sessions|adapters\/[^/]+\/login-sessions)$/.test(url.pathname);
+      if (!local && !cloud) return;
+      loginObservers.push((async () => {
+        const body = await response.json();
+        if (typeof body.sessionId === "string" && /^[a-zA-Z0-9_-]+$/.test(body.sessionId)) loginSessions.set(`${url.pathname}/${body.sessionId}${cloud ? "/cancel" : ""}`, local ? "DELETE" : "POST");
+      })().catch(() => {}));
+    });
+    await page.goto(origin, { waitUntil: "domcontentloaded" });
+    try { await api.get("/api/companies"); }
+    catch (authError) {
+      if (!(authError instanceof ConnectionFailure) || !["api_status_401", "api_status_403"].includes(authError.message)) throw authError;
+      if (!config.browser.headed) throw new ConnectionBlock("awaiting_user", "board_login_required");
+      evidence.assisted = true;
+      evidence.waits.push({ kind: "board_login", startedAt: new Date().toISOString() });
+      await checkpoint("awaiting_board_login");
+      console.log(`[provider-connections] Sign in to Paperclip at ${origin} in the QA browser; this test resumes automatically.`);
+      const deadline = Date.now() + config.browser.loginTimeoutMs;
+      let ready = false;
+      while (Date.now() < deadline) { assertActive(); try { await api.get("/api/companies"); ready = true; break; } catch { await connectionDelay(1000); } }
+      if (!ready) throw new ConnectionBlock("awaiting_user", "board_login_deadline_reached");
+      evidence.waits.at(-1)!.finishedAt = new Date().toISOString();
+    }
+    const registry = new FixtureRegistry();
+    const attachedCompany = config.target.mode === "attach" ? config.target.companyId : undefined;
+    registry.register<Row>({ id: "connection-company", setup: async () => {
+      if (attachedCompany) {
+        const company = await api.get(`/api/companies/${attachedCompany}`);
+        if (!company.name.startsWith("Connection QA") || (await api.get<Row[]>(`/api/companies/${company.id}/agents`)).some(agent => agent.status !== "terminated") || (await api.get(`/api/companies/${company.id}/ai-connections`)).connections.some((connection: Row) => connection.status !== "revoked")) throw new ConnectionBlock("blocked_target", "attach_requires_empty_dedicated_qa_company");
+        return company;
+      }
+      return api.post("/api/companies", { name: `Connection QA ${nonce}`, description: "Disposable provider-connections QA fixture", budgetMonthlyCents: config.budgetCents });
+    }, teardown: async company => {
+      await Promise.all(loginObservers);
+      for (const [route, method] of loginSessions) {
+        try { await api.response(route, method); }
+        catch (error) { if (!(error instanceof ConnectionFailure) || error.message !== "api_status_404") throw error; }
+      }
+      const agents = await api.get<Row[]>(`/api/companies/${company.id}/agents`);
+      for (const agent of agents.filter(agent => agent.name === `Connection QA ${nonce}`)) {
+        await api.patch(`/api/agents/${agent.id}`, { status: "paused" });
+        const liveRuns = await api.get<Row[]>(`/api/companies/${company.id}/heartbeat-runs?agentId=${agent.id}&limit=100`);
+        const active = liveRuns.filter(run => run.agentId === agent.id && ["queued", "running"].includes(run.status));
+        for (const run of active) await api.post(`/api/heartbeat-runs/${run.id}/cancel`);
+      }
+      if (config.retainCompany) { evidence.cleanupRetained = true; return; }
+      // The fixture verified the company was empty before running. Revoke all
+      // newly created test connections even if a later UI assertion failed.
+      const connections = (await api.get(`/api/companies/${company.id}/ai-connections`)).connections;
+      for (const connection of connections.filter((connection: Row) => connection.status !== "revoked")) await api.delete(`/api/tool-connections/${connection.id}`);
+      if ((await api.get(`/api/companies/${company.id}/ai-connections`)).connections.some((connection: Row) => connection.status !== "revoked")) throw new ConnectionFailure("connection_revocation_failed");
+      if (!attachedCompany) { await api.post(`/api/companies/${company.id}/archive`); evidence.companyArchived = true; }
+      else for (const agent of agents.filter(agent => agent.name === `Connection QA ${nonce}`)) await api.delete(`/api/agents/${agent.id}`);
+    } });
+    registry.register<Row>({ id: "connection-environment", dependencies: ["connection-company"], setup: async values => {
+      const company = values.get("connection-company") as Row;
+      const environments = await api.get<Row[]>(`/api/companies/${company.id}/environments`);
+      const selectedId = config.target.mode === "attach" ? config.target.environmentId : undefined;
+      const environment = selectedId ? environments.find(env => env.id === selectedId) : environments.find(env => env.status === "active" && (execution.environment.id === "local" ? env.driver === "local" : env.driver === "sandbox" && env.config?.provider === "daytona"));
+      if (!environment || (execution.environment.id === "local" ? environment.driver !== "local" : environment.config?.provider !== "daytona")) throw new ConnectionBlock("blocked_target", "configured_execution_environment_required");
+      if (execution.profile.generation === "native") {
+        const experimental = await api.get("/api/instance/settings/experimental");
+        if (!experimental.enableNativeRunner) {
+          if (config.target.mode === "attach") throw new ConnectionBlock("blocked_target", "native_runner_disabled_on_target");
+          await api.patch("/api/instance/settings/experimental", { enableNativeRunner: true });
+        }
+      }
+      return environment;
+    } });
+    fixtures = await registry.setupAll();
+    const company = fixtures.values.get("connection-company") as Row;
+    evidence.companyId = company.id;
+    const environment = fixtures.values.get("connection-environment") as Row;
+    const observed = await runConnectionFlow({ page, api, execution, config, settings, company, environment, secret, nonce, privateDir, evidence, checkpoint, assertActive, observeRun: run => observedRuns.set(run.id, run) });
+    Object.assign(result, observed);
+    evidence.outcome = "passed";
+    if (!connectionEvidencePasses(evidence)) throw new ConnectionFailure("connection_proof_incomplete");
+    // Only the fixture's synthetic task is captured, after authentication.
+    const expectedPath = `/${company.issuePrefix}/issues/${observed.issueIdentifier}`;
+    const url = new URL(page.url());
+    if (url.origin !== origin || url.pathname !== expectedPath) throw new ConnectionFailure("final_screenshot_route_mismatch");
+    const screenshot = await page.screenshot({ path: path.join(privateDir, "final-state.png"), fullPage: true });
+    result.screenshots = [{ id: "final-state", label: "Verified connection task and follow-up", file: "final-state.png", sha256: createHash("sha256").update(screenshot).digest("hex") }];
+    result.status = "passed";
+  } catch (caught) {
+    const error = stopped ? new ConnectionBlock("blocked_target", stopped) : caught;
+    if (error instanceof AggregateError) result.cleanup = "failed";
+    evidence.outcome = error instanceof ConnectionBlock ? error.reason : "failed";
+    result.failureClass = error instanceof ConnectionBlock ? "permanent_infrastructure" : classifyFailure(error);
+    result.error = safeConnectionFailure(error, evidence.phase);
+  } finally {
+    clearTimeout(deadlineTimer);
+    try { await fixtures?.teardown(); if (result.cleanup !== "failed") result.cleanup = "passed"; }
+    catch (cleanupError) { result.cleanup = "failed"; result.status = "failed"; result.failureClass = "cleanup_failure"; result.error = `${result.error ? `${result.error}; ` : ""}fixture_cleanup_failed`;
+      const causes = cleanupError instanceof AggregateError ? cleanupError.errors : [cleanupError];
+      console.log(`[provider-connections] Cleanup: ${causes.map(cause => safeConnectionFailure(cause, "cleanup")).join(", ")}`); }
+    // Project errors before a managed target removes its DB/logs.
+    if (diagnosticApi && observedRuns.size) {
+      evidence.runDiagnostics = [];
+      for (const observed of observedRuns.values()) {
+        let run = observed; let log: unknown;
+        try { run = await diagnosticApi.get(`/api/heartbeat-runs/${observed.id}`); observedRuns.set(observed.id, run); } catch { /* Keep the last durable observation. */ }
+        try { log = await diagnosticApi.get(`/api/heartbeat-runs/${observed.id}/log?limitBytes=2000000`); } catch { /* Mark missing log evidence explicitly. */ }
+        evidence.runDiagnostics.push(connectionRunDiagnostic(run, log));
+      }
+    }
+    try { await browser?.close(); }
+    catch { result.cleanup = "failed"; result.status = "failed"; result.failureClass = "cleanup_failure"; result.error = "qa_browser_cleanup_failed"; }
+    result.finishedAt = new Date().toISOString();
+    result.durationMs = Date.parse(result.finishedAt) - Date.parse(startedAt);
+    result.runIds = [...observedRuns.keys()];
+    // Usage is projected to numeric billing below; opaque provider output is never retained.
+    const usage = { runs: [...observedRuns.values()].map(run => ({ usage: run.usageJson })) };
+    result.runtimeUsage = buildRuntimeUsage({ environmentId: execution.environment.id, runs: [...observedRuns.values()], fallbackFinishedAt: result.finishedAt });
+    result.billing = summarizeExecutionBilling({ ...result, usage });
+    // UI environment probes may incur cost without heartbeat usage receipts.
+    result.billing.complete = false;
+    result.billing.llm.costStatus = result.billing.llm.runsWithTokenUsage ? "partial" : "unavailable";
+    result.matcherResults = gradeConnectionEvidence(evidence);
+    await safeWrite(path.join(privateDir, "result.json"), result, secrets);
+    const manifest = await packageEvidence({ privateDir, uploadDir: evidenceDir, secrets, expectPassScreenshot: false });
+    if (manifest.leaks.length || manifest.missing.length) { result.status = "failed"; result.failureClass = "secret_leak"; result.error = "evidence_packaging_failed"; evidence.outcome = "failed"; }
+    await safeWrite(path.join(evidenceDir, "result.json"), result, secrets);
+    await safeWrite(path.join(evidenceDir, "evidence-manifest.json"), manifest, secrets);
+    await safeWrite(path.join(summaryDir, "progress.json"), { executionId: execution.id, ...evidence }, secrets);
+    await rm(privateDir, { recursive: true, force: true });
+    process.removeListener("SIGINT", interrupt);
+    process.removeListener("SIGTERM", interrupt);
+  }
+  return result;
+}
+
+export async function runConnectionCampaign(input: { executions: readonly MatrixExecution[]; catalog: readonly MatrixExecution[]; configFile?: string; repositoryRoot: string }) {
+  if (input.executions.some(e => e.task.flow !== "provider_connection")) throw new ConnectionFailure("Select provider-connections separately from other suites.");
+  const config = await loadConnectionConfig(input.configFile);
+  const campaignId = `connections-${new Date().toISOString().replace(/[:.]/g, "-")}-${randomBytes(3).toString("hex")}`;
+  const summaryDir = path.join(input.repositoryRoot, "tests/runner-e2e/results", campaignId);
+  await mkdir(summaryDir, { recursive: true, mode: 0o700 });
+  const results: RunnerE2EResult[] = [];
+  const target = await startConnectionTarget(config, input.executions, input.repositoryRoot);
+  try {
+    console.log(`[provider-connections] ${input.executions.length} selected cells; target ${target.origin}; one browser/account at a time; max ${config.maxRuns} task runs and ${config.budgetCents} cents per fixture. UI probe spend may be unreported.`);
+    for (const execution of input.executions) {
+      console.log(`[provider-connections] Starting ${execution.id}`);
+      const result = await connectionAttempt(execution, config, target.origin, summaryDir, input.repositoryRoot);
+      results.push(result);
+      console.log(`[provider-connections] ${result.providerConnection!.outcome}: ${execution.id}${result.error ? ` (${result.error})` : ""}`);
+      // Every completed/blocked cell is durable before the next login begins.
+      await writeConnectionCampaignReport(input, campaignId, summaryDir, results);
+      if (result.error === "test_interrupted" || result.providerConnection!.outcome === "awaiting_user") break;
+    }
+  } finally { await target.stop(); }
+  console.log(`[provider-connections] Report: ${path.join(summaryDir, "dashboard.html")}`);
+  if (results.some(result => result.status !== "passed" || result.cleanup !== "passed")) process.exitCode = 1;
+  return summaryDir;
+}
+
+export async function writeConnectionCampaignReport(input: { executions: readonly MatrixExecution[]; catalog: readonly MatrixExecution[] }, campaignId: string, summaryDir: string, results: RunnerE2EResult[]) {
+  const generatedAt = new Date().toISOString();
+  const expected = input.executions.map(e => e.id);
+  const campaign = buildRunnerCampaign({ campaignId, generatedAt, expected, results });
+  await safeWrite(path.join(summaryDir, "campaign.json"), campaign, []);
+  await safeWrite(path.join(summaryDir, "normalized-results.json"), { ...campaign, results: results.map(result => ({ ...result, evidenceValid: result.failureClass !== "secret_leak", evidenceErrors: result.failureClass === "secret_leak" ? ["evidence_packaging_failed"] : [] })) }, []);
+  await stageDashboardBrandAssets(summaryDir);
+  await safeWrite(path.join(summaryDir, "connection-coverage.json"), {
+    required: input.catalog.filter(e => e.suite.id === "provider-connections").length,
+    selected: expected.length, attempted: results.length, passed: results.filter(r => r.status === "passed" && r.cleanup === "passed").length,
+    blocked: results.filter(r => ["awaiting_user", "missing_credential", "blocked_target"].includes(r.providerConnection!.outcome)).length,
+    targets: [...new Set(results.map(r => r.providerConnection!.target.origin))],
+    gaps: [...connectionScopeGaps, ...connectionHarnesses.flatMap(h => connectionModes.filter(mode => !supportsConnection(h, mode)).map(mode => ({ adapter: h.adapter, mode, status: "unsupported" })))],
+  }, []);
+  const dashboard = renderRunnerE2EDashboard({ title: `Live provider connections · ${campaignId}`, generatedAt, expected, catalog: input.catalog, campaign,
+    entries: results.map(result => ({ result, valid: result.failureClass !== "secret_leak", errors: result.failureClass === "secret_leak" ? ["evidence_packaging_failed"] : [], evidenceBaseHref: [result.suiteId, result.profileId, result.environmentId, result.caseId, "attempt-1"].join("/") })) });
+  await writeFile(path.join(summaryDir, "dashboard.html"), dashboard, { mode: 0o600 });
+}

tests/runner-e2e/connection-support.test.ts

@@ -0,0 +1,176 @@
+import { describe, expect, it, vi } from "vitest";
+import { mkdtemp, writeFile, rm } from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import { AGENT_ADAPTER_TYPES } from "../../packages/shared/src/constants.js";
+import { runnerMatrix } from "./catalog.js";
+import { parseRunnerSelectors, selectRunnerExecutions, buildMatrixJobs } from "./selectors.js";
+import { connectionHarnesses, connectionScopeGaps, connectionCell, supportsConnection } from "./connection-cases.js";
+import { parseConnectionConfig, targetOrigin, selectedSecret, resolveConnectionSecret, resolveConnectionSettings } from "./connection-config.js";
+import { connectionCheckpoints, connectionEvidencePasses, connectionRunDiagnostic, createConnectionProof, verifyConnectionArtifact, verifyConnectionRun, completedConnectionArtifactRun, connectionProbeChecks, type ConnectionEvidence } from "./connection-evidence.js";
+import { validateRetainedRunnerResult } from "./result-validation.js";
+import { verifyConnectionTarget } from "./connection-target.js";
+import { safeConnectionFailure } from "./connection-launch.js";
+import { buildRunnerCampaign } from "./history.js";
+import { renderCaseOutcome } from "./case-outcome.js";
+import type { RunnerE2EResult } from "./types.js";
+
+const cells = runnerMatrix.filter(e => e.suite.id === "provider-connections");
+const cell = (method: string) => cells.find(e => e.id === `provider-connections.connection-codex-native.local.agent-${method}`)!;
+
+describe("live connection coverage", () => {
+  it("inventories every built-in adapter and keeps remote exclusions explicit", () => {
+    const inventory = new Set<string>(["paperclip_runner", ...connectionHarnesses.map(h => h.adapter), ...connectionScopeGaps.map(h => h.adapter)]);
+    expect(AGENT_ADAPTER_TYPES.filter(type => !inventory.has(type))).toEqual([]);
+    expect(connectionHarnesses.some(h => h.adapter === "hermes_local")).toBe(true);
+    expect(connectionScopeGaps.find(h => h.adapter === "hermes_gateway")?.status).toBe("excluded");
+  });
+  it("includes both UI entry points and actual runner variants without invented auth capabilities", () => {
+    expect(cells.length).toBe(116);
+    expect(new Set(cells.map(e => connectionCell(e).entry))).toEqual(new Set(["agent", "apps"]));
+    expect(cells.filter(e => connectionCell(e).harness.id === "grok").every(e => ["subscription", "api-key"].includes(connectionCell(e).mode))).toBe(true);
+    expect(supportsConnection(connectionHarnesses.find(h => h.id === "gemini")!, "subscription")).toBe(false);
+  });
+  it("never runs on --all or advertises a preinstalled subscription token as a credential", () => {
+    expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(e => e.suite.id === "provider-connections")).toBe(false);
+    expect(cell("subscription").requiredCredentials).toEqual([]);
+    expect(() => buildMatrixJobs(cells)).toThrow(/not admitted/);
+    expect(parseRunnerSelectors(["--suite", "provider-connections", "--connection-config", "/private/qa.json"]).connectionConfig).toBe("/private/qa.json");
+  });
+});
+
+describe("target and selected credential boundaries", () => {
+  it("accepts local/staging origins and rejects credential-bearing or redirectable target strings", () => {
+    expect(targetOrigin("http://127.0.0.1:3109/")).toBe("http://127.0.0.1:3109");
+    expect(targetOrigin("https://staging.example.test")).toBe("https://staging.example.test");
+    for (const url of ["http://staging.example.test", "https://user:password@example.test", "https://example.test/path", "https://example.test/?token=secret", "https://example.test/#secret", "file:///tmp/qa"]) expect(() => targetOrigin(url)).toThrow();
+  });
+  it("requires an exact staging revision and rejects unknown configuration instead of ignoring typos", () => {
+    expect(() => parseConnectionConfig({ target: { mode: "attach", baseURL: "https://example.test" } })).toThrow();
+    expect(() => parseConnectionConfig({ target: { mode: "managed-local", baseURL: "https://example.test" } })).toThrow();
+    expect(() => parseConnectionConfig({ broswer: {} })).toThrow();
+    expect(() => parseConnectionConfig({ browser: { loginTimeoutMs: Infinity } })).toThrow();
+    expect(parseConnectionConfig({}).browser.headed).toBe(true);
+  });
+  it("pins the target revision before permitting credential selection", async () => {
+    const config = parseConnectionConfig({ target: { mode: "attach", baseURL: "https://staging.example.test", expectedCommit: "a".repeat(40), deploymentMode: "authenticated" } });
+    const fetch = vi.spyOn(globalThis, "fetch").mockResolvedValue(new Response(JSON.stringify({ commit: "b".repeat(40), deploymentMode: "authenticated" })));
+    try {
+      await expect(verifyConnectionTarget("https://staging.example.test", config)).rejects.toThrow("target_identity_mismatch");
+      expect(fetch.mock.calls[0]?.[1]).toMatchObject({ redirect: "error" });
+      fetch.mockResolvedValue(new Response(JSON.stringify({ commit: "a".repeat(40), deploymentMode: "authenticated" })));
+      await expect(verifyConnectionTarget("https://staging.example.test", config)).resolves.toMatchObject({ commit: "a".repeat(40) });
+    } finally { fetch.mockRestore(); }
+  });
+  it("reads literal selected keys without sourcing shell commands or returning unrelated secrets", async () => {
+    expect(selectedSecret("UNRELATED=do-not-read\nexport OPENAI_API_KEY='chosen-value'\n", "OPENAI_API_KEY")).toBe("chosen-value");
+    expect(selectedSecret("OPENAI_API_KEY=one\nOPENAI_API_KEY=two", "OPENAI_API_KEY")).toBe("two");
+    expect(() => selectedSecret("OPENAI_API_KEY=$(touch /tmp/should-not-exist)", "OPENAI_API_KEY")).toThrow();
+    const root = await mkdtemp(path.join(os.tmpdir(), "connection-secret-test-"));
+    try {
+      const file = path.join(root, "keys");
+      await writeFile(file, "OPENAI_API_KEY=selected-only\nUNRELATED=private\n", { mode: 0o600 });
+      const config = parseConnectionConfig({ secretFile: file });
+      expect(await resolveConnectionSecret(config, "OPENAI_API_KEY", {})).toBe("selected-only");
+      expect(await resolveConnectionSecret(config, undefined, {})).toBeUndefined();
+      await expect(resolveConnectionSecret(config, "XAI_API_KEY", {})).rejects.toThrow("missing_XAI_API_KEY");
+      await writeFile(path.join(root, "public"), "OPENAI_API_KEY=private", { mode: 0o644 });
+      await expect(resolveConnectionSecret(parseConnectionConfig({ secretFile: path.join(root, "public") }), "OPENAI_API_KEY", {})).rejects.toThrow(/owner_only/);
+    } finally { await rm(root, { recursive: true, force: true }); }
+  });
+  it("never resolves an API key for a subscription or defaults a gateway to the direct provider's credential", () => {
+    expect(resolveConnectionSettings(cell("subscription"), parseConnectionConfig({})).credentialEnv).toBeUndefined();
+    expect(resolveConnectionSettings(cell("api-key"), parseConnectionConfig({})).credentialEnv).toBe("OPENAI_API_KEY");
+    expect(() => resolveConnectionSettings(cell("responses"), parseConnectionConfig({ routes: { responses: { model: "qa-alias", baseURL: "https://gateway.example.test/v1" } } }))).toThrow(/credential_reference/);
+    const config = parseConnectionConfig({ routes: { responses: { model: "qa-alias", baseURL: "https://gateway.example.test/v1", credentialEnv: "GATEWAY_QA_KEY" } } });
+    expect(resolveConnectionSettings(cell("responses"), config).credentialEnv).toBe("GATEWAY_QA_KEY");
+  });
+});
+
+function passingEvidence(): ConnectionEvidence {
+  return { version: 1, outcome: "passed", phase: "finished", assisted: true, authFreshness: "signed-in", target: { mode: "attach", origin: "https://staging.example.test", commit: "a".repeat(40), deploymentMode: "authenticated" }, entry: "agent", method: "subscription", checkpoints: Object.fromEntries(connectionCheckpoints.map(key => [key, true])), waits: [] };
+}
+
+describe("independent qualification", () => {
+  it("retains closed failure signals across split log chunks without messages or reasoning", () => {
+    const sentinel = "SENTINEL_CREDENTIAL_DO_NOT_RETAIN";
+    const rows = [
+      { type: "acpx.text_delta", channel: "thought", text: 'HTTP 429 '+sentinel },
+      { type: "acpx.error", childStderrTail: 'ApiError: {"error":{"code":503,"message":"model overloaded '+sentinel+'"}}' },
+      { type: "acpx.tool_call", status: "failed", text: "Error checking existing file: Resource not found: "+sentinel },
+    ].map(row => JSON.stringify(row)).join("\n")+"\n";
+    const split = Math.floor(rows.length / 2);
+    const log = { content: [rows.slice(0, split), rows.slice(split)].map(chunk => JSON.stringify({ chunk })).join("\n") };
+    const diagnostic = connectionRunDiagnostic({ id: "run-1", status: "failed" }, log);
+    expect(diagnostic).toEqual({ runId: "run-1", status: "failed", signals: ["provider_overloaded", "missing_file_conversion"], logAvailable: true });
+    expect(JSON.stringify(diagnostic)).not.toContain(sentinel);
+    expect(connectionRunDiagnostic({ id: "run-2", status: "running" })).toEqual({ runId: "run-2", status: "unknown", signals: [], logAvailable: false });
+  });
+  it("distinguishes terminal quota and unsupported configuration from generic failures", () => {
+    expect(connectionRunDiagnostic({ id: "quota", status: "failed", error: 'HTTP 429 RESOURCE_EXHAUSTED' }).signals).toEqual(["provider_quota"]);
+    expect(connectionRunDiagnostic({ id: "config", status: "failed", error: 'session/set_config_option: Method not found (-32601)' }).signals).toEqual(["unsupported_session_configuration"]);
+    expect(connectionRunDiagnostic({ id: "generic", status: "failed", error: 'unknown provider error' }).signals).toEqual([]);
+    expect(connectionRunDiagnostic({ id: "unattributed-503", status: "failed", error: 'HTTP 503' }).signals).toEqual([]);
+  });
+  it("waits for disposition repair and requires the successful artifact-producing run", () => {
+    const expected = { agentId: "a", connectionId: "c", method: "api_key", runtimeMode: "legacy" };
+    const first = { id: "first", agentId: "a", status: "succeeded", runtimeMode: "legacy", contextSnapshot: { aiConnection: { connectionId: "c", method: "api_key" } } };
+    const repair = { ...first, id: "repair" };
+    const file = { originalFilename: "proof.json", createdByAgentId: "a", originatingRunId: "repair" };
+    expect(completedConnectionArtifactRun([first, { ...repair, status: "running" }], [file], "proof.json", expected)).toBeUndefined();
+    expect(completedConnectionArtifactRun([first, repair], [file], "proof.json", expected)).toBe(repair);
+    expect(completedConnectionArtifactRun([first, { ...repair, status: "cancelled" }], [file], "proof.json", expected)).toBeUndefined();
+    expect(completedConnectionArtifactRun([first, repair], [{ ...file, createdByAgentId: "someone-else" }], "proof.json", expected)).toBeUndefined();
+    expect(completedConnectionArtifactRun([first, repair], [file], "followup.json", expected)).toBeUndefined();
+    expect(completedConnectionArtifactRun([first, repair], [file], "proof.json", { ...expected, connectionId: "another" })).toBeUndefined();
+  });
+  it("retains probe codes without provider text, URLs, or credential-bearing fields", () => {
+    const sentinel = "SENTINEL_CREDENTIAL_DO_NOT_RETAIN";
+    const checks = connectionProbeChecks({ checks: [
+      { code: "hermes_cli_not_found", level: "error", message: sentinel, detail: `https://provider.test/?code=${sentinel}` },
+      { code: sentinel, level: "error" }, { code: "safe_code", level: sentinel }, null,
+    ] });
+    expect(checks).toEqual([{ code: "hermes_cli_not_found", level: "error" }]);
+    expect(JSON.stringify(checks)).not.toContain(sentinel);
+  });
+  it("rejects plausible prose, wrong bytes and wrong run attribution", () => {
+    const proof = createConnectionProof();
+    expect(verifyConnectionArtifact(Buffer.from(JSON.stringify(proof.expected)), proof.expected)).toBe(true);
+    expect(verifyConnectionArtifact(Buffer.from("Provider routing works"), proof.expected)).toBe(false);
+    expect(verifyConnectionArtifact(Buffer.from(JSON.stringify({ ...proof.expected, total: proof.expected.total + 1 })), proof.expected)).toBe(false);
+    const expected = { agentId: "a", connectionId: "c", method: "subscription", runtimeMode: "native", model: "m" };
+    const run = { agentId: "a", status: "succeeded", runtimeMode: "native", contextSnapshot: { aiConnection: { connectionId: "c", method: "subscription" } } };
+    expect(verifyConnectionRun(run, expected)).toBe(true);
+    expect(verifyConnectionRun(run, { ...expected, connectionId: "ambient" })).toBe(false);
+    expect(verifyConnectionRun(run, { ...expected, method: "api_key" })).toBe(false);
+  });
+  it("requires every checkpoint and never passes absent evidence or a human deadline", () => {
+    expect(connectionEvidencePasses(passingEvidence())).toBe(true);
+    for (const key of connectionCheckpoints) {
+      const evidence = passingEvidence(); delete evidence.checkpoints[key];
+      expect(connectionEvidencePasses(evidence)).toBe(false);
+    }
+    expect(connectionEvidencePasses({ ...passingEvidence(), outcome: "awaiting_user" })).toBe(false);
+    expect(connectionEvidencePasses({ ...passingEvidence(), target: { ...passingEvidence().target, commit: null } })).toBe(false);
+  });
+  it("withholds raw Playwright call logs including filled secrets, codes, and callback URLs", () => {
+    const sentinel = "SENTINEL_CREDENTIAL_DO_NOT_RETAIN";
+    for (const message of [`locator.fill: value ${sentinel}`, `https://provider.test/callback?code=${sentinel}`, `password=${sentinel}`, "lowercasesecret"]) {
+      const safe = safeConnectionFailure(new Error(message), "connection_creation");
+      expect(safe).toBe("connection_creation_failed"); expect(safe).not.toContain(sentinel);
+    }
+  });
+  it("keeps blocked authentication incomplete in the existing campaign/report", () => {
+    const execution = cell("subscription");
+    const result: RunnerE2EResult = { schema: "paperclip.runner-e2e.result/v2", executionId: execution.id, suiteId: execution.suite.id, suiteDefinitionHash: execution.suiteDefinitionHash,
+      attempt: 1, status: "failed", failureClass: "permanent_infrastructure", error: "provider_login_deadline_reached", profileId: execution.profile.id, environmentId: "local", caseId: execution.task.id, provider: "codex", model: "model", runtimeMode: "native",
+      startedAt: "2026-10-03T00:00:00Z", finishedAt: "2026-10-03T00:01:00Z", durationMs: 60_000, cleanup: "passed", providerConnection: { ...passingEvidence(), outcome: "awaiting_user", checkpoints: { target: true } } };
+    const campaign = buildRunnerCampaign({ campaignId: "test", generatedAt: result.finishedAt, expected: [execution.id], results: [result] });
+    expect(() => validateRetainedRunnerResult({ ...result, status: "passed" })).toThrow(/providerConnection/);
+    expect(() => validateRetainedRunnerResult({ ...result, status: "passed", providerConnection: undefined })).toThrow(/providerConnection/);
+    expect(campaign.passed).toBe(0); expect(campaign.incomplete).toBe(1); expect(campaign.complete).toBe(false);
+    expect(renderCaseOutcome(result, false, [])).toContain("Blocked: awaiting user");
+    const cleanupFailed = buildRunnerCampaign({ campaignId: "failed-cleanup", generatedAt: result.finishedAt, expected: [execution.id], results: [{ ...result, cleanup: "failed", failureClass: "cleanup_failure" }] });
+    expect(cleanupFailed.failed).toBe(1); expect(cleanupFailed.incomplete).toBe(0);
+  });
+});

tests/runner-e2e/connection-target.ts

@@ -0,0 +1,93 @@
+import { spawn, type ChildProcess } from "node:child_process";
+import { randomBytes } from "node:crypto";
+import { mkdir, mkdtemp, rm } from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import type { APIRequestContext } from "@playwright/test";
+import { reserveRunnerE2EServerPort } from "./ports.js";
+import { resolvePaperclipRunnerBinaryForHarness } from "./harness-env.js";
+import { runnerE2ETypeScriptProcessArgs } from "./web-server-command.js";
+import { ConnectionBlock, ConnectionFailure, targetOrigin, type ConnectionConfig } from "./connection-config.js";
+import type { MatrixExecution } from "./types.js";
+
+export const connectionDelay = (ms: number) => new Promise<void>(resolve => setTimeout(resolve, ms));
+
+/** No raw response bodies, callback URLs, headers, or Playwright diagnostics escape. */
+export class ConnectionApi {
+  constructor(readonly request: APIRequestContext, readonly origin: string) {}
+  async response(route: string, method = "GET", data?: unknown) {
+    if (!route.startsWith("/api/") || new URL(route, this.origin).origin !== this.origin) throw new ConnectionFailure("invalid_api_route");
+    try {
+      const response = await this.request.fetch(`${this.origin}${route}`, { method, ...(data === undefined ? {} : { data }), maxRedirects: 0, timeout: 30_000 });
+      if (!response.ok()) throw new ConnectionFailure(`api_status_${response.status()}`);
+      return response;
+    } catch (error) {
+      if (error instanceof Error && /^api_status_\d+$/.test(error.message)) throw error;
+      throw new ConnectionFailure("api_request_failed");
+    }
+  }
+  async get<T = any>(route: string): Promise<T> { return (await this.response(route)).json(); }
+  async post<T = any>(route: string, data?: unknown): Promise<T> { return (await this.response(route, "POST", data)).json(); }
+  async patch<T = any>(route: string, data: unknown): Promise<T> { return (await this.response(route, "PATCH", data)).json(); }
+  async delete(route: string) { await this.response(route, "DELETE"); }
+  async bytes(route: string) { return (await this.response(route)).body(); }
+}
+
+async function stopOwnedServer(child: ChildProcess) {
+  if (!child.pid) return;
+  const exited = () => child.exitCode !== null || child.signalCode !== null;
+  const kill = (signal: NodeJS.Signals) => {
+    try { process.kill(process.platform === "win32" ? child.pid! : -child.pid!, signal); }
+    catch (error) { if ((error as NodeJS.ErrnoException).code !== "ESRCH") throw error; }
+  };
+  kill("SIGTERM");
+  const deadline = Date.now() + 45_000;
+  while (!exited() && Date.now() < deadline) await connectionDelay(100);
+  // The group is exclusively the server this function spawned, never an attached target.
+  kill("SIGKILL");
+}
+
+export async function startConnectionTarget(config: ConnectionConfig, executions: readonly MatrixExecution[], repositoryRoot: string) {
+  if (config.target.mode === "attach") return { origin: targetOrigin(config.target.baseURL), stop: async () => {} };
+  const root = await mkdtemp(path.join(os.tmpdir(), "paperclip-connection-e2e-"));
+  const home = path.join(root, "paperclip-home");
+  const instance = `connection-${randomBytes(6).toString("hex")}`;
+  const port = await reserveRunnerE2EServerPort();
+  await mkdir(home, { mode: 0o700 });
+  // Only bootstrap essentials are inherited. Selected provider keys enter via UI.
+  const env: NodeJS.ProcessEnv = Object.fromEntries(["PATH", "HOME", "USER", "LOGNAME", "SHELL", "TMPDIR", "LANG", "LC_ALL", "NODE_EXTRA_CA_CERTS"].flatMap(key => process.env[key] ? [[key, process.env[key]!]] : []));
+  Object.assign(env, {
+    PAPERCLIP_HOME: home, PAPERCLIP_INSTANCE_ID: instance, PAPERCLIP_CONFIG: path.join(home, "instances", instance, "config.json"),
+    PAPERCLIP_RUNNER_E2E_TEMP_ROOT: root, PAPERCLIP_RUNNER_E2E_PORT: String(port), PAPERCLIP_RUNNER_E2E_SERVER_LOG: path.join(root, "server.log"),
+    PAPERCLIP_RUNNER_BINARY: resolvePaperclipRunnerBinaryForHarness(executions, repositoryRoot),
+    PAPERCLIP_VITE_CACHE_DIR: path.join(root, "vite-cache"), PAPERCLIP_ANNOUNCEMENTS_ENABLED: "false",
+    PAPERCLIP_AGENT_JWT_SECRET: randomBytes(48).toString("hex"), PAPERCLIP_DECISION_SIGNING_SECRET: randomBytes(48).toString("hex"),
+    PAPERCLIP_TOOL_ACTION_SIGNING_SECRET: randomBytes(48).toString("hex"), BETTER_AUTH_SECRET: randomBytes(48).toString("hex"),
+  });
+  const child = spawn(process.execPath, runnerE2ETypeScriptProcessArgs(repositoryRoot, path.join(repositoryRoot, "tests/runner-e2e/server.ts")), { cwd: repositoryRoot, env, detached: process.platform !== "win32", stdio: "ignore" });
+  let failed = false;
+  child.once("error", () => { failed = true; });
+  const stop = async () => { await stopOwnedServer(child); await rm(root, { recursive: true, force: true }); };
+  const origin = `http://127.0.0.1:${port}`;
+  try {
+    const deadline = Date.now() + 180_000;
+    while (Date.now() < deadline) {
+      if (failed || child.exitCode !== null) throw new ConnectionBlock("blocked_target", "managed_server_start_failed");
+      try { const response = await fetch(`${origin}/api/health`, { redirect: "error", signal: AbortSignal.timeout(1500) }); if (response.ok) return { origin, stop }; } catch { /* Bounded startup. */ }
+      await connectionDelay(500);
+    }
+    throw new ConnectionBlock("blocked_target", "managed_server_start_timeout");
+  } catch (error) { await stop(); throw error; }
+}
+
+export async function verifyConnectionTarget(origin: string, config: ConnectionConfig) {
+  let health: any;
+  try {
+    const response = await fetch(`${targetOrigin(origin)}/api/health`, { redirect: "error", signal: AbortSignal.timeout(15_000) });
+    if (!response.ok) throw new ConnectionFailure();
+    health = await response.json();
+  } catch { throw new ConnectionBlock("blocked_target", "target_health_unavailable_or_redirected"); }
+  if (typeof health.commit !== "string" || !/^[a-f0-9]{40}$/.test(health.commit)) throw new ConnectionBlock("blocked_target", "target_revision_unavailable");
+  if (config.target.mode === "attach" && (health.commit !== config.target.expectedCommit || health.deploymentMode !== config.target.deploymentMode)) throw new ConnectionBlock("blocked_target", "target_identity_mismatch");
+  return { commit: health.commit as string, deploymentMode: String(health.deploymentMode) };
+}

tests/runner-e2e/gemini-acp-filesystem.smoke.ts

@@ -0,0 +1,82 @@
+/** Opt-in installed-CLI compatibility check. No real provider credentials or inference. */
+import { createServer } from "node:http";
+import { mkdir, mkdtemp, readFile, realpath, rm, writeFile } from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import { createAcpRuntime, createAgentRegistry, createRuntimeStore } from "../../packages/adapter-utils/node_modules/acpx/dist/runtime.js";
+
+const argument = process.argv.indexOf("--agent-command");
+const command = argument < 0 ? "gemini --acp" : process.argv[argument + 1];
+if (!command) throw new Error("agent_command_required");
+const root = await mkdtemp(path.join(os.tmpdir(), "paperclip-gemini-filesystem-smoke-"));
+const content = '{"nonce":"synthetic-file-smoke","total":28}\n';
+const results: Array<{ case: string; passed: boolean; completedTools: number; failedTools: number; missingFileSignal: boolean }> = [];
+let active: { calls: number; file: string; tool: string };
+const server = createServer(async (request, response) => {
+  // Never retain request bodies, headers, provider prompts or model output.
+  for await (const _chunk of request) { /* Drain the synthetic request. */ }
+  const parts = ++active.calls === 1
+    ? [{ functionCall: { name: active.tool, args: { file_path: active.file, ...(active.tool === "write_file" ? { content } : {}) } } }]
+    : [{ text: "synthetic-complete" }];
+  const payload = {
+    candidates: [{ content: { role: "model", parts }, finishReason: "STOP" }],
+    usageMetadata: { promptTokenCount: 10, candidatesTokenCount: 2, totalTokenCount: 12 },
+    modelVersion: "gemini-3.1-flash-lite",
+  };
+  if (request.url?.includes(":streamGenerateContent")) {
+    response.writeHead(200, { "content-type": "text/event-stream" });
+    response.end(`data: ${JSON.stringify(payload)}\n\n`);
+  } else {
+    response.writeHead(200, { "content-type": "application/json" });
+    response.end(JSON.stringify(request.url?.includes(":countTokens") ? { totalTokens: 10 } : payload));
+  }
+});
+await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve));
+const origin = `http://127.0.0.1:${(server.address() as { port: number }).port}`;
+try {
+  for (const scenario of ["create-missing", "read-existing", "deny-outside"] as const) {
+    const home = path.join(root, scenario, "home");
+    const workspace = path.join(root, scenario, "workspace");
+    await mkdir(path.join(home, ".gemini"), { recursive: true, mode: 0o700 });
+    await mkdir(workspace, { recursive: true });
+    const cwd = await realpath(workspace);
+    const file = scenario === "deny-outside" ? path.join(await realpath(root), "outside.json") : path.join(cwd, "proof.json");
+    if (scenario !== "create-missing") await writeFile(file, content);
+    await writeFile(path.join(home, ".gemini", "settings.json"), JSON.stringify({
+      security: { auth: { selectedType: "gemini-api-key" } }, telemetry: { enabled: false },
+    }), { mode: 0o600 });
+    active = { calls: 0, file, tool: scenario === "read-existing" ? "read_file" : "write_file" };
+    const runtime = createAcpRuntime({
+      cwd, sessionStore: createRuntimeStore({ stateDir: path.join(root, scenario, "state") }),
+      agentRegistry: createAgentRegistry({ overrides: { gemini: command } }),
+      permissionMode: "approve-all", nonInteractivePermissions: "deny", inheritProcessEnv: false,
+      onAgentStderr: () => { /* Child diagnostics may contain untrusted material. */ },
+    });
+    const handle = await runtime.ensureSession({
+      sessionKey: `gemini-filesystem-${scenario}`, agent: "gemini", mode: "oneshot", cwd,
+      sessionOptions: { env: {
+        HOME: home, PATH: process.env.PATH ?? "", GEMINI_MODEL: "gemini-3.1-flash-lite",
+        GEMINI_API_KEY: "synthetic-invalid-key", GOOGLE_GEMINI_BASE_URL: origin,
+      } },
+    });
+    try {
+      let completedTools = 0; let failedTools = 0; let missingFileSignal = false;
+      for await (const event of runtime.runTurn({ handle, text: "Run the synthetic file operation, then stop.", mode: "prompt", requestId: scenario, timeoutMs: 20_000 })) {
+        if (event.type !== "tool_call") continue;
+        if (event.status === "completed") completedTools++;
+        if (event.status === "failed") {
+          failedTools++;
+          missingFileSignal ||= /Error checking existing file: (?:Resource not found|Internal error)/.test(event.text ?? "");
+        }
+      }
+      const bytes = await readFile(file, "utf8").catch(() => null);
+      const passed = bytes === content && (scenario === "deny-outside" ? failedTools > 0 && completedTools === 0 : failedTools === 0 && completedTools > 0);
+      results.push({ case: scenario, passed, completedTools, failedTools, missingFileSignal });
+    } finally { await runtime.close({ handle, reason: "synthetic filesystem smoke complete" }); }
+  }
+  console.log(JSON.stringify({ realProviderCalls: 0, results }, null, 2));
+  if (results.some((result) => !result.passed)) process.exitCode = 1;
+} finally {
+  await new Promise<void>((resolve) => server.close(() => resolve()));
+  await rm(root, { recursive: true, force: true });
+}

tests/runner-e2e/history.test.ts

@@ -31,6 +31,7 @@ import {
 import { renderRunnerHistoryIndex } from "./history-index.js";
 import { renderPublicCampaignSummary } from "./public-summary-image.js";
 import { isPublicRunnerScreenshotRoute } from "./screenshot-policy.js";
+import { connectionCheckpoints, type ConnectionEvidence } from "./connection-evidence.js";
 import type { MatrixExecution, RunnerE2EResult } from "./types.js";
 
 const temporaryDirectories: string[] = [];
@@ -63,6 +64,7 @@ function result(execution: MatrixExecution, status: "passed" | "failed") {
     finishedAt: "2026-08-28T00:00:01.000Z",
     durationMs: 1_000,
     cleanup: "passed",
+    ...(execution.task.flow === "provider_connection" ? { providerConnection: { version: 1, outcome: status, phase: "finished", assisted: false, authFreshness: "signed-in", target: { mode: "attach", origin: "http://127.0.0.1:3100", commit: "a".repeat(40), deploymentMode: "local_trusted" }, entry: "agent", method: "api-key", checkpoints: Object.fromEntries(connectionCheckpoints.map(key => [key, true])), waits: [] } satisfies ConnectionEvidence } : {}),
   } satisfies RunnerE2EResult;
 }
 

tests/runner-e2e/launch.ts

@@ -1083,6 +1083,13 @@ async function main() {
     return;
   }
 
+  if (executions.some(execution => execution.task.flow === "provider_connection")) {
+    const { runConnectionCampaign } = await import("./connection-launch.js");
+    await runConnectionCampaign({ executions, catalog: runnerMatrix, configFile: options.connectionConfig, repositoryRoot });
+    return;
+  }
+  if (options.connectionConfig) throw new Error("--connection-config is only supported by the provider-connections suite");
+
   // Keep admission before local-env loading and credential checks. Pending
   // profiles remain discoverable, but cannot reach a provider.
   assertRunnerE2EPrerequisites(executions);

tests/runner-e2e/report-assets.ts

@@ -0,0 +1,19 @@
+import path from "node:path";
+import { copyFile, mkdir } from "node:fs/promises";
+const repositoryRoot = path.resolve(import.meta.dirname, "../..");
+
+export async function stageDashboardBrandAssets(output: string) {
+  const assets = path.join(output, "assets");
+  await mkdir(assets, { recursive: true });
+  await Promise.all([
+    copyFile(
+      path.join(repositoryRoot, "ui/public/favicon-32x32.png"),
+      path.join(assets, "favicon-32x32.png"),
+    ),
+    copyFile(
+      path.join(repositoryRoot, "ui/public/fonts/InterVariable.woff2"),
+      path.join(assets, "InterVariable.woff2"),
+    ),
+  ]);
+}
+

tests/runner-e2e/report.ts

@@ -1,3 +1,4 @@
+import { stageDashboardBrandAssets } from "./report-assets.js";
 import { discoverReportCatalog } from "./report-catalog.js";
 import path from "node:path";
 import {
@@ -147,20 +148,6 @@ async function stageDashboardEvidence(
   return staged;
 }
 
-async function stageDashboardBrandAssets(output: string) {
-  const assets = path.join(output, "assets");
-  await mkdir(assets, { recursive: true });
-  await Promise.all([
-    copyFile(
-      path.join(repositoryRoot, "ui/public/favicon-32x32.png"),
-      path.join(assets, "favicon-32x32.png"),
-    ),
-    copyFile(
-      path.join(repositoryRoot, "ui/public/fonts/InterVariable.woff2"),
-      path.join(assets, "InterVariable.woff2"),
-    ),
-  ]);
-}
 
 async function main() {
   const root = path.resolve(

tests/runner-e2e/result-validation.ts

@@ -1,3 +1,4 @@
+import { connectionCheckpoints, connectionEvidencePasses, connectionRunSignals } from "./connection-evidence.js";
 import type { RunnerE2EResult } from "./types.js";
 
 type Rule = (value: unknown, at: string) => void;
@@ -240,6 +241,20 @@ const fields = {
       tasks: array(object), agents: array(object), comments: array(object), interactions: array(object), documents: array(object), attachments: optional(array(object)), runs: array(object) })),
     checks: array(shape({ id: string, passed: boolean, notReached: optional(string), evidence: array(string), detail: string })),
   })),
+  providerConnection: optional(shape({
+    version: integer,
+    outcome: oneOf("running", "passed", "failed", "awaiting_user", "missing_credential", "blocked_target"),
+    phase: string, assisted: boolean, authFreshness: oneOf("signed-in", "signed-out"),
+    target: shape({ mode: oneOf("managed-local", "attach"), origin: url, commit: nullable(string), deploymentMode: nullable(string) }),
+    entry: oneOf("apps", "agent"), method: string, checkpoints: shape(Object.fromEntries(connectionCheckpoints.map(key => [key, optional(boolean)]))),
+    waits: array(shape({ kind: oneOf("board_login", "provider_login"), startedAt: date, finishedAt: optional(date) })),
+    artifactChecks: optional(array(shape({ filename: string, runId: string, sha256: string, fields: object, exactFields: boolean }))),
+    inputTransport: optional(oneOf("prompt_base64")),
+    setupChecks: optional(array(shape({ code: string, level: oneOf("info", "warn", "error") }))),
+    runDiagnostics: optional(array(shape({ runId: string, status: oneOf("succeeded", "failed", "cancelled", "interrupted", "timed_out", "unknown"), signals: array(oneOf(...connectionRunSignals)), logAvailable: boolean }))),
+    creationFailure: optional(shape({ status: integer, code: oneOf("ai_connection_api_key_rejected", "ai_connection_verification_failed", "connection_request_rejected") })),
+    connectionId: optional(string), agentId: optional(string), companyId: optional(string), artifactSha256: optional(string), model: optional(string), cleanupRetained: optional(boolean), companyArchived: optional(boolean),
+  })),
   completionQuality: optional(array(shape({
     name: string, purpose: optional(oneOf("product", "calibration")), expectedPass: boolean, passed: boolean, status: oneOf("completed", "failed", "pending"), config: object, configHash: string, evidenceHash: string,
     criteria: array(shape({ id: string, passed: boolean, rationale: string, evidenceIds: array(string) })),
@@ -260,4 +275,6 @@ export function validateRetainedRunnerResult(
   value: unknown,
 ): asserts value is RunnerE2EResult {
   result(value, "result");
+  const checked = value as RunnerE2EResult;
+  if (checked.suiteId === "provider-connections" && checked.status === "passed" && (!checked.providerConnection || !connectionEvidencePasses(checked.providerConnection))) invalid("result.providerConnection");
 }

tests/runner-e2e/selectors.ts

@@ -16,6 +16,7 @@ export interface RunnerSelectorOptions {
   debug: boolean;
   maxParallel: number;
   maxAutomaticRetries: number;
+  connectionConfig?: string;
 }
 
 export class RunnerSelectorError extends Error {}
@@ -55,6 +56,10 @@ export function parseRunnerSelectors(
     else if (flag === "--headed") options.headed = true;
     else if (flag === "--ui") options.ui = true;
     else if (flag === "--debug") options.debug = true;
+    else if (flag === "--connection-config") {
+      options.connectionConfig = valueFor(args, index, flag);
+      index += 1;
+    }
     else if (flag === "--max-parallel") {
       const value = valueFor(args, index, flag);
       index += 1;
@@ -200,6 +205,7 @@ export function selectRunnerExecutions(
 export function buildMatrixJobs(
   executions: readonly MatrixExecution[],
 ): MatrixJob[] {
+  if (executions.some(e => e.task.flow === "provider_connection")) throw new RunnerSelectorError("Provider connection journeys use an explicit local/staging connection configuration; they are not admitted to paid CI matrix jobs yet.");
   return executions
     .map((execution) => ({
       executionId: execution.id,

tests/runner-e2e/types.ts

@@ -5,6 +5,7 @@ export const CREDENTIAL_NAMES = [
   "KIMI_MODEL_API_KEY",
   "XAI_API_KEY",
   "GROK_AUTH_JSON",
+  "GEMINI_API_KEY",
   "DAYTONA_API_KEY",
   "CURSOR_AUTH_TOKEN",
   "COPILOT_GITHUB_TOKEN",
@@ -15,6 +16,7 @@ export type RunnerGeneration = "legacy" | "native";
 export type RunnerEnvironmentId = "local" | "daytona";
 export type RunnerTaskWorkMode = "standard" | "planning" | "ask";
 export type RunnerTaskFlow =
+  | "provider_connection"
   | "blocker_guidance"
   | "everyday_workflow"
   | "context_integrity"
@@ -323,6 +325,7 @@ export interface RunnerE2EResult {
     sha256?: string;
   }>;
   firstTask?: import("./first-task-scoring.js").FirstTaskEvidence;
+  providerConnection?: import("./connection-evidence.js").ConnectionEvidence;
   completionQuality?: import("./completion-quality.js").CompletionQualityRecord[];
   firstTaskQuality?: import("./first-task-quality.js").FirstTaskQuality;
   cleanup: "not_started" | "passed" | "failed";

tests/runner-e2e/user-actions.ts

@@ -8,9 +8,10 @@ export async function createTaskThroughUi(input: {
   prompt: string;
   workMode: "standard" | "planning" | "ask";
   projectName?: string;
+  attachments?: readonly string[];
 }) {
   const issuesUrl = `/${encodeURIComponent(input.issuePrefix)}/issues`;
-  const newTask = input.page.getByRole("button", { name: "New Task" }).first();
+  const newTask = input.page.getByRole("button", { name: /^New task$/i }).first();
   let bootstrapError: unknown;
   for (let bootstrapAttempt = 1; bootstrapAttempt <= 3; bootstrapAttempt += 1) {
     try {
@@ -33,40 +34,74 @@ export async function createTaskThroughUi(input: {
     );
   }
   await newTask.click();
-  await input.page.getByPlaceholder("Task title").fill(input.title);
+  const dialog = input.page.getByRole("dialog");
+  const titleInput = dialog.getByPlaceholder("Task title");
+  const hasTitleInput = await titleInput.count() > 0;
+  if (hasTitleInput) await titleInput.fill(input.title);
   await input.page
     .getByRole("dialog")
     .getByRole("textbox", { name: "editable markdown", exact: true })
     .fill(input.prompt);
   if (input.workMode !== "standard") {
-    await input.page
-      .getByRole("dialog")
-      .locator(`[data-issue-work-mode-chip="standard"]`)
-      .click();
-    await input.page
-      .locator(`[data-issue-work-mode="${input.workMode}"]`)
-      .click();
+    const legacyMode = dialog.locator('[data-issue-work-mode-chip="standard"]');
+    if (await legacyMode.count()) {
+      await legacyMode.click();
+      await input.page.locator(`[data-issue-work-mode="${input.workMode}"]`).click();
+    } else {
+      await dialog.getByRole("button", { name: "Add to composer", exact: true }).click();
+      await input.page.getByRole("menuitem", { name: input.workMode === "planning" ? /^Plan mode/ : /^Ask mode/ }).click();
+    }
   }
   await input.page
-    .getByRole("button", { name: "Assignee", exact: true })
+    .getByRole("button", { name: /^(?:Select )?Assignee$/i })
     .click();
   await input.page
-    .getByPlaceholder("Search assignees...")
+    .getByPlaceholder(/^Search assignees(?:…|\.\.\.)?$/i)
     .fill(input.agentName);
   await input.page.getByText(input.agentName, { exact: true }).last().click();
   if (input.projectName) {
-    const dialog = input.page.getByRole("dialog");
     // Selecting the assignee advances focus to this selector and opens it.
     // Focus is idempotent here; clicking would toggle an already-open popover
     // closed before the search field can be filled.
-    await dialog.getByRole("button", { name: "Project", exact: true }).focus();
-    await dialog.getByPlaceholder("Search projects...").fill(input.projectName);
-    await dialog.getByText(input.projectName, { exact: true }).last().click();
+    const projectTrigger = dialog.getByRole("button", { name: "Project", exact: true })
+      .or(dialog.getByRole("button", { name: input.projectName, exact: true }));
+    await projectTrigger.focus();
+    const projectSearch = input.page.getByPlaceholder(/^Search projects(?:…|\.\.\.)?$/i);
+    await expect(projectSearch).toBeVisible({ timeout: 2_000 }).catch(async () => {
+      if (await projectTrigger.getAttribute("aria-expanded") !== "true") await projectTrigger.click();
+      await expect(projectSearch).toBeVisible();
+    });
+    await projectSearch.fill(input.projectName);
+    await input.page.getByText(input.projectName, { exact: true }).last().click();
   }
   const submittedAtMs = Date.now();
+  if (input.attachments?.length) {
+    const chooser = input.page.waitForEvent("filechooser");
+    const upload = dialog.getByRole("button", { name: "Upload", exact: true });
+    if (await upload.count()) await upload.click();
+    else {
+      await dialog.getByRole("button", { name: "Add to composer", exact: true }).click();
+      await input.page.getByRole("menuitem", { name: /^Files and images/ }).click();
+    }
+    await (await chooser).setFiles([...input.attachments]);
+    // Upload finishes as part of task creation; the dialog retains the selected files.
+  }
+  const created = input.page.waitForResponse(response => response.request().method() === "POST" && /^\/api\/companies\/[^/]+\/issues$/.test(new URL(response.url()).pathname));
   await input.page
-    .getByRole("button", { name: "Create Task", exact: true })
+    .getByRole("button", { name: /^Create task$/i })
     .click();
+  const response = await created;
+  if (!response.ok()) throw new Error("Task creation request failed");
+  if (!hasTitleInput && input.title) {
+    // Prompt-only creation generates a title. Set only fixture naming metadata
+    // through the public API; execution and results remain entirely real.
+    const issue = await response.json();
+    const renamed = await input.page.request.patch(new URL(`/api/issues/${issue.id}`, response.url()).href, { data: { title: input.title } });
+    if (!renamed.ok()) throw new Error("Fixture task naming failed");
+  }
+  // The create response precedes staged-file uploads. The production dialog
+  // closes after those uploads finish; navigating earlier can cancel them.
+  await expect(dialog).not.toBeVisible({ timeout: 30_000 });
   return submittedAtMs;
 }