Files
PaperClipAI/server/src/__tests__/ai-connection-router.test.ts
T
DottaandPaperclip 13709b4c7a feat(connections): let agents configure connections within delegated authority (#17737)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Connections provide governed tools, chat channels, model accounts,
and gateways.
> - Several management routes require a human board actor even when an
agent has the required grant.
> - Default agents must configure these resources within the responsible
user's authority.
> - This pull request adds a shared authorization path to the existing
REST and native API surface.
> - Human authentication, execution approvals, and local host boundaries
remain enforced.
> - The benefit is that a standard agent can complete authorized setup
with a bounded human handoff.

## Linked Issues or Issue Description

**Subsystem affected**

Server authorization, connection services, shared contracts, native API
discovery, database schema.

**Problem or motivation**

Connection administration is available in the web app but often
unavailable to agents. Agent grants, responsible-user authority,
credential ownership, and actor attribution need separate checks.
Opening gateway-token creation also requires preserving the issuing
agent's delegated authority during token use.

**Proposed solution**

Use the existing authorization evaluator for every connection family.
Enforce current responsible-user authority, effective trust, grant
scope, resource visibility, and target-agent access. Keep existing agent
grants unchanged. Expose authorized configuration through default
`search_api` and `call_api`. Save human authentication steps and let
agents continue after completion.

**Roadmap alignment**

This change completes the API access path for the existing Connected
Apps, Personal & Shared AI Accounts, and governed MCP Gateway
capabilities in `ROADMAP.md`. It follows the maintainer's requested
design. Related work: #11899 enforces connection execution access and
creator/manager rules; this PR extends agent management across
connection families. It does not replace that execution-policy work.

## What Changed

- Add shared connection management context and mandatory
responsible-user, trust, scope, and target checks.
- Allow authorized agents to configure tool, chat/email, GitHub/Slack,
AI, profile, and gateway resources through existing routes.
- Preserve human credential ownership and record the actual agent,
responsible user, and run or key in audit data.
- Preserve saved browser authentication handoffs and callback session
checks.
- Bind agent-issued durable gateway tokens to their responsible user and
recheck delegated authority at use.
- Add explicit REST transport and management capability metadata,
built-in guidance, and sensitive-response handling.
- Intersect caller and endpoint-agent Slack test/invitation policies,
including independent and shared rates. Require the addressed human’s
current endpoint identity link before completing identity handoffs.
- Add production-authenticated integration scenarios. Companion #17739
contains the saved setup UI (including the required human
import/rotation step), product and permission documentation,
catalog/Vercel credential and callback ownership, and operation maps to
keep each review below 100 files. Both PRs must ship together; merge
this implementation first, then its companion.

## Verification

- Current implementation head:
`2ecd6451d0d0c8d82121a4e3fa4ce94c5befbbd6`, including master
`ea727dc3c2`, 99 files. The latest follow-up fixes the two valid
findings from the preceding f9 review. Exact-head Greptile has completed
at5/5 with no findings and Superagent passed. Full exact-head Linux CI
workflow40446606056 completed SUCCESS; only the queued external
Paperclip Review remains pending; earlier Linux passes do not qualify
this successor.
- Latest preserved-stack verification passed **2,165 tests** in fresh
PostgreSQL: **70 actual-auth scenarios**, 27 OpenAPI contracts, 2,022
native API tests and 46 saved GitHub wizard cases. **141 existing
gateway/policy acceptance tests** also passed. Workspace
typecheck/build, maps and token gates passed. The first workspace
typecheck was blocked by sandbox tsx IPC permissions; its authorized
rerun passed. This is focused local evidence; the full Linux CI matrix
is still required.
- Issuer/target gateway policy regressions were red before the fix. Both
decisions now intersect, including Block/Ask first/rates, same
credential identity and current token/user/agent authority after
approval. Invocations retain their token and actual requesting agent;
deleted tokens fail closed. Thirteen actual-auth cases cover revocation,
narrowed scope, approvals, shared rates and board compatibility.
Intermediate missing-token attribution, review-context and
implicit-session actor failures remain in the qualification record; all
final cases pass.
- Activity pagination now retains a bounded cursor across private-only
windows. The actual-JWT regression reaches three older readable events
after 27 private ones without exposing private details. A native
device-login regression was red for plaintext receipts; local check and
test-call results now use existing protected-response handling. The
passing actual-auth case proves encrypted persistence, exact replay,
redacted history and no response asset.

- Previous-head `5b55195593` Linux CI initially failed e2e shard 4/8 in
the existing connection-intent continuation test: periodic recovery
queued a repair at 01:21:47, the scripted journey completed the task at
01:21:48, and the extra queued run was cancelled at 01:21:49 because the
task was terminal. The assertion selected that latest cancelled run.
GitHub refused the first attempt while the workflow was nonterminal.
After completion, the single evidence-supported failed-job rerun was
accepted as **attempt 2** on the same head (workflow 40431202466). The
failed shard and final E2E aggregate passed on attempt 2, and the
complete Linux workflow for that head is successful. The original
failure remains part of qualification. No additional unchanged-code
rerun will be used for this failure.

- Companion CI also has an unchanged login-cleanup idleness timeout,
retained in its qualification record. That same 16-case suite passed in
this parent Linux run and a fresh local database. Both preceding
workflows passed after their single failed-job reruns as attempt 2; the
original failures remain recorded. No additional unchanged-code rerun
was used.

- Migration `0380_broken_aqueduct.sql` was regenerated through the
repository workflow after master took 0379 for project Git
authorization. It adds the responsible-user column with `IF NOT EXISTS`,
preserves upstream migrations, and satisfies snapshot retention and
migration safety checks.
- Upstream published Copilot v62 and v63 identities remain byte-for-byte
unchanged. The combined guidance is now v64, digest
`sha256:22e31f95ef972bcead98a4a707c9f33433558fd111e390e027a2bbb4346e112a`.
The normal shared TypeScript/Rust identity check and **61 Copilot
installation tests passed**. The old source-identity assertion first
failed against unchanged published v63; regenerating a new v64 preserves
the published release.
- Workspace `pnpm -r typecheck` and `pnpm build`, map validation, and UI
token gates passed on the final combined stack after the upstream merge.
Node 24.22.1 / pnpm 9.15.4 were used.
- **Earlier 82 API/OpenAPI tests passed** in fresh PostgreSQL 18: 55
assembled-server scenarios and 27 contracts. They use real persistent
agent keys, run JWTs, human sessions, production defaults, and
deterministic provider fixtures. No actor injection establishes access.
- The new email regression was red before per-account scope filtering.
The Arcade regression was red before human ownership and inventory-key
fixes; it also verifies actual-agent audit identity and a human-session
inventory read. The activity regression measured **200 authorization
checks before the fix and one afterward** for the same one-item page.
- **82 GitHub UI tests passed**, including all three new
failed-replacement regressions that were red before their fix. The UI
preserves an earlier copy only after confirming its committed
generation. Changed or unreadable results block copying and reconnecting
until reconciled. Human-only secret responses now include their
nonsecret committed generation.
- Private test status now rechecks the originating principal and linked
task, including mandatory agent/responsible-user privacy. Cross-agent
tests require the full caller tool access decision and the same
credential identity, regardless of auth label; caller and target
policies/approvals and rate limits apply before execution and after
human approval. A profile-only precondition was found to reject valid
allow-policy and scoped-grant access and has been removed. Both
permitted-access cases and a no-auth-label identity regression were red
before the correction; caller-specific Block still prevents provider
execution. The private-status regression was also red before its fix.
- The tool-activity regression measured 100 capability decisions and 102
run captures on a 100-event page before the fix. It now uses two task
decisions and at most two captures while retaining the same 50 visible
events, private-event filtering, and 403 after audit-grant revocation.
The regression was red before request-context reuse and task-decision
sharing; final server typecheck/build also pass.
- Generic chat configure/reconnect now hands agent-supplied verification
credentials to the addressed human before provider work or vault
changes. Actual key/JWT regressions were red for Slack signing-key
replacement and generic channel secret-reference rebinding. They now
prove unchanged refs and vaulted signing value, human setup, permitted
agent bot-token repair, and governed test messages. All 82 API/OpenAPI
cases pass in fresh PostgreSQL. Eight native receipt protections and
final server typecheck/build also pass.
- The preceding review found that old healthy state could retire a
credential-entry card before a human changed anything. The actual-JWT
regression was red. Saved field generations now change only with
protected values and commit with refs plus a verification fence; retries
retain the baseline. Old health, bot-token repair and unchanged entry
stay pending. New signing verification and current authority permit
continuation. A separate red race proved that user revocation during
provider authentication could return 403 after refs changed; authority
is now checked before persistence and inside the credential transaction.
The final 82 API/OpenAPI cases, four neighboring real-database
Slack/GitHub lifecycle cases, and final server typecheck/build pass.
- A preceding full run crossed a rate-window boundary. Database rows
prove one consumed slot at 23:35 and another at 23:36 for the same
policy/bucket. Only Date is frozen in the rate test cases; transport and
asynchronous timers remain real. Production rate behavior is unchanged,
and the final full run passes. An intermediate fixture failure that
overwrote hidden generation metadata with an API projection is retained
in the qualification record.
- Linux CI on an earlier head found two rate-admission assertions that
omitted the newly persisted bucket key. The failures were reproduced
locally; both assertions now verify the exact actor/tool bucket and all
49 policy-service tests pass. The companion qualification record retains
the initial and intermediate failures. Catalog creator attribution is
preserved in the companion to keep this implementation review at 99
files.
- The latest review reproduced three regressions before repair: complete
human replacements failed on unavailable old signing secrets, fresh
GitHub links incorrectly selected reconnect, and different
credential-entry requests shared an idempotency key. Each old protected
ref now has a narrow known-unavailable fallback; unexpected resolver
failures and company/scope violations still abort. Fresh imports keep
their credential form, configured endpoints keep reconnect, and
canonical field sets/completion modes distinguish handoffs while
matching retries retain their baseline. All 55 actual-auth scenarios and
27 separately run OpenAPI contracts pass, as do server typecheck/build.
Companion #17739 also passes all 41 wizard tests including actual
fresh-form submission, UI typecheck, maps and token gates. Earlier red
results and a mistyped OpenAPI filename are retained in qualification.
- The configured GitHub handoff regression was red in the
browser-component tests and actual-auth link/persistence tests. Saved
links now name requested fields and all/any completion mode; the
companion form submits those supplied fields through reconnect and
retains other saved credentials. Webhook/client-secret reconnect remains
human-only; first-setup secret restrictions remain. All 46 wizard tests
pass, including combined modes and legacy links. A duplicate review
referenced the older companion bf760912; the current form is already
fixed, linked in the reply, and its thread is resolved.
- Linux CI on bf760912 failed automatic Slack provisioning because vault
persistence cleared an already verified manifest-update challenge. The
same regression was reproduced locally. The provider-owned helper now
preserves proof only for the current signing fingerprint and callback
URL. Eight existing compatibility cases pass, including this case and
initial GitHub secret rejection. The first uncommitted GitHub-field
variant also failed that initial boundary and was narrowed to reconnect;
both failures remain in qualification.
- The upstream merge conflicted only in generated migration metadata.
Upstream 0379 is preserved, normal generation/pruning produced 0380, and
two repeated applications of the idempotent additive SQL passed. The
final combined stack passed **128 tests** (55 actual-auth, 27 OpenAPI
and 46 wizard cases) in fresh PostgreSQL. Eight Google UI cases also
pass; the two protocol cases could not initialize embedded PostgreSQL in
this fresh local attempt. Their earlier passing evidence remains
historical; fresh Linux CI must execute them. Vercel personal credential
attribution is preserved in companion #17739 with its other
ownership/consent follow-ups, keeping this PR at 99 files.
- The suite covers fresh defaults, restricted users/agents, scopes,
personal isolation, token revocation, human handoffs, callback replay,
execution approvals, authorized self-targeted policy changes,
self-approval denial, and current authority before commit. Reproduce API
checks with `pnpm exec vitest run
server/src/__tests__/connection-management.integration.test.ts
server/src/__tests__/openapi-routes.test.ts`, using a fresh database
through `PAPERCLIP_CONNECTION_MANAGEMENT_TEST_DATABASE_URL` or the
default embedded test database.
- Companion #17739 also records consent separately from token refresh so
saved Google choices survive refresh. Its eight service-picker and two
local-HTTP OAuth tests pass; both new UI cases were red before the fix.
Normal migration generation confirms no SQL change is needed for the
additive JSON metadata.
- The integration module shares one assembled server, authentication
factory, and provider lifecycle to preserve the
HTTP-to-vault-to-task-continuation proof. A reused external database
caused a Slack fixture identity conflict; the fresh final run passed.
Earlier fixture/transport and contract-update failures remain in the
companion qualification record.
- The broad local `pnpm test:run` attempt remains blocked by macOS
embedded PostgreSQL's host semaphore limit. It is not a clean pass. The
fresh Linux CI matrix must complete. Provider fixtures do not prove live
setup or an empirical LLM success rate.
- A current-head completed Paperclip assessment is not yet recorded. Its
earlier external recovery boundary has not been bypassed, and no
incomplete assessment is reported as passing. The user authorized both
merges when checks pass; merge this implementation first, then replay,
retarget, and requalify companion #17739 against the squash-merged
master.

- After the master refresh, local workspace typecheck/build, **97
actual-auth/OpenAPI cases** and **141 gateway/policy cases** pass. The
original fixed database port was occupied; Docker assigned a free
localhost port. The first neighbor setup command could not resolve root
`tsx`, leaving its schema uninitialized; the built database package
migrated the separate fixture and all unchanged cases passed. Both setup
failures are retained in qualification. Fresh Linux CI remains required;
no full local test-sweep pass is claimed.

- The f9 Greptile review scored **2/5** for two reproduced findings:
Slack test sends skipped the caller’s tool policy, and healthy App
credentials completed an unconfirmed identity handoff. Both are fixed in
this head and have resolved evidence threads
([Slack](https://github.com/paperclipai/paperclip/pull/17737#discussion_r4384927765),
[identity](https://github.com/paperclipai/paperclip/pull/17737#discussion_r4384927895)).
Actual key/JWT cases cover twelve Slack caller/target/recipient
combinations; the human-session GitHub OAuth callback alone remains
pending until explicit confirmation. All **99 API/OpenAPI tests** (72
actual-auth + 27 contracts), workspace typecheck and build pass. On the
combined companion, all four focused cases (including the twelve Slack
combinations), typecheck/build and maps also pass. The continuation
suite’s 50 embedded-PG cases were skipped because its database could not
start locally; fresh Linux CI must run them. Initial fixture
credential-collision and audit-projection assertion failures are
retained with the red reproductions in the companion qualification
record. No full local sweep or live consent is claimed.

## Risks

- The maintainer explicitly confirmed authorized policy administration,
including self-targeted changes that loosen controls. These writes
require management grants and the responsible user’s current authority.
Agents cannot approve their own governed action requests or edit their
own principal permission grants. The actual-auth regression exercises
both permitted administration and those denials.
- This is a broad authorization change. Incorrect scope or owner
resolution could deny permitted setup or expose another user's
resources. Integration cases cover both permitted and denied requests
with actual authentication.
- The additive nullable gateway-token column does not infer identities
for legacy clients or backfill agent grants. Legacy agent tokens without
a responsible user fail closed.
- Provider fixtures test supported setup and callback contracts. They do
not prove live provider availability.

## Model Used

- OpenAI Codex, GPT-6 family, with reasoning, repository tools, and code
execution. The runtime does not expose a more specific deployment ID or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass — focused suites above;
full local sweep remains blocked as documented
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green — exact `2ecd6451d0` Linux CI
completed SUCCESS; external Paperclip Review is still queued
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups —
exact `2ecd6451d0` review completed; both f9 findings have resolved
evidence replies
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 08:20:24 -05:00

514 lines
49 KiB
TypeScript

import { enableRunFailureReporting } from "./helpers/run-failure-reporting.js";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { randomUUID } from "node:crypto";
import { mkdir } from "node:fs/promises";
import { createIsolatedTestHome } from "./helpers/isolated-test-home.js";
import path from "node:path";
import { and, eq } from "drizzle-orm";
import { createDb, closeRegisteredClients, companies, agents, authUsers, companyMemberships, connectionGrants, plugins, aiConnectionRouterCursors, aiConnectionTaskPins, toolConnections, companySecrets, heartbeatRuns, issues, dotAgentBindings, grokBotRoutines } from "@paperclipai/db";
import { startEmbeddedPostgresTestDatabase } from "@paperclipai/db/test-embedded-postgres";
import type { AiConnectionPoolMember, AiConnectionRouterRequest, AiConnectionRouterResult, PaperclipPluginManifestV1 } from "@paperclipai/shared";
import { aiConnectionRouterService, AiConnectionPoolExhausted, poolMemberRuntimeConfig, applyAiConnectionRouterTaskSettings } from "../services/ai-connection-router.js";
import { aiConnectionSessionCompatibilityInputs } from "../services/ai-connection-session.js";
import { projectPaperclipRunnerTaskConfig, resolvePaperclipRunnerNativeProviderInput } from "../services/native-runtime/provider-profile.js";
import { aiConnectionService } from "../services/ai-connections.js";
import { instanceSettingsService } from "../services/instance-settings.js";
import { managedAiSessionFingerprintConfig, prepareManagedAiRuntime } from "../services/ai-connection-runtime.js";
import { secretService } from "../services/secrets.js";
import type { PluginWorkerManager } from "../services/plugin-worker-manager.js";
import express from "express";
import request from "supertest";
import { toolAccessRoutes } from "../routes/tool-access.js";
import { aiConnectionRoutes } from "../routes/ai-connections.js";
import { errorHandler } from "../middleware/index.js";
import { heartbeatService, buildEffectiveRunSessionConfigMetadata, resolveTaskSessionConfigFreshness } from "../services/heartbeat.js";
import { connectionIntentService } from "../services/connection-intents.js";
import { legacyExecutionNeedsReconciliation } from "../services/legacy-execution-recovery.js";
import { getServerAdapter, registerServerAdapter, unregisterServerAdapter } from "../adapters/index.js";
import { waitForPendingRunFailureReports } from "../services/run-failure-report.js";
import { dotRunnerBroker } from "../services/dot-runner-broker.js";
const captureRunFailure = vi.hoisted(() => vi.fn());
vi.mock("../sentry.js", async (importOriginal) => ({
...await importOriginal<typeof import("../sentry.js")>(),
captureRunFailure,
}));
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
const externalTestDatabaseUrl = process.env.PAPERCLIP_TEST_DATABASE_URL?.trim();
let db: ReturnType<typeof createDb>;
let home: string;
let isolatedHome: ReturnType<typeof createIsolatedTestHome> | undefined;
const companyId = randomUUID(), otherCompanyId = randomUUID(), agentId = randomUUID(), secondAgentId = randomUUID(), pluginId = randomUUID();
const pluginKey = "fixture.ai-router";
const manifest: PaperclipPluginManifestV1 = { id: pluginKey, apiVersion: 1, version: "0.1.0", displayName: "Fixture router", description: "Fixture", author: "Tests", categories: ["connector"], capabilities: ["ai.connections.route"], aiConnectionRouter: { name: "AI connection pool", description: "Use saved connections" }, entrypoints: { worker: "worker.js" } };
let members: AiConnectionPoolMember[];
const proposal = vi.fn(async (_id: string, _method: string, request: AiConnectionRouterRequest): Promise<AiConnectionRouterResult> => {
const next = (request.memberOrder.indexOf(request.lastMemberId ?? "") + 1) % request.memberOrder.length;
const order = request.pinnedMemberId ? [request.pinnedMemberId] : [...request.memberOrder.slice(next), ...request.memberOrder.slice(0, next)];
const selected = order.find(id => request.candidates.some(candidate => candidate.member.id === id));
return selected ? { kind: "selected", memberId: selected } : { kind: "unavailable", skipped: {} };
});
const worker = { isRunning: () => true, call: proposal } as unknown as PluginWorkerManager;
const defaultProposal = proposal.getMockImplementation()!;
const service = () => aiConnectionRouterService(db, worker);
const resolve = (poolId: string, taskKey: string, extra = {}) => service().resolve({ companyId, poolId, agentId, userId: "alice", adapterType: "paperclip_runner", taskKey, ...extra });
const makePool = async (config = {}) => service().save(pluginKey, { companyId, config: { name: randomUUID(), enabled: true, mode: "round_robin", thresholdPercent: 90, members, ...config } }, "alice");
beforeAll(async () => {
isolatedHome = createIsolatedTestHome();
home = isolatedHome.home;
vi.stubEnv("PAPERCLIP_HOME", home); vi.stubEnv("PAPERCLIP_INSTANCE_ID", "router-fixture");
if (externalTestDatabaseUrl) db = createDb(externalTestDatabaseUrl);
else { database = await startEmbeddedPostgresTestDatabase("paperclip-router-db-"); db = createDb(database.connectionString); }
await db.insert(companies).values([{ id: companyId, name: "Router fixture", issuePrefix: "RTF" }, { id: otherCompanyId, name: "Other", issuePrefix: "RTO" }]);
await db.insert(agents).values([agentId, secondAgentId].map(id => ({ id, companyId, name: "Router agent", adapterType: "paperclip_runner" })));
await db.insert(authUsers).values(["alice", "bob"].map(id => ({ id, name: id, email: `${id}@router.example.test`, createdAt: new Date(), updatedAt: new Date() }))).onConflictDoNothing();
await db.insert(companyMemberships).values(["alice", "bob"].map(principalId => ({ companyId, principalId, principalType: "user", status: "active", membershipRole: "member" })));
await db.insert(plugins).values({ id: pluginId, pluginKey, packageName: pluginKey, version: "0.1.0", manifestJson: manifest, status: "ready" });
const accounts = aiConnectionService(db);
const codex = await accounts.save(companyId, "alice", { provider: "openai", method: "api_key", ownership: "personal", name: "Codex", apiKey: "fixture", agentIds: [], allAgents: true }, "fixture-codex");
const claude = await accounts.save(companyId, "alice", { provider: "anthropic", method: "subscription", ownership: "personal", name: "Claude", loginSessionId: "fixture", agentIds: [], allAgents: true }, "fixture-claude");
members = [
{ id: randomUUID(), binding: { mode: "delegated", provider: "openai", method: "api_key", ...codex }, profile: { provider: "codex", model: "gpt-5.6-sol", effort: "low" } },
{ id: randomUUID(), binding: { mode: "delegated", provider: "anthropic", method: "subscription", ...claude }, profile: { provider: "acpx", acpxAgent: "claude", model: "claude-sonnet-5" } },
];
}, 90000);
afterAll(async () => {
await waitForPendingRunFailureReports();
if (externalTestDatabaseUrl) await closeRegisteredClients(externalTestDatabaseUrl);
await db?.$client.end();
await database?.cleanup(); vi.unstubAllEnvs(); isolatedHome?.cleanup();
});
describe("durable, authorized connection routing", () => {
it("defaults off and does not expose selectable pools before opt-in", async () => {
expect((await instanceSettingsService(db).getExperimental()).enableAiConnectionRouters).toBe(false);
expect((await service().catalog())[0]?.availability).toEqual({ available: false, reason: "Ask your instance operator to enable AI connection routing." });
await expect(makePool()).rejects.toThrow("instance operator");
expect(await service().selectable(companyId, "alice")).toEqual([]);
await instanceSettingsService(db).updateExperimental({ enableAiConnectionRouters: true });
});
it("registers native connectors only for declared installed routers", async () => {
const entry = (await service().catalog())[0]!;
expect(entry.name).toBe("AI connection pool");
expect(entry.aiConnectionRouter).toEqual({ pluginKey });
expect(entry.availability?.available).toBe(true);
await db.update(plugins).set({ status: "disabled" }).where(eq(plugins.id, pluginId));
try {
expect((await service().catalog())[0]?.availability?.available).toBe(false);
await db.update(plugins).set({ status: "uninstalled" }).where(eq(plugins.id, pluginId));
expect(await service().catalog()).toEqual([]);
} finally { await db.update(plugins).set({ status: "ready" }).where(eq(plugins.id, pluginId)); }
});
it("native connector removal retains durable pins and ordinary updates cannot bypass pool revisions", async () => {
const pool = await makePool(); const selected = await resolve(pool.id, "catalog-removal");
const app = express(); app.use(express.json());
app.use((req, _res, next) => { req.actor = { type: "board", userId: "alice", source: "local_implicit", companyIds: [companyId], memberships: [{ companyId, status: "active", membershipRole: "owner" }] } as typeof req.actor; next(); });
app.use("/api", toolAccessRoutes(db)); app.use("/api", aiConnectionRoutes(db)); app.use(errorHandler);
expect((await request(app).patch(`/api/tool-connections/${pool.id}`).send({ name: "Bypass" })).status).toBe(400);
expect((await request(app).delete(`/api/tool-connections/${pool.id}`)).status).toBe(400);
expect((await request(app).delete(`/api/companies/${companyId}/ai-connection-pools/${pool.id}`).send({ expectedRevision: pool.revision + 1 })).status).toBe(409);
expect((await request(app).delete(`/api/companies/${companyId}/ai-connection-pools/${pool.id}`).send({ expectedRevision: pool.revision })).status).toBe(200);
expect((await service().list(companyId)).some(row => row.id === pool.id)).toBe(false);
expect(await db.select().from(aiConnectionTaskPins).where(eq(aiConnectionTaskPins.poolId, pool.id))).toHaveLength(1);
expect(await resolve(pool.id, "catalog-removal", { persisted: selected })).toEqual(selected);
expect((await db.select().from(toolConnections).where(eq(toolConnections.id, members[0]!.binding.connectionId)))[0]?.status).toBe("active");
});
it("requires company connection management authority on pool configuration routes", async () => {
await db.update(companyMemberships).set({ membershipRole: "owner" }).where(and(eq(companyMemberships.companyId, companyId), eq(companyMemberships.principalId, "alice")));
const app = express(); app.use(express.json());
let actor = { type: "board", userId: "alice", source: "session", companyIds: [companyId], memberships: [{ companyId, status: "active", membershipRole: "owner" }] };
app.use((req, _res, next) => { req.actor = actor as typeof req.actor; next(); }); app.use("/api", aiConnectionRoutes(db)); app.use(errorHandler);
const pool = await makePool(); const url = `/api/companies/${companyId}/ai-connection-pools`;
expect((await request(app).get(url)).status).toBe(200);
expect((await request(app).get(`/api/companies/${otherCompanyId}/ai-connection-pools`)).status).toBe(403);
expect((await request(app).delete(`/api/companies/${otherCompanyId}/ai-connection-pools/${pool.id}`).send({ expectedRevision: pool.revision })).status).toBe(403);
await db.update(companyMemberships).set({ membershipRole: "viewer" }).where(and(eq(companyMemberships.companyId, companyId), eq(companyMemberships.principalId, "alice")));
actor = { ...actor, memberships: [{ companyId, status: "active", membershipRole: "viewer" }] };
expect((await request(app).get(url)).status).toBe(403);
expect((await request(app).post(url).send({ pluginKey, config: pool })).status).toBe(403);
expect((await request(app).get(`${url}/${pool.id}/inspection`)).status).toBe(403);
expect((await request(app).delete(`${url}/${pool.id}`).send({ expectedRevision: pool.revision })).status).toBe(403);
actor = { type: "agent", agentId, companyId, source: "agent_jwt" } as unknown as typeof actor;
expect((await request(app).get(url)).status).toBe(403);
expect((await request(app).delete(`${url}/${pool.id}`).send({ expectedRevision: pool.revision })).status).toBe(403);
await db.update(companyMemberships).set({ membershipRole: "owner" }).where(and(eq(companyMemberships.companyId, companyId), eq(companyMemberships.principalId, "alice")));
actor = { type: "board", userId: "alice", source: "session", companyIds: [companyId], memberships: [{ companyId, status: "active", membershipRole: "owner" }] };
expect((await request(app).delete(`${url}/${pool.id}`).send({ expectedRevision: pool.revision + 1 })).status).toBe(409);
expect((await request(app).delete(`${url}/${pool.id}`).send({ expectedRevision: pool.revision })).status).toBe(200);
expect((await request(app).get(url)).body.some((entry: { id: string }) => entry.id === pool.id)).toBe(false);
await db.update(companyMemberships).set({ membershipRole: "member" }).where(and(eq(companyMemberships.companyId, companyId), eq(companyMemberships.principalId, "alice")));
});
it("deletes pools without erasing pins, cursors, or admitted recovery evidence", async () => {
const pool = await makePool(); const selected = await resolve(pool.id, "retained");
await expect(service().remove(otherCompanyId, pool.id, pool.revision, "alice")).rejects.toThrow("not found");
await expect(service().remove(companyId, pool.id, pool.revision + 1, "alice")).rejects.toThrow("reload before deleting");
await service().remove(companyId, pool.id, pool.revision, "alice");
expect((await service().list(companyId)).some(entry => entry.id === pool.id)).toBe(false);
expect((await service().selectable(companyId, "alice")).some(entry => entry.id === pool.id)).toBe(false);
expect(await db.select().from(aiConnectionTaskPins).where(eq(aiConnectionTaskPins.poolId, pool.id))).toHaveLength(1);
const [cursor] = await db.select().from(aiConnectionRouterCursors).where(eq(aiConnectionRouterCursors.poolId, pool.id)); expect(cursor?.version).toBe(1);
await expect(resolve(pool.id, "new")).rejects.toThrow("choose another AI connection");
expect(await resolve(pool.id, "retained", { persisted: selected })).toEqual(selected);
const { name, enabled, mode, thresholdPercent, members: savedMembers } = pool;
await expect(service().save(pluginKey, { companyId, id: pool.id, expectedRevision: pool.revision + 1, config: { name, enabled, mode, thresholdPercent, members: savedMembers } }, "alice")).rejects.toThrow("not found");
});
it("does not commit a selection when its pool is deleted during a proposal", async () => {
const pool = await makePool();
proposal.mockImplementationOnce(async () => { await service().remove(companyId, pool.id, pool.revision, "alice"); return { kind: "selected", memberId: members[0]!.id }; });
await expect(resolve(pool.id, "deleted-during-selection")).rejects.toThrow("choose another AI connection");
expect(await db.select().from(aiConnectionTaskPins).where(eq(aiConnectionTaskPins.poolId, pool.id))).toHaveLength(0);
const [cursor] = await db.select().from(aiConnectionRouterCursors).where(eq(aiConnectionRouterCursors.poolId, pool.id)); expect(cursor?.version).toBe(0);
});
it("allows cleanup while experimental routing and its plugin are disabled", async () => {
const pool = await makePool();
await instanceSettingsService(db).updateExperimental({ enableAiConnectionRouters: false });
await db.update(plugins).set({ status: "disabled" }).where(eq(plugins.id, pluginId));
try { await expect(service().remove(companyId, pool.id, pool.revision, "alice")).resolves.toEqual({ ok: true }); }
finally { await instanceSettingsService(db).updateExperimental({ enableAiConnectionRouters: true }); await db.update(plugins).set({ status: "ready" }).where(eq(plugins.id, pluginId)); }
});
it("rotates new tasks across qualified backends and retains pins on turns, resets and service restart", async () => {
const pool = await makePool();
expect((await resolve(pool.id, "one")).memberId).toBe(members[0]!.id);
expect((await resolve(pool.id, "two")).memberId).toBe(members[1]!.id);
expect((await resolve(pool.id, "one", { overrides: { model: "claude-sonnet-5", modelReasoningEffort: "high" } })).memberId).toBe(members[0]!.id);
expect((await aiConnectionRouterService(db, worker).resolve({ companyId, poolId: pool.id, agentId, userId: "alice", adapterType: "paperclip_runner", taskKey: "one" })).memberId).toBe(members[0]!.id);
const [cursor] = await db.select().from(aiConnectionRouterCursors).where(eq(aiConnectionRouterCursors.poolId, pool.id)); expect(cursor?.version).toBe(2);
});
it("commits one allocation for duplicate starts and serializes concurrent tasks", async () => {
const pool = await makePool();
const same = await Promise.all(Array.from({ length: 8 }, () => resolve(pool.id, "duplicate")));
expect(new Set(same.map(s => s.memberId)).size).toBe(1);
const many = await Promise.all(Array.from({ length: 8 }, (_, n) => resolve(pool.id, `concurrent-${n}`)));
expect(many.filter(s => s.memberId === members[0]!.id)).toHaveLength(4);
const pins = await db.select().from(aiConnectionTaskPins).where(eq(aiConnectionTaskPins.poolId, pool.id)); expect(pins).toHaveLength(9);
const [cursor] = await db.select().from(aiConnectionRouterCursors).where(eq(aiConnectionRouterCursors.poolId, pool.id)); expect(cursor?.version).toBe(9);
});
it("rolls back both pin and cursor on a failure during commit", async () => {
const pool = await makePool();
await db.execute(`CREATE FUNCTION fixture_router_crash() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN RAISE EXCEPTION 'fixture commit crash'; END $$`);
await db.execute(`CREATE TRIGGER fixture_router_crash BEFORE UPDATE ON ai_connection_router_cursors FOR EACH ROW EXECUTE FUNCTION fixture_router_crash()`);
try { await expect(resolve(pool.id, "crash")).rejects.toThrow(); } finally { await db.execute(`DROP TRIGGER fixture_router_crash ON ai_connection_router_cursors`); await db.execute(`DROP FUNCTION fixture_router_crash()`); }
expect(await db.select().from(aiConnectionTaskPins).where(eq(aiConnectionTaskPins.poolId, pool.id))).toHaveLength(0);
const selected = await resolve(pool.id, "crash"); expect(selected.memberId).toBe(members[0]!.id);
const [cursor] = await db.select().from(aiConnectionRouterCursors).where(eq(aiConnectionRouterCursors.poolId, pool.id)); expect(cursor?.version).toBe(1);
});
it("rechecks config revisions after a proposal and preserves removed member snapshots", async () => {
const pool = await makePool(); const old = await resolve(pool.id, "existing");
const saved = await service().save(pluginKey, { companyId, id: pool.id, expectedRevision: pool.revision, config: { name: "Edited", enabled: true, mode: "round_robin", thresholdPercent: 90, members: [members[1]!] } }, "alice");
await expect(service().save(pluginKey, { companyId, id: pool.id, expectedRevision: pool.revision, config: saved }, "alice")).rejects.toThrow();
expect((await resolve(pool.id, "existing")).binding).toEqual(old.binding);
expect((await resolve(pool.id, "future")).memberId).toBe(members[1]!.id);
});
it("keeps account and harness pinned through independent composer fallback", async () => {
const pool = await makePool();
const first = await resolve(pool.id, "composer");
const changed = await resolve(pool.id, "composer", { overrides: { model: "claude-sonnet-5", modelReasoningEffort: "high" } });
expect(changed.binding).toEqual(first.binding); expect(changed.runtimeConfig).toMatchObject({ provider: "codex", model: "gpt-5.6-sol", modelReasoningEffort: "high" }); expect(changed.notes).toHaveLength(1);
const claude = poolMemberRuntimeConfig(members[1]!, "paperclip_runner", { model: "gpt-5.6-sol", modelReasoningEffort: "ultra" });
expect(claude.config).toMatchObject({ provider: "acpx", acpxAgent: "claude", model: "claude-sonnet-5" }); expect(claude.notes).toHaveLength(2);
const claudeEffort = poolMemberRuntimeConfig(members[1]!, "paperclip_runner", { modelReasoningEffort: "xhigh" });
expect(claudeEffort.config).toMatchObject({ provider: "acpx", acpxAgent: "claude", modelReasoningEffort: "xhigh" });
expect(claudeEffort.notes).toEqual([]);
const overrides = applyAiConnectionRouterTaskSettings({ model: "claude-sonnet-5", effort: "invalid", cwd: "/fixture" }, changed);
const native = resolvePaperclipRunnerNativeProviderInput({ backend: "codex_app_server", adapterConfig: projectPaperclipRunnerTaskConfig("codex_app_server", changed.runtimeConfig, overrides) });
expect(native).toMatchObject({ provider: "codex", model: "gpt-5.6-sol", codexReasoningEffort: "high" });
expect(overrides).toMatchObject({ cwd: "/fixture" }); expect(overrides).not.toHaveProperty("effort");
});
it("retries a changed config proposal without advancing the abandoned selection", async () => {
const pool = await makePool();
proposal.mockImplementationOnce(async () => {
await service().save(pluginKey, { companyId, id: pool.id, expectedRevision: pool.revision, config: { name: pool.name, enabled: true, mode: pool.mode, thresholdPercent: pool.thresholdPercent, members: [members[1]!] } }, "alice");
return { kind: "selected", memberId: members[0]!.id };
});
expect((await resolve(pool.id, "revision-conflict")).memberId).toBe(members[1]!.id);
const [cursor] = await db.select().from(aiConnectionRouterCursors).where(eq(aiConnectionRouterCursors.poolId, pool.id)); expect(cursor?.version).toBe(1);
});
it("fails closed when the experimental flag changes during a proposal", async () => {
const pool = await makePool();
proposal.mockImplementationOnce(async () => { await instanceSettingsService(db).updateExperimental({ enableAiConnectionRouters: false }); return { kind: "selected", memberId: members[0]!.id }; });
try { await expect(resolve(pool.id, "disabled-in-flight")).rejects.toThrow("disabled during allocation"); expect(await db.select().from(aiConnectionTaskPins).where(eq(aiConnectionTaskPins.poolId, pool.id))).toHaveLength(0); }
finally { await instanceSettingsService(db).updateExperimental({ enableAiConnectionRouters: true }); }
});
it("uses only compatible legacy members and makes reassignment an independent pin", async () => {
const pool = await makePool();
expect((await resolve(pool.id, "legacy", { adapterType: "claude_local" })).memberId).toBe(members[1]!.id);
expect((await resolve(pool.id, "legacy", { agentId: secondAgentId, adapterType: "codex_local" })).memberId).toBe(members[0]!.id);
const pins = await db.select().from(aiConnectionTaskPins).where(eq(aiConnectionTaskPins.poolId, pool.id)); expect(pins).toHaveLength(2);
});
it("adopts a valid existing account and requires explicit reset for an unrelated session", async () => {
const pool = await makePool();
expect((await resolve(pool.id, "adopt", { requireExisting: true, existingGrantId: members[1]!.binding.grantId })).memberId).toBe(members[1]!.id);
await expect(resolve(pool.id, "foreign-session", { requireExisting: true, existingGrantId: randomUUID() })).rejects.toThrow("Reset");
expect((await resolve(pool.id, "foreign-session")).memberId).toBe(members[0]!.id);
});
it("rejects cross-company access and unauthorized plugin proposals", async () => {
const pool = await makePool();
await expect(resolve(pool.id, "foreign", { companyId: otherCompanyId })).rejects.toThrow();
proposal.mockResolvedValueOnce({ kind: "selected", memberId: randomUUID() });
await expect(resolve(pool.id, "bad-proposal")).rejects.toThrow("unauthorized");
expect(await db.select().from(aiConnectionTaskPins).where(eq(aiConnectionTaskPins.poolId, pool.id))).toHaveLength(0);
});
it("does not grant members to another responsible user or move revoked pins", async () => {
const pool = await makePool(); const selected = await resolve(pool.id, "revoke");
await expect(resolve(pool.id, "revoke", { userId: "bob" })).rejects.toThrow();
await db.update(connectionGrants).set({ status: "revoked" }).where(eq(connectionGrants.id, selected.binding.mode === "responsible_user" ? "" : selected.binding.grantId));
try { await expect(resolve(pool.id, "revoke")).rejects.toThrow(); expect((await resolve(pool.id, "new")).memberId).toBe(members[1]!.id); }
finally { await db.update(connectionGrants).set({ status: "active" }).where(eq(connectionGrants.id, members[0]!.binding.grantId)); }
});
it("never probes pure rotation, caches authorized observations, and invalidates on credential change", async () => {
const spy = vi.spyOn(globalThis, "fetch").mockImplementation(async () => new Response(JSON.stringify({ five_hour: { utilization: 12 } })));
try {
const pure = await makePool(); await resolve(pure.id, "pure"); expect(spy).not.toHaveBeenCalled();
const aware = await makePool({ mode: "usage_aware", members: [members[1]!] }); await resolve(aware.id, "aware"); expect(spy).toHaveBeenCalledTimes(1); await resolve(aware.id, "aware"); expect(spy).toHaveBeenCalledTimes(1);
expect((await service().inspect(companyId, aware.id, "alice"))[members[1]!.id]?.checkedAt).not.toBeNull();
const [grant] = await db.select().from(connectionGrants).where(eq(connectionGrants.id, members[1]!.binding.grantId)); const ref = grant!.credentialSecretRefs.find(ref => ref.configPath === "ai.credential")!;
await secretService(db).rotate(ref.secretId, { value: "fixture-refreshed", preserveAiSessionEpoch: true }, { userId: "alice" });
expect((await service().inspect(companyId, aware.id, "alice"))[members[1]!.id]?.checkedAt).toBeNull();
await resolve(aware.id, "aware"); expect(spy).toHaveBeenCalledTimes(2);
proposal.mockResolvedValueOnce({ kind: "exhausted", retryAt: new Date(Date.now() + 60_000).toISOString(), skipped: {} });
await expect(resolve(aware.id, "aware")).rejects.toBeInstanceOf(AiConnectionPoolExhausted);
const pins = await db.select().from(aiConnectionTaskPins).where(eq(aiConnectionTaskPins.poolId, aware.id)); expect(pins).toHaveLength(1);
} finally { spy.mockRestore(); }
});
it("shares an in-flight usage observation across concurrent allocations", async () => {
const [grant] = await db.select().from(connectionGrants).where(eq(connectionGrants.id, members[1]!.binding.grantId));
const ref = grant!.credentialSecretRefs.find(ref => ref.configPath === "ai.credential")!;
await secretService(db).rotate(ref.secretId, { value: "fixture-shared-probe", preserveAiSessionEpoch: true }, { userId: "alice" });
const fetch = vi.spyOn(globalThis, "fetch").mockImplementation(async () => {
await new Promise(resolve => setTimeout(resolve, 100));
return new Response(JSON.stringify({ five_hour: { utilization: 12 } }));
});
try {
const pool = await makePool({ mode: "usage_aware", members: [members[1]!] });
await Promise.all(Array.from({ length: 6 }, (_, index) => resolve(pool.id, `shared-probe-${index}`)));
expect(fetch).toHaveBeenCalledTimes(1);
} finally { fetch.mockRestore(); }
});
it("expires a usage observation at a reported reset", async () => {
const [grant] = await db.select().from(connectionGrants).where(eq(connectionGrants.id, members[1]!.binding.grantId));
const ref = grant!.credentialSecretRefs.find(ref => ref.configPath === "ai.credential")!;
await secretService(db).rotate(ref.secretId, { value: "fixture-reset-cache", preserveAiSessionEpoch: true }, { userId: "alice" });
const spy = vi.spyOn(globalThis, "fetch").mockImplementation(async () => new Response(JSON.stringify({ five_hour: { utilization: 12, resets_at: new Date(Date.now() + 200).toISOString() } })));
try {
const pool = await makePool({ mode: "usage_aware", members: [members[1]!] }); await resolve(pool.id, "reset-cache"); expect(spy).toHaveBeenCalledTimes(1);
await new Promise(resolve => setTimeout(resolve, 250)); await resolve(pool.id, "reset-cache"); expect(spy).toHaveBeenCalledTimes(2);
} finally { spy.mockRestore(); }
});
it("treats unfinished probes as unknown within the shared selection budget", async () => {
const [grant] = await db.select().from(connectionGrants).where(eq(connectionGrants.id, members[1]!.binding.grantId));
const ref = grant!.credentialSecretRefs.find(ref => ref.configPath === "ai.credential")!;
await secretService(db).rotate(ref.secretId, { value: "fixture-slow-probe", preserveAiSessionEpoch: true }, { userId: "alice" });
const spy = vi.spyOn(globalThis, "fetch").mockImplementation(() => new Promise<Response>(() => {}));
try { const pool = await makePool({ mode: "usage_aware", members: [members[1]!] }); const start = Date.now(); expect((await resolve(pool.id, "slow-probe")).memberId).toBe(members[1]!.id); expect(Date.now() - start).toBeLessThan(20_000); expect(proposal.mock.calls.at(-1)?.[2].candidates[0]?.usage).toBeUndefined(); }
finally { spy.mockRestore(); }
}, 25_000);
it("persists run-key affinity before policy calls and reuses it across allocation recovery", async () => {
await instanceSettingsService(db).updateExperimental({ enableAiConnectionRouters: true, enableWorktreeRunExecution: true });
const pool = await makePool({ members: [members[1]!] });
const [agent] = await db.insert(agents).values({ companyId, name: "Run-key fixture", adapterType: "claude_local", adapterConfig: { cwd: home, engine: "cli" }, runtimeConfig: { aiConnection: { mode: "router", connectionId: pool.id }, heartbeat: { enabled: false } } }).returning();
let affinity: string | undefined;
const execute = vi.fn(); registerServerAdapter({ ...getServerAdapter("claude_local"), execute });
const heartbeat = heartbeatService(db, { pluginWorkerManager: worker, runtimeEnv: {} });
proposal.mockImplementationOnce(async (_id, _method, request) => {
const [run] = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.agentId, agent!.id), eq(heartbeatRuns.status, "running")));
expect(run?.contextSnapshot?.aiRouterTaskKey).toBe(run?.id);
expect(request.taskKey).toBe(run?.id);
affinity = run!.id;
throw new Error("fixture policy interruption");
});
try {
const run = await heartbeat.wakeup(agent!.id, { source: "on_demand", reason: "manual", triggerDetail: "manual", contextSnapshot: { responsibleUserId: "alice", wakeReason: "manual" } });
expect(run).not.toBeNull();
await expect.poll(async () => (await heartbeat.getRun(run!.id))?.status, { timeout: 20_000 }).toBe("failed");
expect(affinity).toBe(run!.id);
expect((await heartbeat.getRun(run!.id))?.contextSnapshot?.aiRouterTaskKey).toBe(affinity);
expect(execute).not.toHaveBeenCalled();
const input = { agentId: agent!.id, adapterType: "claude_local" };
const committed = await resolve(pool.id, affinity!, input);
expect(await aiConnectionRouterService(db, worker).resolve({ companyId, poolId: pool.id, agentId: agent!.id, userId: "alice", adapterType: "claude_local", taskKey: affinity! })).toEqual(committed);
const [cursor] = await db.select().from(aiConnectionRouterCursors).where(eq(aiConnectionRouterCursors.poolId, pool.id)); expect(cursor?.version).toBe(1);
expect(await db.select().from(aiConnectionTaskPins).where(eq(aiConnectionTaskPins.poolId, pool.id))).toHaveLength(1);
} finally { proposal.mockReset(); proposal.mockImplementation(defaultProposal); unregisterServerAdapter("claude_local"); await heartbeat.drainActiveRunExecutions(); }
}, 30_000);
it("defers a pinned task before provider work and schedules a retry with its affinity intact", async () => {
await instanceSettingsService(db).updateExperimental({ enableAiConnectionRouters: true, enableWorktreeRunExecution: true });
const pool = await makePool({ members: [members[1]!] });
const [agent] = await db.insert(agents).values({ companyId, name: "Quota fixture", adapterType: "claude_local", adapterConfig: { cwd: home, engine: "cli" }, runtimeConfig: { aiConnection: { mode: "router", connectionId: pool.id }, heartbeat: { enabled: false } } }).returning();
const [issue] = await db.insert(issues).values({ companyId, title: "Quota fixture", status: "todo", assigneeAgentId: agent!.id, responsibleUserId: "alice", createdByUserId: "alice" }).returning();
const pinned = await resolve(pool.id, issue!.id, { agentId: agent!.id, adapterType: "claude_local" });
await service().save(pluginKey, { companyId, id: pool.id, expectedRevision: pool.revision, config: { name: pool.name, enabled: true, mode: "usage_aware", thresholdPercent: 90, members: [members[1]!] } }, "alice");
const execute = vi.fn(); registerServerAdapter({ ...getServerAdapter("claude_local"), execute });
const fetch = vi.spyOn(globalThis, "fetch").mockImplementation(async () => new Response(JSON.stringify({ five_hour: { utilization: 100 } })));
proposal.mockResolvedValueOnce({ kind: "exhausted", retryAt: new Date(Date.now() + 60_000).toISOString(), skipped: { [members[1]!.id]: "Exhausted" } });
const heartbeat = heartbeatService(db, { pluginWorkerManager: worker, runtimeEnv: {} });
try {
const run = await heartbeat.wakeup(agent!.id, { source: "assignment", reason: "issue_assigned", payload: { issueId: issue!.id }, contextSnapshot: { issueId: issue!.id, wakeReason: "issue_assigned", responsibleUserId: "alice" }, triggerDetail: "system" });
expect(run).not.toBeNull();
await expect.poll(async () => (await heartbeat.getRun(run!.id))?.status, { timeout: 20_000 }).toBe("cancelled");
const cancelled = await heartbeat.getRun(run!.id); expect(cancelled?.errorCode).toBe("ai_connection_pool_exhausted");
expect(cancelled?.resultJson).toMatchObject({
executionRecovery: { kind: "ai_connection_wait", providerWorkStarted: false },
cancellation: {
source: "control_plane", expected: true, initiator: { type: "system" },
reason: "Waiting for an AI connection pool account",
recordedAt: cancelled!.finishedAt!.toISOString(),
},
});
expect(legacyExecutionNeedsReconciliation(cancelled!)).toBe(false);
expect(legacyExecutionNeedsReconciliation({ ...cancelled!, resultJson: { executionRecovery: { kind: "ai_connection_wait", providerWorkStarted: true } } })).toBe(true);
await expect.poll(async () => (await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.agentId, agent!.id), eq(heartbeatRuns.scheduledRetryReason, "ai_connection_pool_wait")))).length, { timeout: 5000 }).toBe(1);
const [retry] = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.agentId, agent!.id), eq(heartbeatRuns.scheduledRetryReason, "ai_connection_pool_wait")));
await waitForPendingRunFailureReports();
expect(captureRunFailure.mock.calls.filter(([report]) => report.runId === run!.id)).toHaveLength(0);
expect(retry?.resultJson?.cancellation).toBeUndefined();
expect(retry?.contextSnapshot?.aiRouterTaskKey).toBe(issue!.id); expect(retry?.contextSnapshot?.executionRetryAccounting).toMatchObject({ failureRetries: 0 }); expect(execute).not.toHaveBeenCalled();
const [pin] = await db.select().from(aiConnectionTaskPins).where(and(eq(aiConnectionTaskPins.poolId, pool.id), eq(aiConnectionTaskPins.agentId, agent!.id))); expect(pin?.selection).toEqual(pinned);
const [cursor] = await db.select().from(aiConnectionRouterCursors).where(eq(aiConnectionRouterCursors.poolId, pool.id)); expect(cursor?.version).toBe(1);
await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date() }).where(eq(heartbeatRuns.agentId, agent!.id));
} finally { fetch.mockRestore(); proposal.mockReset(); proposal.mockImplementation(defaultProposal); unregisterServerAdapter("claude_local"); await heartbeat.drainActiveRunExecutions(); }
}, 30_000);
it("keeps unexpected pool policy failures reportable before provider work", async () => {
await instanceSettingsService(db).updateExperimental({ enableAiConnectionRouters: true, enableWorktreeRunExecution: true });
const pool = await makePool({ members: [members[1]!] });
const [agent] = await db.insert(agents).values({ companyId, name: "Unexpected router failure", adapterType: "claude_local", adapterConfig: { cwd: home, engine: "cli" }, runtimeConfig: { aiConnection: { mode: "router", connectionId: pool.id }, heartbeat: { enabled: false } } }).returning();
const [issue] = await db.insert(issues).values({ companyId, title: "Router failure fixture", status: "todo", assigneeAgentId: agent!.id, responsibleUserId: "alice", createdByUserId: "alice" }).returning();
const execute = vi.fn(); registerServerAdapter({ ...getServerAdapter("claude_local"), execute });
proposal.mockRejectedValueOnce(new Error("Synthetic unexpected router policy failure"));
const heartbeat = heartbeatService(db, { pluginWorkerManager: worker, runtimeEnv: {} });
try {
const run = await heartbeat.wakeup(agent!.id, { source: "assignment", reason: "issue_assigned", payload: { issueId: issue!.id }, contextSnapshot: { issueId: issue!.id, wakeReason: "issue_assigned", responsibleUserId: "alice" }, triggerDetail: "system" });
expect(run).not.toBeNull();
await expect.poll(async () => (await heartbeat.getRun(run!.id))?.status, { timeout: 20_000 }).toBe("failed");
const failed = await heartbeat.getRun(run!.id);
expect(failed?.errorCode).toBe("configuration_incomplete");
expect(failed?.resultJson).toMatchObject({ configurationIncomplete: { reason: "ai_connection_unavailable" } });
expect(failed?.resultJson?.cancellation).toBeUndefined();
expect(execute).not.toHaveBeenCalled();
await expect.poll(() => captureRunFailure.mock.calls.filter(([report]) => report.runId === run!.id)).toHaveLength(1);
expect(captureRunFailure).toHaveBeenCalledWith(expect.objectContaining({ runId: run!.id, runStatus: "failed", errorCode: "configuration_incomplete" }));
} finally { proposal.mockReset(); proposal.mockImplementation(defaultProposal); unregisterServerAdapter("claude_local"); await heartbeat.drainActiveRunExecutions(); }
}, 30_000);
it("repairs the failed pool account without replacing the pool or adopting the configured provider", async () => {
const pool = await makePool({ members: [members[1]!] });
const [agent] = await db.insert(agents).values({ companyId, name: "Pool repair", adapterType: "paperclip_runner", adapterConfig: { provider: "codex", model: "gpt-5.6-sol" }, runtimeConfig: { aiConnection: { mode: "router", connectionId: pool.id } } }).returning();
const [issue] = await db.insert(issues).values({ companyId, title: "Pool repair", status: "in_progress", assigneeAgentId: agent!.id, responsibleUserId: "alice", createdByUserId: "alice" }).returning();
const selection = await resolve(pool.id, issue!.id, { agentId: agent!.id });
const runtime = await prepareManagedAiRuntime(db, { companyId, agentId: agent!.id, responsibleUserId: "alice", adapterType: "paperclip_runner", config: selection.runtimeConfig, binding: selection.binding });
const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId: agent!.id, status: "failed", errorCode: "acpx_auth_required", responsibleUserId: "alice", startedAt: new Date(), contextSnapshot: { issueId: issue!.id, aiRouterTaskKey: issue!.id, aiRouterSelection: selection, aiConnection: { ...runtime!.attribution, identity: runtime!.identity } } }).returning();
try {
const intents = connectionIntentService(db);
const card = await intents.requestForRunAuthFailure(run!.id);
expect(card?.service).toBe("anthropic"); expect(card?.state).toBe("needs_user_action");
const options = await intents.setupOptions(card!.interactionId!);
expect(options.aiConnection).toEqual(selection.binding); expect(options.aiConnectionRequiresAdoption).toBeUndefined();
const [failedGrant] = await db.select().from(connectionGrants).where(eq(connectionGrants.id, members[1]!.binding.grantId)); expect(failedGrant?.status).toBe("needs_reauthorization");
await db.update(connectionGrants).set({ status: "active" }).where(eq(connectionGrants.id, members[1]!.binding.grantId));
await db.update(toolConnections).set({ healthStatus: "ok" }).where(eq(toolConnections.id, members[1]!.binding.connectionId));
expect(await intents.complete(card!.interactionId!, members[1]!.binding.connectionId, "alice")).toMatchObject({ status: "accepted" });
const [savedAgent] = await db.select().from(agents).where(eq(agents.id, agent!.id)); expect(savedAgent?.runtimeConfig.aiConnection).toEqual({ mode: "router", connectionId: pool.id });
expect((await resolve(pool.id, issue!.id, { agentId: agent!.id })).binding).toEqual(selection.binding);
await db.update(agents).set({ runtimeConfig: { aiConnection: { mode: "router", connectionId: randomUUID() } } }).where(eq(agents.id, agent!.id));
await expect(intents.setupOptions(card!.interactionId!)).rejects.toThrow("configuration changed");
} finally {
await runtime?.cleanup();
await db.update(connectionGrants).set({ status: "active" }).where(eq(connectionGrants.id, members[1]!.binding.grantId));
await db.update(toolConnections).set({ healthStatus: "ok" }).where(eq(toolConnections.id, members[1]!.binding.connectionId));
}
});
it("rejects disabled routing but recovers server-owned native evidence after disable/uninstall", async () => {
const pool = await makePool(); const selected = await resolve(pool.id, "recovery");
await instanceSettingsService(db).updateExperimental({ enableAiConnectionRouters: false });
try { await expect(resolve(pool.id, "recovery")).rejects.toMatchObject({ status: 422, details: { code: "ai_connection_router_disabled" } }); expect(await resolve(pool.id, "recovery", { persisted: selected })).toEqual(selected); }
finally { await instanceSettingsService(db).updateExperimental({ enableAiConnectionRouters: true }); }
await db.update(plugins).set({ status: "disabled" }).where(eq(plugins.id, pluginId));
try { await expect(resolve(pool.id, "new")).rejects.toThrow("plugin"); expect(await resolve(pool.id, "recovery", { persisted: selected })).toEqual(selected); await db.delete(toolConnections).where(eq(toolConnections.id, pool.id)); expect(await resolve(pool.id, "recovery", { persisted: selected })).toEqual(selected); }
finally { await db.update(plugins).set({ status: "ready" }).where(eq(plugins.id, pluginId)); }
});
it("resumes a responsible-user session through a delegated pool using real managed account configurations", async () => {
await aiConnectionService(db).setDefault(companyId, "alice", members[0]!.binding.grantId!);
const pool = await makePool({ members: [members[0]!] });
const selected = await resolve(pool.id, "managed-mode-adoption", { requireExisting: true, existingGrantId: members[0]!.binding.grantId });
const directBinding = { mode: "responsible_user" as const, provider: "openai" as const, method: "api_key" as const };
const prepare = { companyId, agentId, responsibleUserId: "alice", adapterType: "paperclip_runner", config: selected.runtimeConfig };
const before = await prepareManagedAiRuntime(db, { ...prepare, binding: directBinding });
const after = await prepareManagedAiRuntime(db, { ...prepare, binding: selected.binding });
try {
expect(before.config.managedAiConnection.mode).toBe("responsible_user");
expect(after.config.managedAiConnection.mode).toBe("delegated");
expect(after.sessionIdentity).toBe(before.sessionIdentity);
const common = { adapterType: "paperclip_runner", issueOverrides: null, workspaceConfig: {}, environment: {}, environmentEnv: null, projectEnv: null, routineEnv: null, runtimeSkills: [], agentConfigRevision: null };
const previous = await buildEffectiveRunSessionConfigMetadata({ ...common, effectiveAdapterConfig: before.config, managedAiHome: before.home, agentRuntimeConfig: { aiConnection: directBinding } });
const currentInput = { ...common, effectiveAdapterConfig: after.config, managedAiHome: after.home, agentRuntimeConfig: { aiConnection: { mode: "router", connectionId: pool.id } } };
const current = await buildEffectiveRunSessionConfigMetadata(currentInput);
const alternatives = aiConnectionSessionCompatibilityInputs({ ...currentInput, originalIssueOverrides: null, binding: selected.binding, router: true, storedIdentity: before.sessionIdentity, sessionIdentity: after.sessionIdentity, credentialIdentity: after.identity, revisions: [] });
const compatibleConfigMetadata = await Promise.all(alternatives.map(candidate => buildEffectiveRunSessionConfigMetadata({ ...currentInput, ...candidate })));
expect(resolveTaskSessionConfigFreshness({ hasTaskSession: true, configuredModel: String(selected.runtimeConfig.model), taskSessionParams: { __paperclipConfiguredModel: selected.runtimeConfig.model, __paperclipConfigFingerprint: previous.fingerprint, __paperclipConfigFingerprintVersion: previous.version, __paperclipConfigCategoryFingerprints: previous.categoryFingerprints }, configMetadata: current, compatibleConfigMetadata }).reset).toBe(false);
} finally { await before.cleanup(); await after.cleanup(); }
});
it("admits a healthy pool member when the saved Grok connection is held", async () => {
const original = await instanceSettingsService(db).getExperimental();
await instanceSettingsService(db).updateExperimental({ enableAiConnectionRouters: true,
enableNativeRunner: true, enableGrokBot: true, enablePublicMcp: true, enableWorktreeRunExecution: true });
const pool = await makePool({ members: [members[0]!] });
const bindingId = randomUUID();
const workspace = path.join(home, "routed-grok-work");
await mkdir(workspace);
const [agent] = await db.insert(agents).values({ companyId, name: "Routed Grok fixture", status: "idle",
adapterType: "paperclip_runner", adapterConfig: { provider: "grok_bot", externalBindingId: bindingId, cwd: workspace },
runtimeConfig: { aiConnection: { mode: "router", connectionId: pool.id }, heartbeat: { enabled: false, wakeOnDemand: true } } }).returning();
await db.insert(dotAgentBindings).values({ id: bindingId, companyId, agentId: agent.id,
operatorId: "alice", providerKind: "grok_bot", status: "connected" });
const [grant] = await db.select().from(connectionGrants).where(eq(connectionGrants.id, members[0]!.binding.grantId));
const ref = grant!.credentialSecretRefs.find(ref => ref.configPath === "ai.credential")!;
await db.insert(grokBotRoutines).values({ companyId, agentId: agent.id, bindingId, bindingGeneration: 1,
botName: "Unused held Bot", secretId: ref.secretId, secretVersion: 1,
recoveryRequiredAt: new Date(), recoveryReason: "work_session" });
expect(await dotRunnerBroker(db).canStartNewWork(companyId, agent.id, bindingId)).toBe(false);
const readiness = vi.spyOn(dotRunnerBroker(db), "canStartNewWork");
const beforeNative = vi.fn(async () => {
await db.update(agents).set({ status: "paused" }).where(eq(agents.id, agent.id));
throw new Error("fixture stopped before provider start");
});
const heartbeat = heartbeatService(db, { pluginWorkerManager: worker, runtimeEnv: {},
beforeNativeRuntimeSelection: beforeNative,
nativeSessionBackendFactory: () => { throw new Error("Unexpected provider start"); } });
const [issue] = await db.insert(issues).values({ companyId, title: "Healthy routed task", status: "todo",
assigneeAgentId: agent.id, responsibleUserId: "alice", createdByUserId: "alice" }).returning();
try {
const run = await heartbeat.wakeup(agent.id, { source: "assignment", reason: "issue_assigned", triggerDetail: "system",
payload: { issueId: issue.id }, contextSnapshot: { issueId: issue.id, responsibleUserId: "alice" } });
expect(run).not.toBeNull();
await heartbeat.drainActiveRunExecutions();
const saved = await heartbeat.getRun(run!.id);
expect(saved?.contextSnapshot?.aiRouterSelection).toMatchObject({ poolId: pool.id,
memberId: members[0]!.id, runtimeConfig: { provider: "codex" } });
expect(saved?.error).toContain("fixture stopped before provider start");
expect(beforeNative).toHaveBeenCalledExactlyOnceWith(run!.id);
expect(readiness).not.toHaveBeenCalled();
} finally {
await db.update(agents).set({ status: "paused" }).where(eq(agents.id, agent.id));
await heartbeat.drainActiveRunExecutions();
readiness.mockRestore();
await instanceSettingsService(db).updateExperimental(original);
}
}, 30_000);
it("keeps session identity and fingerprints stable on authenticated refresh, but changes them on manual replacement", async () => {
const input = { companyId, agentId, responsibleUserId: "alice", adapterType: "paperclip_runner", binding: members[0]!.binding, config: { provider: "codex", model: "gpt-5.6-sol" } };
const first = await prepareManagedAiRuntime(db, input);
const [grant] = await db.select().from(connectionGrants).where(eq(connectionGrants.id, members[0]!.binding.grantId)); const ref = grant!.credentialSecretRefs.find(ref => ref.configPath === "ai.credential")!;
try {
await secretService(db).rotate(ref.secretId, { value: "fixture-new-token", preserveAiSessionEpoch: true }, { userId: "alice" });
const refreshed = await prepareManagedAiRuntime(db, input);
try { expect(refreshed.identity).not.toBe(first.identity); expect(refreshed.sessionIdentity).toBe(first.sessionIdentity); expect(managedAiSessionFingerprintConfig(refreshed.config, refreshed.home)).toEqual(managedAiSessionFingerprintConfig(first.config, first.home)); } finally { await refreshed.cleanup(); }
await secretService(db).rotate(ref.secretId, { value: "fixture-replacement" }, { userId: "alice" });
const replaced = await prepareManagedAiRuntime(db, input); try { expect(replaced.sessionIdentity).not.toBe(first.sessionIdentity); } finally { await replaced.cleanup(); }
const [secret] = await db.select().from(companySecrets).where(and(eq(companySecrets.companyId, companyId), eq(companySecrets.id, ref.secretId))); expect(secret!.aiSessionEpoch).toBeGreaterThan(0);
} finally { await first.cleanup(); }
});
});
enableRunFailureReporting();