mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-07 16:35:48 +01:00
First PR: runtime. UI: #15728. Qualification: #15729.
## Thinking Path
> - Paperclip manages agent work through tasks and governed connections.
> - Chat connections already bind external conversations to those tasks.
> - Speko handles voice and telephony, while the selected Paperclip
agent does the work.
> - A call needs durable session authority, ordered follow-ups, and safe
result delivery.
> - This PR adds an explicit voice transport to the existing connection
runtime.
> - The UI and qualification suites follow in linked PRs. Speko stays
experimental.
## Linked Issues or Issue Description
**Problem or motivation**
Phone and browser voice conversations cannot use the chat connection
task lifecycle. A separate voice runtime is needed to preserve task
authority, retries, and reply history.
**Proposed solution**
Add company-scoped voice sessions, signed callbacks, durable replies,
inbound intake, and an agent-facing outbound-call tool. Reuse existing
tasks, queues, secrets, and publications.
**Roadmap alignment**
This extends the experimental chat connector entry points under Agent
Chat and Connected Apps. The project owner requested this feature.
Refs #11418 for the original provider catalog proposal. This
implementation uses the newer chat connection infrastructure rather than
an MCP catalog-only integration. Refs #15721 for remote OpenCode
startup.
## What Changed
- Add additive voice schema, shared contracts, session APIs, and
provider callbacks.
- Bind browser, inbound, and outbound sessions to an endpoint, task, and
selected agent.
- Add low-trust guest intake in an explicitly selected sandbox and keep
ordinary approval rules.
- Submit task work through session-scoped tools and push approved
replies through Speko POST /v1/calls/{id}/messages.
- Distinguish provider acceptance from confirmed spoken playback.
Reconcile reports without downloading recordings.
- Add protocol, migration, persistence, provider, and
delivery-diagnostic tests.
## Verification
- Master 3389bea708 is integrated. Database/shared builds, server
typecheck, migration upgrade/replay checks, and 46
communication-guidance/provider/protocol tests passed.
- The final persistence suite passed 85 real-database tests. Five
focused native continuation/authorization regressions passed. Five
callback ownership/rotation tests passed after the last runtime repair.
- Review repairs cover reconnect ownership, current caller authority,
duplicate delivery, multipart answers, task-page answers, guest question
handling, and saved sandbox recovery.
- A real OpenCode/OpenRouter DeepSeek task ran in Daytona and saved a
correct disk-space answer. This did not establish live-call playback.
- Full local typecheck/build passed before the final repairs; focused
typechecks passed after them. Latest-head CI runs the repository gates.
- The separate integrated browser qualification passed two journeys and
failed browser answer retrieval. The feature remains experimental. The
report in #15729 records this failure and outstanding real-provider
acceptance.
- No lockfiles are included. Speko migrations follow master as
0324–0330.
## Risks
- Voice sessions add migrations and provider-dependent lifecycle
behavior. Migrations preserve existing chat and AI transports.
- Callback origins must be reachable HTTPS. Provider keys and signing
secrets remain vaulted.
- Unverified callers receive task-scoped low-trust authority. Caller ID
does not grant private task access.
- Outbound telephony incurs provider charges. Provider acceptance is not
proof that the caller heard an answer.
- Keep the feature behind the existing experimental chat setting until
the real browser, inbound, outbound, and delayed-result acceptance
matrix passes.
## Model Used
OpenAI Codex, GPT-6, with tool use and code execution. The exact serving
model ID and context window are not exposed in this session.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
260 lines
9.8 KiB
TypeScript
260 lines
9.8 KiB
TypeScript
import { z } from "zod";
|
||
import { aiProviderRoutingSchema, aiRoutingHarness, isAiRoutingCompatible, type AiProviderRouting } from "./ai-provider-routing.js";
|
||
|
||
/** Runtime authentication is a separate transport, never a tool or channel. */
|
||
export const connectionPurposeTransportSchema = z.discriminatedUnion(
|
||
"connectionPurpose",
|
||
[
|
||
z.object({
|
||
connectionPurpose: z.literal("tool"),
|
||
transport: z.enum(["mcp_remote", "rest_api", "local_stdio"]),
|
||
}),
|
||
z.object({
|
||
connectionPurpose: z.literal("channel"),
|
||
transport: z.enum(["chat_sdk", "rest_api", "voice"]),
|
||
config: z.object({ provider: z.string().optional() }).passthrough().optional(),
|
||
}).refine(
|
||
(connection) => connection.transport === "voice" ? connection.config?.provider === "speko" : connection.transport === "chat_sdk" ? connection.config?.provider !== "speko" : connection.config?.provider === "agentmail",
|
||
{ message: "Channel transport must match its provider: voice for Speko, REST for AgentMail, Chat SDK otherwise", path: ["config", "provider"] },
|
||
),
|
||
z.object({
|
||
connectionPurpose: z.literal("ai"),
|
||
transport: z.literal("runtime_auth"),
|
||
}),
|
||
],
|
||
);
|
||
export type ConnectionPurposeTransport = z.infer<
|
||
typeof connectionPurposeTransportSchema
|
||
>;
|
||
|
||
export const AI_PROVIDERS = [
|
||
"anthropic",
|
||
"openai",
|
||
"openrouter",
|
||
"xai",
|
||
"google",
|
||
] as const;
|
||
export const aiProviderSchema = z.enum(AI_PROVIDERS);
|
||
export const aiAuthMethodSchema = z.enum(["subscription", "api_key"]);
|
||
export type AiProvider = z.infer<typeof aiProviderSchema>;
|
||
export type AiAuthMethod = z.infer<typeof aiAuthMethodSchema>;
|
||
const requirement = { provider: aiProviderSchema, method: aiAuthMethodSchema };
|
||
export const aiConnectionBindingSchema = z.discriminatedUnion("mode", [
|
||
z.object({
|
||
provider: aiProviderSchema,
|
||
// Retained on the wire for older servers during rolling upgrades. The
|
||
// responsible user's provider default determines the actual run method.
|
||
method: aiAuthMethodSchema,
|
||
mode: z.literal("responsible_user"),
|
||
}).strict(),
|
||
z
|
||
.object({
|
||
...requirement,
|
||
mode: z.literal("shared"),
|
||
connectionId: z.string().uuid(),
|
||
grantId: z.string().uuid(),
|
||
})
|
||
.strict(),
|
||
z
|
||
.object({
|
||
...requirement,
|
||
// Explicit personal selection. Human access still applies on every run.
|
||
mode: z.literal("delegated"),
|
||
connectionId: z.string().uuid(),
|
||
grantId: z.string().uuid(),
|
||
})
|
||
.strict(),
|
||
]);
|
||
export type AiConnectionBinding = z.infer<typeof aiConnectionBindingSchema>;
|
||
export const aiConnectionMetadataSchema = z.object({ ...requirement, routing: aiProviderRoutingSchema.optional() }).strict();
|
||
export type AiConnectionMetadata = z.infer<typeof aiConnectionMetadataSchema>;
|
||
|
||
/** Existing integrations only. This table describes compatibility, never routing. */
|
||
export const AI_CONNECTION_CAPABILITIES: Record<
|
||
AiProvider,
|
||
{
|
||
name: string;
|
||
methods: Partial<
|
||
Record<AiAuthMethod, { adapters: readonly string[]; envKey: string }>
|
||
>;
|
||
}
|
||
> = {
|
||
google: { name: "Google", methods: { api_key: { adapters: ["gemini_local"], envKey: "GEMINI_API_KEY" } } },
|
||
anthropic: {
|
||
name: "Claude",
|
||
methods: {
|
||
subscription: {
|
||
adapters: ["claude_local"],
|
||
envKey: "CLAUDE_CODE_OAUTH_TOKEN",
|
||
},
|
||
api_key: { adapters: ["claude_local"], envKey: "ANTHROPIC_API_KEY" },
|
||
},
|
||
},
|
||
openai: {
|
||
name: "OpenAI",
|
||
methods: {
|
||
subscription: { adapters: ["codex_local"], envKey: "CODEX_HOME" },
|
||
api_key: { adapters: ["codex_local"], envKey: "OPENAI_API_KEY" },
|
||
},
|
||
},
|
||
openrouter: {
|
||
name: "OpenRouter",
|
||
methods: {
|
||
api_key: { adapters: ["opencode_local"], envKey: "OPENROUTER_API_KEY" },
|
||
},
|
||
},
|
||
xai: {
|
||
name: "Grok",
|
||
methods: {
|
||
subscription: { adapters: ["grok_local"], envKey: "GROK_HOME" },
|
||
api_key: { adapters: ["grok_local"], envKey: "XAI_API_KEY" },
|
||
},
|
||
},
|
||
};
|
||
export function isAiConnectionCompatible(
|
||
requirement: AiConnectionMetadata | AiConnectionBinding,
|
||
adapterType: string,
|
||
model?: unknown,
|
||
runnerProvider?: unknown,
|
||
acpxAgent?: unknown,
|
||
): boolean {
|
||
adapterType = aiRoutingHarness(adapterType, runnerProvider, acpxAgent);
|
||
if ("routing" in requirement && requirement.routing) return requirement.method === "api_key" && isAiRoutingCompatible(requirement.routing, adapterType);
|
||
// A fixed binding contains identity only. The service checks authoritative
|
||
// connection metadata before resolving credentials or running the harness.
|
||
if ("mode" in requirement && requirement.mode !== "responsible_user" && requirement.method === "api_key")
|
||
return ["claude_local", "codex_local", "opencode_local", "hermes_local", "gemini_local", "grok_local"].includes(adapterType);
|
||
const methods = AI_CONNECTION_CAPABILITIES[requirement.provider].methods;
|
||
const candidates = "mode" in requirement && requirement.mode === "responsible_user"
|
||
? Object.values(methods)
|
||
: requirement.method ? [methods[requirement.method]] : [];
|
||
return (
|
||
candidates.some((method) => method?.adapters.includes(adapterType)) &&
|
||
(requirement.provider !== "openrouter" ||
|
||
(typeof model === "string" && model.startsWith("openrouter/")))
|
||
);
|
||
}
|
||
export type AiConnectionUnavailableReason =
|
||
| "responsible_user_missing"
|
||
| "membership_missing"
|
||
| "default_missing"
|
||
| "connection_missing"
|
||
| "connection_unavailable"
|
||
| "incompatible"
|
||
| "access_denied"
|
||
| "credential_missing";
|
||
export interface AiConnectionAttribution {
|
||
/** Billing account at execution time; never resolved again from mutable settings. */
|
||
subscriptionId?: string;
|
||
connectionId: string;
|
||
grantId: string;
|
||
provider: AiProvider;
|
||
method: AiAuthMethod;
|
||
mode: AiConnectionBinding["mode"];
|
||
responsibleUserId: string | null;
|
||
}
|
||
export type AiConnectionResolution =
|
||
| { ok: true; attribution: AiConnectionAttribution }
|
||
| { ok: false; reason: AiConnectionUnavailableReason; message: string };
|
||
|
||
export interface AiManagedConnectionSummary {
|
||
id: string;
|
||
grantId: string;
|
||
companyId: string;
|
||
provider: AiProvider;
|
||
method: AiAuthMethod;
|
||
name: string;
|
||
routing?: AiProviderRouting;
|
||
accountLabel?: string;
|
||
ownership: "personal" | "shared";
|
||
ownerUserId?: string;
|
||
ownerName?: string;
|
||
isDefault: boolean;
|
||
status: "connected" | "needs_attention" | "expired" | "revoked";
|
||
unavailableReason?: string;
|
||
usageProbeSupported?: boolean;
|
||
}
|
||
export interface AiConnectionList {
|
||
currentUserId: string;
|
||
pools?: import("./ai-connection-router.js").AiConnectionPool[];
|
||
canManageConnections: boolean;
|
||
connections: AiManagedConnectionSummary[];
|
||
}
|
||
export const createAiConnectionSchema = z
|
||
.object({
|
||
...requirement,
|
||
name: z.string().trim().min(1).max(160),
|
||
ownership: z.enum(["personal", "shared"]),
|
||
routing: aiProviderRoutingSchema.optional(),
|
||
apiKey: z.string().trim().min(1).max(32768).optional(),
|
||
loginSessionId: z.string().max(128).optional(),
|
||
connectionId: z.string().uuid().optional(),
|
||
agentIds: z.array(z.string().uuid()).max(1000).default([]),
|
||
allAgents: z.boolean().default(false),
|
||
})
|
||
.strict()
|
||
.superRefine((v, ctx) => {
|
||
if (!AI_CONNECTION_CAPABILITIES[v.provider].methods[v.method])
|
||
ctx.addIssue({ code: "custom", message: "Unsupported sign-in method" });
|
||
if (v.routing && (v.method !== "api_key" || (v.routing.kind === "openrouter" && v.provider !== "openrouter") || (v.routing.kind === "bedrock" && v.provider !== "anthropic")))
|
||
ctx.addIssue({ code: "custom", message: "Routing requires the matching provider and API authentication." });
|
||
if (v.routing && ["gateway", "local"].includes(v.routing.kind) && v.provider !== (v.routing.protocol === "messages" ? "anthropic" : "openai"))
|
||
ctx.addIssue({ code: "custom", message: "The provider must match the endpoint’s API format." });
|
||
if (v.routing?.auth === "none") {
|
||
if (v.apiKey || v.loginSessionId) ctx.addIssue({ code: "custom", message: "Provide only the selected authentication method." });
|
||
return;
|
||
}
|
||
if (
|
||
v.method === "api_key"
|
||
? !v.apiKey || Boolean(v.loginSessionId)
|
||
: !v.loginSessionId || Boolean(v.apiKey)
|
||
) {
|
||
ctx.addIssue({
|
||
code: "custom",
|
||
message:
|
||
"Provide exactly the credential for the selected sign-in method",
|
||
});
|
||
}
|
||
});
|
||
export type CreateAiConnection = z.infer<typeof createAiConnectionSchema>;
|
||
|
||
export const aiConnectionLoginIntentSchema = z
|
||
.object({
|
||
provider: z.enum(["anthropic", "openai", "xai"]),
|
||
method: z.literal("subscription"),
|
||
name: z.string().trim().min(1).max(160),
|
||
ownership: z.enum(["personal", "shared"]),
|
||
connectionId: z.string().uuid().optional(),
|
||
agentIds: z.array(z.string().uuid()).max(1000).default([]),
|
||
allAgents: z.boolean().default(false),
|
||
})
|
||
.strict();
|
||
export type AiConnectionLoginIntent = z.infer<
|
||
typeof aiConnectionLoginIntentSchema
|
||
>;
|
||
|
||
export const localAiConnectionSchema = aiConnectionLoginIntentSchema.extend({
|
||
localSessionId: z.string().uuid().optional(),
|
||
});
|
||
export const localAiLoginStartSchema = aiConnectionLoginIntentSchema.extend({ restart: z.boolean().optional() });
|
||
export interface LocalAiLoginStatus {
|
||
status: "ready" | "sign_in_required" | "expired";
|
||
authorizationUrl?: string;
|
||
code?: string;
|
||
error?: string;
|
||
}
|
||
export interface LocalAiLoginAttempt {
|
||
sessionId: string;
|
||
/** Grok still uses terminal sign-in until it has an in-app login runner. */
|
||
command?: string;
|
||
expiresAt: string;
|
||
}
|
||
|
||
/** Preview-era copies of rotating local credentials must be reconnected. */
|
||
export function aiSubscriptionNeedsIsolatedLogin(config: Record<string, unknown> | undefined): boolean {
|
||
const metadata = aiConnectionMetadataSchema.safeParse(config?.ai);
|
||
return metadata.success && metadata.data.method === "subscription" &&
|
||
(metadata.data.provider === "openai" || metadata.data.provider === "xai") &&
|
||
config?.aiIsolatedSubscription !== true;
|
||
}
|