Files
PaperClipAI/server/src/services/chat-provider-lifecycle.ts
T
DottaandPaperclip 3ec7309d91 feat(voice): add experimental Speko task connection runtime (#15727)
First PR: runtime. UI: #15728. Qualification: #15729.

## Thinking Path

> - Paperclip manages agent work through tasks and governed connections.
> - Chat connections already bind external conversations to those tasks.
> - Speko handles voice and telephony, while the selected Paperclip
agent does the work.
> - A call needs durable session authority, ordered follow-ups, and safe
result delivery.
> - This PR adds an explicit voice transport to the existing connection
runtime.
> - The UI and qualification suites follow in linked PRs. Speko stays
experimental.

## Linked Issues or Issue Description

**Problem or motivation**

Phone and browser voice conversations cannot use the chat connection
task lifecycle. A separate voice runtime is needed to preserve task
authority, retries, and reply history.

**Proposed solution**

Add company-scoped voice sessions, signed callbacks, durable replies,
inbound intake, and an agent-facing outbound-call tool. Reuse existing
tasks, queues, secrets, and publications.

**Roadmap alignment**

This extends the experimental chat connector entry points under Agent
Chat and Connected Apps. The project owner requested this feature.

Refs #11418 for the original provider catalog proposal. This
implementation uses the newer chat connection infrastructure rather than
an MCP catalog-only integration. Refs #15721 for remote OpenCode
startup.

## What Changed

- Add additive voice schema, shared contracts, session APIs, and
provider callbacks.
- Bind browser, inbound, and outbound sessions to an endpoint, task, and
selected agent.
- Add low-trust guest intake in an explicitly selected sandbox and keep
ordinary approval rules.
- Submit task work through session-scoped tools and push approved
replies through Speko POST /v1/calls/{id}/messages.
- Distinguish provider acceptance from confirmed spoken playback.
Reconcile reports without downloading recordings.
- Add protocol, migration, persistence, provider, and
delivery-diagnostic tests.

## Verification

- Master 3389bea708 is integrated. Database/shared builds, server
typecheck, migration upgrade/replay checks, and 46
communication-guidance/provider/protocol tests passed.
- The final persistence suite passed 85 real-database tests. Five
focused native continuation/authorization regressions passed. Five
callback ownership/rotation tests passed after the last runtime repair.
- Review repairs cover reconnect ownership, current caller authority,
duplicate delivery, multipart answers, task-page answers, guest question
handling, and saved sandbox recovery.
- A real OpenCode/OpenRouter DeepSeek task ran in Daytona and saved a
correct disk-space answer. This did not establish live-call playback.
- Full local typecheck/build passed before the final repairs; focused
typechecks passed after them. Latest-head CI runs the repository gates.
- The separate integrated browser qualification passed two journeys and
failed browser answer retrieval. The feature remains experimental. The
report in #15729 records this failure and outstanding real-provider
acceptance.
- No lockfiles are included. Speko migrations follow master as
0324–0330.

## Risks

- Voice sessions add migrations and provider-dependent lifecycle
behavior. Migrations preserve existing chat and AI transports.
- Callback origins must be reachable HTTPS. Provider keys and signing
secrets remain vaulted.
- Unverified callers receive task-scoped low-trust authority. Caller ID
does not grant private task access.
- Outbound telephony incurs provider charges. Provider acceptance is not
proof that the caller heard an answer.
- Keep the feature behind the existing experimental chat setting until
the real browser, inbound, outbound, and delayed-result acceptance
matrix passes.

## Model Used

OpenAI Codex, GPT-6, with tool use and code execution. The exact serving
model ID and context window are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-09 21:54:11 -05:00

515 lines
17 KiB
TypeScript

import type {
ChatProvider,
ChatResourceAvailability,
} from "@paperclipai/shared";
export type ChatProviderLifecycleEffect =
| {
kind: "resource";
provider: ChatProvider;
providerEventId: string;
providerResourceId: string;
/** Telegram basic-group id superseded by this supergroup id. */
previousProviderResourceId?: string;
parentProviderResourceId?: string;
resourceType: string;
label: string;
providerUrl?: string;
availability: ChatResourceAvailability;
providerOrder?: {
sequence?: string;
occurredAt?: string;
};
metadata?: Record<string, unknown>;
}
| {
kind: "endpoint";
provider: ChatProvider;
providerEventId: string;
availability: "available" | "attention" | "revoked";
reason: string;
providerOrder?: {
sequence?: string;
occurredAt?: string;
};
metadata?: Record<string, unknown>;
};
export interface ParseChatProviderLifecycleInput {
provider: ChatProvider;
headers?: Headers | Record<string, string | undefined>;
payload: unknown;
/** The provider-verified bot identity stored on the endpoint. */
botExternalId?: string | null;
}
function record(value: unknown): Record<string, unknown> | null {
return value && typeof value === "object" && !Array.isArray(value)
? (value as Record<string, unknown>)
: null;
}
function string(value: unknown): string | null {
return typeof value === "string" && value.trim() ? value.trim() : null;
}
function identifier(value: unknown): string | null {
if (typeof value === "number" && Number.isFinite(value)) return String(value);
return string(value);
}
function header(
headers: ParseChatProviderLifecycleInput["headers"],
name: string,
): string | null {
if (!headers) return null;
if (headers instanceof Headers) return headers.get(name);
const target = name.toLowerCase();
const entry = Object.entries(headers).find(
([key]) => key.toLowerCase() === target,
);
return entry ? string(entry[1]) : null;
}
function numericSequence(value: unknown): string | undefined {
if (typeof value === "number" && Number.isFinite(value)) return String(value);
const candidate = string(value);
return candidate && /^\d+(?:\.\d+)?$/.test(candidate) ? candidate : undefined;
}
function occurredAt(value: unknown): string | undefined {
const candidate = string(value);
if (!candidate) return undefined;
const timestamp = Date.parse(candidate);
return Number.isFinite(timestamp)
? new Date(timestamp).toISOString()
: undefined;
}
function parseSlackLifecycle(
input: ParseChatProviderLifecycleInput,
): ChatProviderLifecycleEffect[] {
const payload = record(input.payload);
if (!payload) return [];
const envelopeId =
string(payload.event_id) ?? string(payload.trigger_id) ?? "slack:lifecycle";
const event = record(payload.event) ?? payload;
const type = string(event.type);
if (!type) return [];
const sequence = numericSequence(event.event_ts ?? payload.event_time);
const providerOrder = sequence ? { sequence } : undefined;
if (type === "app_uninstalled" || type === "tokens_revoked") {
return [
{
kind: "endpoint",
provider: "slack",
providerEventId: envelopeId,
availability: "revoked",
providerOrder,
reason:
type === "app_uninstalled"
? "Slack app was uninstalled"
: "Slack app credentials were revoked",
},
];
}
if (type === "member_joined_channel" || type === "member_left_channel") {
const user = identifier(event.user);
const channel = identifier(event.channel);
if (!channel || !input.botExternalId || user !== input.botExternalId)
return [];
return [
{
kind: "resource",
provider: "slack",
providerEventId: envelopeId,
providerResourceId: channel,
resourceType: "channel",
label: channel,
availability:
type === "member_joined_channel" ? "available" : "unavailable",
providerOrder,
metadata: {
source: "membership_event",
...(string(event.channel_type)
? { channelType: string(event.channel_type) }
: {}),
},
},
];
}
// Slack emits these bot-self events when the installed app leaves a public
// or private channel. They are distinct from member_left_channel and do not
// include a user field because the authenticated bot is the member that
// left. Without them, `/remove @bot` can leave Paperclip's reach inventory
// incorrectly available until a later manual reconciliation.
if (type === "channel_left" || type === "group_left") {
const channel = identifier(event.channel);
if (!channel) return [];
return [
{
kind: "resource",
provider: "slack",
providerEventId: envelopeId,
providerResourceId: channel,
resourceType: "channel",
label: channel,
availability: "unavailable",
providerOrder,
metadata: { source: type },
},
];
}
if (
type === "channel_archive" ||
type === "group_archive" ||
type === "channel_unarchive" ||
type === "group_unarchive" ||
type === "channel_deleted" ||
type === "channel_rename" ||
type === "group_rename"
) {
const channelValue = record(event.channel);
const channelId =
identifier(channelValue?.id) ?? identifier(event.channel) ?? null;
if (!channelId) return [];
const availability: ChatResourceAvailability =
type === "channel_deleted"
? "removed"
: type === "channel_archive" || type === "group_archive"
? "unavailable"
: "available";
return [
{
kind: "resource",
provider: "slack",
providerEventId: envelopeId,
providerResourceId: channelId,
resourceType: "channel",
label: string(channelValue?.name) ?? channelId,
availability,
providerOrder,
metadata: { source: type },
},
];
}
return [];
}
function parseGitHubLifecycle(
input: ParseChatProviderLifecycleInput,
): ChatProviderLifecycleEffect[] {
const payload = record(input.payload);
if (!payload) return [];
const event = header(input.headers, "x-github-event");
const delivery =
header(input.headers, "x-github-delivery") ?? "github:lifecycle";
const action = string(payload.action);
if (event === "installation") {
const installation = record(payload.installation);
const installationId = identifier(installation?.id);
if (action === "deleted" || action === "suspend") {
return [
{
kind: "endpoint",
provider: "github",
providerEventId: delivery,
availability: action === "deleted" ? "revoked" : "attention",
reason:
action === "deleted"
? "GitHub App installation was removed"
: "GitHub App installation was suspended",
metadata: installationId ? { installationId } : undefined,
},
];
}
if (
action === "created" ||
action === "unsuspend" ||
action === "new_permissions_accepted"
) {
return [
{
kind: "endpoint",
provider: "github",
providerEventId: delivery,
availability: "available",
reason:
action === "unsuspend"
? "GitHub App installation was unsuspended"
: "GitHub App installation is available",
metadata: installationId ? { installationId } : undefined,
},
];
}
return [];
}
if (event !== "installation_repositories") return [];
// A repository-selection callback is one canonical reconciliation trigger,
// regardless of how many repositories GitHub includes in its delta. The
// service deliberately re-reads the complete installation inventory rather
// than trusting these callback-local additions/removals.
return [
{
kind: "endpoint",
provider: "github",
providerEventId: delivery,
availability: "available",
reason: "GitHub App repository access changed",
metadata: {
repositoriesAdded: Array.isArray(payload.repositories_added)
? payload.repositories_added.length
: 0,
repositoriesRemoved: Array.isArray(payload.repositories_removed)
? payload.repositories_removed.length
: 0,
},
},
];
}
function teamsResourceType(payload: Record<string, unknown>): string {
const conversation = record(payload.conversation);
const conversationType = string(conversation?.conversationType)
?.toLowerCase()
.replace(/[^a-z]/g, "");
if (conversationType === "personal") return "direct_message";
if (conversationType === "group" || conversationType === "groupchat")
return "group_chat";
if (conversationType === "channel") return "channel";
if (conversation?.isGroup === false) return "direct_message";
const channelData = record(payload.channelData);
if (record(channelData?.team) || record(channelData?.channel))
return "channel";
return "group_chat";
}
function teamsBotIdentityMatches(
botExternalId: string,
memberId: unknown,
): boolean {
const canonical = (value: unknown): string | null => {
const id = identifier(value)?.toLowerCase();
return id?.replace(/^28:/, "") ?? null;
};
const expected = canonical(botExternalId);
return expected !== null && canonical(memberId) === expected;
}
function teamsResourceId(payload: Record<string, unknown>): string | null {
const conversation = record(payload.conversation);
const channelData = record(payload.channelData);
const channel = record(channelData?.channel);
// Teams root replies append `;messageid=...` to the Bot Framework
// conversation id. Use its stable base as the access-control resource so
// installation events and later message ingress address the same row.
const conversationId = identifier(conversation?.id);
return (
conversationId?.replace(/;messageid=[^;]+/i, "") ?? identifier(channel?.id)
);
}
function teamsResourceLabel(
payload: Record<string, unknown>,
fallback: string,
): string {
const conversation = record(payload.conversation);
const channelData = record(payload.channelData);
const channel = record(channelData?.channel);
const team = record(channelData?.team);
return (
string(channel?.name) ??
string(conversation?.name) ??
string(team?.name) ??
fallback
);
}
function parseTeamsLifecycle(
input: ParseChatProviderLifecycleInput,
): ChatProviderLifecycleEffect[] {
const payload = record(input.payload);
if (!payload) return [];
const type = string(payload.type);
const eventId = identifier(payload.id) ?? "teams:lifecycle";
const eventOccurredAt = occurredAt(payload.timestamp);
const providerOrder = eventOccurredAt
? { occurredAt: eventOccurredAt }
: undefined;
const resourceId = teamsResourceId(payload);
if (!resourceId) return [];
if (type === "installationUpdate") {
const action = string(payload.action)?.toLowerCase();
if (
!["add", "add-upgrade", "remove", "remove-upgrade"].includes(action ?? "")
)
return [];
const channelData = record(payload.channelData);
const team = record(channelData?.team);
return [
{
kind: "resource",
provider: "microsoft-teams",
providerEventId: eventId,
providerResourceId: resourceId,
parentProviderResourceId: identifier(team?.id) ?? undefined,
resourceType: teamsResourceType(payload),
label: teamsResourceLabel(payload, resourceId),
availability: action?.startsWith("remove") ? "removed" : "available",
providerOrder,
metadata: { source: "installation_update", action },
},
];
}
if (type === "conversationUpdate" && input.botExternalId) {
const added = Array.isArray(payload.membersAdded)
? payload.membersAdded.map(record).filter(Boolean)
: [];
const removed = Array.isArray(payload.membersRemoved)
? payload.membersRemoved.map(record).filter(Boolean)
: [];
const joined = added.some((member) =>
teamsBotIdentityMatches(input.botExternalId!, member?.id),
);
const left = removed.some((member) =>
teamsBotIdentityMatches(input.botExternalId!, member?.id),
);
if (!joined && !left) return [];
return [
{
kind: "resource",
provider: "microsoft-teams",
providerEventId: eventId,
providerResourceId: resourceId,
resourceType: teamsResourceType(payload),
label: teamsResourceLabel(payload, resourceId),
availability: left ? "unavailable" : "available",
providerOrder,
metadata: { source: "conversation_membership" },
},
];
}
return [];
}
function parseTelegramLifecycle(
input: ParseChatProviderLifecycleInput,
): ChatProviderLifecycleEffect[] {
const payload = record(input.payload);
const migrationMessage =
record(payload?.message) ?? record(payload?.channel_post);
const migrationChat = record(migrationMessage?.chat);
const migrationChatId = identifier(migrationChat?.id);
const migrateToChatId = identifier(migrationMessage?.migrate_to_chat_id);
const migrateFromChatId = identifier(migrationMessage?.migrate_from_chat_id);
const migrationFromId = migrateToChatId ? migrationChatId : migrateFromChatId;
const migrationToId =
migrateToChatId ?? (migrateFromChatId ? migrationChatId : null);
if (
payload &&
migrationMessage &&
migrationChat &&
migrationFromId &&
migrationToId
) {
const title =
string(migrationChat.title) ??
[string(migrationChat.first_name), string(migrationChat.last_name)]
.filter(Boolean)
.join(" ") ??
string(migrationChat.username) ??
migrationToId;
const updateId = numericSequence(payload.update_id);
return [
{
kind: "resource",
provider: "telegram",
providerEventId: `telegram:${identifier(payload.update_id) ?? "migration"}`,
providerResourceId: migrationToId,
previousProviderResourceId: migrationFromId,
resourceType: "chat",
label: title || migrationToId,
availability: "available",
providerOrder: updateId ? { sequence: updateId } : undefined,
metadata: {
source: "chat_migration",
migratedFrom: migrationFromId,
migratedTo: migrationToId,
},
},
];
}
const membership = record(payload?.my_chat_member);
const chat = record(membership?.chat);
const member = record(membership?.new_chat_member);
const chatId = identifier(chat?.id);
const status = string(member?.status);
const updateId = numericSequence(payload?.update_id);
if (!payload || !membership || !chat || !member || !chatId || !status)
return [];
const available =
status === "member" ||
status === "administrator" ||
(status === "restricted" && member.is_member === true);
const unavailable =
status === "left" || status === "kicked" || status === "restricted";
if (!available && !unavailable) return [];
const title =
string(chat.title) ??
[string(chat.first_name), string(chat.last_name)]
.filter(Boolean)
.join(" ") ??
string(chat.username) ??
chatId;
return [
{
kind: "resource",
provider: "telegram",
providerEventId: `telegram:${identifier(payload.update_id) ?? "lifecycle"}`,
providerResourceId: chatId,
resourceType: string(chat.type) === "private" ? "direct_message" : "chat",
label: title || chatId,
availability: available ? "available" : "unavailable",
providerOrder: updateId ? { sequence: updateId } : undefined,
metadata: { source: "my_chat_member", memberStatus: status },
},
];
}
/**
* Parse provider installation and membership events only after the native
* adapter has verified the webhook. The returned effects contain no provider
* credentials and are safe to persist in Paperclip's lifecycle ledger.
*/
export function parseChatProviderLifecycle(
input: ParseChatProviderLifecycleInput,
): ChatProviderLifecycleEffect[] {
switch (input.provider) {
case "speko":
return []; // Voice lifecycle is reconciled against persisted sessions.
case "imessage-photon": return []; // Authenticated gRPC events own lifecycle.
case "agentmail": return [];
case "slack":
return parseSlackLifecycle(input);
case "github":
return parseGitHubLifecycle(input);
case "discord":
// Discord server/channel reach is reconciled from the Bot REST API on
// connect, resume, and reconnect. Gateway membership events are not yet
// treated as an authorization source.
return [];
case "microsoft-teams":
return parseTeamsLifecycle(input);
case "telegram":
return parseTelegramLifecycle(input);
}
}