Files
PaperClipAI/server/src/services/voice/speko-protocol.ts
T
DottaandPaperclip 3ec7309d91 feat(voice): add experimental Speko task connection runtime (#15727)
First PR: runtime. UI: #15728. Qualification: #15729.

## Thinking Path

> - Paperclip manages agent work through tasks and governed connections.
> - Chat connections already bind external conversations to those tasks.
> - Speko handles voice and telephony, while the selected Paperclip
agent does the work.
> - A call needs durable session authority, ordered follow-ups, and safe
result delivery.
> - This PR adds an explicit voice transport to the existing connection
runtime.
> - The UI and qualification suites follow in linked PRs. Speko stays
experimental.

## Linked Issues or Issue Description

**Problem or motivation**

Phone and browser voice conversations cannot use the chat connection
task lifecycle. A separate voice runtime is needed to preserve task
authority, retries, and reply history.

**Proposed solution**

Add company-scoped voice sessions, signed callbacks, durable replies,
inbound intake, and an agent-facing outbound-call tool. Reuse existing
tasks, queues, secrets, and publications.

**Roadmap alignment**

This extends the experimental chat connector entry points under Agent
Chat and Connected Apps. The project owner requested this feature.

Refs #11418 for the original provider catalog proposal. This
implementation uses the newer chat connection infrastructure rather than
an MCP catalog-only integration. Refs #15721 for remote OpenCode
startup.

## What Changed

- Add additive voice schema, shared contracts, session APIs, and
provider callbacks.
- Bind browser, inbound, and outbound sessions to an endpoint, task, and
selected agent.
- Add low-trust guest intake in an explicitly selected sandbox and keep
ordinary approval rules.
- Submit task work through session-scoped tools and push approved
replies through Speko POST /v1/calls/{id}/messages.
- Distinguish provider acceptance from confirmed spoken playback.
Reconcile reports without downloading recordings.
- Add protocol, migration, persistence, provider, and
delivery-diagnostic tests.

## Verification

- Master 3389bea708 is integrated. Database/shared builds, server
typecheck, migration upgrade/replay checks, and 46
communication-guidance/provider/protocol tests passed.
- The final persistence suite passed 85 real-database tests. Five
focused native continuation/authorization regressions passed. Five
callback ownership/rotation tests passed after the last runtime repair.
- Review repairs cover reconnect ownership, current caller authority,
duplicate delivery, multipart answers, task-page answers, guest question
handling, and saved sandbox recovery.
- A real OpenCode/OpenRouter DeepSeek task ran in Daytona and saved a
correct disk-space answer. This did not establish live-call playback.
- Full local typecheck/build passed before the final repairs; focused
typechecks passed after them. Latest-head CI runs the repository gates.
- The separate integrated browser qualification passed two journeys and
failed browser answer retrieval. The feature remains experimental. The
report in #15729 records this failure and outstanding real-provider
acceptance.
- No lockfiles are included. Speko migrations follow master as
0324–0330.

## Risks

- Voice sessions add migrations and provider-dependent lifecycle
behavior. Migrations preserve existing chat and AI transports.
- Callback origins must be reachable HTTPS. Provider keys and signing
secrets remain vaulted.
- Unverified callers receive task-scoped low-trust authority. Caller ID
does not grant private task access.
- Outbound telephony incurs provider charges. Provider acceptance is not
proof that the caller heard an answer.
- Keep the feature behind the existing experimental chat setting until
the real browser, inbound, outbound, and delayed-result acceptance
matrix passes.

## Model Used

OpenAI Codex, GPT-6, with tool use and code execution. The exact serving
model ID and context window are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-09 21:54:11 -05:00

67 lines
6.4 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
import { createHash, createHmac, timingSafeEqual } from "node:crypto";
import { z } from "zod";
import { respondIssueThreadInteractionSchema } from "@paperclipai/shared";
export const SPEKO_CALLBACK_LIMIT = 64 * 1024;
export const SPEKO_CALLBACK_TOLERANCE_SECONDS = 300;
const id = z.string().regex(/^[A-Za-z0-9_:-]{1,200}$/);
const common = { session_id: id, tool_call_id: id, idempotency_key: z.string().max(401) };
export const spekoToolEnvelopeSchema = z.discriminatedUnion("tool", [
z.object({ ...common, tool: z.literal("submit_request"), args: z.object({ text: z.string().trim().min(1).max(16_000) }).strict() }),
z.object({ ...common, tool: z.literal("answer_question"), args: respondIssueThreadInteractionSchema.pick({ answers: true }).extend({ interactionId: z.string().uuid() }).strict() }),
z.object({ ...common, tool: z.literal("get_updates"), args: z.object({ cursor: z.number().int().min(0).max(Number.MAX_SAFE_INTEGER), repeat: z.boolean().optional() }).strict() }),
]).refine((v) => v.idempotency_key === `${v.session_id}:${v.tool_call_id}`, "Invalid tool identity");
export type SpekoToolEnvelope = z.infer<typeof spekoToolEnvelopeSchema>;
export interface SpekoSigningKey {
secret: string;
/** Old keys must have an explicit grace deadline. Do not retain them indefinitely. */
validUntil?: Date;
}
export class SpekoProtocolError extends Error {
constructor(readonly code: "body_too_large" | "invalid_signature" | "invalid_envelope") { super(code); }
}
/** Standard Webhooks verification over the ORIGINAL body; no parsed-body fallback. */
export function verifySpekoSignature(input: {
body: Buffer;
headers: Record<string, string | string[] | undefined>;
keys: readonly SpekoSigningKey[];
now?: number;
}): { webhookId: string } {
if (input.body.length > SPEKO_CALLBACK_LIMIT) throw new SpekoProtocolError("body_too_large");
const webhookId = input.headers["webhook-id"], timestamp = input.headers["webhook-timestamp"], signatures = input.headers["webhook-signature"];
const now = input.now ?? Date.now();
if (typeof webhookId !== "string" || !/^[A-Za-z0-9_:-]{1,401}$/.test(webhookId)
|| typeof timestamp !== "string" || !/^\d{1,12}$/.test(timestamp)
|| Math.abs(now / 1000 - Number(timestamp)) > SPEKO_CALLBACK_TOLERANCE_SECONDS
|| typeof signatures !== "string" || signatures.length > 4096) throw new SpekoProtocolError("invalid_signature");
const candidates = signatures.split(" ").filter((v) => /^v1,[A-Za-z0-9+/]{43}=$/.test(v)).map((v) => Buffer.from(v.slice(3), "base64"));
let valid = false;
for (const key of input.keys) {
if (key.validUntil && key.validUntil.getTime() <= now || !/^whsec_[A-Za-z0-9+/]{43}=$/.test(key.secret)) continue;
const expected = createHmac("sha256", Buffer.from(key.secret.slice(6), "base64")).update(`${webhookId}.${timestamp}.`).update(input.body).digest();
for (const candidate of candidates) valid = timingSafeEqual(expected, candidate) || valid;
}
if (!valid) throw new SpekoProtocolError("invalid_signature");
return { webhookId };
}
export function verifySpekoToolRequest(input: Parameters<typeof verifySpekoSignature>[0]): { envelope: SpekoToolEnvelope; webhookId: string; fingerprint: string } {
const { webhookId } = verifySpekoSignature(input);
let envelope: SpekoToolEnvelope;
try { envelope = spekoToolEnvelopeSchema.parse(JSON.parse(input.body.toString("utf8"))); }
catch { throw new SpekoProtocolError("invalid_envelope"); }
const fingerprint = createHash("sha256").update(JSON.stringify(envelope)).digest("hex");
return { envelope, webhookId, fingerprint };
}
export function spekoToolDefinitions(callbackUrl: string) {
const url = new URL(callbackUrl);
if (url.protocol !== "https:" || url.username || url.password || url.hash || url.search) throw new Error("Speko requires an explicit public HTTPS callback URL");
const source = { kind: "webhook" as const, url: url.href, method: "POST" as const, responseMode: "sync" as const, timeoutMs: 3500, headers: { authorization: "Bearer {{paperclip_session_token}}" } };
return [
{ name: "submit_request", description: "Delegate work, factual questions, corrections, and follow-up instructions to the assigned Paperclip agent on this task. Include the caller’s words and any clarified details. Submit once per new request, including while prior work runs. The agent has its own tools and execution environment: delegate questions about its computer, files, disk space, or capabilities. You may handle greetings, quick acknowledgments and clarification immediately without this tool. Acceptance means saved, not completed. Approved phone answers are pushed into the live call; get_updates can inspect status or repeat a delivered answer.", parameters: { type: "object", properties: { text: { type: "string", minLength: 1, maxLength: 16000 } }, required: ["text"], additionalProperties: false }, source },
{ name: "answer_question", description: "Answer the exact pending question returned by get_updates, using its question and option IDs. Ask for clarification if the answer is ambiguous. Cannot approve actions.", parameters: { type: "object", properties: { interactionId: { type: "string", format: "uuid" }, answers: { type: "array", minItems: 1, maxItems: 20, items: { type: "object", properties: { questionId: { type: "string" }, optionIds: { type: "array", items: { type: "string" } }, otherText: { type: "string", maxLength: 16000 } }, required: ["questionId", "optionIds"], additionalProperties: false } } }, required: ["interactionId", "answers"], additionalProperties: false }, source },
{ name: "get_updates", description: "Retrieve the assigned Paperclip agent’s approved answers for this live call. Use cursor=0 initially and then the returned cursor. Approved telephone answers are pushed into the live call automatically. Use this tool for a caller-requested status check, permitted task questions, or an explicit repeat; do not continuously poll for completion. Empty updates means pending, not finished. Read each returned answer once. Set repeat=true only when the caller explicitly asks to hear the last delivered answer again. The work state distinguishes queueing, execution, and awaiting publication; it is not an answer or proof of completion.", parameters: { type: "object", properties: { cursor: { type: "integer", minimum: 0 }, repeat: { type: "boolean", description: "Explicit caller request to repeat the last delivered answer" } }, required: ["cursor"], additionalProperties: false }, source },
];
}