mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-21 16:27:18 +02:00
Third PR: depends on #15728 and #15727. The user has reviewed and authorized merging this experimental feature stack. ## Thinking Path > - Paperclip manages agent work through tasks and governed connections. > - Chat connections already bind external conversations to those tasks. > - Speko handles voice and telephony, while the selected Paperclip agent does the work. > - A call needs durable session authority, ordered follow-ups, and safe result delivery. > - This PR adds repeatable qualification for the Speko voice task integration. > - This PR depends on #15728, which depends on #15727. Speko stays experimental. ## Linked Issues or Issue Description **Problem or motivation** Phone and browser voice conversations cannot use the chat connection task lifecycle. A separate voice runtime is needed to preserve task authority, retries, and reply history. **Proposed solution** Add company-scoped voice sessions, signed callbacks, durable replies, inbound intake, and an agent-facing outbound-call tool. Reuse existing tasks, queues, secrets, and publications. **Roadmap alignment** This extends the experimental chat connector entry points under Agent Chat and Connected Apps. The project owner requested this feature. Refs #11418 for the original provider catalog proposal. This implementation uses the newer chat connection infrastructure rather than an MCP catalog-only integration. Refs #15721 for remote OpenCode startup. ## What Changed - Add a real Paperclip server/database E2E fixture that stubs only external Speko transport and the execution provider. - Add authenticated multi-user and cross-company qualification journeys. - Add Storybook interaction, accessibility, responsive, and visual-suite integration. - Add an E01–E16 runbook, component-to-story inventory, and current redacted qualification report. - Keep historical private call traces out of the PR. Provider proof tools are opt-in. ## Verification - All 21 standalone proof tests and browser client/server syntax checks pass. They cover signatures, redaction, cleanup retry, final audio saves, bounded report transport and late server write ordering. No live calls were placed. - All 27 focused UI/controller regressions, UI typecheck, token gates, and 28 map/catalog structural tests pass. The shared connection cache test covers pause/resume invalidation during the 30-second fresh period. - Storybook builds 157 Speko production-component stories. The complete prior 156-story set passed all 624 checks on UI source249cbc616a. After the final cache repair, all 152 affected page/launcher checks passed across both themes, 390/1200 widths and reduced motion. Real microphone access and external provider requests are blocked. - Hosted Storybook origin isolation and representative narrow-screen stories pass. Provider requests remain blocked. - Source annotations follow the current generated-catalog contract. No generated catalog or lockfile is included. Three inherited GitHub-heading tests passed after master reconciliation. - Commit905bc5da5dhas 55 passing checks, including required verify/E2E gates, and a fresh Greptile 5/5 review with zero unresolved findings. Local fixture success does not establish provider or spoken-playback acceptance. - The separate integrated browser run passed two journeys and failed browser answer retrieval. Real browser/telephone playback and timing acceptance remain outstanding. See the dated qualification report and E01–E16 runbook. Speko stays experimental. ## Risks - Voice sessions add migrations and provider-dependent lifecycle behavior. Migrations preserve existing chat and AI transports. - Callback origins must be reachable HTTPS. Provider keys and signing secrets remain vaulted. - Unverified callers receive task-scoped low-trust authority. Caller ID does not grant private task access. - Outbound telephony incurs provider charges. Provider acceptance is not proof that the caller heard an answer. - Keep the feature behind the existing experimental chat setting until the real browser, inbound, outbound, and delayed-result acceptance matrix passes. ## Model Used OpenAI Codex, GPT-6, with tool use and code execution. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
98 lines
7.5 KiB
JavaScript
98 lines
7.5 KiB
JavaScript
/** One-session loopback browser qualification harness. No Paperclip authority. */
|
|
import { createServer } from 'node:http';
|
|
import { once } from 'node:events';
|
|
import { readFile, writeFile, mkdir } from 'node:fs/promises';
|
|
import { randomBytes, createHash } from 'node:crypto';
|
|
import { homedir } from 'node:os';
|
|
import { resolve, dirname } from 'node:path';
|
|
import { build } from 'esbuild';
|
|
import { createReportSaver } from './cleanup.mjs';
|
|
import { createProof, createProofServer, DELAY_MS } from './proof.mjs';
|
|
const key = process.env.SPEKO_MCP_API_KEY;
|
|
if (!key) throw new Error('Credential required in environment');
|
|
const state = JSON.parse(await readFile(`${homedir()}/.paperclip/speko-proof/state.json`, 'utf8'));
|
|
const out = resolve(process.env.SPEKO_PROOF_OUTPUT ?? `${homedir()}/.paperclip/speko-proof/browser-${Date.now()}`);
|
|
await mkdir(dirname(out), { recursive: true, mode: 0o700 }); await mkdir(out, { mode: 0o700 });
|
|
const sourceDigests = {};
|
|
for (const file of ['browser-server.mjs', 'browser-client.mjs', 'cleanup.mjs', 'browser.html', 'proof.mjs', 'application-notification.txt']) sourceDigests[file] = createHash('sha256').update(await readFile(new URL(file, import.meta.url))).digest('hex');
|
|
await writeFile(resolve(out, 'source.json'), JSON.stringify({ startedAt: new Date().toISOString(), sourceDigests }), { mode: 0o600 });
|
|
const bundle = await build({ entryPoints: [new URL('./browser-client.mjs', import.meta.url).pathname], bundle: true, write: false, platform: 'browser', format: 'esm' });
|
|
// Disk writes stay serialized even when a browser request times out.
|
|
const saveReport = createReportSaver(async body => {
|
|
await writeFile(resolve(out, 'browser-report.json'), JSON.stringify(body, null, 2), {mode: 0o600});
|
|
await writeFile(resolve(out, 'proof.json'), JSON.stringify(proof.evidence(), null, 2), {mode: 0o600});
|
|
});
|
|
const inputs = new Map();
|
|
for (const name of ['start', 'followup']) {
|
|
const data = await readFile(new URL(`../../.paperclip-local/speko-proof-audio/${name}.pcm`, import.meta.url));
|
|
if (!data.length || data.length % 2 || data.length > 48000 * 2 * 10) throw new Error('Invalid synthetic fixture');
|
|
inputs.set(name, data);
|
|
}
|
|
const port = Number(process.env.SPEKO_BROWSER_PORT ?? 3199);
|
|
const origin = `http://127.0.0.1:${port}`;
|
|
const launch = randomBytes(24).toString('hex'), csrf = randomBytes(24).toString('hex'), cookie = randomBytes(24).toString('hex');
|
|
const proof = createProof({ signingSecret: state.signingSecret, onEvent: (event) => { if (event.kind === 'request_accepted') startTime = Date.now(); } });
|
|
const hook = createProofServer(proof);
|
|
let sessionId, attempted = false, startTime, ending, stopped = false;
|
|
async function provider(path, body) {
|
|
const r = await fetch(`https://api.speko.dev${path}`, { method: 'POST', headers: { Authorization: `Bearer ${key}`, 'content-type': 'application/json' }, body: JSON.stringify(body), signal: AbortSignal.timeout(20000) });
|
|
if (!r.ok) throw new Error(`Provider returned ${r.status}`);
|
|
return r.json();
|
|
}
|
|
async function endSession() {
|
|
if (!sessionId) return;
|
|
ending ??= provider(`/v1/calls/${sessionId}/end`, {});
|
|
try { await ending; } catch (error) { ending = undefined; throw error; }
|
|
}
|
|
const server = createServer(async (req, res) => {
|
|
const reply = (code, data, type = 'application/json') => { res.writeHead(code, { 'content-type': type, 'cache-control': 'no-store', 'x-content-type-options': 'nosniff' }); res.end(type === 'application/json' ? JSON.stringify(data) : data); };
|
|
try {
|
|
if (req.headers.host !== `127.0.0.1:${port}`) return reply(403, { error: 'Wrong host' });
|
|
if (req.method === 'GET' && req.url === `/launch/${launch}`) {
|
|
res.setHeader('set-cookie', `proof=${cookie}; HttpOnly; SameSite=Strict; Path=/`);
|
|
res.writeHead(302, { location: '/', 'referrer-policy': 'no-referrer' }); return res.end();
|
|
}
|
|
if (!req.headers.cookie?.split(';').some((part) => part.trim() === `proof=${cookie}`)) return reply(401, { error: 'Launch this isolated proof from its private link' });
|
|
if (req.method === 'GET' && req.url === '/') return reply(200, (await readFile(new URL('./browser.html', import.meta.url), 'utf8')).replace('CONFIG_JSON', JSON.stringify({ csrf })), 'text/html');
|
|
if (req.method === 'GET' && req.url === '/client.js') return reply(200, bundle.outputFiles[0].contents, 'text/javascript');
|
|
if (req.method === 'GET' && ['/input/start', '/input/followup'].includes(req.url)) return reply(200, inputs.get(req.url.split('/').at(-1)), 'application/octet-stream');
|
|
if (req.method !== 'POST' || req.headers.origin !== origin || req.headers['x-proof-csrf'] !== csrf) return reply(403, { error: 'Same-origin request required' });
|
|
const limit = req.url === '/audio' ? 20 * 1024 * 1024 : 256 * 1024;
|
|
const chunks = []; let size = 0;
|
|
for await (const chunk of req) { size += chunk.length; if (size > limit) return reply(413, { error: 'Request too large' }); chunks.push(chunk); }
|
|
const raw = Buffer.concat(chunks);
|
|
if (req.url === '/audio') { await writeFile(resolve(out, 'received.webm'), raw, { mode: 0o600 }); return reply(200, { saved: true }); }
|
|
const body = JSON.parse(raw.toString('utf8'));
|
|
if (req.url === '/session') {
|
|
if (attempted) return reply(409, { error: 'One session per probe. Creation is never blindly retried.' });
|
|
attempted = true;
|
|
await writeFile(resolve(out, 'intent.json'), JSON.stringify({ action: 'create_browser_session', at: new Date().toISOString() }), { mode: 0o600 });
|
|
const session = await provider('/v1/sessions', { mode: 'cascade', agentId: state.agentId, ttlSeconds: 120, maxDurationSeconds: 180, constraints: { allowedProviders: { llm: ['openai:gpt-4.1'] } }, systemPrompt: await readFile(new URL('./application-notification.txt', import.meta.url), 'utf8') });
|
|
sessionId = session.sessionId;
|
|
await writeFile(resolve(out, 'session.json'), JSON.stringify({ sessionId }), { mode: 0o600 });
|
|
return reply(200, { sessionId, transportToken: session.transportToken, transportUrl: session.transportUrl });
|
|
}
|
|
if (req.url === '/evidence') {
|
|
const evidence = proof.evidence();
|
|
// A server-owned clock, bound to the first signed submission, schedules
|
|
// the synthetic completion hint; the page cannot advance it.
|
|
return reply(200, { acceptedRequests: evidence.acceptedRequests, resultReturned: evidence.resultReturned, notificationReady: startTime !== undefined && Date.now() >= startTime + DELAY_MS + 1000,
|
|
expectedNumber: evidence.expectedSyntheticResult?.match(/Verification number (\d+)/)?.[1] ?? null });
|
|
}
|
|
if (req.url === '/report') { await saveReport(body); return reply(200, { saved: true }); }
|
|
if (req.url === '/end') { await endSession(); return reply(200, { ended: true }); }
|
|
return reply(404, { error: 'Not found' });
|
|
} catch { return reply(502, { error: 'Probe request failed; inspect private evidence. Do not retry session creation.' }); }
|
|
});
|
|
try {
|
|
hook.listen(3198, '127.0.0.1'); await once(hook, 'listening');
|
|
server.listen(port, '127.0.0.1'); await once(server, 'listening');
|
|
} catch (error) { hook.close(); server.close(); throw error; }
|
|
console.log(JSON.stringify({ url: `${origin}/launch/${launch}`, output: out }));
|
|
async function stop() {
|
|
if (stopped) return; stopped = true;
|
|
try { await endSession(); } finally { proof.close(); hook.close(); hook.closeAllConnections(); server.close(); server.closeAllConnections(); clearTimeout(cap); }
|
|
}
|
|
const cap = setTimeout(() => { void stop(); }, 270000);
|
|
process.once('SIGINT', () => { void stop(); }); process.once('SIGTERM', () => { void stop(); });
|