11 Commits

Include renames

Author	SHA1	Message	Date
DottaandPaperclip	7afee8f062	fix(server): schedule direct GitHub webhook recovery from saved deadlines (#17666) ... <!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/218/changes/paperclip/218 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - GitHub callbacks can fail before the app records an inbox row. > - Direct GitHub integrations need bounded recovery from provider delivery history. > - The chat timer checks for due recovery every second, even without an eligible integration. > - Existing recovery checkpoints already store the next scan time and runtime epoch. > - This pull request schedules those saved obligations and wakes on committed configuration changes. > - It removes the final chat maintenance interval while preserving authenticated recovery. ## Linked Issues or Issue Description **What happened?** The app queries due GitHub recovery endpoints every second. This still runs when no direct GitHub integration needs recovery. Other chat maintenance duties already have independent workers in the preceding stack. **What did you expect?** Reconstruct existing recovery deadlines at startup and after committed changes. Run only when an eligible endpoint reaches its saved deadline. Make no recurring SQL or HTTP calls for unconfigured, disabled, or Cloud-relayed integrations. **Steps to reproduce** Start an instance without an active direct GitHub integration. Leave it idle. The chat maintenance interval continues to call processFailedGitHubWebhookDeliveries each second. Follows merged #17664. Related #15981 owns the original GitHub webhook ingress/recovery separation. #17662 preserves review-check recovery, which is a separate obligation. No duplicate scheduling migration was found in the related-PR search. ## What Changed - Put deadline reconstruction, scopes, and registration in the GitHub provider owner. Read existing recovery checkpoints in pages of 100. Reload current eligibility before each dispatch. - Dispatch batches of five through the existing shared helper. Preserve company fairness and the default limit of 25 for other consumers. - Notify inside configuration, credential, checkpoint, installation-lifecycle, and Cloud-registration transactions. A signed installation deletion reproduced a missing wake before the fix. - Detect credential-only epoch changes during a future backoff. Preserve the existing floor reset without adopting old events into new credentials. - Gate every new recovery HTTP call on dispatch admission and shutdown. Join in-flight work and retain its existing outcome/uncertainty settlement. Other queues remain independent. - Remove the final chat reconciliation interval and its unused coordinator. Update the old coordinator fixtures; preserve the voice retention worker acceptance. - Restore known-unsent redelivery claims after drain/shutdown, guarded by exact attempt identity. Preserve uncertain outcomes after transport starts and never overwrite a newer receipt. - Keep all existing redelivery, source, authority, lease, receipt, and callback checks. Add no table, migration, or external wake contract. ## Verification Current source: `ab00fc5b30d51d6c85891536599e7bdfd0c822fe`. - **751 distinct cases pass** at `3d7f163bc5259b60f9bd19b7fc90b402f0a6e925`: 274 complete GitHub review cases, 294 other GitHub-named cases, 30 publication cases, 145 voice/retention cases, and eight startup cases. - All 19 new recovery worker cases and all six stop-before-transport variants pass. Coverage includes configured eligibility, callback-only admission, committed configuration changes, outer transaction settlement, restart/drain/standby, saved deadlines, fair batches, independent queues, held HTTP, joined shutdown, known-unsent restoration, and newer-receipt fencing. - The first workflow filter omitted the separate recovery group. The saved case list exposed the gap. The second selection ran the remaining GitHub cases in a new database. Passed-name sets are disjoint. The two commands filtered 1,504 and 1,484 unrelated cases respectively. No full chat or repository-suite claim is made. - Database build, server typecheck/build, module boundaries, and migration order pass. Five fresh PostgreSQL fixtures receive the unchanged parent migration history through 0403. This slice adds no database migration or table. - The parent browser-fixture repair is carried with both complete patches preserved. At predecessor `836acef0370d42323fdfdbd6d62a805b2469f1d4`, entire server, package and UI source trees and owned hashes match the 751-case acceptance. The current follow-up adds three test cases and audit evidence; production source remains unchanged. The parent has merged. - [The owning audit](https://github.com/paperclipai/paperclip/blob/ab00fc5b30d51d6c85891536599e7bdfd0c822fe/doc/plans/2026-10-25-github-webhook-recovery-sleep-audit.md) preserves earlier failures and exact acceptance scope. Fresh exact-head Greptile and all CI gates remain required. ## Risks - Direct GitHub endpoints still require the existing one-minute external history cadence, with five-minute failure/claim backoff and provider backoff up to one day. Cloud relay endpoints are excluded. This does not promise network-idle direct integrations. - The existing history, envelope, attempt, and lease limits remain. Provider history never substitutes for a genuine authenticated callback. - Supported producers share the current database owner's process-local signals. Independent SQL writers and other server processes need their own integration. Unknown transaction outcomes remain held by the existing reconciliation protocol. - Already-started HTTP keeps its existing timeout and is joined during shutdown. No later request starts after drain or stop; durable attempt uncertainty is preserved. - Host-sleep eligibility and Cloud alarm coverage stay unchanged. Local worker tests do not prove host/database suspension or fleet savings. - Each remaining sleep slice needs its own current review and CI checks before merge. ## Model Used OpenAI Codex (GPT-6), with reasoning and code execution. The exact serving model ID and context-window size were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Parent startup-test correction: #17664 fixes its legacy coordinator boundary marker. This slice removes that coordinator and timer entirely, so its existing absence assertions supersede that obsolete inspection. All other parent and upstream changes are preserved. At predecessor `836acef0370d42323fdfdbd6d62a805b2469f1d4`, complete server, package and UI trees match the 751-case acceptance; all eight startup cases also pass after the explicit conflict resolution. Fresh review and CI are required for this new head. ## Required review finding: orphan assignment investigation The reported orphan endpoint cannot be committed through the existing schema: assigned_agent_id is NOT NULL, its agent FK is ON DELETE RESTRICT, and a validated nondeferrable composite FK enforces the same company. A first reproduction failed at fixture mutation with PostgreSQL 23502 before reaching the worker; that failure remains recorded. Three new real-PostgreSQL cases verify NULL/missing/deleted assignments are rejected, then prove the preserved valid endpoint recovers once, saves its next deadline, and makes no SQL reads during a 15-second pre-deadline interval. All 22 recovery-scheduling cases pass (19 previous plus three new); unselected cases are not counted. The preceding style-only commit preserved production behavior and non-comment tokens, and the prior 751-case evidence retains its original scope (754 distinct cases across both selections). A new review must assess this evidence before merge; the old conditional verdict is not waived. Actual parent squash ancestry is carried with whole-tree identity before this test-only follow-up. Repository style follow-up removes seven new code-comment lines from the recovery ingress owner and constraint tests. Their explanations remain in the audit. Both files retain identical non-comment TypeScript token streams, preserving all guards, assertions and deadlines. Production eligibility SQL is unchanged. Fresh current-head CI and reviews remain required. ## CI test corrections The original strict deletion assertion reproduced CI's PostgreSQL-18 failure locally: the forbidden delete reports 23001 (restrict violation), versus 23503 on PostgreSQL 17. The corrected assertion accepts only these integrity codes for deletion, retains exact codes for NULL/missing assignments, and verifies the original agent and assignment remain. All 22 recovery cases pass on embedded PostgreSQL 18; the same three integrity probes pass on PostgreSQL 17. A delayed real announcement response reproduced the attachment E2E's filechooser timeout: the dismiss handler tried to click outside an already-open modal menu. It now closes the menu, dismisses through the UI, and restores that menu so the original action resumes. Controlled late-announcement acceptance passes; the temporary fault is removed. All 11 complete attachment receipt journeys pass, retries zero and original assertions/budgets unchanged, using a fresh supported external PostgreSQL-17 database with canonical isolated onboarding/server/UI/storage. An earlier embedded-cluster startup failure occurred before tests and remains recorded. Production server typecheck passes; production source and schema are unchanged. Fresh current-head CI and review are required. --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 20:05:31 -05:00
DottaandPaperclip	1ed0af4186	fix(server): wake native chat progress from committed changes (#17639) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Chat users receive coarse progress updates while native runs execute. > - A recurring chat pass scans saved run events even when no progress changes. > - This traffic prevents an idle database from staying idle. > - Events, lineage changes, and publication state already identify when projection can change. > - This PR gives native progress its own commit-driven worker and schedules only unfinished work. > - Idle progress stops scanning while restart and recovery preserve delivery. ## Linked Issues or Issue Description Refs #17606. This is slice 29 of the recurring-server-work stack, based on `codex/sleep-28-chat-run-milestones`. It owns native chat progress only. Related historical optimization: #16845. The parent and this PR must land in stack order. **What happened?** Native chat progress is still called by the one-second chat reconciliation loop. It repeatedly selects run events with no new work. Its cursor also truncates PostgreSQL timestamp precision, which can repeat a rejected full page forever. **Expected behavior** Committed events and eligibility changes should wake the projection owner. Existing rows should reconstruct work after restart. A settled worker should have no recurring database activity. Full batches, lock contention, and errors must retain recovery. **Steps to reproduce** Start an otherwise idle instance with chat history. Observe the native progress lane in the recurring chat coordinator. For the cursor defect, create a full page of suppressed candidates sharing a microsecond timestamp and one eligible later candidate. The old cursor fails to advance. ## What Changed - Extract native progress from lifecycle milestones into its owning service. Keep the public combined API and budget compatible. - Register a scoped worker with the existing delivery coordinator and scheduler. Remove native progress from the recurring chat coordinator. - Notify before allowlisted event writes in both persistence owners. Preserve outer commit, rollback, duplicate detection, and the ordinary non-progress log fast path. - Extend the existing lineage/authority notification boundary once. Failed-publication replay and operator retry carry their issue scope. First working publications signal progress so commit ordering cannot strand an already-inspected event. - Limit notification payloads to company/task IDs so wider caller inputs never serialize database handles or private fields. Load task-binding fields only for progress event kinds; other native/recovery events retain their narrow lock query. - Preserve current safe phrases, cadence, phase deduplication, provider policy, authorization, and final-output precedence. - Retry concrete issue/run lock contention. Disarm deleted or terminal candidates. Preserve scoped dirty work across batches, failures, drain and shutdown. - Preserve PostgreSQL timestamp precision in pagination. Add a bounded regression for the former infinite rejected-page scan. - Document the producer audit, lifecycle contract, failures, and remaining limits in `doc/plans/2026-10-25-native-chat-progress-sleep-audit.md`. ## Verification Published head: `b7ee820e49fd06c780b36b79d9b1c0883557fb73`, targeting master after #17606 merged at `5f0da562dbfdf3229f3cd4bb3dc742ab541986b4`. A normal ancestry merge preserves the complete 18-path native-progress patch byte-for-byte relative to its verified parent, with all six unrelated current master paths intact. No new production change was introduced during final reconciliation. - **163 distinct selected cases pass on the verified unchanged patch:** 137 across eight complete suites (native progress, atomic event append, all five milestone worker suites, and app admission wiring), plus 26 real chat integration cases selected by `native progress`. The other 1,595 chat integration cases were skipped, not passed. - The full worker suites cover restart, committed events, trace reprojection, duplicate/no-progress events, first-working publication ordering, concrete lock contention, issue scope, backlog, drain/resume, failures and shutdown. Settled history leaves no recurring SQL. The real chat cases verify operator retry and replay, microsecond pagination, authority, cadence, phase deduplication, final/question precedence and redaction. - The sole earlier parent conflict preserves both this slice's issue-scoped retry wake and the exact voice conversation scope. The current parent's milestone producers, onboarding completion guard, identity query/read-scope coverage, migration 0397 and voice cleanup/retention repairs remain present. - Server production typechecking, build and module-boundary checks pass. The selected server tests live under the excluded `src/__tests__` path and were executed separately. Each database owner uses a new isolated app-migrated PostgreSQL fixture. - [Current-parent acceptance and failure ledger](https://github.com/paperclipai/paperclip/blob/b7ee820e49fd06c780b36b79d9b1c0883557fb73/doc/plans/2026-10-25-native-chat-progress-sleep-audit.md#current-milestone-parent-acceptance-october-28) records current counts and scope. Older combined checks and failures retain their original revision scope and are not added to 163. No new table, dependency, workflow, migration, or scheduler contract is introduced. - **Exact-head Greptile is 5/5 and all 55 current checks are clean.** No Ponytail review is present, which is an explicitly authorized nonblocker; no failing verdict is waived. Older head/parent reviews are not current passes. Actual failing Ponytail findings must be fixed. Missing/stopped/paused turns without a verdict are authorized nonblockers; active unfinished reviews retain the original per-head timer with a 25-minute monitor cutoff respecting the 30-minute maximum. - No complete local repository matrix, live provider tenant, external wake, physical host/database suspension or savings claim is made. Following stack duties remain separate. ## Risks - Signals are process-local under the existing single active writer contract. Independent SQL writers or another active process need an explicit wake integration. - Allowlisted direct log writes now use a transaction. Native bound runs register transaction intent; unrelated direct event types retain their atomic-statement path. - The shared eligibility boundary wakes both lifecycle and native projection. It does not decide provider policy or copy event payloads. - The existing 24-hour selection window, event-time cadence and authority checks remain. A suppressed event is not retried just because wall-clock time passes. - This establishes worker quiet, not whole-instance suspension. Cloud alarm coverage and conservative sleep gates do not change. Other chat maintenance remains on its existing cadence. - This actual-master reconciliation still requires fresh exact-head review and CI before protected merge. Subsequent slices retain their own gates. ## Model Used OpenAI Codex, GPT-6 family, with repository inspection, code execution and browser-test tools. The exact serving variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 11:22:29 -05:00
DottaandPaperclip	5f0da562db	fix(server): wake chat lifecycle milestones from committed changes (#17606) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Connected chats show safe notices when a run starts, waits, finishes, or fails. > - A recurring scan projected these notices even when no eligible work existed. > - Eligibility depends on committed run state, conversation lineage, and current authority. > - Existing database rows can reconstruct that work after restart. > - This PR wakes a scoped worker from those writes and removes lifecycle polling. > - Native progress events keep their separate existing owner until the next migration. ## Linked Issues or Issue Description **What happened?** The chat maintenance timer repeatedly scanned run lifecycle state. Empty and settled instances still queried the database. A run-status event alone cannot replace the scan: prompt receipts, continuation receipts, identity changes, and restored authority can make a run eligible later. **What did you expect?** Committed changes wake projection. Startup reconstructs eligible work. A completed pass stops until another relevant change arrives. Failed work retains recovery without scanning settled history forever. **Steps to reproduce** Configure an automatic chat endpoint. Finish its work, or leave it unused. Observe recurring lifecycle projection queries from the chat maintenance callback. Stacked on #17592. Related foundation: #15625. This PR owns lifecycle milestones only. ## What Changed - Preserve upstream coalesced Cloud membership writes together with locked membership restoration and its committed chat notification. - Signal milestone eligibility from the real Slack account-link transaction. - Register the lifecycle worker with the existing delivery coordinator. Remove lifecycle projection from the periodic chat callback. Keep native progress in a separately named lane and preserve the explicit combined API. - Reconstruct company/issue scopes from the existing selector. Drain bounded batches with company fairness. Retry failures, retain writes through drain/resume, and disarm settled work. - Signal from supported run, presentation, interaction, receipt, lineage, conversation, recovery, and authority writers. Preserve their transaction ownership, including nested savepoints and uncertain settlement. No new table is added. - Preserve provider authorization, authored-final suppression, responsible-user continuity, deduplication, the 24-hour selection window, and raw-output restrictions. Keep provider interpretation in its owner. - Preserve PostgreSQL timestamp precision in lifecycle pagination. Rejected submillisecond candidates must not repeat the same page and starve later notices. - Document producer coverage and alternate-writer dispositions in `doc/plans/2026-10-25-chat-run-milestone-sleep-audit.md`. ## Verification Published head: `1d056f68c5b1005933b9e12de8745bf1c4cbeeca`, targeting master after #17592 merged at `7d75ae0569e41786640cf09647ae8849411d2f22`. A normal merge integrates the actual landed parent. The entire 77-path milestone patch is binary-identical to the locally verified preparation across that ancestry reconciliation; both unrelated master paths are preserved. The four earlier producer/test integration conflicts retain scoped milestone and voice notices, current onboarding completion admission, and expanded identity query/read-scope coverage. - **330 distinct selected cases pass on the verified unchanged patch:** 305 across 14 complete suites (all five milestone worker suites, run identity, board claim, Cloud provisioning, auth session routes, Cloud actor authentication, stale-lock recovery, GitHub credentials, project Git authorization, and app admission wiring), plus 25 selected real chat producer cases. The remaining 1,590 chat integration cases were skipped, not passed. - The real producer cases cover saved milestone discovery, GitHub and Slack identity changes, account-link rollback/reconfirmation, ordinary ingress reactivation, post-close action/delivery receipt settlement, all ten authority-restoration paths including Cloud synchronization, and changed native review metadata. - Worker coverage verifies outer commit, rollback, uncertain-write recovery, restart, bounded batching/fairness, timestamp precision, retries, drain/resume, and settled SQL quiet. Authorization and existing publication deduplication remain with their owning services. - Server production typechecking, build, and module-boundary checks pass. The tests above were executed separately. Production typechecking excludes `src/__tests__` and does not replace those runs. Each database owner uses its own newly created and app-migrated PostgreSQL fixture. - [Current-parent acceptance and preserved failure ledger](https://github.com/paperclipai/paperclip/blob/1d056f68c5b1005933b9e12de8745bf1c4cbeeca/doc/plans/2026-10-25-chat-run-milestone-sleep-audit.md#current-parent-integration-and-acceptance-october-28) records commands, counts, producer scope, conflict dispositions, and limits. Earlier combined-stack counts and failures keep their original revision scope; local acceptance does not erase old CI failures. - No new migration, queue table, dependency, workflow, or shared scheduler contract is introduced. Parent migration 0397, bounded consent recovery, voice cleanup/retention repairs, and runner checkpoint restoration remain present. - **Exact-head Greptile is 5/5 and all 55 current checks are clean.** No Ponytail review is present, which is an explicitly authorized nonblocker; no failing review is waived. Older reviews and parent checks are not current passes. Actual failing Ponytail findings must be fixed; an absent/stopped/paused turn without a verdict is an authorized nonblocker. Active unfinished Ponytail uses its original request/start timer, with a 25-minute monitor cutoff respecting the user's 30-minute maximum. - No complete local repository matrix, live provider tenant, external alarm, physical host/database suspension, or fleet savings claim is made. Native progress remains a separate following slice. ## Risks - A future eligibility writer must notify the existing projection boundary. Independent SQL writers or another active server process are outside this process-local signal contract. - Authority and lineage remain sensitive. The worker uses the existing selectors and publication policy; it does not grant execution or disclosure authority. - Native progress and other publication maintenance still have recurring work. This PR removes one reason, not the entire chat maintenance timer. - Worker quiet does not prove database suspension, host sleep, or external alarm coverage. Cloud sleep blockers remain unchanged. - Merge eligibility requires passing current-head CI and Greptile 5/5 after this actual-master reconciliation. Subsequent stack slices retain their own gates. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context-window size are not exposed. Used repository inspection, reasoning, shell execution, PostgreSQL tests, browser E2Es and GitHub tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details available in this session) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run the documented focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 11:07:54 -05:00
DottaandPaperclip	8049fbf51d	fix(server): schedule voice call reports from saved deadlines (#17586) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Voice calls retain reports and project guest transcripts into intake tasks. > - Chat maintenance repeatedly scans recent calls even when no report is due. > - Empty scans keep the tenant database active. > - Existing session and report rows already contain the work and its deadlines. > - This pull request moves report refresh to the shared coordinator and removes the final voice maintenance lane. > - Idle voice configuration and old call history no longer cause report scans. ## Linked Issues or Issue Description Refs #17583, #17580, #17578, #17576. This is slice 25 of the recurring-server sleep stack. It now follows merged #17583 and targets master. **What existing behavior does this improve?** Voice call-report refresh and guest transcript projection. **Current behavior** The chat interval scans terminal calls for reports. It repeats the scan even when there are no eligible calls or all saved deadlines are in the future. **Proposed behavior** Committed session transitions and signed callbacks wake a scoped worker. Startup reconstructs existing rows. Later refreshes use saved deadlines. The worker becomes quiet after all eligible windows expire. **Reason and benefit** Stop empty database scans without losing late transcript and billing updates. Keep the existing 24-hour refresh policy, provider response ordering, and guest-task trust restrictions. ## What Changed - Add a voice-owned report worker using the existing scoped deadline helper. No queue table or field is added; the existing session table receives a report-window index. - Save each attempt and its retry before provider I/O. Compare that ownership on settlement so an older response cannot overwrite a later callback or claim. - Preserve available reports when a response is incomplete or stale. Preserve their finite refresh window. - Route voice-change and reply call-ended writes through transaction-aware report signals. - Preserve guest transcript locks and use the observed document revision when updating an existing transcript. - Register the worker in app startup. Remove voice reconciliation from the chat interval. Keep explicit one-shot diagnostics. - Document producer scope, lifecycle behavior, tests, and external wake limits. ## Verification Published head: `758cfe15c3e6283d38ba512c8416ccce4d200862`. The follow-up after `c28de398` changes only two documentation files; the verified production, migration and test source is unchanged. It normally merges landed retention/master `980775e0e956eecdc28c73c9b4aca0407ad45e1a`. The 20 owned paths remain the complete slice, including officially regenerated migration 0397, its snapshot and required pruning of snapshot 0392. The non-database reviewed patch was preserved during ancestry integration; the only subsequent source edit makes one existing report test’s provider responses call-specific. All production report behavior and unrelated master changes are preserved. No shared worker, workflow or lockfile change belongs to reports. All 58 current-head checks are now clean, including the previously failing approval shard; exact-head Greptile is 5/5 with no new actionable issues. Ponytail’s current turn stopped without a verdict, an authorized nonblocking state. The prior conditional documentation requirement was fixed; old CI failures remain recorded below. - **Prior-head evidence at `9bf20467df`: 303 distinct targeted cases passed**, not rerun wholesale for this integration: all 132 voice integration cases; 170 cases across 11 complete report, retention, cleanup, recovery, company-removal, shared-worker and startup/hostname suites; and one complete snapshot/schema drift case. This includes 15 report, 13 retention and 14 cleanup lifecycle cases. Voice integration and the four other database owners use separate initially empty databases and migrate themselves. Report cases use their dedicated database after the setup repair. - Five real connection saves over 205 sessions with saved future report deadlines selected 1,025 session rows before the notification repair. They now select zero and preserve the original deadline. The report notification helper accepts only exact session scopes. Configuration, authority and connection hints retain their existing live recovery/reply/cleanup behavior but do not rebuild report history. Real session terminal writes, callback hints, report claims and receipts still reactivate reports. - Database/server typechecks and builds, migration numbering/safety, and module-boundary checks pass. Final server types/build also pass after integrating the retention review repairs. The parent 205-admission, five-configuration-save test still observes zero retained-history rows and preserves its deadline. - The old eager reader selected all 105 saved-report headers before yielding and failed the 100-row budget. Registered reconstruction now uses the existing paged contract and yields at 100 rows per scope refresh. A partial scan can restart on the same owner and reconstruct every saved deadline. At due time all 105 report IDs are fetched exactly once. Future deadlines remain quiet between actual retries. - The company-fairness case places a small company deterministically on page two and requires it in the next 25-work batch after discovery, with all 105 reports processed before the failed provider's retry. Guarded mutations catch a missing cursor reset and bypassed company rotation. Original source was restored before final acceptance. - At prior head `9bf20467df`, migration 0396 was generated with `pnpm db:generate` on the parent 0395 lineage. It creates the partial `(ended_at, id)` report-window index concurrently and replayably through the existing migration runner. All 394 historical SQL files and the journal prefix are unchanged; snapshots 0392 through 0396 remain. Fresh application migration and two replay invocations succeed with a valid index. The real plan case retains 20,001 old sessions, 205 recent sessions, precision-safe keysets, and fewer than 1,000 rows removed by filtering on each page. - Existing real hangup/callback/reply-error producers, callback ownership fencing, monotonic report results, saved retry before provider contact, guest transcript projection after a document lock, transaction rollback/nesting, drain and shutdown remain covered. No table, field or provider protocol is added. Available reports retain the existing finite 24-hour refresh policy. - [Current-parent acceptance and failure ledger](https://github.com/paperclipai/paperclip/blob/9bf20467dfd50b81848e1c8dd7ddc905d206418f/doc/plans/2026-10-25-voice-call-report-sleep-audit.md#current-parent-reconstruction-and-migration-acceptance-october-28) records the failed fresh fixture, eager read, fairness-clock attempts, guards, commands, counts and limits. Older 174/138-case and workspace-wide results belong to their stated heads. No new complete local repository suite, live provider, Cloud wake, host/database suspension or savings claim is made. - [Report configuration-scope repair](https://github.com/paperclipai/paperclip/blob/9bf20467dfd50b81848e1c8dd7ddc905d206418f/doc/plans/2026-10-25-voice-call-report-sleep-audit.md#configuration-scope-review-repair-october-28) records the current 303 cases and measured rescan repair. The prior `80207ba6bb` head had Ponytail 5/5; that review is stale after this change. Fresh exact-head Greptile 5/5 and all CI remain required. Actual failing Ponytail findings must be fixed; a stopped or paused turn without a verdict is an authorized nonblocker. Active no-verdict review waits are capped at 30 minutes from the original request/start (25-minute monitor cutoff). Older scores and parent checks are not inferred as current passes. - **Current-master verification: 185 distinct cases pass**: final complete voice integration 132, complete report/cleanup/retention 15+24+13, and snapshot/schema drift 1. Each database owner used its own empty fixture database. The first combined run had 183 passing cases and one report-test failure; it is preserved in the audit. A deliberately earlier-ended competing call reproduces the old one-shot mock consuming the intended response. The corrected test selects responses by provider session ID, verifies independent costs for both calls, then rejects stale target data. The isolated repair and full 132-case suite pass; production behavior and timeouts are unchanged. These 185 cases are not added to the older 303 count. - **Current migration lineage:** official `pnpm db:generate` after actual master creates `0397_known_madrox.sql`; the reviewed concurrent/idempotent report-index SQL is retained byte-for-byte. All 395 existing SQL files and journal entries are unchanged; snapshots 0393–0397 remain and the new snapshot links parent 0396. The application migrator boots a fresh database, repairs a deliberately invalid index during two concurrent replay invocations, records one history row, and replays again successfully. Database build, server build/typecheck, migration checks and module boundaries pass. [Full current-master acceptance and failure ledger](https://github.com/paperclipai/paperclip/blob/c28de398fef074f650eff532952ab6f5563739a8/doc/plans/2026-10-25-voice-call-report-sleep-audit.md#landed-master-migration-and-acceptance-october-28). - **Review follow-up and preserved CI timeout:** Ponytail’s prior `c28de398` review was 4/5 advisory, with no must-fix production defect and a request to fix the stale design-doc migration reference before shipping. That document now names migration 0397 and links current 185-case acceptance; older 302/303-case evidence is explicitly historical. The optional suggestions about one-shot diagnostic inventory cost and inherited test-clock helpers remain disclosed. CI run `40703490340` failed the unchanged approval-queue backoff test on its existing 15-second timeout and then an activity-log FK in teardown (2,097 pass/one fail/one skip in that shard). The complete unchanged 28-case approval suite passes on a new isolated database; the exact CI timing cause remains unproven. No timeout, workflow or production logic was changed, and no old-head rerun or failure waiver is claimed. This documentation head requires fresh CI and exact-head Greptile. [Review and timeout ledger](https://github.com/paperclipai/paperclip/blob/758cfe15c3e6283d38ba512c8416ccce4d200862/doc/plans/2026-10-25-voice-call-report-sleep-audit.md#review-documentation-and-preserved-remote-timeout-october-28). ## Risks - Process-local commit signals do not observe independent SQL writers or another server process. - Available reports still refresh for 24 hours. The provider's complete flag does not promise immutable final data. - Transcript projection retries remain within the existing report window. This change does not add indefinite projection repair. - One-shot diagnostic `reconcile(limit)` inventories the eligible window before slicing; it remains a manual diagnostic cost, not recurring idle work. Some inherited test helpers advance fake time during assertion retries, an optional reliability follow-up noted by review. - A local deadline cannot wake a stopped host. Cloud alarms and sleep eligibility remain unchanged. No deployed sleep or cost saving is claimed. - This PR depends on the preceding stack. Final master reconciliation and remote review gates remain required. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, code editing, and local test execution. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 09:41:15 -05:00
DottaandPaperclip	980775e0e9	fix(server): schedule inbound voice history retention by deadline (#17583) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Incoming voice calls leave history after provider cleanup is confirmed. > - The chat maintenance tick repeatedly deletes old admission history. > - Each eligible row already contains the dates needed for its retention deadline. > - This pull request schedules that deadline and wakes on committed changes. > - Idle retention stops querying while the existing seven-day policy remains enforced. ## Linked Issues or Issue Description **What existing behavior does this improve?** Inbound voice history retention. This PR follows merged #17580, which owns physical call cleanup, and now targets master. Call reports remain a separate migration. **Current behavior** Voice maintenance scans admission history repeatedly. It deletes confirmed, terminal, unbound admissions when both expiry and last update are at least seven days old. **Proposed behavior** Load existing retention deadlines at startup. Wake from the existing admission transaction notifications. Delete at the later timestamp plus seven days. Lock and recheck current eligibility before deletion. **Reason and benefit** Configured idle endpoints and future history do not cause recurring retention SQL. Failed deletion retains a retry. Restart reconstructs the same saved deadlines. ## What Changed - Add a voice-owned retention worker using the existing scoped deadline helper and coordinator. Yield after 100 history rows per scope refresh and reset reconstruction on registration. Keep complete one-shot inventory reads. No table, field, or migration is added. - Extend `notifyVoiceInbound` to notify retention before the same committed writes that already notify cleanup. Callers need no new recovery wrapper. - Preserve confirmation, terminal-state, session, task, and seven-day eligibility conditions. Round PostgreSQL submillisecond deadlines up. - Lock and recheck the current row before deletion. New bindings or moved dates invalidate stale candidates. - Remove history pruning from recurring voice maintenance. Keep the explicit one-shot diagnostic path. Leave call reports for their own slice. - Document producer ownership, retry/drain behavior, tests, and external wake limitations. ## Verification Published head: `53f2fafcb64d104887eb4455c292b48e6dd10253`. It includes the normal merge of actual landed master `71b448b9cb841e6fa18fe883585430c5081245da` after cleanup #17580. That ancestry integration at `768c650365` preserved the complete binary diff of all 13 reviewed retention paths. This follow-up changes only the existing cleanup fairness test and its failure ledger, bringing the final scope to 14 paths; all production source is unchanged from that integration. All unrelated master changes, cleanup repair, migration `0396_tan_paladin.sql`, and gateway clock repair are preserved. Database files, shared worker, workflows and lockfile match actual master. The earlier 282-case/type/build evidence below belongs to unchanged retention source at its stated head; current repair acceptance is 37 distinct complete cleanup and retention lifecycle cases. All 58 current-head checks are now clean, including the Canary Dry Run; Greptile is exact-head 5/5 with no new actionable issues. The current Ponytail turn stopped without a verdict, which is an authorized nonblocking state. The earlier failing CI remains preserved below. - **282 distinct targeted cases pass**: all 127 voice integration cases and all 155 cases in ten complete retention, cleanup, recovery, company-removal, shared-worker, startup and hostname suites. These include 13 retention and 14 cleanup lifecycle cases. Voice integration, parent recovery and company removal start on separate empty databases and migrate themselves. Retention uses its dedicated database after the setup repair; cleanup uses its already-migrated dedicated database. - Ponytail's prior exact-head 4/5 review found configuration-driven history rescans. Five real connection saves selected 1,025 rows over 205 future admissions before repair. Retention now receives only admission hints and startup reconstruction; the same saves select zero admission rows and retain the original deadline. Cleanup still receives its full configuration and authority scopes. Restrictive binding foreign keys prevent implicit unbinding outside the admission-write boundary. - Successful deletion now returns its confirmed result without reloading the deleted row. A real SQL-count case and a guarded negative probe cover that path. Changed or not-yet-due history still reloads current eligibility and its deadline. - Server typecheck, build and module-boundary checks pass. No schema, driver, shared scheduler contract or provider protocol changes are added by this slice. - A real 105-row future-history case failed before the reader repair because startup selected 105 rows rather than yielding at 100. The existing paged shared-worker contract now bounds each scope refresh. The same-owner restart case reconstructs every deadline and deletes all 105 eligible rows at the original deadline. It verifies SQL quiet before that deadline and after settlement. Removing only cursor reset fails at five rows instead of 100 after restart; guarded source was restored before final acceptance. - Existing cases retain the later-of-two-timestamps seven-day policy, submillisecond rounding, current-state confirmation and binding checks, concurrent binding lock, rollback/outer commit, first write during drain, failure retry and shutdown. Real signed ingress, cleanup confirmation and maintenance-lane removal remain covered by the complete voice suite. - The fresh retention fixture first failed on missing schema and now applies the application migrator. The initial parent merge duplicated a test import and failed to parse before any voice case ran. Both failures are retained in the audit; the corrected complete suites pass. - Prior head `daca270a99` had a failed server (4/12) check, run 40557790710/job 122386307340. Its actual log reports a runner shutdown and cancellation before completion. This is incomplete acceptance, not a pass or a proven product flake. New-head CI must pass independently. - [Current-parent acceptance and failure ledger](https://github.com/paperclipai/paperclip/blob/68b62b6d13f28ac5f17f35b98adf5488752f9b97/doc/plans/2026-10-25-voice-inbound-retention-sleep-audit.md#current-parent-paging-and-restart-acceptance-october-28) records the exact scope, commands, counts and preserved failures. Earlier 218/133/134-case and full-workspace results belong to their stated heads. No new full local repository suite, live provider, Cloud wake, host/database suspension or savings claim is made. - [Notification review repair and current 282-case evidence](https://github.com/paperclipai/paperclip/blob/68b62b6d13f28ac5f17f35b98adf5488752f9b97/doc/plans/2026-10-25-voice-inbound-retention-sleep-audit.md#configuration-rescan-review-repair-october-28) records the new source. The prior head had green CI and a 4/5 Ponytail review; neither is accepted for this new commit. Fresh exact-head Greptile 5/5 and all CI remain required. Completed actual failing Ponytail findings must be repaired; a stopped or paused turn without a verdict is not a merge blocker under the user’s authorization. An active Ponytail review waits at most 30 minutes from its original request (25-minute monitor cutoff). - **Preserved current-master CI failures:** run `40697141057` failed server shard 7/12 in cleanup’s fairness test and Canary Dry Run on an npm `rollup-4.66.0.tgz` 404. A fixed final admission UUID reproduces the same fairness assertion: the second company was sometimes beyond the first 100-row page, so expecting it among the first 25 calls was invalid. The corrected test controls time and proves service within the next 25-call batch after discovery, all 105 successful calls before the failed provider’s saved retry, exactly one retry at its deadline, and idle SQL after settlement. Bypassing company rotation fails the corrected assertion; the shared helper was restored byte-for-byte. Both complete cleanup (24) and retention (13) suites pass after restoration. An intermediate 23/24 run incorrectly demanded idle before reconstruction finished; the audit preserves that failure and its corrected pending-intent wait. No production behavior, timeouts, dependencies or workflow gates changed. The exact npm metadata and tarball now return 200; new-head canary CI must pass independently. Prior Greptile/Ponytail 5/5 on `768c650365` are stale for this new head. [Full CI repair ledger](https://github.com/paperclipai/paperclip/blob/53f2fafcb64d104887eb4455c292b48e6dd10253/doc/plans/2026-10-25-voice-inbound-retention-sleep-audit.md#current-master-ci-repair-october-28). ## Risks - Supported admission writers must use the existing transaction notification boundary. Independent SQL writers and additional processes need a separate wake contract. - Retention never deletes unconfirmed cleanup or session/task-bound history. Those records remain with their owners. - The in-process deadline does not wake stopped compute. This PR does not extend Cloud alarms or relax sleep eligibility, and does not claim deployed host/database suspension. - Call-report reconciliation and other chat maintenance still need separate migrations. ## Model Used OpenAI Codex with reasoning, shell, and code-editing tools. The exact model version and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green for this new head - [x] Greptile is 5/5 on this new head with no unresolved actionable findings - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 08:57:58 -05:00
DottaandPaperclip	71b448b9cb	fix(server): schedule inbound voice cleanup from durable obligations (#17580) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Incoming voice calls need approval, expiry, and safe provider cleanup. > - The remaining voice maintenance tick scans admissions every second, including idle endpoints. > - Admission status does not prove that the provider call has ended. > - Existing admission rows can retain the cleanup owner, confirmation, and retry deadline. > - This pull request replaces the cleanup scan with committed changes and actual deadlines. > - Idle cleanup stops querying while unfinished calls keep a recoverable path to termination. ## Linked Issues or Issue Description **What existing behavior does this improve?** Inbound voice admission cleanup. Its reply-delivery parent #17578 has landed; this PR targets master. Live-session recovery is in #17576. It preserves separate ownership for those services. **Current behavior** The voice maintenance tick scans expired admissions even when no cleanup is due. A session can commit before the separate approved-admission write. Expiry can then terminate that authorized call. A rejected guest call can be labeled ended before provider termination succeeds. Seven-day retention can delete that unfinished cleanup. **Proposed behavior** Wake cleanup from committed admission and session changes. Schedule the original expiry or saved retry. Claim current state under the admission lock before provider I/O. Commit approval with session reservation. Keep unconfirmed termination durable and transfer bound calls to live-session recovery. **Reason and benefit** Configured idle endpoints and confirmed history cause no recurring admission cleanup SQL or provider requests. Approval, retries, crash recovery, and cleanup remain supported. History retention and call reports stay separate migration items. ## What Changed - Recover terminal and rejected legacy admissions whose session binding was lost. Copy known termination or bind a proven matching live session for cleanup without granting approval. - Register inbound cleanup with the shared coordinator and scoped deadline worker. Yield after each 100-row scope page and reset cursors on registration. Remove only cleanup from periodic voice maintenance. Keep complete manual reconciliation. - Add commit-aware notifications at admission, approval, rejection, guest binding, pending-tool expiry, lifecycle confirmation, and session termination writes. - Commit authorized session binding and admission approval together. Recheck cleanup ownership under the same admission lock. Recover older interrupted bindings only from matching session evidence. - Add nullable cleanup confirmation, next-attempt, and provider-account fields to the existing admission table. No new queue table is added. Generated migration 0396 backfills only provable historical termination in committed 1,000-row keyset batches and adds a concurrent partial index. - Save the retry before provider contact. An unconfirmed end receipt retains cleanup. Bound calls remain owned by live-session recovery. - Verify the signed provider account before using a rotated key for unbound cleanup. A different account cannot inspect or end the saved call. Legacy admissions require their original credential fingerprint. - Prevent history pruning and company deletion from erasing unfinished cleanup. Company deletion locks endpoints before the voice owner's check, matching ingress serialization. - Document producer coverage, ownership, deadlines, verification, and remaining sleep limits. ## Verification Published head: `a4e7ccf976fc5793a8914e4e4861aa080d43a4da`. The reply parent #17578 has landed. A normal merge integrates actual master `e7ea2e2961863171956e70fb05d3d1f3790f7909`, preserves its unrelated changes, and regenerates this slice’s migration after landed migration 0395. Fresh exact-head reviews and CI are required. - The previous Ponytail 3/5 finding is repaired: legacy terminal admissions no longer enter approval-only repair, and rejected admissions with valid matching live sessions retain their rejection and persist the binding needed for session-scoped cleanup confirmation. Five regressions failed with ownership conflicts before the repair. Five conflicting-session regressions prevent an unrelated leg from being claimed. No cleanup operation grants approval or starts a provider call. - **24 complete cleanup lifecycle cases and 122 complete voice-store integration cases pass** on the repair. After actual-master integration and migration regeneration, the complete 24-case cleanup suite passes again on a new empty PostgreSQL 17 database, and the database migration integration test passes separately. The cleanup owner, voice-store test and admission schema are byte-identical to that tested repair; the cleanup fixture only changes the migration filename. Migration numbering/safety, database build, production server typecheck and build pass. - Migration `0396_tan_paladin.sql` was generated with `pnpm db:generate`. Every landed SQL migration and the complete parent journal prefix are unchanged. Generated snapshots retain exactly 0392–0396, pruning 0391. The reviewed cleanup SQL, concurrent idempotent index and committed 1,000-row historical backfill are retained byte for byte under the new generated filename, registered for top-level execution. The real application migrator boots a fresh database, repairs an interrupted concurrent index, serializes replays and verifies eligible-history backfill across actual transactions. - The preceding published head `48d8356e73308579470c182e759eead6e83173d2` received Greptile 5/5 but failed CI server shard 9/12: one existing connected MCP rate-limit test expected two real-clock calls to share a fixed 60-second bucket. A controlled window rollover reproduced its exact failure. That existing fixture now freezes only `Date` around the same-window 429 assertion, verifies acceptance after advancing one window, and restores the real clock in `finally`. All 92 gateway acceptance cases pass. Production rate limiting, timeouts and assertions are unchanged. This adds one existing CI fixture to the slice; the original failed run remains recorded. - The first regression invocation skipped the suite because it used the wrong database variable; that is not a pass. The corrected invocation reproduced five ownership conflicts. The first fixed cleanup run passed 22 and failed two new low-level deletion test calls on a retained-session foreign key; the regressions now check the cleanup deletion guard at its owning boundary, with production deletion ordering unchanged. The final full suite passes all 24. - The first complete voice-store invocation passed 121 and timed out on one transcript case at the unchanged 15-second limit. That case passed in isolation on a fresh database. One subsequent complete invocation, with builds finished, passed all 122 without source, assertion or timeout changes. The underlying timeout cause is unproven; the original failure remains recorded. A new PostgreSQL startup also failed from exhausted host semaphore capacity; fresh databases on the already-owned server were used without removing shared IPC resources. - Prior source `b8b12199b9` passed 266 distinct cases and carried three guarded mutation proofs; those older results are not added to this repair’s case count. Its CI run 40641168596 failed E2E shard 3 when legacy cancellation hit a one-second PostgreSQL lock timeout. GitHub initially rejected a retry while the run was active, then accepted one targeted retry after completion; attempt 2 passed on that same old head. The lock holder and underlying cause remain unproven. The unchanged Node policy checker’s older failure on ignored generated Copilot manifests and passing tracked-input mirror are also retained in the audit; no workflow, checker, vendor manifest or ignore rule was modified. - [Current repair and failure record](https://github.com/paperclipai/paperclip/blob/a4e7ccf976fc5793a8914e4e4861aa080d43a4da/doc/plans/2026-10-25-voice-inbound-cleanup-sleep-audit.md#legacy-terminal-admissions-missing-their-session-binding--october-28). Current-head CI and Greptile 5/5 gate merge. A completed failing Ponytail verdict must be fixed; missing or interrupted reviews do not block, and active pending review waits are capped at 30 minutes under the maintainer’s explicit instruction. No new full local repository suite, real provider, Cloud alarm, host/database suspension or savings claim is made. ## Risks - The migration distinguishes admission status from physical termination. Legacy ended guest rejections without proof remain cleanup obligations. Missing original credentials can keep those old obligations unresolved. - Future writers must use the existing notification boundary. Independent SQL writers and multiple processes need their own wake contract. - Pending inbound cleanup now blocks company deletion instead of allowing cascade deletion to lose the provider obligation. Existing voice-history deletion restrictions remain. This deliberately includes an active anonymous inbound ring and an old call whose termination cannot be verified. An operator must stop ingress and settle the call before deletion; this slice adds no abandonment override and never treats missing credentials as proof of termination. This is the retained cleanup ownership policy, not a claim that every deletion attempt can complete immediately. - History retention and report processing still run in the remaining maintenance lane. This PR does not extend Cloud wake coverage or claim measured host/database suspension. ## Model Used OpenAI Codex (GPT-6), with reasoning and code execution. The exact serving model ID and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 08:30:15 -05:00
DottaandPaperclip	e93732c3d4	fix(server): deliver voice replies from commits and saved deadlines (#17578) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Phone calls receive approved answers from the existing chat publication queue. > - The chat maintenance timer checks for voice replies every second, including when no reply is ready. > - A reply can be saved before its publication commits. An early wake alone can miss that final transition. > - Existing session, reply, publication, and action rows already record delivery and recovery work. > - This pull request uses committed changes and saved deadlines to drive phone reply delivery. > - Empty reply queues become quiet while authorization, ordering, and uncertain-send protection remain in place. ## Linked Issues or Issue Description Stacked on #17576. Related delivery infrastructure: #15625. Related voice connection work: #11418. **What existing behavior does this improve?** Background phone reply delivery and recovery of abandoned dispatch intents. **Current behavior** Chat maintenance calls voice reply delivery every second. It also scans old dispatch intents. Publication transport starts a direct push before the final publication commit. The timer masks that timing gap. **Proposed behavior** Committed publication, reply, session, and configuration changes wake the shared coordinator. Ready replies drain in bounded passes. Saved rate-limit and dispatch-expiry deadlines retain recovery. An empty queue or a permanent delivery barrier has no recurring reply scan. **Reason and benefit** Remove recurring SQL from idle phone reply delivery without dropping replies, replaying uncertain sends, or adding a queue table. **Breaking changes** No API or schema change. Explicit manual recovery remains available. Inbound admission cleanup, history retention, and reports remain on their current paths for separate migrations. ## What Changed - Extract ordered reply state and multipart policy within the voice owner. Retain current authorization and atomic dispatch claims. - Register a scoped reply worker with startup reconstruction, fair work batches, failure backoff, drain/resume handling and shutdown ownership. - Extend the existing shared dispatcher with normalized partial-page results to preserve the bounded parent recovery contract. Existing complete-array consumers retain their calls. - Signal before relevant writes in their existing transactions. Both normal publication completion and operator resolution wake the actual conversation after commit. - Bound reply inventory to 100 total headers per refresh. Preserve partial scans, concurrent changes and newer denied-call restoration obligations across reconstruction. - Track abandoned dispatch actions independently of live calls. Quarantine expired intents without resending them. - Use existing active-call authorization checks to reactivate replies that were previously denied. Healthy call inspections do not rescan empty reply queues. - Remove the periodic reply scan and the early direct publication push. Keep provider requests outside database transactions. ## Verification Published head: `1435cab10be3f95c38ab6672d2a890ae49e9d300`. Voice recovery #17576 has landed at master `e1ba77f06771a27114b3999004b0ca5e3c172dd4`. A normal merge reconciles squash ancestry while preserving the complete reviewed 31-file reply patch byte for byte and all unrelated master changes. The PR now targets master; fresh exact-head CI and Greptile are required. The prior 449-case/typecheck/build results below cover the identical owned source, not a new full-suite run. - **449 distinct targeted cases pass**: 111 complete voice-store/reply; 19 complete parent recovery; and 319 complete tool-gateway, secret-proposal, shared dispatcher/coordinator/batching/startup/hostname cases. Production server typecheck and build pass. The policy checker and its Node tests pass. Later edits record evidence only. - Two real normal-publication/operator-resolution tests seed 105 unrelated calls and 105 future dispatches. Five publications selected **1,065/1,060 inventory rows before repair**; both paths now pass the same **125-row total budget** and deliver every answer once. This counts selected SQL results, not physical tuple visits or host savings. - Publication commits now signal their actual conversation. The voice reader returns at most 100 total session/action headers per refresh, preserves phase/cursor state and retries, and resets on worker registration. Current-state checks and dispatch claims still own provider effects. The 105-action case measures the page cap, restarts the same owner during reconstruction, waits at saved future deadlines and quarantines every abandoned action without resending. - A real 403 fixture proves that an older broad scan cannot erase a denied call first observed behind its cursor. Its owner callback still emits the committed restoration wake after scan completion. The existing actual-service authority-restoration and final-publication timing tests remain in the full suite. They are not replaced by direct helper calls. - Guarded removal of conversation scoping, cursor reset or the denial revision fence fails the intended budget, 100-versus-5 restart or missing committed-wake assertion; source is restored. An initial shadowed fixture name and an inadequate due-action reset probe are retained as failed attempts in the [current audit](https://github.com/paperclipai/paperclip/blob/d2c0a7de8c3e9cac8382b1268f3f5d883a2067b3/doc/plans/2026-10-25-voice-reply-delivery-sleep-audit.md#publication-inventory-and-paging-acceptance-october-28). - Parent recovery still passes paging, restart, behind-cursor insertion, feature changes, actual disablement, quiet health observations, current authority, exact-scope failure fences and wildcard-failure isolation through the shared worker. Four existing array-based worker consumers retain their API and pass their full suites. - The two voice fixtures self-migrate separate empty databases. Legacy tool-review/secrets suites use separate migrated clones. Earlier full-local attempts remain limited by embedded PostgreSQL setup and are not passing full-matrix claims. Older counts apply only to their stated revisions. Fresh exact-head Greptile, Ponytail if present, security and all CI remain merge gates. No live-provider, multi-writer, external-wake, deployed-sleep or savings claim is made. ## Risks - Missing a supported producer can strand work. Tests cover real enqueue, retrieval, normal publication, and operator publication paths. Startup reconstructs existing records. - Signals cover the process-local database owner. Independent SQL writers or other replicas need their own wake contract. - Authority restoration relies on the existing live-call recovery check. This dependency must be preserved if that owner changes later. - A provider acceptance receipt does not prove speech. Unknown outcomes remain blocked from automatic resend. - This change does not extend Cloud alarms, allow sleep during live calls, or prove deployed suspension or cost savings. ## Model Used OpenAI GPT-6 via Codex, with repository inspection, reasoning, code execution and test tools. The runtime does not expose a more specific deployment identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used with the available version and capability details - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the relevant issue template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket identifier - [x] I have run the selected tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 07:50:27 -05:00
DottaandPaperclip	e1ba77f067	fix(server): schedule live voice recovery from session changes and deadlines (#17576) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Voice calls retain durable session state and provider identities. > - The chat maintenance timer scans live voice sessions even when no call exists. > - Session writes, saved expiry and active-call inspection deadlines identify real work. > - This pull request moves live-session recovery to the shared coordinator. > - Idle session recovery becomes quiet without losing exact-call cleanup or current authorization checks. ## Linked Issues or Issue Description Refs #17572. This PR is stacked on receiver discovery. It owns only live voice session recovery. Reply delivery, inbound admission cleanup, inbound history retention and call reports remain separate migration slices. **What happened?** The one-second chat maintenance lane reads voice settings and scans live sessions even when an instance has no calls or only terminal history. **What did you expect?** Reconstruct live calls on startup. Wake on committed session or configuration changes. Inspect active calls and retry cleanup at actual deadlines. Stop querying when no live obligation remains. **Steps to reproduce** Start an instance with no live voice sessions. Let startup settle. The old maintenance method continues to read settings and session inventory every chat maintenance pass. ## What Changed - Add a voice-owned bounded deadline reader and scoped worker on the existing coordinator. Retain concurrent writes across pages and reset partial scans on restart. No tables or migrations are added. - Separate generic connection health observations from configuration changes. Existing receiver and command consumers retain their notifications. Voice eligibility and provider operations remain in the voice owner. - Signal existing session transactions and configuration/feature changes. Keep provider calls outside database transactions. Standalone session writes use one small transaction helper. - Preserve saved expiry, the two-minute unknown-creation grace, active-call authorization and missed-hangup inspection, and exact-provider cleanup. Recovery never creates a provider call. - Retry transient authority lookup failures instead of treating them as confirmed revocation. - Fence expiration against a provider identity bound after discovery. A late known call remains eligible for cleanup. - Preserve work through rollback, drain, restart, concurrent refresh, backoff and shutdown. Page inventory and rotate bounded batches across companies. - Leave the four other voice maintenance duties in the periodic lane. - Document producer coverage and the supported writer contract in `doc/idle-voice-session-recovery.md` and the dated audit. ## Verification Published head: `f0b2e3cc22a18e7d08b10d4fc02aae4a19a52f30`. Receiver #17572 has landed in master at `dce004259394486f6361d0c0dd5a33a5272acc0a`. This normal merge reconciles squash ancestry; the complete 22-file voice recovery patch is byte-for-byte identical to the reviewed and tested `ad79c7e805` patch, while all unrelated master changes are preserved. The PR now targets master. Fresh exact-head CI and Greptile are running; interrupted Ponytail turns are not completed verdicts under the maintainer’s clarified merge policy, and an actual failing verdict still blocks. - **210 distinct targeted cases pass on the current repair**: 19 complete recovery lifecycle cases and 191 complete voice-store, coordinator, batching, startup and hostname cases. Production server typecheck and build pass. Later edits record evidence only. - CI `40633667141` on prior head `1b6323c164` failed the feature-change assertion with `34050 < 25000`; its verification aggregate also failed. The test now disables automatically advancing fake time, drives 20 controlled scheduler turns and requires all 205 calls to close before an unrelated inspection deadline can mask a lost wake. Removing generation replay fails with missing closures. The repair does not benchmark database/host speed or raise the elapsed-time bound. - A real committed exact-call regression reproduces wildcard-backoff starvation. The worker keeps the broad retry but processes independent exact hints and cached due calls. Each check reloads current state and authority. Failed exact scopes remain fenced. Removing wildcard isolation fails the intended assertion; a successful broad retry then leaves the worker SQL-quiet. Full-row and lean queries now share the state/scope/cursor predicate. - Earlier acceptance on `712f3862b5` passed 725 distinct targeted cases. Its measurements remain useful history: five real metadata writes and one health check over 205 future calls selected 1,230 rows before the inventory repair and zero afterward, without changing the saved deadline. Those older test counts are not added to the current 210-case count. - Both external voice fixtures self-migrate; the current full voice-store run uses a separate initially empty database. Earlier fixture/method/observer failures, source mutations and partial edit attempts remain in the [current audit](https://github.com/paperclipai/paperclip/blob/ad79c7e805367fc7e2df3ea53c942ca6c7d27365/doc/plans/2026-10-25-voice-maintenance-sleep-audit.md#wildcard-failure-and-deterministic-generation-acceptance-october-28). Source is restored after guarded mutations. - Older CI `40632057899` failed because its trusted policy script was absent from the checkout, while build separately received a runner shutdown signal. E2E shards succeeded but their aggregate failed on policy. The current checkout includes the landed policy tooling and its checker/tests pass locally. The shutdown cause is unproven; neither old failed run is counted as passing. Earlier full-local attempts stopped on embedded PostgreSQL setup failures and did not pass. - Fresh exact-head Greptile, security and all CI must pass before merge. A completed failing Ponytail verdict blocks; an interrupted turn without a verdict does not. No full current local repository matrix, live provider acceptance, multi-writer support, Cloud wake, host suspension or savings is claimed. ## Risks - Process-local hints cover the existing active writer and shared clients. Independent SQL writers or another replica need explicit wake integration. - Membership, task binding, phone preferences and other authority inputs remain checked on the existing 30-second active-call inspection deadline and on use. This is not an instant cleanup guarantee for every authority mutation. - Unknown creation retains its existing maximum-duration grace. Cleanup failures retain retries. A provider request is never replayed merely because its creation response is unknown. - Genuine configuration changes retain bounded reconstruction. Metadata writes can probe parent eligibility, while health-only observations leave voice recovery quiet. Paging state is process-local and reconstructs from durable rows after restart. - Active calls, the other voice maintenance owners and connector transports remain real obligations. This does not expand Cloud alarm coverage, relax sleep eligibility or prove deployed suspension. - The receiver and Discord parents have landed. Current-master compatibility and exact-head CI and Greptile remain merge gates. ## Model Used OpenAI GPT-6 via Codex, with repository inspection, reasoning, code execution and test tools. The runtime does not expose a more specific deployment identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket identifier - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 07:38:48 -05:00
DottaandPaperclip	2cbca4ceb0	fix: retain channel tools across task continuations (#16806) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Chat channels give an agent tools for its assigned task. > - Each later run must resolve the task and its accepted requester again. > - Some native runs store the task in a database column without the older JSON fields. > - Tool discovery read only the JSON fields. GitHub calls also required a new inbound comment. > - This pull request uses the durable task binding and accepted request history. > - Later runs retain their channel tools while current access checks still apply. ## Linked Issues or Issue Description **What happened?** A GitHub task can find `begin_review` on its first run, then receive an empty tool list on a later native run. If discovery is repaired alone, a call can still fail because the later run has no new inbound comment. Slack and voice tool access also read the older JSON task fields. **Expected behavior** A later run for the same task can find and call its assigned channel tools. The current requester, task, company, agent, endpoint, and access rules must still allow the call. Revoked access must fail. **Steps to reproduce** 1. Start a GitHub task and accept its linked requester. 2. Confirm that the runtime can find `begin_review`. 3. Start a later native run with `nativeIssueId`, accepted parent identity, and no JSON task ID or new wake comment. 4. Search for `begin_review` again. Before this change, the search returns an empty list. 5. Repair discovery alone and call `read_pull_request`. Before the request-history fix, the call fails with no active conversation capability. **Paperclip version or commit** Reproduced in source tests before this patch. The branch includes master at `fa50cab5685b9cdbb26ad9c610cc5b6c373d3ff4`. **Additional context** This is a core run-binding bug. The tests use PostgreSQL and synthetic provider responses. They do not send live reviews, Slack messages, or phone calls. ## What Changed - Resolve run task IDs from durable columns, with legacy JSON fields as a fallback. - Use the same binding in the gateway, access policy, identity service, and channel runtime preparation. - Recover accepted channel request history across later runs. Keep company, task, agent, and requester boundaries. - Recover older wake sources through verified parent runs, including after same-user board follow-ups. Clear the board message’s own legacy wake fallback. Keep cycle checks without a fixed limit on valid task history. - Retain the GitHub origin after a board comment from the same accepted user. Skip GitHub origin lookup results from other providers. - Keep ordinary board tools available when the requester changes. Accept legacy user comments without an author-type field. Still reject explicit malformed task IDs. - Prevent a later failure reply from replacing an earlier final GitHub answer. Recheck this rule under the publication lease. - Rotate the native tool contract to replace retained sessions with an older tool list. - Add continuation, publication, revocation, and boundary tests. Update the access and feature maps. ## Verification - Red: the later GitHub assigned-tool search returned an empty list before the binding fix. - Red: the later GitHub read failed with no active conversation capability before the request-history fix. - Green: runtime MCP discovery, GitHub read/begin/submit review, and response publication pass for monitor and same-user board continuations. - Green: Slack history, retained access controls, revocation, and voice gateway continuation pass. - Green: 35 channel, failure-publication, legacy-origin, long-history, and provenance tests pass. Both review findings had failing tests before their fixes. - Green: 107 gateway, operation-credential, identity, and provenance tests pass. These cover both first-run CI failures without changing their tests. - Green: 122 runtime, discovery-policy, and native-session tests pass; 2 unrelated tests are skipped. - Map checks and whitespace checks pass. - Updated to master in `cfcaba22628772d7fe1fb32b57ff7031c7072781`. The sole conflict was two tests added at the same location. Both the continuation test and the publication 403/401 test are retained. - Current local verification: all 179 gateway, credential, requester-identity, access-policy, runtime-context, and session-resume tests passed. Repository typecheck and build passed. Map and whitespace checks passed. - The new review regression first returned no source for a same-user board follow-up over legacy wake metadata. After the repair, all 27 provenance and channel cases passed. The combined GitHub discovery/read/begin/submit path and revoked access are covered. - The broader local channel run passed 176 cases and failed four timing/setup cases: a 15-second Slack authorization timeout, webhook-verification polling, a Slack acknowledgement deadline, and Discord setup polling. All four passed in isolation without source, assertion, or deadline changes. The earlier failure record is retained; the exact load/order cause is not established. Full CI must pass on the latest commit. - Current head: `7b9a08dcbc7b8140b589742f136944270b0496bb`. Repository typecheck and build pass on this commit. Greptile completed on this exact commit with 5/5; all four review threads are resolved. The final-head [CI run](https://github.com/paperclipai/paperclip/actions/runs/39278921653) passed: 55 checks passed and 2 intentionally skipped, with no failures or pending checks. This includes the full sharded test matrix, browser tests, typecheck, build, and Runner verification. Results were delayed during the [GitHub Actions incident](https://www.githubstatus.com/incidents/8tzbpcnl75jp). An additional unsharded local full test run has not completed; the completed full-suite evidence is CI. - Earlier verification: the first PR commit failed two server suites. Those failures were fixed and verified before this update. The original 4 GiB sandbox could not complete full local typechecking or tests; the current checkout completed typechecking and build. The full test suite runs through the CI shards. No full local test-suite pass or live provider delivery is claimed. ## Risks - Request history can restore the wrong authority if its scope is too broad. The resolver checks each accepted parent against the task, company, agent, and requester. Tests cover invalid and cyclic history. - The native contract change starts a new provider session for older contracts. - No database migration or public API schema changes are required. Each tool call still checks live access. ## Model Used - OpenAI Codex, GPT-6 family. The runtime does not expose the exact deployment model ID or context-window size. The agent used reasoning, repository tools, code edits, and local test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-19 11:20:39 -05:00
DottaandPaperclip	40ecacff4c	fix(voice): accept authenticated Cloud callers (#16254) ... ## Thinking Path > - Paperclip manages agent work through company-scoped tasks. > - Speko voice connections bind a caller to that work. > - Private voice routes must recognize an authenticated human caller. > - Cloud sign-in uses a trusted Cloud actor, rather than a local session actor. > - The voice caller check rejected that existing authentication type. > - This pull request accepts authenticated Cloud callers and retains the current access checks. > - Cloud users can load call history and start private voice sessions. ## Linked Issues or Issue Description Refs #15727, #15728, #15729. **What happened?** A signed-in Cloud user sees “Sign in to start a private voice conversation” when the Speko connection loads call activity. The same check prevents the user from starting a private voice session. **Expected behavior** The trusted Cloud user identity should have the same voice caller access as a local authenticated session. The current company and user identity checks must still apply. **Steps to reproduce** 1. Sign in to a Cloud workspace with experimental chat connectors enabled. 2. Connect a Speko persona to an existing agent. 3. Open its call activity, or start a browser voice session. 4. Observe the sign-in rejection despite the authenticated workspace session. **Paperclip version or commit** Reproduced on the voice route in master at b8ffeb973a. **Deployment mode** Paperclip Cloud with trusted tenant authentication. ## What Changed - Accept `cloud_tenant` and `session` actors with a user ID as private voice callers. - Use current company membership for Cloud owners, including owners with temporary administrator access. Retain local administrator authority, company isolation, and credential-source restrictions. - Add route regressions for call history, session creation, administrator authority, missing identities, company boundaries, and rejected caller types. - Add Cloud authentication qualification steps to the Speko runbook. ## Verification - Before the fix: 2 Cloud regressions failed; 9 boundary and local-user tests passed. - After the fix: all 100 voice caller and real-database persistence regressions passed. The Cloud-owner route test covers history, session creation, persisted caller authority, and membership revocation. - Repository typecheck and build passed with the repository’s pinned dependencies. - All current-head CI gates passed. Greptile reviewed f84fee2605 at 5/5 with no remaining findings. The repaired local full repository test run is still in progress. The first local full-suite attempt failed on missing native database library aliases; those aliases were repaired only in the isolated test dependency. No skipped run is counted as database proof. - On Cloud, verify that signed-in call activity loads and private session creation reaches media setup. Real conversation and spoken playback acceptance remain separate. ## Risks The change accepts an existing trusted Cloud actor type. Authentication still runs in the current middleware. Raw Cloud headers, board API keys, MCP OAuth, Cloud control credentials, and agent callers cannot establish private voice caller authority. No migration, UI change, credential, transcript, or lockfile is included. ## Model Used OpenAI Codex, GPT-6 family, with tool use and code execution. The exact serving variant and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-13 21:39:56 -05:00
DottaandPaperclip	3ec7309d91	feat(voice): add experimental Speko task connection runtime (#15727) ... First PR: runtime. UI: #15728. Qualification: #15729. ## Thinking Path > - Paperclip manages agent work through tasks and governed connections. > - Chat connections already bind external conversations to those tasks. > - Speko handles voice and telephony, while the selected Paperclip agent does the work. > - A call needs durable session authority, ordered follow-ups, and safe result delivery. > - This PR adds an explicit voice transport to the existing connection runtime. > - The UI and qualification suites follow in linked PRs. Speko stays experimental. ## Linked Issues or Issue Description **Problem or motivation** Phone and browser voice conversations cannot use the chat connection task lifecycle. A separate voice runtime is needed to preserve task authority, retries, and reply history. **Proposed solution** Add company-scoped voice sessions, signed callbacks, durable replies, inbound intake, and an agent-facing outbound-call tool. Reuse existing tasks, queues, secrets, and publications. **Roadmap alignment** This extends the experimental chat connector entry points under Agent Chat and Connected Apps. The project owner requested this feature. Refs #11418 for the original provider catalog proposal. This implementation uses the newer chat connection infrastructure rather than an MCP catalog-only integration. Refs #15721 for remote OpenCode startup. ## What Changed - Add additive voice schema, shared contracts, session APIs, and provider callbacks. - Bind browser, inbound, and outbound sessions to an endpoint, task, and selected agent. - Add low-trust guest intake in an explicitly selected sandbox and keep ordinary approval rules. - Submit task work through session-scoped tools and push approved replies through Speko POST /v1/calls/{id}/messages. - Distinguish provider acceptance from confirmed spoken playback. Reconcile reports without downloading recordings. - Add protocol, migration, persistence, provider, and delivery-diagnostic tests. ## Verification - Master 3389bea708 is integrated. Database/shared builds, server typecheck, migration upgrade/replay checks, and 46 communication-guidance/provider/protocol tests passed. - The final persistence suite passed 85 real-database tests. Five focused native continuation/authorization regressions passed. Five callback ownership/rotation tests passed after the last runtime repair. - Review repairs cover reconnect ownership, current caller authority, duplicate delivery, multipart answers, task-page answers, guest question handling, and saved sandbox recovery. - A real OpenCode/OpenRouter DeepSeek task ran in Daytona and saved a correct disk-space answer. This did not establish live-call playback. - Full local typecheck/build passed before the final repairs; focused typechecks passed after them. Latest-head CI runs the repository gates. - The separate integrated browser qualification passed two journeys and failed browser answer retrieval. The feature remains experimental. The report in #15729 records this failure and outstanding real-provider acceptance. - No lockfiles are included. Speko migrations follow master as 0324–0330. ## Risks - Voice sessions add migrations and provider-dependent lifecycle behavior. Migrations preserve existing chat and AI transports. - Callback origins must be reachable HTTPS. Provider keys and signing secrets remain vaulted. - Unverified callers receive task-scoped low-trust authority. Caller ID does not grant private task access. - Outbound telephony incurs provider charges. Provider acceptance is not proof that the caller heard an answer. - Keep the feature behind the existing experimental chat setting until the real browser, inbound, outbound, and delayed-result acceptance matrix passes. ## Model Used OpenAI Codex, GPT-6, with tool use and code execution. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-09 21:54:11 -05:00