DottaandPaperclip
e93732c3d4
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Phone calls receive approved answers from the existing chat publication queue. > - The chat maintenance timer checks for voice replies every second, including when no reply is ready. > - A reply can be saved before its publication commits. An early wake alone can miss that final transition. > - Existing session, reply, publication, and action rows already record delivery and recovery work. > - This pull request uses committed changes and saved deadlines to drive phone reply delivery. > - Empty reply queues become quiet while authorization, ordering, and uncertain-send protection remain in place. ## Linked Issues or Issue Description Stacked on #17576. Related delivery infrastructure: #15625. Related voice connection work: #11418. **What existing behavior does this improve?** Background phone reply delivery and recovery of abandoned dispatch intents. **Current behavior** Chat maintenance calls voice reply delivery every second. It also scans old dispatch intents. Publication transport starts a direct push before the final publication commit. The timer masks that timing gap. **Proposed behavior** Committed publication, reply, session, and configuration changes wake the shared coordinator. Ready replies drain in bounded passes. Saved rate-limit and dispatch-expiry deadlines retain recovery. An empty queue or a permanent delivery barrier has no recurring reply scan. **Reason and benefit** Remove recurring SQL from idle phone reply delivery without dropping replies, replaying uncertain sends, or adding a queue table. **Breaking changes** No API or schema change. Explicit manual recovery remains available. Inbound admission cleanup, history retention, and reports remain on their current paths for separate migrations. ## What Changed - Extract ordered reply state and multipart policy within the voice owner. Retain current authorization and atomic dispatch claims. - Register a scoped reply worker with startup reconstruction, fair work batches, failure backoff, drain/resume handling and shutdown ownership. - Extend the existing shared dispatcher with normalized partial-page results to preserve the bounded parent recovery contract. Existing complete-array consumers retain their calls. - Signal before relevant writes in their existing transactions. Both normal publication completion and operator resolution wake the actual conversation after commit. - Bound reply inventory to 100 total headers per refresh. Preserve partial scans, concurrent changes and newer denied-call restoration obligations across reconstruction. - Track abandoned dispatch actions independently of live calls. Quarantine expired intents without resending them. - Use existing active-call authorization checks to reactivate replies that were previously denied. Healthy call inspections do not rescan empty reply queues. - Remove the periodic reply scan and the early direct publication push. Keep provider requests outside database transactions. ## Verification Published head: `1435cab10be3f95c38ab6672d2a890ae49e9d300`. Voice recovery #17576 has landed at master `e1ba77f06771a27114b3999004b0ca5e3c172dd4`. A normal merge reconciles squash ancestry while preserving the complete reviewed 31-file reply patch byte for byte and all unrelated master changes. The PR now targets master; fresh exact-head CI and Greptile are required. The prior 449-case/typecheck/build results below cover the identical owned source, not a new full-suite run. - **449 distinct targeted cases pass**: 111 complete voice-store/reply; 19 complete parent recovery; and 319 complete tool-gateway, secret-proposal, shared dispatcher/coordinator/batching/startup/hostname cases. Production server typecheck and build pass. The policy checker and its Node tests pass. Later edits record evidence only. - Two real normal-publication/operator-resolution tests seed 105 unrelated calls and 105 future dispatches. Five publications selected **1,065/1,060 inventory rows before repair**; both paths now pass the same **125-row total budget** and deliver every answer once. This counts selected SQL results, not physical tuple visits or host savings. - Publication commits now signal their actual conversation. The voice reader returns at most 100 total session/action headers per refresh, preserves phase/cursor state and retries, and resets on worker registration. Current-state checks and dispatch claims still own provider effects. The 105-action case measures the page cap, restarts the same owner during reconstruction, waits at saved future deadlines and quarantines every abandoned action without resending. - A real 403 fixture proves that an older broad scan cannot erase a denied call first observed behind its cursor. Its owner callback still emits the committed restoration wake after scan completion. The existing actual-service authority-restoration and final-publication timing tests remain in the full suite. They are not replaced by direct helper calls. - Guarded removal of conversation scoping, cursor reset or the denial revision fence fails the intended budget, 100-versus-5 restart or missing committed-wake assertion; source is restored. An initial shadowed fixture name and an inadequate due-action reset probe are retained as failed attempts in the [current audit](https://github.com/paperclipai/paperclip/blob/d2c0a7de8c3e9cac8382b1268f3f5d883a2067b3/doc/plans/2026-10-25-voice-reply-delivery-sleep-audit.md#publication-inventory-and-paging-acceptance-october-28). - Parent recovery still passes paging, restart, behind-cursor insertion, feature changes, actual disablement, quiet health observations, current authority, exact-scope failure fences and wildcard-failure isolation through the shared worker. Four existing array-based worker consumers retain their API and pass their full suites. - The two voice fixtures self-migrate separate empty databases. Legacy tool-review/secrets suites use separate migrated clones. Earlier full-local attempts remain limited by embedded PostgreSQL setup and are not passing full-matrix claims. Older counts apply only to their stated revisions. Fresh exact-head Greptile, Ponytail if present, security and all CI remain merge gates. No live-provider, multi-writer, external-wake, deployed-sleep or savings claim is made. ## Risks - Missing a supported producer can strand work. Tests cover real enqueue, retrieval, normal publication, and operator publication paths. Startup reconstructs existing records. - Signals cover the process-local database owner. Independent SQL writers or other replicas need their own wake contract. - Authority restoration relies on the existing live-call recovery check. This dependency must be preserved if that owner changes later. - A provider acceptance receipt does not prove speech. Unknown outcomes remain blocked from automatic resend. - This change does not extend Cloud alarms, allow sleep during live calls, or prove deployed suspension or cost savings. ## Model Used OpenAI GPT-6 via Codex, with repository inspection, reasoning, code execution and test tools. The runtime does not expose a more specific deployment identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used with the available version and capability details - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the relevant issue template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket identifier - [x] I have run the selected tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
9.7 KiB
Voice reply delivery without idle polling
Phone replies wake from committed publication, reply, session, and configuration
changes. The application registers the worker with the shared delivery coordinator.
The one-second chat maintenance lane no longer calls pushReplies. Inbound voice
admission, report reconciliation, and other chat duties remain separate migrations.
Durable truth and delivery boundaries
The existing voice sessions, voice replies, chat publications, and chat actions remain authoritative. No table or migration is added. The worker reconstructs active phone sessions and outstanding dispatch intents at startup. Notifications are scoped hints, never permission to replay a provider action.
A reply inserted while its publication is streaming is not yet deliverable. Both normal publication completion and operator resolution signal the endpoint before committing its published state. Reply insertion and authorized question retrieval signal their session before writing. The existing transaction boundary holds notifications until outer commit and reconciles uncertain outcomes.
The dispatcher retains current authorization, session locking, ordered reply and multipart barriers, and its atomic action claim. Provider calls remain outside SQL transactions. An accepted receipt is distinct from spoken playback. Unknown outcomes never authorize resending the answer or appending its remaining parts.
Deadlines and disarming
- A ready ordered reply is immediate work; a bounded pass drains up to eight parts.
- A definitive rate limit retains its saved retry deadline.
- A dispatch without a durable receipt retains its existing strict 30-second uncertainty deadline. These actions are tracked independently of live sessions, so ending a call cannot strand its abandoned dispatch intent.
- Database or provider-path failures retain per-entry backoff. Scoped inventory failures retry their scope; healthy companies retain fair processing access.
- No ready reply, a presented question awaiting retrieval, accepted history, or an unknown/failed outcome leaves no recurring reply scan.
Live-call recovery already checks current authority before provider inspection. Reply delivery remembers sessions denied by its authorization check. A successful subsequent live-call authorization signals only those blocked sessions. This preserves restoration after membership/task authority changes without adding notifications to every authorization table or scanning healthy empty reply queues on every live-call inspection. Startup reconstructs this in-memory state from existing sessions. This dependency must remain explicit if active-call recovery is changed later; a notification alone never bypasses reauthorization.
Lifecycle and limits
The coordinator coalesces changes, serializes work, preserves dirty scopes across
failure and drain, and stops work on shutdown. Startup loads identities in pages;
processing uses fair bounded batches. Manual pushReplies and one-shot reconcile
remain available for explicit diagnostics/tests and must not become periodic lanes.
Signals cover the supported process-local database owner. Independent SQL writers or other processes require their own notification/reconciliation contract. These changes do not extend Cloud wake coverage, permit sleep during a live call, or prove deployed instance/database suspension.
Both live-session recovery and reply delivery use scoped-deadline-work.ts for
coalescing, fair batching, scope reconstruction, retries and drain retention.
Their adapters keep domain state and provider operations in the voice owner.
The production-wiring regression covers authority returning before live recovery
observes the denial; recovery still ends calls when their authority is absent.
Current-parent paging contract acceptance, October 27
This slice normally integrates bounded voice recovery #17576 at 712f3862b5,
including receiver 6e88ca0201 and the landed foundations. The original 30-file
reply scope adds one shared contract path, scoped-deadline-work.ts. No database
file changes. The existing helper could represent only a complete inventory;
using that contract for the repaired parent would erase its bounded-pass guarantee.
The real 205-call parent regression fails before the integration repair: startup executes three inventory queries (100, 100, 5 rows) instead of one 100-row query. The shared helper now accepts either its existing complete array or a normalized page result. It retains page generations and seen identities, keeps existing entries until reconstruction completes, limits dirty scopes per pass, and replays writes received during reconstruction. It owns no SQL or voice/provider policy. Recovery delegates to its owner refresh and resets partial owner state on worker registration. Existing complete-array consumers keep their public calls. Connection observations retain the parent's separate receiver/command notification path; actual configuration wakes both voice owners.
All 445 distinct targeted cases pass: 18 complete parent recovery lifecycle cases, 108 complete voice-store/reply integrations, 224 complete tool-gateway and secret-proposal cases, and 95 shared dispatcher/coordinator/batching/startup/ hostname cases. The helper's single existing test also ran in the initial 19-case green group; it is counted only in the 95-case group. The 224 cases exercise the four other production consumers of the shared helper, including complete-array empty/expiry/recovery/projection, transaction and drain behavior. Server production typecheck and server build pass. The voice fixtures self-migrate separate initially empty databases. The two other database suites use separate migrated clones. Expected injected provider/wakeup failures and listener warnings remain in logs. Only documentation edits follow these passing production checks.
This acceptance preserves the parent's startup paging, restart, behind-cursor insert, concurrent feature change, health-only quiet, true disablement, current authority, failure, drain and shutdown contracts. Existing real publication, operator resolution, uncertain send, multipart/question ordering, authority restoration and 105-abandoned-dispatch tests remain in the 108-case suite. Fresh exact-head reviews and all CI remain required. No full local matrix, live provider acceptance, Cloud wake, deployed suspension or savings is claimed.
Publication inventory and paging acceptance, October 28
The security review correctly found an endpoint-wide read per publication. Two real publication-path regressions seed 105 unrelated active phone calls and 105 future dispatch intents on the same endpoint. Five normal publications select 1,065 inventory rows; five operator resolutions select 1,060. Both exceed the 125-row total budget. The corrected paths pass that same budget and deliver each answer once. These measurements count selected row results from executed queries, not physical PostgreSQL tuple visits or a deployment cost benchmark.
Normal publication completion and operator resolution now notify their actual conversation inside the existing transaction. The voice owner handles that normalized conversation scope. Its reader returns at most 100 total session and action headers per refresh, with a saved phase/cursor for the next pass. Startup, configuration changes and abandoned actions retain finite reconstruction. The shared dispatcher retains generations and observed IDs across pages; it also preserves the parent wildcard-failure isolation. SQL and provider decisions stay inside the voice owner. No schema, table, provider-specific shared branch or per-route recovery wrapper is introduced.
The abandoned-dispatch case now proves a 100-row startup cap, restart of the same owner before its scan completes, restoration of all 105 saved deadlines, no SQL before the deadline, and quarantine without resending. A separate owner-level real-database case reproduces a new denied call behind an older scan cursor: completion of the old scan must not erase the later restoration obligation. Denial revisions fence that cleanup. The case confirms a real 403, then observes the committed wake. The existing actual-service authority-restoration case still verifies the complete production callback path; the new cache test is not a replacement for that integration coverage.
Guarded mutations broaden publication hints back to endpoints, omit owner reset, and omit the denial revision fence. All fail the corresponding row-budget, 100-versus-5 restart, or missing committed-wake assertion, with source restored. An earlier reset probe using already-due actions did not detect the missing reset because its own action writes rediscovered the backlog. That incomplete probe is retained as such; future deadlines make the strengthened regression deterministic. The first amplification fixture also used a shadowed publication function name; that fixture failure preceded the actual 1,065/1,060-row baseline.
The final branch normally integrates voice parent ad79c7e805. Its original
30-file scope plus the existing shared dispatcher remains 31 files, with no DB
changes. All 449 distinct targeted cases pass: 111 complete voice-store/reply,
19 complete parent recovery, and 319 complete shared/legacy-consumer/startup cases.
Both voice fixtures self-migrate separate initially empty databases; the two
legacy DB suites use separate migrated clones. Production server typecheck and
build pass. The current policy checker and its Node tests pass. Only evidence
edits follow these checks. Older 445/786-case totals belong to their stated source
revisions and are not added here. Current exact-head reviews and all CI remain
required. No full local matrix, live provider, external wake, deployed suspension,
physical query-cost or savings result is claimed.