DottaandPaperclip
8049fbf51d
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Voice calls retain reports and project guest transcripts into intake tasks. > - Chat maintenance repeatedly scans recent calls even when no report is due. > - Empty scans keep the tenant database active. > - Existing session and report rows already contain the work and its deadlines. > - This pull request moves report refresh to the shared coordinator and removes the final voice maintenance lane. > - Idle voice configuration and old call history no longer cause report scans. ## Linked Issues or Issue Description Refs #17583, #17580, #17578, #17576. This is slice 25 of the recurring-server sleep stack. It now follows merged #17583 and targets master. **What existing behavior does this improve?** Voice call-report refresh and guest transcript projection. **Current behavior** The chat interval scans terminal calls for reports. It repeats the scan even when there are no eligible calls or all saved deadlines are in the future. **Proposed behavior** Committed session transitions and signed callbacks wake a scoped worker. Startup reconstructs existing rows. Later refreshes use saved deadlines. The worker becomes quiet after all eligible windows expire. **Reason and benefit** Stop empty database scans without losing late transcript and billing updates. Keep the existing 24-hour refresh policy, provider response ordering, and guest-task trust restrictions. ## What Changed - Add a voice-owned report worker using the existing scoped deadline helper. No queue table or field is added; the existing session table receives a report-window index. - Save each attempt and its retry before provider I/O. Compare that ownership on settlement so an older response cannot overwrite a later callback or claim. - Preserve available reports when a response is incomplete or stale. Preserve their finite refresh window. - Route voice-change and reply call-ended writes through transaction-aware report signals. - Preserve guest transcript locks and use the observed document revision when updating an existing transcript. - Register the worker in app startup. Remove voice reconciliation from the chat interval. Keep explicit one-shot diagnostics. - Document producer scope, lifecycle behavior, tests, and external wake limits. ## Verification Published head: `758cfe15c3e6283d38ba512c8416ccce4d200862`. The follow-up after `c28de398` changes only two documentation files; the verified production, migration and test source is unchanged. It normally merges landed retention/master `980775e0e956eecdc28c73c9b4aca0407ad45e1a`. The 20 owned paths remain the complete slice, including officially regenerated migration 0397, its snapshot and required pruning of snapshot 0392. The non-database reviewed patch was preserved during ancestry integration; the only subsequent source edit makes one existing report test’s provider responses call-specific. All production report behavior and unrelated master changes are preserved. No shared worker, workflow or lockfile change belongs to reports. All 58 current-head checks are now clean, including the previously failing approval shard; exact-head Greptile is 5/5 with no new actionable issues. Ponytail’s current turn stopped without a verdict, an authorized nonblocking state. The prior conditional documentation requirement was fixed; old CI failures remain recorded below. - **Prior-head evidence at `9bf20467df`: 303 distinct targeted cases passed**, not rerun wholesale for this integration: all 132 voice integration cases; 170 cases across 11 complete report, retention, cleanup, recovery, company-removal, shared-worker and startup/hostname suites; and one complete snapshot/schema drift case. This includes 15 report, 13 retention and 14 cleanup lifecycle cases. Voice integration and the four other database owners use separate initially empty databases and migrate themselves. Report cases use their dedicated database after the setup repair. - Five real connection saves over 205 sessions with saved future report deadlines selected 1,025 session rows before the notification repair. They now select zero and preserve the original deadline. The report notification helper accepts only exact session scopes. Configuration, authority and connection hints retain their existing live recovery/reply/cleanup behavior but do not rebuild report history. Real session terminal writes, callback hints, report claims and receipts still reactivate reports. - Database/server typechecks and builds, migration numbering/safety, and module-boundary checks pass. Final server types/build also pass after integrating the retention review repairs. The parent 205-admission, five-configuration-save test still observes zero retained-history rows and preserves its deadline. - The old eager reader selected all 105 saved-report headers before yielding and failed the 100-row budget. Registered reconstruction now uses the existing paged contract and yields at 100 rows per scope refresh. A partial scan can restart on the same owner and reconstruct every saved deadline. At due time all 105 report IDs are fetched exactly once. Future deadlines remain quiet between actual retries. - The company-fairness case places a small company deterministically on page two and requires it in the next 25-work batch after discovery, with all 105 reports processed before the failed provider's retry. Guarded mutations catch a missing cursor reset and bypassed company rotation. Original source was restored before final acceptance. - At prior head `9bf20467df`, migration 0396 was generated with `pnpm db:generate` on the parent 0395 lineage. It creates the partial `(ended_at, id)` report-window index concurrently and replayably through the existing migration runner. All 394 historical SQL files and the journal prefix are unchanged; snapshots 0392 through 0396 remain. Fresh application migration and two replay invocations succeed with a valid index. The real plan case retains 20,001 old sessions, 205 recent sessions, precision-safe keysets, and fewer than 1,000 rows removed by filtering on each page. - Existing real hangup/callback/reply-error producers, callback ownership fencing, monotonic report results, saved retry before provider contact, guest transcript projection after a document lock, transaction rollback/nesting, drain and shutdown remain covered. No table, field or provider protocol is added. Available reports retain the existing finite 24-hour refresh policy. - [Current-parent acceptance and failure ledger](https://github.com/paperclipai/paperclip/blob/9bf20467dfd50b81848e1c8dd7ddc905d206418f/doc/plans/2026-10-25-voice-call-report-sleep-audit.md#current-parent-reconstruction-and-migration-acceptance-october-28) records the failed fresh fixture, eager read, fairness-clock attempts, guards, commands, counts and limits. Older 174/138-case and workspace-wide results belong to their stated heads. No new complete local repository suite, live provider, Cloud wake, host/database suspension or savings claim is made. - [Report configuration-scope repair](https://github.com/paperclipai/paperclip/blob/9bf20467dfd50b81848e1c8dd7ddc905d206418f/doc/plans/2026-10-25-voice-call-report-sleep-audit.md#configuration-scope-review-repair-october-28) records the current 303 cases and measured rescan repair. The prior `80207ba6bb` head had Ponytail 5/5; that review is stale after this change. Fresh exact-head Greptile 5/5 and all CI remain required. Actual failing Ponytail findings must be fixed; a stopped or paused turn without a verdict is an authorized nonblocker. Active no-verdict review waits are capped at 30 minutes from the original request/start (25-minute monitor cutoff). Older scores and parent checks are not inferred as current passes. - **Current-master verification: 185 distinct cases pass**: final complete voice integration 132, complete report/cleanup/retention 15+24+13, and snapshot/schema drift 1. Each database owner used its own empty fixture database. The first combined run had 183 passing cases and one report-test failure; it is preserved in the audit. A deliberately earlier-ended competing call reproduces the old one-shot mock consuming the intended response. The corrected test selects responses by provider session ID, verifies independent costs for both calls, then rejects stale target data. The isolated repair and full 132-case suite pass; production behavior and timeouts are unchanged. These 185 cases are not added to the older 303 count. - **Current migration lineage:** official `pnpm db:generate` after actual master creates `0397_known_madrox.sql`; the reviewed concurrent/idempotent report-index SQL is retained byte-for-byte. All 395 existing SQL files and journal entries are unchanged; snapshots 0393–0397 remain and the new snapshot links parent 0396. The application migrator boots a fresh database, repairs a deliberately invalid index during two concurrent replay invocations, records one history row, and replays again successfully. Database build, server build/typecheck, migration checks and module boundaries pass. [Full current-master acceptance and failure ledger](https://github.com/paperclipai/paperclip/blob/c28de398fef074f650eff532952ab6f5563739a8/doc/plans/2026-10-25-voice-call-report-sleep-audit.md#landed-master-migration-and-acceptance-october-28). - **Review follow-up and preserved CI timeout:** Ponytail’s prior `c28de398` review was 4/5 advisory, with no must-fix production defect and a request to fix the stale design-doc migration reference before shipping. That document now names migration 0397 and links current 185-case acceptance; older 302/303-case evidence is explicitly historical. The optional suggestions about one-shot diagnostic inventory cost and inherited test-clock helpers remain disclosed. CI run `40703490340` failed the unchanged approval-queue backoff test on its existing 15-second timeout and then an activity-log FK in teardown (2,097 pass/one fail/one skip in that shard). The complete unchanged 28-case approval suite passes on a new isolated database; the exact CI timing cause remains unproven. No timeout, workflow or production logic was changed, and no old-head rerun or failure waiver is claimed. This documentation head requires fresh CI and exact-head Greptile. [Review and timeout ledger](https://github.com/paperclipai/paperclip/blob/758cfe15c3e6283d38ba512c8416ccce4d200862/doc/plans/2026-10-25-voice-call-report-sleep-audit.md#review-documentation-and-preserved-remote-timeout-october-28). ## Risks - Process-local commit signals do not observe independent SQL writers or another server process. - Available reports still refresh for 24 hours. The provider's complete flag does not promise immutable final data. - Transcript projection retries remain within the existing report window. This change does not add indefinite projection repair. - One-shot diagnostic `reconcile(limit)` inventories the eligible window before slicing; it remains a manual diagnostic cost, not recurring idle work. Some inherited test helpers advance fake time during assertion retries, an optional reliability follow-up noted by review. - A local deadline cannot wake a stopped host. Cloud alarms and sleep eligibility remain unchanged. No deployed sleep or cost saving is claimed. - This PR depends on the preceding stack. Final master reconciliation and remote review gates remain required. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, code editing, and local test execution. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
6.6 KiB
Voice call reports and idle work
Voice report reconciliation uses the shared delivery coordinator and scheduler.
It no longer runs from the chat maintenance interval. Existing
chat_voice_sessions and chat_voice_reports rows own its durable state; this
change adds no queue table, provider endpoint, or independent timer. A concurrent
partial index supports bounded reconstruction of the existing report window.
Trigger and deadline
A terminal session with a provider call ID remains eligible for report refresh
for 24 hours after ended_at, inclusive. A missing report is due immediately.
An existing report uses its saved next_check_at. The retry starts at 60 seconds
and doubles to one hour, as before. Available reports remain refreshable during
this finite window: provider availability is not a promise that transcript or
billing data can no longer change.
Session transitions and signed callbacks notify the report worker before their
writes, inside the owning database transaction. The reply sender's call_ended
result does the same. Healthy active-session observations do not activate report
work. Nested transactions wait for the outer commit; rollback does not publish
new work. The existing work-signals protocol owns uncertain outcomes.
Startup reconstructs eligible sessions in pages ordered by (ended_at, id),
yielding after at most 100 rows per scope refresh. Re-registering the same owner
resets partial scan cursors. The explicit one-shot inventory still loads every
page when requested.
A partial index selects terminal sessions with provider IDs. The query uses
its ended-at range to restrict reads to the current 24-hour window. The cursor retains PostgreSQL timestamp precision. Committed report hints reload the exact changed session. Configuration,
authority and connection metadata changes do not alter report eligibility or
saved retries, so they do not wake report reconstruction. Each cached entry carries the
saved report deadline or the first millisecond after the refresh window ends,
whichever comes first. Submillisecond PostgreSQL timestamps round in the safe
direction. Expiring the window removes the cached obligation without a provider
request. Configured idle endpoints, active calls, old terminal history, and
sessions without provider IDs do not cause recurring report SQL or requests.
Claims, responses, and projection
The worker locks and rechecks the session and report, then saves its attempt and next retry before contacting the provider. Network I/O happens outside that transaction. Settlement compares the saved attempt and retry deadline. A newer claim or callback that moves the deadline invalidates the older response. Incomplete or older responses cannot overwrite a retained available report. Eight failures can mark a pending report unavailable; failure cannot downgrade an already available report.
Guest transcripts remain confined to the call's quarantined intake task. The worker reads the retained report, keeps the task's source-trust metadata, and uses the existing document revision contract. A document lock or conflicting edit leaves the saved report and retry intact. After the lock clears, the next successful refresh retries projection. The existing 24-hour eligibility policy still applies; this PR does not add indefinite transcript repair or extend it.
Provider credentials, call-report parsing, and delivery diagnostics remain in the voice integration. The shared scheduler receives only normalized work and deadlines. No provider-specific policy is added to app startup.
Lifecycle and limitations
The shared scoped worker coalesces notifications, processes fair company batches, and retains dirty scopes across drain or failed reads. Shutdown joins in-flight work. Restart reconstructs the persisted retry instead of immediately repeating the provider GET. The explicit one-shot voice reconciliation method remains for diagnostics; production maintenance does not call it periodically.
The supported deployment has one active server process writing voice session and report state through its owning database client. Active-active servers and out-of-band SQL writers are not supported by this worker. Process-local signals do not reach another process; such a deployment needs a durable cross-process wake contract before removing its reconciliation backstop. This PR neither adds that transport nor asserts that startup reconstruction covers concurrent writers. This is worker quiet, not a claim of physical host sleep. Cloud alarm coverage and conservative sleep eligibility are unchanged; a local deadline cannot wake stopped compute.
Verification
voice-call-reports.test.ts exercises real PostgreSQL claims, configured-idle
quiet, finite expiry, timestamp precision, retries and restart, committed writes,
rollback and nesting, drain/resume, callback races, concurrent claims, backlog
fairness, result ordering, and shutdown. The full voice integration suite checks
real hangup/callback/reply-error producers and transcript projection after a lock.
The app wiring suite protects removal of the voice maintenance lane.
The audit and evidence ledger is voice call-report scheduling.
The historical parent acceptance records the earlier 302 distinct passing cases, the bounded startup and same-owner restart regressions, deterministic company fairness, generated migration 0396 and the preserved failures. The report owner keeps its finite provider refresh obligation; this is not an unconditional idle scan or proof of deployed suspension.
The subsequent configuration-scope repair
records the earlier 303-case acceptance at 9bf20467df. Five real connection saves over 205 saved
reports select zero session rows, after reproducing 1,025 selected rows before
the repair. The notification helper accepts only an exact session scope.
The landed-master acceptance
records the current 185-case acceptance and generated migration
0397_known_madrox.sql, after landed cleanup migration 0396. It preserves all
existing SQL and journal entries, retains snapshots 0393 through 0397, and checks
fresh application migration plus concurrent replay and invalid-index recovery.
The older 302/303-case sections above are historical evidence, not the current
migration filename or a claim that those full groups were rerun on this head.