mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-29 12:59:13 +01:00
fix(server): wake interaction continuations from committed responses (#17653)
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/204/changes/paperclip/204 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip coordinates agent work and human approvals. > - Human decisions already persist the continuation action that resumes an agent. > - A response can commit before its callback or publication finishes. > - Periodic maintenance repeatedly discovers those rows, including when idle. > - Committed writes should wake the existing owner; saved retries and claims supply real deadlines. > - Startup reconstructs durable work, and settled history becomes quiet. ## Linked Issues or Issue Description Refs #17651, which is merged. This existing continuation slice targets master and removes its duty from periodic chat maintenance. **Problem:** A saved `interaction_wakeup` action can survive a committed human response whose callback crashes. Without producer coverage, removing maintenance strands the continuation. Repeated empty maintenance also keeps the database active. **Expected behavior:** The outer transaction's settlement wakes the continuation owner. It handles ready conversation publications, dispatches under existing authority, preserves interrupted/retry work, and becomes quiet after settlement. ## What Changed - Register continuation recovery with the shared scoped deadline coordinator. Reconstruct existing issued/processing actions and retain the existing retry and stale-claim deadlines. - Notify from both enqueue producers and owner-local claim/settlement writes. Preserve outer commit/rollback and avoid moving provider or heartbeat calls into database transactions. - Give the existing publication queue an optional company/endpoint/conversation scope. The continuation owner gives ready cards in that conversation a bounded delivery opportunity before claiming the action; FIFO, capacity, ambiguity and provider authority remain with the queue. - Carry admission/cancellation through publication preparation and check before every new send. Join in-flight requests and retain pending actions/publications through drain, standby, shutdown and resume. - Remove only recurring continuation recovery; keep the manual API. Add no table. A concurrent partial index on existing pending continuation actions and matching company/id keyset paging prevent retained history from being scanned. Parent #17651's indexed confirmation discovery and current master are retained. ## Verification Published head: `a1d16f84b3fae361d5d648670a5964486370c3a3`, on actual landed terminal-confirmation parent `ee67d2ece9bda157dfad9bdf25175b1d004ed341`. - **75 distinct current cases pass:** 32 selected real PostgreSQL chat cases (14 continuation lifecycle/producer/stop cases, one new continuation query-cost case, and 17 parent confirmation/query-cost cases; 1,671 skipped), and all 43 cases in complete interaction-publication and app/startup suites. Database build, server typecheck/build, module boundaries and fresh migrations through 0399 pass. The original landed-parent matrix passed 74 cases before the additional query-cost repair; those are not added again. - The new real-SQL regression seeds 20,000 completed/unrelated actions and one pending processing claim. Before repair it examines 20,345 rows and hits 20,421 buffers. The final startup and company-scoped queries must examine at most two action rows and fewer than 20 action buffers; empty scoped recovery must also stay below 20 buffers. Planner settings are unchanged. Migration `0399_ambitious_onslaught.sql` is concurrent/idempotent and uses the existing top-level migration runner; prior SQL/journal entries and the five newest snapshots are preserved. - The matrix covers callback recovery, concurrent responses, malformed prefixes, more than one page, saved retries/stale claims, outer rollback/commit, deferred/future/paused publication states, restart, drain and stop before each new provider request. Quiet assertions inspect real selects/transactions across virtual idle time. - Normal integration `db5a84c0f8b73a0ce863f4d2fbaee0a115cf044f` preserves the complete 12-path owned binary patch and all 12 unrelated current-master paths, including managed Cloud sandbox changes. A verified semantic base resolves squash-ancestry conflicts; both normal parents and the exact tree are checked. The subsequent index/keyset repair is a deliberate additional change, tested separately; the final patch is not claimed identical to the original 12-file implementation. No force push or rebase occurs. - The same 74 pre-index cases passed on the earlier prepared and actual landed parents. The new query-cost regression failed before its repair, then passes on the repaired source. Earlier 57-case and broad acceptance retain their original revision scopes; repeated cases are not added to the current count. The audit-only follow-up leaves all owned server source/test hashes unchanged. - [The owner audit](https://github.com/paperclipai/paperclip/blob/a1d16f84b3fae361d5d648670a5964486370c3a3/doc/plans/2026-10-25-interaction-continuation-sleep-audit.md#acceptance-on-the-actual-landed-terminal-confirmation-parent) retains the original missing-signal regression, the skipped standalone publication suite, and the review's stop-before-send failures with their corrections. Skips and earlier failing revisions are not relabelled as current passes. - **Fresh exact-head CI and Greptile 5/5 without unresolved actionable findings remain required.** The former remote head's reviews are stale. No full local repository suite, live provider or deployed sleep result is claimed. ## Risks - Process-local transaction hints cover supported application writers and shared pools. Independent SQL writers and other replicas do not gain a new wake guarantee. - The publication queue's existing FIFO, capacity, retry/ambiguity and authorization contracts still govern effects. In-flight provider requests can settle after Stop; the fence prevents a subsequent request from starting. - Missing a supported producer can defer recovery until another change or startup. Producer and rollback tests cover this slice's supported paths. - Worker quiet is separate from host/database suspension. External alarm coverage and Cloud sleep gates are unchanged. ## Model Used OpenAI Codex, using repository inspection, code execution and real PostgreSQL tests. The exact model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
287b04d72f
commit
063cfd4381
17 files changed
+936
-69
No files matched your search
@@ -2296,3 +2296,33 @@ owed cron occurrence; fixing that fixture clock passes all 28 plugin tests.
|
||||
Thirteen selected chat lifecycle cases pass after the retry-notice wait fix.
|
||||
Initial Greptile is 5/5 without findings. Fresh full acceptance and corrected-head
|
||||
remote gates remain required; see the slice-32 audit for all retained failures.
|
||||
|
||||
## Slice 32 accepted-head verification and slice 33 start
|
||||
|
||||
At `8b43e6222706491f2d8044af3bf86358085069af`, #17651 passes all
|
||||
53 remote checks and fresh Greptile 5/5, with no review threads. The corrected
|
||||
full chat/startup run passed all 1,512 tests on a fresh database in 463.63 seconds;
|
||||
final server typecheck also passed. Prior failed runs remain documented above.
|
||||
The PR remains unmerged. Latest fetched master is `67aa87a93b`; final stack
|
||||
reconciliation and older unresolved gates/reviews remain outstanding.
|
||||
|
||||
Slice 33 starts from that accepted head and owns interaction continuation
|
||||
recovery only. The owner audit is in
|
||||
`2026-10-25-interaction-continuation-sleep-audit.md`. GitHub history maintenance
|
||||
and host-sleep eligibility remain separate work.
|
||||
|
||||
Slice 33 now implements interaction continuation recovery with committed action
|
||||
signals, saved retry/claim deadlines, and a scoped call into the existing
|
||||
publication dispatcher before claiming continuation work. It preserves paused
|
||||
and future-retry publication behavior. Eleven new real-database cases pass,
|
||||
including outer rollback/commit, historical prefixes, and concurrent response
|
||||
commit. Existing Telegram acceptance also passes. Broad acceptance and fresh
|
||||
PR gates remain pending; no merge or full-stack completion is claimed.
|
||||
|
||||
Slice 33 is [#17653](https://github.com/paperclipai/paperclip/pull/17653), draft
|
||||
and unmerged. Initial broad acceptance passed 1,523 tests with 14 skipped;
|
||||
workspace types/build passed. Review found that publication preparation could
|
||||
start another send after drain/standby/shutdown. Three regressions reproduced
|
||||
it. Per-send admission/abort checks now pass all 14 new lifecycle cases, and the
|
||||
previously skipped publication suite plus startup guards pass 27 cases with the
|
||||
external test database. Fresh corrected-head verification remains required.
|
||||
@@ -0,0 +1,255 @@
|
||||
# Interaction continuation delivery migration
|
||||
|
||||
Date: 2026-10-25. Inspected parent implementation at aeaaed7b3e17e1377433165a38fef558acacb631 (#17651). This is an owner audit, not an implemented migration.
|
||||
|
||||
## Obligation
|
||||
|
||||
`chat-channels/interactions/continuations.ts` owns durable `chat_actions.kind = interaction_wakeup` delivery. The existing row already stores issue, interaction, assignee, actor provenance, source comment/run, and optional plan-review context. No table is needed.
|
||||
|
||||
Current selector: issued with null result is due now; issued with result is due at updatedAt + CONFIRMATION_WAKEUP_RETRY_BACKOFF_MS (currently 30,000ms); processing is reclaimable at updatedAt + 30,000ms. Claim repeats the predicate. A valid result marks processed; invalid payload marks failed; invalid current issue/interaction/assignee marks processed with an explicit no-longer-applicable code. A refused wake returns issued/deferred. Exceptions return issued/failed with an incremented attemptCount. Existing acceptance deliberately retries beyond five failures; do not silently cap/discard it.
|
||||
|
||||
The owner coalesces against an existing non-failed/non-cancelled/non-skipped `agent_wakeup_requests` row with interaction-specific idempotency. A different current durable agent recipient is accounted for as reassignment coalescing. It retains actor provenance, inbound comment lineage, forced fresh-session plan context, and source run/comment. These are not scheduler responsibilities.
|
||||
|
||||
## Actual producers
|
||||
|
||||
1. `enqueueTerminalIssueInteractionChatPublications` in `chat-interaction-publications.ts` creates issued actions. It is used inside interaction resolution transactions, native review retirement transactions, and callback race handling. The helper inserts continuation state before it iterates terminal publication inserts. Audit every caller's transaction ownership; do not emit a root post-write hint and assume those writes are atomic.
|
||||
2. `createTelegramConfirmationActions(...).settleTerminalConfirmationAction` inserts the same issued action in its settlement transaction. Its interaction-resolution publication insert is in that transaction too. It can lose insert ownership to the generic producer; onConflictDoNothing is intentional.
|
||||
3. The continuation worker's claims, deferrals, failures and completion mutate scheduling state. Explicit/manual recovery may run beside the registered worker. Preserve CAS and stale-claim deadlines across those paths and transaction failure.
|
||||
|
||||
Search found no other non-test insertion of interaction_wakeup. Native question answers have a separate question-delivery owner and the generic producer explicitly excludes answered ask_user_questions.
|
||||
|
||||
## Existing invocation and ordering
|
||||
|
||||
`processPublicationMaintenance` still calls `processPendingInteractionWakeups` after scheduling GitHub maintenance. The explicit publication queue calls it only when maintenance=true, after awaiting all publication tasks owned by that invocation. Production publication scheduling uses maintenance=false. Remove only the periodic continuation call; preserve the explicit/manual contract.
|
||||
|
||||
Do not merely start an independent action worker on insert: a continuation can become visible in the same commit as a still-pending resolution card. Existing Telegram acceptance asserts the provider card edit happens before the continuation wake. The current periodic path is not itself proof that this ordering is robust. Establish an explicit owner predicate/receipt for owed publication delivery, together with its terminal, cancelled, ambiguous, retry, removed-binding and no-publication cases. Waiting on publication completion must use its committed changes rather than repeatedly poll it. The owner must allow continuation when the prompt was never externally visible and no resolution publication is owed. Avoid turning permanent publication failure into indefinite loss of the user's accepted decision.
|
||||
|
||||
The shared `notifyChatPublicationWork(tx, optionalIssueScope)` already signals delivery changes, and terminal publication transitions often supply the issue scope. Audit all release/resolution writers before depending on that fanout. `onPublicationsSettled` currently stores only one callback used by publication refill; do not overwrite it with another consumer.
|
||||
|
||||
## Mechanism and verification still to choose
|
||||
|
||||
Reuse the shared coordinator/scheduler. Keep action eligibility, publication sequencing and dispatch in the continuation owner; app startup only registers it. Scope committed hints by issue/company (or exact action when useful), restore existing issued/processing debt at startup, retain work during drain and changes during a sweep, and register actual existing claim/retry deadlines. Do not scan completed history periodically.
|
||||
|
||||
Tests should use the real accept/reject/board producers and preserve existing provider edit-before-wake, duplicate durable request, reassignment coalescing, deferred delivery, repeated failures and plan/source provenance assertions. Add idle first-write, no recurring SQL after settlement, rollback/outer commit, restart, failed attempt retry, paused/drain resume, bounded progress, and changed publication state wake coverage. Inspect current code again before implementation.
|
||||
|
||||
## Independent scopes
|
||||
|
||||
GitHub history maintenance remains another PR. Host sleep proof currently fails closed on any retained chat action/publication/delivery rows; this worker migration does not relax that gate or claim external Cloud alarm coverage. All earlier stack findings and final-master checks remain required.
|
||||
|
||||
## Ordering refinement from the current queue
|
||||
|
||||
The explicit publication queue only attempts eligible, due work, then wakes continuations. A retry with a future deadline, paused/attention endpoint, explicit-publication endpoint, or unrelated FIFO blocker can be skipped. Therefore requiring every resolution publication to become `published` would strengthen the old behavior into an indefinite agent-wake dependency and could lose useful functionality. Preserve the opportunity for ready resolution publication before the wake while leaving provider retry and ambiguous-delivery recovery with the publication owner.
|
||||
|
||||
A concrete option to evaluate is a narrowly scoped queue drain for the continuation's issue/interaction before claiming its action. Reuse the queue's eligibility, FIFO, endpoint slots and waitForCompletion contract; do not call provider transport directly or start a second dispatcher. Keep the drain bounded and scoped so an unrelated failing provider cannot block all interaction wakes. Do it before the 30-second action claim so a slow provider attempt does not expire a newly acquired action claim. The generic and Telegram continuation producers must signal before their writes inside existing transactions. Inspect all caller transaction ownership and prove the real callback path before removing maintenance.
|
||||
|
||||
## Slice 33 implementation
|
||||
|
||||
Implemented on parent `8b43e6222706491f2d8044af3bf86358085069af`.
|
||||
The owner now registers a scoped deadline worker. It restores issued/processing
|
||||
rows in 100-row UUID pages, attempts up to 25 entries per sweep with company
|
||||
rotation, and schedules existing retry and stale-claim deadlines. Completed
|
||||
history has no recurring deadline. Total startup inventory and memory still grow
|
||||
with unfinished actions; page size bounds individual reads, not total inventory.
|
||||
|
||||
Both enqueue producers register intent before inserting. The generic producer
|
||||
uses a nested transaction around its action insert, preserving caller-owned
|
||||
outer commit and also supporting its existing root-client contract. All 20
|
||||
production call sites of the generic helper pass a transaction. Telegram
|
||||
settlement already owns a transaction. The continuation owner uses one local
|
||||
transaction helper for claim, failure, deferral, and settlement writes, with an
|
||||
exact action hint. Provider publication and heartbeat dispatch stay outside
|
||||
these transactions. Signals add opted-in transaction/XID work; they are not free
|
||||
or a cross-process notification mechanism.
|
||||
|
||||
The publication queue gained an optional company/endpoint/conversation scope.
|
||||
Its existing eligibility, FIFO, credential fencing, endpoint concurrency and
|
||||
ambiguity quarantine remain authoritative. Registered continuation recovery
|
||||
attempts up to 25 ready publications in the originating conversation before
|
||||
claiming the action. It waits for that endpoint's existing task when necessary,
|
||||
without joining unrelated endpoint work merely because the scoped queue is
|
||||
empty. Shared endpoint capacity can still require waiting for a slot. This is
|
||||
not a guarantee that every card in an arbitrary backlog becomes published before
|
||||
wake. Paused/attention, future retry, explicit mode, and FIFO-blocked publications
|
||||
retain the old queue semantics and do not indefinitely block continuation.
|
||||
The explicit/manual publication-maintenance contract remains unchanged. Only
|
||||
the recurring continuation invocation was removed.
|
||||
|
||||
The continuation worker rechecks drain/cancellation after publication preparation
|
||||
before taking a new action claim. Interrupted dispatch and provider ambiguity
|
||||
remain governed by existing claims, idempotency, and publication recovery. No
|
||||
new table, expiry policy, attempt cap, or host-sleep exemption was added.
|
||||
|
||||
## Verification evidence
|
||||
|
||||
- `/tmp/sleep33-missing-signal.log`: the real callback commits its response and
|
||||
then hits the existing injected settlement crash. Before enqueue notifications
|
||||
are added, the durable continuation remains issued instead of processed.
|
||||
- `/tmp/sleep33-signaled.log`: the new regression plus existing comprehensive
|
||||
Telegram acceptance pass (2 cases). Existing acceptance includes card edit
|
||||
before continuation, duplicate callbacks, board attribution, durable wake
|
||||
coalescing, deferral, and more than five failures.
|
||||
- `/tmp/sleep33-lifecycle.log`: 65 selected integration cases pass, including
|
||||
seven new lifecycle modes. The two other named suites were filtered out by
|
||||
the test-name selection and are not claimed as run.
|
||||
- `/tmp/sleep33-transactions.log`: 10 new cases pass: accept/reject, drain,
|
||||
restart, dispatch deferral/failure, paused endpoint, future publication retry,
|
||||
stale processing claim, and outer transaction rollback/commit. Idle assertions
|
||||
advance 120 seconds with no root selects or transactions. Server typecheck
|
||||
passed before this command.
|
||||
- `/tmp/sleep33-concurrent.log`: 101 malformed issued actions are explicitly
|
||||
failed; valid continuation work progresses. A second real callback commits
|
||||
while the first dispatch is blocked; both continuations settle and become
|
||||
quiet. This is one additional case.
|
||||
|
||||
Full acceptance, workspace typecheck/build, and remote review/checks are still
|
||||
required. These local results do not prove actual hosting or database suspension.
|
||||
|
||||
## Review correction: stop before each publication
|
||||
|
||||
At implementation head `6d01f4c730`, workspace typecheck/build passed. Full chat
|
||||
and startup acceptance passed 1,523 tests in 465.91 seconds, but the 14 standalone
|
||||
publication tests skipped because their suite supported only embedded PostgreSQL.
|
||||
This is not evidence those 14 tests passed. The suite now accepts an explicitly
|
||||
named disposable `_test` external database and closes its own client afterward.
|
||||
|
||||
Greptile review was 4/5 and found a valid stop-boundary gap: the continuation
|
||||
checked cancellation/admission after preparing publications, but that preparation
|
||||
could start another send after drain, standby, or shutdown began. The three new
|
||||
real-queue regressions each reproduced the second publication becoming published
|
||||
when it should remain pending (`/tmp/sleep33-stop-before.log`). They hold the
|
||||
first provider edit in flight, change the lifecycle state, and release the edit.
|
||||
|
||||
The existing queue now accepts an optional `canContinue` callback. The
|
||||
continuation owner passes its current admission/abort check through preparation.
|
||||
The queue checks before entry, each batch, each reservation, and each newly
|
||||
started send. Already-started work is joined and settled. Deferred actions and
|
||||
publications survive resume/restart. Default callers retain their current
|
||||
behavior, with shutdown also checked before invoking a reserved send.
|
||||
|
||||
All 14 new continuation cases pass after this correction
|
||||
(`/tmp/sleep33-stop-fixed.log`). The previously skipped publication suite and
|
||||
startup guards pass separately, 27 cases total
|
||||
(`/tmp/sleep33-publication-startup.log`). Fresh broad acceptance and remote gates
|
||||
on the corrected head remain required. No failed run has been relabelled passing.
|
||||
|
||||
|
||||
## Current prepared integration — October 28
|
||||
|
||||
The existing 12-path slice is normally merged onto prepared confirmation parent
|
||||
`9e9fff2be989d8447eb34a7fcba78076a26bdd27` at
|
||||
`22530681c6a4a3d11ea14356d751cecf45035ba8`. The complete original edits and context
|
||||
are identical after excluding only blob index hashes/hunk offsets. The proof is
|
||||
`/tmp/sleep33-current-parent-composition-proof.json`. No new implementation is
|
||||
added during this preparation; its predecessors carry current master and the
|
||||
notice discovery/restoration repairs.
|
||||
|
||||
**57 distinct cases pass:** all 14 selected real continuation lifecycle,
|
||||
concurrent-response, outer-commit and stop-before-publication cases (1,687 other
|
||||
chat cases skipped), plus 43 across the complete interaction-publication and
|
||||
app/startup suites. These run after server typechecking/build and module boundary
|
||||
checks, which also pass. Two isolated PostgreSQL fixtures keep the chat lifecycle
|
||||
and publication consumers independent; the publication owner applies its own
|
||||
migrations. Commands, fixtures and tested revision are in
|
||||
`/tmp/sleep33-current-acceptance-command.json`; logs/exit receipts are
|
||||
`/tmp/sleep33-current-{types,build,boundaries,continuations,consumers}.*`.
|
||||
|
||||
The older 14-case publication count and broad acceptance above retain their
|
||||
original revisions. The current complete publication/startup group has 43 cases;
|
||||
none are skipped. This preparation is local and unpublished, with both preceding
|
||||
PRs still required to land before final integration and fresh exact-head checks.
|
||||
No full local repository, live-provider or physical sleep claim is made.
|
||||
|
||||
|
||||
### Local acceptance on the indexed terminal-confirmation parent
|
||||
|
||||
Normal merge `71ec3c00d15cd8081514f0d2eb99d5a1728425b1` adopts published terminal
|
||||
confirmation parent `a35e78d076c1b618bc643391b7520fa6952fc996`, including its
|
||||
concurrent partial index, company/id keyset discovery and current master
|
||||
`644937c93150c423f12674992c7ea1e8b5dbc98c`. Both complete sets of added/removed
|
||||
patch edits are preserved, excluding only blob hashes and hunk line metadata.
|
||||
The two shared files (chat integration tests and confirmation actions) merge
|
||||
normally; every unrelated parent blob is retained. Proof:
|
||||
`/tmp/sleep33-indexed-parent-composition-proof.json`.
|
||||
|
||||
**74 distinct cases pass on this combined source:** 31 selected chat cases
|
||||
(14 continuation lifecycle/stop/retry/outer-commit cases plus 17 parent
|
||||
confirmation lifecycle/query-cost cases; 1,671 skipped), and all 43 cases across
|
||||
complete interaction-publication and app/startup suites. Server typechecking,
|
||||
build and module boundaries pass. Fresh isolated PostgreSQL databases use the
|
||||
parent's complete migration history through 0398. Its five newest snapshots
|
||||
remain 0394–0398, with no schema changes from this slice.
|
||||
|
||||
Commands, actual fixtures and tested revision:
|
||||
`/tmp/sleep33-indexed-parent-acceptance-command.json`. Logs/exit receipts:
|
||||
`/tmp/sleep33-indexed-parent-{types,build,boundaries,migrate,lifecycle,consumers}.*`.
|
||||
The previous 57-case matrix retains its original revision scope; repeated cases
|
||||
are not added to the current total. The following audit-only commit leaves all
|
||||
owned product/test sources unchanged.
|
||||
|
||||
This is local preparation. Terminal confirmations #17651 must actually merge
|
||||
before this slice is reconciled with its landed squash parent and published.
|
||||
The existing remote head's review is not a review of this integration. Fresh
|
||||
exact-head CI and Greptile 5/5 without actionable findings remain required.
|
||||
No full local repository suite, live provider or physical suspension result is
|
||||
inferred from this acceptance.
|
||||
|
||||
|
||||
### Acceptance on the actual landed terminal-confirmation parent
|
||||
|
||||
Terminal confirmations #17651 landed at
|
||||
`ee67d2ece9bda157dfad9bdf25175b1d004ed341`. Normal integration
|
||||
`db5a84c0f8b73a0ce863f4d2fbaee0a115cf044f` preserves the complete original
|
||||
12-path continuation patch and all 12 unrelated master paths, including managed
|
||||
Cloud sandbox selection changes. Historical squash-ancestry conflicts are
|
||||
resolved against the verified semantic parent `a35e78d076c1b618bc643391b7520fa6952fc996`;
|
||||
both normal merge parents and the exact composition tree are checked. Proof:
|
||||
`/tmp/sleep33-landed-parent-publication-proof.json`. All 74 original selected
|
||||
cases pass again on that actual-parent integration, with fresh PostgreSQL,
|
||||
database build, server typechecking/build and module boundaries.
|
||||
|
||||
Before publication, inspection found the same retained-history scan that the
|
||||
parent's review exposed in confirmation recovery. This owner also filtered
|
||||
`chat_actions` without a matching work index. The new actual-SQL regression
|
||||
fails at **20,345 examined rows and 20,421 shared buffers** for one pending claim,
|
||||
with 20,000 completed/unrelated actions present. This red result is retained in
|
||||
`/tmp/sleep33-cost-red.log` and its nonzero exit receipt.
|
||||
|
||||
Repair `a4a207ff23116c3fb739ad614c4c254992b1cdac` adds the generated
|
||||
`0399_ambitious_onslaught.sql` partial index on `(company_id, id)` for
|
||||
`interaction_wakeup` actions in `issued`/`processing` state. It is concurrent
|
||||
and idempotent, using the existing top-level migration runner. Keyset ordering
|
||||
and its composite cursor match that index; fixed predicates remain SQL literals
|
||||
for prepared-plan eligibility. This changes neither claim/retry deadlines nor
|
||||
continuation/publication authority. It adds no table, timer or scheduler.
|
||||
This is a deliberate additional repair after the original 12-path preservation
|
||||
proof, not a claim that the final patch is unchanged from the old implementation.
|
||||
|
||||
The regression captures the owner's real startup and company-scoped SQL, replays
|
||||
it unchanged under `EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON)`, and requires the
|
||||
correct pending ID with at most two action rows examined and fewer than 20 action
|
||||
buffers. Empty scoped recovery must also touch fewer than 20 buffers. No planner
|
||||
setting forces the index. A future processing claim prevents the test from
|
||||
consuming the pending row; the coordinator is stopped before checking empty
|
||||
recovery. The test vacuums/analyzes its owned table so dead tuples from previous
|
||||
cases do not masquerade as retained-history cost. The narrow green result is
|
||||
`/tmp/sleep33-cost-green.log`.
|
||||
|
||||
**75 distinct final cases pass:** 32 chat cases (14 continuation lifecycle,
|
||||
concurrency, malformed backlog, outer commit and stop cases; the new cost case;
|
||||
and 17 parent confirmation/query-cost cases), with 1,671 skipped; plus all 43
|
||||
cases across complete publication and startup suites. Database build, server
|
||||
typecheck/build, module boundaries and fresh migration through 0399 pass. The
|
||||
398-to-399 upgrade and target-branch migration-order gate also pass. All prior
|
||||
SQL/journal entries and retained snapshot bytes are preserved. Generation prunes
|
||||
0394 and retains the five newest snapshots, 0395–0399.
|
||||
|
||||
Actual commands and fixtures:
|
||||
`/tmp/sleep33-cost-final-acceptance-command.json`. Logs/receipts:
|
||||
`/tmp/sleep33-cost-final-{db-build,types,build,boundaries,migrate,lifecycle,consumers}.*`,
|
||||
`/tmp/sleep33-cost-upgrade.*`, and `/tmp/sleep33-cost-migration-order.*`.
|
||||
Earlier 57/74-case runs retain their revision scopes; repeated cases are not
|
||||
added to 75. Original missing-signal, skipped-publication and stop-before-send
|
||||
failures above remain recorded. The audit-only follow-up leaves the final owned
|
||||
source/test/migration hashes unchanged. Fresh exact-head CI and Greptile 5/5
|
||||
with no actionable findings remain required before merge. These tests establish
|
||||
owner recovery/quiet and bounded reads, not deployed physical suspension.
|
||||
@@ -550,6 +550,7 @@ const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([
|
||||
"0396_tan_paladin.sql",
|
||||
"0397_known_madrox.sql",
|
||||
"0398_yielding_random.sql",
|
||||
"0399_ambitious_onslaught.sql",
|
||||
|
||||
|
||||
"0384_funny_adam_warlock.sql",
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
CREATE INDEX CONCURRENTLY IF NOT EXISTS "chat_actions_continuation_work_idx" ON "chat_actions" USING btree ("company_id","id") WHERE "chat_actions"."kind" = 'interaction_wakeup' AND "chat_actions"."status" IN ('issued', 'processing');
|
||||
+130
-2
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"id": "633afb6b-4b89-44ea-81d5-a477a3696eb5",
|
||||
"prevId": "e1ddc6f7-6335-4b3d-aeec-a5df27793874",
|
||||
"id": "9ca01952-f1fa-4c54-a189-3f5a3a3a3761",
|
||||
"prevId": "7f056258-4b1a-4828-b950-33f070fc0c47",
|
||||
"version": "7",
|
||||
"dialect": "postgresql",
|
||||
"tables": {
|
||||
@@ -1350,6 +1350,34 @@
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"activity_log_company_user_issue_actor_idx": {
|
||||
"name": "activity_log_company_user_issue_actor_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "company_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "actor_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "entity_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"where": "\"activity_log\".\"actor_type\" = 'user' AND \"activity_log\".\"entity_type\" = 'issue'",
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
}
|
||||
},
|
||||
"foreignKeys": {
|
||||
@@ -9596,6 +9624,50 @@
|
||||
}
|
||||
},
|
||||
"indexes": {
|
||||
"chat_actions_continuation_work_idx": {
|
||||
"name": "chat_actions_continuation_work_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "company_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"where": "\"chat_actions\".\"kind\" = 'interaction_wakeup' AND \"chat_actions\".\"status\" IN ('issued', 'processing')",
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"chat_actions_confirmation_recovery_idx": {
|
||||
"name": "chat_actions_confirmation_recovery_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "company_id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"where": "\"chat_actions\".\"kind\" = 'confirmation_response' AND \"chat_actions\".\"status\" = 'processing'",
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"chat_actions_provider_action_uq": {
|
||||
"name": "chat_actions_provider_action_uq",
|
||||
"columns": [
|
||||
@@ -14160,6 +14232,24 @@
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"provider_account_id": {
|
||||
"name": "provider_account_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"cleanup_confirmed_at": {
|
||||
"name": "cleanup_confirmed_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"cleanup_next_attempt_at": {
|
||||
"name": "cleanup_next_attempt_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"expires_at": {
|
||||
"name": "expires_at",
|
||||
"type": "timestamp with time zone",
|
||||
@@ -14182,6 +14272,22 @@
|
||||
}
|
||||
},
|
||||
"indexes": {
|
||||
"chat_voice_inbound_cleanup_idx": {
|
||||
"name": "chat_voice_inbound_cleanup_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"where": "\"chat_voice_inbound_calls\".\"cleanup_confirmed_at\" is null",
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"chat_voice_inbound_expiry_idx": {
|
||||
"name": "chat_voice_inbound_expiry_idx",
|
||||
"columns": [
|
||||
@@ -14934,6 +15040,28 @@
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
},
|
||||
"chat_voice_sessions_report_window_idx": {
|
||||
"name": "chat_voice_sessions_report_window_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "ended_at",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
},
|
||||
{
|
||||
"expression": "id",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"where": "\"chat_voice_sessions\".\"state\" in ('ended', 'failed', 'expired') and \"chat_voice_sessions\".\"provider_session_id\" is not null",
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
}
|
||||
},
|
||||
"foreignKeys": {
|
||||
+7
@@ -2780,6 +2780,13 @@
|
||||
"when": 1793216165150,
|
||||
"tag": "0398_yielding_random",
|
||||
"breakpoints": true
|
||||
},
|
||||
{
|
||||
"idx": 399,
|
||||
"version": "7",
|
||||
"when": 1793218102114,
|
||||
"tag": "0399_ambitious_onslaught",
|
||||
"breakpoints": true
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -673,6 +673,9 @@ export const chatActions = pgTable(
|
||||
.defaultNow(),
|
||||
},
|
||||
(table) => [
|
||||
index("chat_actions_continuation_work_idx")
|
||||
.on(table.companyId, table.id)
|
||||
.where(sql`${table.kind} = 'interaction_wakeup' AND ${table.status} IN ('issued', 'processing')`),
|
||||
index("chat_actions_confirmation_recovery_idx")
|
||||
.on(table.companyId, table.id)
|
||||
.where(sql`${table.kind} = 'confirmation_response' AND ${table.status} = 'processing'`),
|
||||
|
||||
@@ -93,6 +93,7 @@ describe("createChatReconciliationCoordinator", () => {
|
||||
expect(publicationMaintenance).not.toContain("await enqueueInboundWakeupPublications(");
|
||||
expect(publicationMaintenance).not.toContain("await enqueueFailedChatRetryPublications(");
|
||||
expect(publicationMaintenance).not.toContain("reconcileTerminalConfirmationActions(");
|
||||
expect(source).toContain("chatChannels.registerInteractionContinuationWork(deliveryWork,");
|
||||
expect(source).toContain("chatChannels.registerTerminalConfirmationWork(deliveryWork,");
|
||||
expect(source).toContain("chatFailedRetryNoticeWork(chatChannels).register(deliveryWork,");
|
||||
});
|
||||
|
||||
@@ -45693,6 +45693,349 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
return { fixture, callbacks, onAction: callbacks.onAction, endpoint, service, wakeup, settlementBarrier, dm, conversation, linkedUserId, principal, interaction, publication, acceptAction, actionEvent };
|
||||
}
|
||||
|
||||
it("bounds interaction continuation recovery reads with completed history", async () => {
|
||||
const context = await telegramConfirmationRecoveryFixture();
|
||||
await expect(context.onAction(context.actionEvent))
|
||||
.rejects.toThrow("injected post-interaction confirmation settlement crash");
|
||||
const [action] = await db.select().from(chatActions).where(and(
|
||||
eq(chatActions.companyId, context.fixture.companyId), eq(chatActions.kind, "interaction_wakeup"),
|
||||
));
|
||||
expect(action).toBeDefined();
|
||||
await db.update(chatActions).set({ status: "processing", updatedAt: new Date() }).where(eq(chatActions.id, action.id));
|
||||
const prefix = `continuation-plan:${randomUUID()}:`;
|
||||
await db.execute(sql`
|
||||
INSERT INTO chat_actions (company_id, endpoint_id, kind, provider_action_id, status)
|
||||
SELECT ${context.fixture.companyId}::uuid, ${context.endpoint.id}::uuid,
|
||||
CASE WHEN n <= 10000 THEN 'interaction_wakeup' ELSE 'unrelated_action' END,
|
||||
${prefix} || n,
|
||||
CASE WHEN n <= 10000 THEN 'processed' ELSE 'processing' END
|
||||
FROM generate_series(1, 20000) AS n
|
||||
`);
|
||||
await db.execute(sql`VACUUM (ANALYZE) chat_actions`);
|
||||
const statements: { sql: string; params: unknown[] }[] = [];
|
||||
const select = db.select.bind(db);
|
||||
const reads = vi.spyOn(db, "select").mockImplementation((...args) => {
|
||||
const selection = select(...args), from = selection.from.bind(selection);
|
||||
vi.spyOn(selection, "from").mockImplementation(table => {
|
||||
const query = from(table);
|
||||
if (table === chatActions) {
|
||||
const execute = query.execute.bind(query);
|
||||
vi.spyOn(query, "execute").mockImplementation(async () => {
|
||||
const statement = query.toSQL();
|
||||
if (statement.params.includes("interaction_wakeup") || statement.sql.includes("'interaction_wakeup'")) {
|
||||
statements.push(statement);
|
||||
}
|
||||
return execute();
|
||||
});
|
||||
}
|
||||
return query;
|
||||
});
|
||||
return selection;
|
||||
});
|
||||
const errors: unknown[] = [];
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true,
|
||||
canReconcile: () => true, onError: error => errors.push(error) });
|
||||
try {
|
||||
await context.service.registerInteractionContinuationWork(coordinator, () => true).ready;
|
||||
expect(statements.length).toBeGreaterThan(0);
|
||||
const startup = statements[0], beforeHint = statements.length;
|
||||
await db.transaction(async tx => {
|
||||
await signalDatabaseWork(tx, DELIVERY_QUEUES.chatInteractionContinuations,
|
||||
JSON.stringify({ companyId: context.fixture.companyId }));
|
||||
});
|
||||
await vi.waitFor(() => expect(statements.length).toBeGreaterThan(beforeHint), { timeout: 5_000 });
|
||||
const scoped = statements.at(-1)!;
|
||||
expect(scoped.params).toContain(context.fixture.companyId);
|
||||
await coordinator.stop(); reads.mockRestore();
|
||||
for (const statement of [startup, scoped]) {
|
||||
expect(await db.$client.unsafe(statement.sql, statement.params as Parameters<typeof db.$client.unsafe>[1]))
|
||||
.toMatchObject([{ id: action.id }]);
|
||||
const result = await db.$client.unsafe(
|
||||
`EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) ${statement.sql}`,
|
||||
statement.params as Parameters<typeof db.$client.unsafe>[1],
|
||||
);
|
||||
type PlanNode = { Plans?: PlanNode[]; [key: string]: unknown };
|
||||
const nodes = (node: PlanNode): PlanNode[] => [node, ...(node.Plans ?? []).flatMap(nodes)];
|
||||
const actionReads = nodes(result[0]["QUERY PLAN"][0].Plan)
|
||||
.filter(node => node["Relation Name"] === "chat_actions");
|
||||
expect(actionReads).toHaveLength(1);
|
||||
expect(Number(actionReads[0]["Actual Rows"]) + Number(actionReads[0]["Rows Removed by Filter"] ?? 0),
|
||||
JSON.stringify({ params: statement.params, actionReads })).toBeLessThanOrEqual(2);
|
||||
expect(Number(actionReads[0]["Shared Hit Blocks"]) + Number(actionReads[0]["Shared Read Blocks"]))
|
||||
.toBeLessThan(20);
|
||||
}
|
||||
await db.update(chatActions).set({ status: "processed" }).where(eq(chatActions.id, action.id));
|
||||
await db.execute(sql`ANALYZE chat_actions`);
|
||||
expect(await db.$client.unsafe(scoped.sql, scoped.params as Parameters<typeof db.$client.unsafe>[1])).toHaveLength(0);
|
||||
const empty = await db.$client.unsafe(
|
||||
`EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) ${scoped.sql}`, scoped.params as Parameters<typeof db.$client.unsafe>[1],
|
||||
);
|
||||
expect(empty[0]["QUERY PLAN"][0].Plan["Shared Hit Blocks"]
|
||||
+ empty[0]["QUERY PLAN"][0].Plan["Shared Read Blocks"]).toBeLessThan(20);
|
||||
expect(errors).toEqual([]);
|
||||
} finally {
|
||||
await coordinator.stop(); reads.mockRestore();
|
||||
await db.delete(chatActions).where(sql`${chatActions.providerActionId} LIKE ${`${prefix}%`}`);
|
||||
}
|
||||
});
|
||||
|
||||
it.each(["accept", "reject", "drain", "restart", "deferred", "failure", "paused endpoint", "future publication", "stale claim"] as const)("wakes interaction continuations from a committed response through %s", async mode => {
|
||||
const context = await telegramConfirmationRecoveryFixture(mode === "reject" ? "reject" : "accept");
|
||||
const errors: unknown[] = [];
|
||||
let canRun = true;
|
||||
let coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => canRun,
|
||||
canReconcile: () => true, onError: error => errors.push(error) });
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
|
||||
const settle = async () => vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(10);
|
||||
expect(idleWorkSnapshot().active).toBe(0); expect(coordinator.nextWakeAt()).toBeNull();
|
||||
}, { timeout: 10_000 });
|
||||
const quiet = async () => {
|
||||
const select = vi.spyOn(db, "select"), transaction = vi.spyOn(db, "transaction");
|
||||
try {
|
||||
await vi.advanceTimersByTimeAsync(120_000);
|
||||
expect(select).not.toHaveBeenCalled(); expect(transaction).not.toHaveBeenCalled();
|
||||
} finally { select.mockRestore(); transaction.mockRestore(); }
|
||||
};
|
||||
const originalWake = context.wakeup.getMockImplementation()!;
|
||||
context.wakeup.mockImplementation(async (...args) => {
|
||||
expect((context.service.runtime as unknown as FakeChatSdkRuntime).endpoints.get(context.endpoint.id)?.edits).toHaveLength(["paused endpoint", "future publication"].includes(mode) ? 0 : 1);
|
||||
return originalWake(...args);
|
||||
});
|
||||
if (mode === "deferred") context.wakeup.mockResolvedValueOnce(null);
|
||||
if (mode === "failure") context.wakeup.mockRejectedValueOnce(new Error("temporary continuation admission failure"));
|
||||
try {
|
||||
await context.service.registerInteractionContinuationWork(coordinator, () => canRun).ready;
|
||||
await settle(); await quiet();
|
||||
if (["drain", "paused endpoint", "future publication"].includes(mode)) canRun = false;
|
||||
if (mode === "restart" || mode === "stale claim") await coordinator.stop();
|
||||
await expect(context.onAction(context.actionEvent)).rejects.toThrow("injected post-interaction confirmation settlement crash");
|
||||
if (["drain", "paused endpoint", "future publication"].includes(mode) || mode === "restart" || mode === "stale claim") {
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
await vi.waitFor(() => expect(idleWorkSnapshot().active).toBe(0));
|
||||
expect(context.wakeup).not.toHaveBeenCalled();
|
||||
await quiet();
|
||||
if (mode === "paused endpoint") await db.update(chatEndpoints).set({ status: "paused" }).where(eq(chatEndpoints.id, context.endpoint.id));
|
||||
if (mode === "future publication") await db.update(chatPublications).set({ state: "retry", nextAttemptAt: new Date(Date.now() + 60_000) })
|
||||
.where(eq(chatPublications.idempotencyKey, `interaction-resolution:${context.interaction.id}:${context.endpoint.id}`));
|
||||
if (mode === "stale claim") await db.update(chatActions).set({ status: "processing", updatedAt: new Date() }).where(and(
|
||||
eq(chatActions.companyId, context.fixture.companyId), eq(chatActions.kind, "interaction_wakeup"),
|
||||
));
|
||||
canRun = true;
|
||||
if (mode === "restart" || mode === "stale claim") {
|
||||
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => canRun,
|
||||
canReconcile: () => true, onError: error => errors.push(error) });
|
||||
await context.service.registerInteractionContinuationWork(coordinator, () => canRun).ready;
|
||||
} else if (coordinator.nextWakeAt() !== null) {
|
||||
await vi.advanceTimersByTimeAsync(Math.max(0, coordinator.nextWakeAt()! - Date.now()));
|
||||
}
|
||||
}
|
||||
if (mode === "stale claim") {
|
||||
const [action] = await db.select().from(chatActions).where(and(
|
||||
eq(chatActions.companyId, context.fixture.companyId), eq(chatActions.kind, "interaction_wakeup"),
|
||||
));
|
||||
expect(coordinator.nextWakeAt()).toBe(action.updatedAt.getTime() + 30_000);
|
||||
await vi.advanceTimersByTimeAsync(Math.max(0, coordinator.nextWakeAt()! - Date.now() - 1));
|
||||
expect(context.wakeup).not.toHaveBeenCalled();
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
}
|
||||
if (mode === "deferred" || mode === "failure") {
|
||||
let retryAt = 0;
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(10);
|
||||
const [action] = await db.select().from(chatActions).where(and(
|
||||
eq(chatActions.companyId, context.fixture.companyId), eq(chatActions.kind, "interaction_wakeup"),
|
||||
));
|
||||
expect(action).toMatchObject({ status: "issued", result: { code: mode === "deferred" ? "interaction_wakeup_deferred" : "interaction_wakeup_failed" } });
|
||||
retryAt = action.updatedAt.getTime() + 30_000;
|
||||
expect(idleWorkSnapshot().active).toBe(0);
|
||||
expect(coordinator.nextWakeAt()).toBe(retryAt);
|
||||
}, { timeout: 10_000 });
|
||||
expect(context.wakeup).toHaveBeenCalledTimes(1);
|
||||
await vi.advanceTimersByTimeAsync(Math.max(0, retryAt - Date.now() - 1));
|
||||
expect(context.wakeup).toHaveBeenCalledTimes(1);
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
}
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(10);
|
||||
expect(await db.select({ status: chatActions.status }).from(chatActions).where(and(
|
||||
eq(chatActions.companyId, context.fixture.companyId), eq(chatActions.kind, "interaction_wakeup"),
|
||||
))).toEqual([{ status: "processed" }]);
|
||||
}, { timeout: 10_000 });
|
||||
expect(context.wakeup).toHaveBeenCalledTimes(mode === "deferred" || mode === "failure" ? 2 : 1);
|
||||
expect(context.wakeup).toHaveBeenCalledWith(context.fixture.assignedAgentId, expect.objectContaining({
|
||||
idempotencyKey: `interaction:${context.interaction.id}:${mode === "reject" ? "rejected" : "accepted"}`, allowRunCoalescing: false,
|
||||
}));
|
||||
await settle(); await quiet();
|
||||
expect(errors).toEqual([]);
|
||||
} finally { await coordinator.stop(); vi.useRealTimers(); await context.service.shutdown(); }
|
||||
});
|
||||
|
||||
it("passes historical continuation prefixes and retains a response committed during dispatch", async () => {
|
||||
const first = await telegramConfirmationRecoveryFixture();
|
||||
const second = await telegramConfirmationRecoveryFixture();
|
||||
await expect(first.onAction(first.actionEvent)).rejects.toThrow("injected post-interaction confirmation settlement crash");
|
||||
const invalidIds = Array.from({ length: 101 }, (_, index) => `00000000-0000-4000-8002-${String(index + 1).padStart(12, "0")}`);
|
||||
await db.insert(chatActions).values(invalidIds.map(id => ({
|
||||
id, companyId: first.fixture.companyId, endpointId: first.endpoint.id,
|
||||
kind: "interaction_wakeup", providerActionId: `invalid-continuation:${id}`, status: "issued", payload: {},
|
||||
})));
|
||||
let release!: () => void;
|
||||
const barrier = new Promise<void>(resolve => { release = resolve; });
|
||||
const originalWake = first.wakeup.getMockImplementation()!;
|
||||
first.wakeup.mockImplementationOnce(async (...args) => { await barrier; return originalWake(...args); });
|
||||
const errors: unknown[] = [];
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true,
|
||||
canReconcile: () => true, onError: error => errors.push(error) });
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
|
||||
const ready = first.service.registerInteractionContinuationWork(coordinator, () => true).ready;
|
||||
try {
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(10);
|
||||
expect(first.wakeup).toHaveBeenCalledTimes(1);
|
||||
}, { timeout: 15_000 });
|
||||
await expect(second.onAction(second.actionEvent)).rejects.toThrow("injected post-interaction confirmation settlement crash");
|
||||
expect(await db.select({ status: chatActions.status }).from(chatActions).where(and(
|
||||
eq(chatActions.companyId, second.fixture.companyId), eq(chatActions.kind, "interaction_wakeup"),
|
||||
))).toEqual([{ status: "issued" }]);
|
||||
release(); await ready;
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(10);
|
||||
expect(first.wakeup).toHaveBeenCalledTimes(2);
|
||||
expect(idleWorkSnapshot().active).toBe(0); expect(coordinator.nextWakeAt()).toBeNull();
|
||||
}, { timeout: 15_000 });
|
||||
expect(await db.select({ id: chatActions.id }).from(chatActions).where(and(
|
||||
inArray(chatActions.id, invalidIds), eq(chatActions.status, "failed"),
|
||||
eq(sql<string>`${chatActions.result}->>'code'`, "interaction_wakeup_payload_invalid"),
|
||||
))).toHaveLength(101);
|
||||
expect(await db.select({ id: chatActions.id }).from(chatActions).where(and(
|
||||
inArray(chatActions.companyId, [first.fixture.companyId, second.fixture.companyId]),
|
||||
eq(chatActions.kind, "interaction_wakeup"), eq(chatActions.status, "processed"),
|
||||
))).toHaveLength(2);
|
||||
const select = vi.spyOn(db, "select"), transaction = vi.spyOn(db, "transaction");
|
||||
try {
|
||||
await vi.advanceTimersByTimeAsync(120_000);
|
||||
expect(select).not.toHaveBeenCalled(); expect(transaction).not.toHaveBeenCalled();
|
||||
} finally { select.mockRestore(); transaction.mockRestore(); }
|
||||
expect(errors).toEqual([]);
|
||||
} finally {
|
||||
release(); await ready; await coordinator.stop(); vi.useRealTimers();
|
||||
await db.delete(chatActions).where(inArray(chatActions.id, invalidIds));
|
||||
await first.service.shutdown(); await second.service.shutdown();
|
||||
}
|
||||
});
|
||||
|
||||
it.each(["drain", "standby", "shutdown"] as const)("stops starting continuation publications during %s", async mode => {
|
||||
const context = await telegramConfirmationRecoveryFixture();
|
||||
await expect(context.onAction(context.actionEvent)).rejects.toThrow("injected post-interaction confirmation settlement crash");
|
||||
const next = await issueThreadInteractionService(db).create(
|
||||
{ id: context.conversation.issueId, companyId: context.fixture.companyId },
|
||||
{ kind: "request_confirmation", continuationPolicy: "wake_assignee", title: "Next decision",
|
||||
payload: { version: 1, prompt: "A second decision after the accepted response?" } },
|
||||
{ agentId: context.fixture.assignedAgentId },
|
||||
);
|
||||
const nextPublication = () => db.select({ state: chatPublications.state }).from(chatPublications).where(
|
||||
eq(chatPublications.idempotencyKey, `interaction:${next.id}:${context.endpoint.id}`));
|
||||
expect(await nextPublication()).toEqual([{ state: "pending" }]);
|
||||
const provider = (context.service.runtime as unknown as FakeChatSdkRuntime).endpoints.get(context.endpoint.id)!;
|
||||
const originalThread = provider.thread.bind(provider);
|
||||
let entered = false;
|
||||
let release!: () => void;
|
||||
const gate = new Promise<void>(resolve => { release = resolve; });
|
||||
const threadSpy = vi.spyOn(provider, "thread").mockImplementation(threadId => {
|
||||
const thread = originalThread(threadId), edit = thread.adapter.editMessage.bind(thread.adapter);
|
||||
thread.adapter.editMessage = async (...args) => { entered = true; await gate; return edit(...args); };
|
||||
return thread;
|
||||
});
|
||||
let canRun = true;
|
||||
const errors: unknown[] = [];
|
||||
let coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => canRun,
|
||||
canReconcile: () => true, onError: error => errors.push(error) });
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
|
||||
const ready = context.service.registerInteractionContinuationWork(coordinator, () => canRun).ready;
|
||||
let stopping: Promise<void> | undefined;
|
||||
try {
|
||||
await vi.waitFor(() => expect(entered).toBe(true), { timeout: 10_000 });
|
||||
if (mode === "shutdown") stopping = coordinator.stop(); else canRun = false;
|
||||
release(); await ready; await stopping;
|
||||
await vi.waitFor(() => expect(idleWorkSnapshot().active).toBe(0));
|
||||
expect(provider.edits).toHaveLength(1);
|
||||
expect(await nextPublication()).toEqual([{ state: "pending" }]);
|
||||
expect(context.wakeup).not.toHaveBeenCalled();
|
||||
if (mode === "shutdown") {
|
||||
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => canRun,
|
||||
canReconcile: () => true, onError: error => errors.push(error) });
|
||||
await context.service.registerInteractionContinuationWork(coordinator, () => canRun).ready;
|
||||
} else {
|
||||
canRun = true;
|
||||
await vi.advanceTimersByTimeAsync(Math.max(0, coordinator.nextWakeAt()! - Date.now()));
|
||||
}
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(10);
|
||||
expect(context.wakeup).toHaveBeenCalledTimes(1);
|
||||
expect(await nextPublication()).toEqual([{ state: "published" }]);
|
||||
expect(idleWorkSnapshot().active).toBe(0); expect(coordinator.nextWakeAt()).toBeNull();
|
||||
}, { timeout: 10_000 });
|
||||
expect(errors).toEqual([]);
|
||||
} finally {
|
||||
release(); await ready; await stopping; await coordinator.stop(); vi.useRealTimers();
|
||||
threadSpy.mockRestore(); await context.service.shutdown();
|
||||
}
|
||||
});
|
||||
|
||||
it("defers continuation dispatch until the outer response transaction commits", async () => {
|
||||
const context = await telegramConfirmationRecoveryFixture();
|
||||
const errors: unknown[] = [];
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true,
|
||||
canReconcile: () => true, onError: error => errors.push(error) });
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
|
||||
const settle = async () => vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(10);
|
||||
expect(idleWorkSnapshot().active).toBe(0); expect(coordinator.nextWakeAt()).toBeNull();
|
||||
}, { timeout: 10_000 });
|
||||
const originalWake = context.wakeup.getMockImplementation()!;
|
||||
context.wakeup.mockImplementation(async (...args) => {
|
||||
expect((context.service.runtime as unknown as FakeChatSdkRuntime).endpoints.get(context.endpoint.id)?.edits).toHaveLength(1);
|
||||
return originalWake(...args);
|
||||
});
|
||||
try {
|
||||
await context.service.registerInteractionContinuationWork(coordinator, () => true).ready;
|
||||
await settle();
|
||||
const [issue] = await db.select().from(issues).where(eq(issues.id, context.conversation.issueId));
|
||||
const accept = async (tx: Parameters<Parameters<Db["transaction"]>[0]>[0]) => {
|
||||
await issueThreadInteractionService(tx as unknown as Db).acceptInteraction(issue, context.interaction.id, {}, { userId: context.linkedUserId });
|
||||
await vi.advanceTimersByTimeAsync(100);
|
||||
expect(context.wakeup).not.toHaveBeenCalled();
|
||||
expect(await db.select({ status: issueThreadInteractions.status }).from(issueThreadInteractions)
|
||||
.where(eq(issueThreadInteractions.id, context.interaction.id))).toEqual([{ status: "pending" }]);
|
||||
expect(await db.select({ id: chatActions.id }).from(chatActions).where(and(
|
||||
eq(chatActions.companyId, context.fixture.companyId), eq(chatActions.kind, "interaction_wakeup"),
|
||||
))).toEqual([]);
|
||||
};
|
||||
await expect(db.transaction(async tx => {
|
||||
await accept(tx);
|
||||
throw new Error("rollback outer continuation decision");
|
||||
})).rejects.toThrow("rollback outer continuation decision");
|
||||
await settle();
|
||||
expect(context.wakeup).not.toHaveBeenCalled();
|
||||
const select = vi.spyOn(db, "select"), transaction = vi.spyOn(db, "transaction");
|
||||
try {
|
||||
await vi.advanceTimersByTimeAsync(120_000);
|
||||
expect(select).not.toHaveBeenCalled(); expect(transaction).not.toHaveBeenCalled();
|
||||
} finally { select.mockRestore(); transaction.mockRestore(); }
|
||||
await db.transaction(accept);
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(10);
|
||||
expect(context.wakeup).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
await settle();
|
||||
expect(await db.select({ status: chatActions.status }).from(chatActions).where(and(
|
||||
eq(chatActions.companyId, context.fixture.companyId), eq(chatActions.kind, "interaction_wakeup"),
|
||||
))).toEqual([{ status: "processed" }]);
|
||||
expect(errors).toEqual([]);
|
||||
} finally { await coordinator.stop(); vi.useRealTimers(); await context.service.shutdown(); }
|
||||
});
|
||||
|
||||
it.each(["accept", "reject", "drain", "restart", "failed selection"] as const)("recovers committed terminal confirmations without polling through %s", async mode => {
|
||||
const context = await telegramConfirmationRecoveryFixture(mode === "reject" ? "reject" : "accept");
|
||||
let canRun = true;
|
||||
|
||||
@@ -1288,6 +1288,7 @@ export async function createApp(
|
||||
chatChannels.registerReceiverWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
||||
chatChannels.registerTeamsConsentExpiryWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
||||
chatChannels.registerTeamsAttemptRecoveryWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
||||
chatChannels.registerInteractionContinuationWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
||||
chatChannels.registerTerminalConfirmationWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
||||
chatChannels.registerCommandWork(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
||||
nativeChatProgressWork(db).register(deliveryWork, () => !isWarmStandby() && !isIdleTaskDrainActive());
|
||||
|
||||
@@ -3350,7 +3350,6 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) {
|
||||
|
||||
async function processPublicationMaintenance(limit = 25) {
|
||||
scheduleGitHubMaintenance();
|
||||
await processPendingInteractionWakeups(limit);
|
||||
}
|
||||
|
||||
// The app refills free endpoint slots on commits and durable deadlines.
|
||||
@@ -3475,7 +3474,8 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) {
|
||||
withCredentialMutationLease, receiptReactionCompletionRunId,
|
||||
});
|
||||
const { processSelectedPublication } = publicationWorker;
|
||||
const { processPendingInteractionWakeups } = createChatInteractionContinuations(db, {
|
||||
const { processPendingInteractionWakeups, registerInteractionContinuationWork } = createChatInteractionContinuations(db, {
|
||||
preparePublications: (scope, canContinue) => processPendingPublications(25, { maintenance: false, scope, canContinue }),
|
||||
redactError, CONFIRMATION_WAKEUP_RETRY_BACKOFF_MS, get heartbeat() { return options.heartbeat; },
|
||||
});
|
||||
const { handleAction, handleModalSubmit, reconcileTerminalConfirmationActions, registerTerminalConfirmationWork } = createChatInteractionCallbacks(db, {
|
||||
@@ -3502,6 +3502,7 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) {
|
||||
const { prepareTeamsBoardSend } = createTeamsBoardSendPreparation(db, { endpointRecord, runtimeFor });
|
||||
const { getPublicationBatchStatus, publishComment, publishBoardMessage } = createChatPublicationRequests(db, { endpointRecord, stageTeamsBoardFileIntent, processPendingSlackBoardMessages, processPendingPublications, prepareTeamsBoardSend });
|
||||
return {
|
||||
registerInteractionContinuationWork,
|
||||
registerTerminalConfirmationWork,
|
||||
loadFailedRetryNoticeScopes,
|
||||
reconcileFailedRetryNotices,
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyChatContinuationWork } from "../../chat-continuation-notifications.js";
|
||||
import { connectionIntentService } from "../../connection-intents.js";
|
||||
import { queueIssueAssignmentWakeup, type IssueAssignmentWakeupDeps } from "../../issue-assignment-wakeup.js";
|
||||
import { chatInteractionPresentation } from "../../chat-interaction-presentation.js";
|
||||
@@ -580,6 +581,7 @@ export function createChatConfirmationActions(db: Db, options: ChatConfirmationO
|
||||
})
|
||||
.onConflictDoNothing();
|
||||
if (shouldWake && issue.assigneeAgentId) {
|
||||
await notifyChatContinuationWork(tx, { companyId: issue.companyId, issueId: issue.id });
|
||||
await tx
|
||||
.insert(chatActions)
|
||||
.values({
|
||||
|
||||
@@ -1,8 +1,14 @@
|
||||
import { notifyChatContinuationWork, type ChatContinuationScope } from "../../chat-continuation-notifications.js";
|
||||
import { DELIVERY_QUEUES } from "../../delivery-work-notifications.js";
|
||||
import type { createDeliveryWorkCoordinator } from "../../delivery-work-coordinator.js";
|
||||
import { registerScopedDeadlineWork } from "../../scoped-deadline-work.js";
|
||||
import {
|
||||
and,
|
||||
asc,
|
||||
desc,
|
||||
eq,
|
||||
inArray,
|
||||
sql,
|
||||
isNotNull,
|
||||
isNull,
|
||||
lte,
|
||||
@@ -32,6 +38,7 @@ import { issueThreadInteractionService } from "../../issue-thread-interactions.j
|
||||
|
||||
type InteractionContinuationsOptions = {
|
||||
heartbeat: IssueAssignmentWakeupDeps;
|
||||
preparePublications: (scope: { companyId: string; endpointId: string; conversationId: string }, canContinue: () => boolean) => Promise<unknown>;
|
||||
redactError: (error: unknown) => string;
|
||||
CONFIRMATION_WAKEUP_RETRY_BACKOFF_MS: number;
|
||||
};
|
||||
@@ -42,6 +49,9 @@ export function createChatInteractionContinuations(db: Db, options: InteractionC
|
||||
|
||||
async function processPendingInteractionWakeups(
|
||||
limit = 100,
|
||||
actionId?: string,
|
||||
preparePublications = false,
|
||||
canContinue: () => boolean = () => true,
|
||||
): Promise<number> {
|
||||
const staleBefore = new Date(Date.now() - 30_000);
|
||||
const retryBefore = new Date(
|
||||
@@ -53,6 +63,7 @@ export function createChatInteractionContinuations(db: Db, options: InteractionC
|
||||
.where(
|
||||
and(
|
||||
eq(chatActions.kind, "interaction_wakeup"),
|
||||
actionId ? eq(chatActions.id, actionId) : undefined,
|
||||
or(
|
||||
and(
|
||||
eq(chatActions.status, "issued"),
|
||||
@@ -72,7 +83,17 @@ export function createChatInteractionContinuations(db: Db, options: InteractionC
|
||||
.limit(limit);
|
||||
let processed = 0;
|
||||
for (const candidate of candidates) {
|
||||
const [claimed] = await db
|
||||
if (preparePublications && candidate.conversationId) {
|
||||
await options.preparePublications({ companyId: candidate.companyId,
|
||||
endpointId: candidate.endpointId, conversationId: candidate.conversationId }, canContinue);
|
||||
}
|
||||
if (!canContinue()) break;
|
||||
const writeAction = <T>(write: (tx: Parameters<Parameters<Db["transaction"]>[0]>[0]) => PromiseLike<T>) =>
|
||||
db.transaction(async tx => {
|
||||
await notifyChatContinuationWork(tx, { companyId: candidate.companyId, actionId: candidate.id });
|
||||
return write(tx);
|
||||
});
|
||||
const [claimed] = await writeAction(tx => tx
|
||||
.update(chatActions)
|
||||
.set({ status: "processing", updatedAt: new Date() })
|
||||
.where(
|
||||
@@ -93,7 +114,7 @@ export function createChatInteractionContinuations(db: Db, options: InteractionC
|
||||
),
|
||||
),
|
||||
)
|
||||
.returning();
|
||||
.returning());
|
||||
if (!claimed) continue;
|
||||
const payload = claimed.payload;
|
||||
const requestedByActorType =
|
||||
@@ -122,7 +143,7 @@ export function createChatInteractionContinuations(db: Db, options: InteractionC
|
||||
!requestedByActorType ||
|
||||
!requestedByActorId
|
||||
) {
|
||||
await db
|
||||
await writeAction(tx => tx
|
||||
.update(chatActions)
|
||||
.set({
|
||||
status: "failed",
|
||||
@@ -134,7 +155,7 @@ export function createChatInteractionContinuations(db: Db, options: InteractionC
|
||||
eq(chatActions.id, claimed.id),
|
||||
eq(chatActions.status, "processing"),
|
||||
),
|
||||
);
|
||||
));
|
||||
continue;
|
||||
}
|
||||
const currentIssue = await db
|
||||
@@ -167,7 +188,7 @@ export function createChatInteractionContinuations(db: Db, options: InteractionC
|
||||
currentIssue.status === "done" ||
|
||||
currentIssue.status === "cancelled"
|
||||
) {
|
||||
await db
|
||||
await writeAction(tx => tx
|
||||
.update(chatActions)
|
||||
.set({
|
||||
status: "processed",
|
||||
@@ -179,7 +200,7 @@ export function createChatInteractionContinuations(db: Db, options: InteractionC
|
||||
eq(chatActions.id, claimed.id),
|
||||
eq(chatActions.status, "processing"),
|
||||
),
|
||||
);
|
||||
));
|
||||
processed += 1;
|
||||
continue;
|
||||
}
|
||||
@@ -325,7 +346,7 @@ export function createChatInteractionContinuations(db: Db, options: InteractionC
|
||||
.limit(1)
|
||||
.then((rows) => rows[0] ?? null));
|
||||
if (!queuedRun && !durable) {
|
||||
await db
|
||||
await writeAction(tx => tx
|
||||
.update(chatActions)
|
||||
.set({
|
||||
status: "issued",
|
||||
@@ -337,10 +358,10 @@ export function createChatInteractionContinuations(db: Db, options: InteractionC
|
||||
eq(chatActions.id, claimed.id),
|
||||
eq(chatActions.status, "processing"),
|
||||
),
|
||||
);
|
||||
));
|
||||
continue;
|
||||
}
|
||||
await db
|
||||
await writeAction(tx => tx
|
||||
.update(chatActions)
|
||||
.set({
|
||||
status: "processed",
|
||||
@@ -358,7 +379,7 @@ export function createChatInteractionContinuations(db: Db, options: InteractionC
|
||||
eq(chatActions.id, claimed.id),
|
||||
eq(chatActions.status, "processing"),
|
||||
),
|
||||
);
|
||||
));
|
||||
processed += 1;
|
||||
} catch (error) {
|
||||
const durable = isUniqueViolation(error)
|
||||
@@ -393,7 +414,7 @@ export function createChatInteractionContinuations(db: Db, options: InteractionC
|
||||
? previousResult.attemptCount
|
||||
: 0;
|
||||
const attemptCount = previousAttempts + 1;
|
||||
await db
|
||||
await writeAction(tx => tx
|
||||
.update(chatActions)
|
||||
.set({
|
||||
status: durable ? "processed" : "issued",
|
||||
@@ -412,7 +433,7 @@ export function createChatInteractionContinuations(db: Db, options: InteractionC
|
||||
eq(chatActions.id, claimed.id),
|
||||
eq(chatActions.status, "processing"),
|
||||
),
|
||||
);
|
||||
));
|
||||
if (!durable) {
|
||||
logger.warn(
|
||||
{
|
||||
@@ -428,5 +449,53 @@ export function createChatInteractionContinuations(db: Db, options: InteractionC
|
||||
}
|
||||
return processed;
|
||||
}
|
||||
return { processPendingInteractionWakeups };
|
||||
function registerInteractionContinuationWork(
|
||||
coordinator: ReturnType<typeof createDeliveryWorkCoordinator>,
|
||||
canRun: () => boolean,
|
||||
) {
|
||||
type Entry = { id: string; companyId: string; issueId: string | null; dueAt: number };
|
||||
const selection = {
|
||||
id: chatActions.id, companyId: chatActions.companyId,
|
||||
issueId: sql<string | null>`${chatActions.payload}->>'issueId'`,
|
||||
status: chatActions.status, result: chatActions.result, updatedAt: chatActions.updatedAt,
|
||||
};
|
||||
const entry = (row: { id: string; companyId: string; issueId: string | null; status: string;
|
||||
result: unknown; updatedAt: Date }): Entry => ({
|
||||
id: row.id, companyId: row.companyId, issueId: row.issueId,
|
||||
dueAt: row.status === "processing" ? row.updatedAt.getTime() + 30_000
|
||||
: row.result === null ? 0 : row.updatedAt.getTime() + CONFIRMATION_WAKEUP_RETRY_BACKOFF_MS,
|
||||
});
|
||||
return registerScopedDeadlineWork(coordinator, {
|
||||
queue: DELIVERY_QUEUES.chatInteractionContinuations,
|
||||
decodeScope: (hint: string): ChatContinuationScope | undefined => hint === "*" ? undefined : JSON.parse(hint),
|
||||
matches: (value: Entry, scope?: ChatContinuationScope) => !scope ||
|
||||
(value.companyId === scope.companyId && (!scope.actionId || value.id === scope.actionId) &&
|
||||
(!scope.issueId || value.issueId === scope.issueId)),
|
||||
load: async (scope?: ChatContinuationScope, signal?: AbortSignal): Promise<Entry[]> => {
|
||||
const entries: Entry[] = [];
|
||||
let cursor: { companyId: string; id: string } | undefined;
|
||||
while (!signal?.aborted) {
|
||||
const rows = await db.select(selection).from(chatActions).where(and(
|
||||
sql`${chatActions.kind} = 'interaction_wakeup' AND ${chatActions.status} IN ('issued', 'processing')`,
|
||||
scope ? eq(chatActions.companyId, scope.companyId) : undefined,
|
||||
scope?.actionId ? eq(chatActions.id, scope.actionId) : undefined,
|
||||
scope?.issueId ? eq(sql<string>`${chatActions.payload}->>'issueId'`, scope.issueId) : undefined,
|
||||
cursor ? sql`(${chatActions.companyId}, ${chatActions.id}) > (${cursor.companyId}, ${cursor.id})` : undefined,
|
||||
)).orderBy(asc(chatActions.companyId), asc(chatActions.id)).limit(100);
|
||||
entries.push(...rows.map(entry));
|
||||
if (rows.length < 100) return entries;
|
||||
cursor = rows[rows.length - 1];
|
||||
}
|
||||
throw signal?.reason ?? new Error("Interaction continuation recovery stopped");
|
||||
},
|
||||
check: async (value: Entry, signal?: AbortSignal) => {
|
||||
await processPendingInteractionWakeups(1, value.id, true, () => canRun() && !signal?.aborted);
|
||||
const [current] = await db.select(selection).from(chatActions).where(and(
|
||||
eq(chatActions.id, value.id), inArray(chatActions.status, ["issued", "processing"]),
|
||||
));
|
||||
return current ? entry(current) : null;
|
||||
},
|
||||
}, canRun);
|
||||
}
|
||||
return { processPendingInteractionWakeups, registerInteractionContinuationWork };
|
||||
}
|
||||
@@ -181,13 +181,22 @@ export function createChatPublicationQueue(db: Db, options: PublicationQueueOpti
|
||||
|
||||
async function processPendingPublications(
|
||||
limit = 25,
|
||||
{ waitForCompletion = true, maintenance = true }: { waitForCompletion?: boolean; maintenance?: boolean } = {},
|
||||
{ waitForCompletion = true, maintenance = true, scope, canContinue = () => true }: {
|
||||
waitForCompletion?: boolean; maintenance?: boolean; canContinue?: () => boolean;
|
||||
scope?: { companyId: string; endpointId: string; conversationId: string };
|
||||
} = {},
|
||||
) {
|
||||
if (isShuttingDown()) return 0;
|
||||
if (isShuttingDown() || !canContinue()) return 0;
|
||||
if (maintenance) {
|
||||
await enqueueInboundWakeupPublications();
|
||||
await reconcileTerminalConfirmationActions(limit);
|
||||
}
|
||||
if (isShuttingDown() || !canContinue()) return 0;
|
||||
const scopeCondition = scope ? and(eq(chatPublications.companyId, scope.companyId),
|
||||
eq(chatPublications.endpointId, scope.endpointId), eq(chatPublications.conversationId, scope.conversationId)) : undefined;
|
||||
const pendingTasks = () => scope
|
||||
? [publicationEndpointTasks.get(scope.endpointId)].filter((task): task is Promise<void> => Boolean(task))
|
||||
: [...publicationEndpointTasks.values()];
|
||||
const now = new Date();
|
||||
const staleBefore = new Date(now.getTime() - 60_000);
|
||||
// A process that disappears after the provider accepted a post but before
|
||||
@@ -196,6 +205,7 @@ export function createChatPublicationQueue(db: Db, options: PublicationQueueOpti
|
||||
// checking the provider conversation.
|
||||
const quarantinedPublications = await db.transaction(async tx => {
|
||||
const condition = and(
|
||||
scopeCondition,
|
||||
sql`not exists (select 1 from chat_endpoints e where e.id = ${chatPublications.endpointId} and e.publication_mode = 'explicit')`,
|
||||
eq(chatPublications.state, "streaming"),
|
||||
lte(chatPublications.updatedAt, staleBefore),
|
||||
@@ -278,7 +288,7 @@ export function createChatPublicationQueue(db: Db, options: PublicationQueueOpti
|
||||
let failed = false;
|
||||
let firstError: unknown;
|
||||
try {
|
||||
while (attemptedIds.length < limit && !failed && !isShuttingDown()) {
|
||||
while (attemptedIds.length < limit && !failed && !isShuttingDown() && canContinue()) {
|
||||
if (publicationEndpointTasks.size >= PUBLICATION_ENDPOINT_CONCURRENCY) {
|
||||
if (!waitForCompletion) break;
|
||||
await Promise.race(publicationEndpointTasks.values());
|
||||
@@ -295,7 +305,7 @@ export function createChatPublicationQueue(db: Db, options: PublicationQueueOpti
|
||||
.select()
|
||||
.from(chatPublications)
|
||||
.where(
|
||||
publicationEligibility({ now, busyEndpointIds, coolingTeamsPublicationIds, attemptedIds }),
|
||||
and(scopeCondition, publicationEligibility({ now, busyEndpointIds, coolingTeamsPublicationIds, attemptedIds })),
|
||||
)
|
||||
.orderBy(
|
||||
asc(chatPublications.createdAt),
|
||||
@@ -308,8 +318,9 @@ export function createChatPublicationQueue(db: Db, options: PublicationQueueOpti
|
||||
// exposing a new head that must be queried before declaring idle.
|
||||
if (busyEndpointIds.some((id) => !publicationEndpointTasks.has(id)))
|
||||
continue;
|
||||
if (!waitForCompletion || publicationEndpointTasks.size === 0) break;
|
||||
await Promise.race(publicationEndpointTasks.values());
|
||||
const pending = pendingTasks();
|
||||
if (!waitForCompletion || pending.length === 0) break;
|
||||
await Promise.race(pending);
|
||||
continue;
|
||||
}
|
||||
let started = false;
|
||||
@@ -318,7 +329,7 @@ export function createChatPublicationQueue(db: Db, options: PublicationQueueOpti
|
||||
publicationEndpointTasks.size >= PUBLICATION_ENDPOINT_CONCURRENCY ||
|
||||
attemptedIds.length >= limit ||
|
||||
failed ||
|
||||
isShuttingDown()
|
||||
(isShuttingDown() || !canContinue())
|
||||
)
|
||||
break;
|
||||
if (publicationEndpointTasks.has(selectedPublication.endpointId))
|
||||
@@ -330,6 +341,7 @@ export function createChatPublicationQueue(db: Db, options: PublicationQueueOpti
|
||||
// endpoint-exclusive credential fence instead of timing out behind it.
|
||||
const task = Promise.resolve()
|
||||
.then(async () => {
|
||||
if (isShuttingDown() || !canContinue()) return;
|
||||
await processSelectedPublication(selectedPublication);
|
||||
await settleExternalConversation(db, selectedPublication.companyId, selectedPublication.issueId).catch((err) => {
|
||||
logger.warn({ err, publicationId: selectedPublication.id }, "Slack conversation settlement deferred to reconciliation");
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
||||
|
||||
export type ChatContinuationScope = { companyId: string; issueId?: string; actionId?: string };
|
||||
|
||||
export async function notifyChatContinuationWork(tx: object, scope: ChatContinuationScope) {
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatInteractionContinuations,
|
||||
JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId, actionId: scope.actionId }));
|
||||
}
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyChatContinuationWork } from "./chat-continuation-notifications.js";
|
||||
import { chatInteractionForm } from "./chat-interaction-forms.js";
|
||||
import { slackInteractionRequiresBoard } from "./chat-channels/slack/cards.js";
|
||||
import { answeredChatQuestionBody, chatInteractionPresentation } from "./chat-interaction-presentation.js";
|
||||
@@ -710,51 +711,54 @@ export async function enqueueTerminalIssueInteractionChatPublications(
|
||||
interaction.resolvedByAgentId === issue.assigneeAgentId
|
||||
)
|
||||
) {
|
||||
await db
|
||||
.insert(chatActions)
|
||||
.values({
|
||||
companyId: interaction.companyId,
|
||||
endpointId: wakeBinding.endpointId,
|
||||
conversationId: wakeBinding.conversationId,
|
||||
kind: "interaction_wakeup",
|
||||
providerActionId: `interaction_wakeup:${interaction.id}`,
|
||||
payload: {
|
||||
version: 1,
|
||||
interactionId: interaction.id,
|
||||
interactionKind: interaction.kind,
|
||||
interactionStatus: interaction.status,
|
||||
issueId: interaction.issueId,
|
||||
agentId: issue.assigneeAgentId,
|
||||
sourceCommentId: interaction.sourceCommentId ?? null,
|
||||
sourceRunId: interaction.sourceRunId ?? null,
|
||||
requestedByActorType,
|
||||
requestedByActorId,
|
||||
...(planTarget
|
||||
? {
|
||||
planReviewInteraction: {
|
||||
id: interaction.id,
|
||||
kind: interaction.kind,
|
||||
status: interaction.status,
|
||||
target: planTarget,
|
||||
acceptedTargetRevision:
|
||||
interaction.status === "accepted" ? planTarget : null,
|
||||
result: interaction.result,
|
||||
},
|
||||
}
|
||||
: {}),
|
||||
...(interaction.status === "accepted" && planTarget
|
||||
? {
|
||||
forceFreshSession: true,
|
||||
workspaceRefreshReason: "accepted_plan_confirmation",
|
||||
}
|
||||
: {}),
|
||||
...(interaction.resolvedByUserId
|
||||
? { requestedByUserId: interaction.resolvedByUserId }
|
||||
: {}),
|
||||
},
|
||||
status: "issued",
|
||||
})
|
||||
.onConflictDoNothing();
|
||||
await db.transaction(async tx => {
|
||||
await notifyChatContinuationWork(tx, { companyId: interaction.companyId, issueId: interaction.issueId });
|
||||
await tx
|
||||
.insert(chatActions)
|
||||
.values({
|
||||
companyId: interaction.companyId,
|
||||
endpointId: wakeBinding.endpointId,
|
||||
conversationId: wakeBinding.conversationId,
|
||||
kind: "interaction_wakeup",
|
||||
providerActionId: `interaction_wakeup:${interaction.id}`,
|
||||
payload: {
|
||||
version: 1,
|
||||
interactionId: interaction.id,
|
||||
interactionKind: interaction.kind,
|
||||
interactionStatus: interaction.status,
|
||||
issueId: interaction.issueId,
|
||||
agentId: issue.assigneeAgentId,
|
||||
sourceCommentId: interaction.sourceCommentId ?? null,
|
||||
sourceRunId: interaction.sourceRunId ?? null,
|
||||
requestedByActorType,
|
||||
requestedByActorId,
|
||||
...(planTarget
|
||||
? {
|
||||
planReviewInteraction: {
|
||||
id: interaction.id,
|
||||
kind: interaction.kind,
|
||||
status: interaction.status,
|
||||
target: planTarget,
|
||||
acceptedTargetRevision:
|
||||
interaction.status === "accepted" ? planTarget : null,
|
||||
result: interaction.result,
|
||||
},
|
||||
}
|
||||
: {}),
|
||||
...(interaction.status === "accepted" && planTarget
|
||||
? {
|
||||
forceFreshSession: true,
|
||||
workspaceRefreshReason: "accepted_plan_confirmation",
|
||||
}
|
||||
: {}),
|
||||
...(interaction.resolvedByUserId
|
||||
? { requestedByUserId: interaction.resolvedByUserId }
|
||||
: {}),
|
||||
},
|
||||
status: "issued",
|
||||
})
|
||||
.onConflictDoNothing();
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -59,6 +59,7 @@ export const DELIVERY_QUEUES = {
|
||||
chatPublications: "chat-publications",
|
||||
chatNativeProgress: "chat-native-progress",
|
||||
chatInboundWakeNotices: "chat-inbound-wake-notices",
|
||||
chatInteractionContinuations: "chat-interaction-continuations",
|
||||
chatTerminalConfirmations: "chat-terminal-confirmations",
|
||||
chatFailedRetryNotices: "chat-failed-retry-notices",
|
||||
chatRunMilestones: "chat-run-milestones",
|
||||
|
||||
Reference in new issue
Block a user