From ec3b558277be5488c92e9b388e25300826caee89 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 29 Oct 2026 02:31:20 -0500 Subject: [PATCH] fix(server): schedule accepted native response recovery (#17696) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs store accepted results and their finalization decisions. > - A completed run can still owe its public answer after an interruption. > - Periodic recovery scans for that answer even when no new work exists. > - This pull request wakes recovery from committed changes and saved retry deadlines. > - The existing owner still verifies the result, source, and current publication authority. > - Settled or dormant response work no longer needs its own periodic scan. ## Linked Issues or Issue Description Refs #17690 (merged). This is slice 48 of the sleep migration stack, now based on master. Related: #17661 handles GitHub terminal publication; this change handles native accepted-response presentation. Governed-review presentation remains a separate duty. **What happened?** Native finalization recovery repeatedly checks already accepted results for a missing answer. Removing that check without complete notifications would lose responses when evidence arrives late, an agent resumes, or admission settles after execution. **Expected behavior** Recover the answer from its saved result when a supported change makes it eligible. Retain real retry deadlines. Stop querying when the answer is selected or cannot currently be published. Reconstruct work after restart. **Steps to reproduce** 1. Commit a native accepted result with no selected response comment. 2. Run finalization recovery while its source is unavailable or locked. 3. Restore that source or release the lock. The answer must appear once without repeating provider execution. ## What Changed - Honor recorded non-create decisions and skip-comment policy in both discovery and recovery. Apply durable conversation/result/continuation guards before recovery locks. Preserve genuine unanswered and legacy recovery. - Scope routine response wakes to the affected issue/run. Coalesced input names its active run; pending-effect settlement names its issue. Sixteen existing global authority/configuration hooks use one explicit company invalidation boundary. No new transaction wrappers or tables are added. - Add an accepted-response worker using the existing delivery coordinator and scoped deadline helper. Preserve saved retry deadlines and pending-effect fences under the owner's lock. - Share the finalization notification boundary. Wake on finalization settlement, exact accepted-result/final-message evidence, agent availability, retry admission, and existing chat authority changes. - Keep lifecycle policy in services. A narrow pre-write status hook lets lifecycle retain ownership of its transaction and status transitions. - Register the worker during enabled maintenance startup. Remove its authority subscription on shutdown. Periodic replay skips this duty on both committed and fresh-commit paths; ordinary finalization and explicit recovery retain their behavior. - Add a partial index and expression statistics for unpresented native runs. Migration 0412 supports concurrent replay and adds no queue table. - Add real PostgreSQL and lifecycle coverage. Preserve the implementation and producer audit in the plan document. ## Verification - Additional real Photon callback acceptance on the published production code disproves the security warning: ten active assertions cause zero response-recovery hints/SELECTs, while three concurrent assertions during an actual attention-to-active recovery cause exactly one wake/SELECT. The existing locked status guard already enforces this boundary. No production change was needed; the resolved finding now has a successful current-head security check. The regression will be retained with the next slice. - Eligibility repair source `892f36e5f66ba1af123034936a7160c64c3ab883` passes **1489 distinct server cases**: 1139 in six complete recovery/finalizer/monitor/summary/workspace suites (81.02 seconds), plus 350 selected chat cases (297.68s). Another 1463 chat cases are outside the pattern; the complete 1813-case file is not claimed. All eleven new regressions are included in that count. Five dedicated fresh PostgreSQL 17 databases migrate through unchanged 0412, with no fixture-source adaptations. Database build, server types/build, boundaries and full-SHA migration ordering pass. All seven test source hashes match. - The exact published baseline first fails eight behavioral assertions while the three genuine-answer cases pass. The same corrected test source passes all eleven after repair: recorded none/reuse remain suppressed; five ineligible history families take zero recovery row locks across repeated issue hints; missing/legacy/create-null answers still publish once; fresh periodic finalization defers presentation to its worker. Earlier fixture setup and variable/compiler failures remain explained in the audit, rather than erased by later success. - Repair source `2fc8b0614aaf9ac7337045de4ced7325f95edc45` passes **1821 server cases**: 1496 in 13 complete producer/lifecycle/authority suites (110.63 seconds), plus 325 selected chat cases (278.99 seconds). Coverage includes all native-session cases, queue admission, status decisions, membership/board access, the complete GitHub workflow, accepted-response lifecycle and real authority restoration. The other 1477 chat cases are outside this selection; the complete 1802-case file is not claimed for this repair run. - Actual PostgreSQL regressions first reproduce a coalesced-input hint missing its run scope and two routine hint paths producing 12 response-history SELECTs each after three notifications. The same assertions pass with run-scoped coalescing and zero response-history SELECTs for unscoped routine hints. Actual membership restoration still publishes the waiting response once. Assertion time limits are unchanged; this measures real driver SQL and persisted producer work. - Eleven dedicated fresh PostgreSQL 17 databases migrate through 0412. One embedded-only wake-queue fixture import temporarily uses its own external database and restores byte for byte. All 14 selected source files match pre-run hashes. This qualifies external-database behavior rather than reproducing embedded CI locally. - Master advanced independently with #18126's Pi archive retry fix. Normal merge `2a535bc61259ef99387369e8b32498f774e5ba2c` preserves both complete binary patches. All accepted server/database/UI/test/CLI trees stay identical; the three independent runner installer/manifest files are retained. All 13 installer tests pass, including stalled header/body cancellation. Combined server types/build, boundaries and full-SHA migration order pass. A subsequent normal merge of #18128 at `e8b78cce5bb557b086bee9ca930eeaab1d09b244` retains its three test-only file changes with both complete patches preserved. The changed HTTPS backfill case passes on a fresh PostgreSQL fixture (1 selected case; 178 outside the pattern), and combined types/build, boundaries and migration order pass again. All 14 repair source hashes remain unchanged. The independent browser fixture changes are preserved; no local browser run is claimed. - The earlier source `9feec170ef8174011fc3c5d3d6d76f0a0f39fea0` passed the 4171-case full baseline before this scope repair. That scope is preserved in the audit rather than represented as a current full rerun. All 0412 database bytes remain unchanged: original concurrent index/statistics/ANALYZE, prior executable history, journal entries and retained snapshots. Five snapshots 0408 through 0412 remain. The populated 0411-to0412 upgrade preserved 410 prior receipts and 38 companies, appended one receipt and none on replay; index/statistic checks passed. - The earlier P2 security failure, Greptile's former conditional dependency on #17702, and the subsequent actual Ponytail eligibility findings are recorded in `doc/plans/2026-10-25-native-response-presentation-sleep-audit.md`. This PR now includes the issue-scoped pending-effect notification fix itself, so it is independently correct before #17702. Fresh published-head CI, Greptile 5/5 and a clean security scan remain required. No deployed-sleep, external-wake or fleet-savings claim is made. ## Risks - The writer contract is process-local. Independent SQL writers and other replicas are not covered by these notifications. - Unpresented backlog reconstruction still has memory and time cost proportional to candidate count. Pages contain at most 100 rows; execution uses the shared fair batch. Selected history is excluded. Health-only signals do not trigger reconstruction. - Publication policy remains with its current owner. Digest, company/run binding, source authorization, selected-answer suppression, and ownership fences remain required. - The rest of native recovery and governed-review presentation still run separately. This PR does not relax Cloud sleep eligibility or add external wake coverage. - The stack parent has merged and its actual ancestry is reconciled. This publication still needs fresh current-head CI and review. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, shell execution, and PostgreSQL tests. The runtime does not expose a more specific model identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip Co-authored-by: Codex --- ...2026-10-24-recurring-server-sleep-stack.md | 111 +++ ...0-25-native-pending-effects-sleep-audit.md | 6 + ...ative-response-presentation-sleep-audit.md | 665 ++++++++++++++++++ packages/db/src/client.ts | 1 + .../0412_tearful_squadron_supreme.sql | 3 + ...{0407_snapshot.json => 0412_snapshot.json} | 160 ++++- packages/db/src/migrations/meta/_journal.json | 7 + packages/db/src/schema/heartbeat_runs.ts | 3 + .../chat-channels.integration.test.ts | 551 ++++++++++++++- .../committed-chat-response-authority.test.ts | 55 ++ .../server-startup-feedback-export.test.ts | 14 +- server/src/board-claim.ts | 6 +- server/src/index.ts | 11 +- server/src/middleware/auth.ts | 4 +- .../modules/run-dispatch/adapters/postgres.ts | 4 +- .../wake-queue/adapters/postgres.test.ts | 26 + .../modules/wake-queue/adapters/postgres.ts | 2 +- server/src/services/access.ts | 14 +- server/src/services/agent-lifecycle.ts | 4 +- .../src/services/chat-channels/endpoints.ts | 4 +- .../chat-channels/failed-run-retry.ts | 4 + .../services/chat-channels/identity-links.ts | 4 +- .../chat-channels/resource-reconciliation.ts | 4 +- .../chat-channels/slack/setup-services.ts | 4 +- server/src/services/chat-github-management.ts | 6 +- .../chat-monitor-continuation.test.ts | 32 +- .../chat-run-milestone-notifications.ts | 12 +- .../src/services/chat-work-notifications.ts | 4 +- .../services/delivery-work-notifications.ts | 1 + server/src/services/durable-chat-wakeup.ts | 16 + .../execution-control-reconciliation.ts | 4 +- server/src/services/heartbeat-run-events.ts | 10 +- server/src/services/heartbeat.ts | 3 + server/src/services/heartbeat/queue.ts | 7 +- server/src/services/heartbeat/recovery.ts | 3 +- .../src/services/heartbeat/run-completion.ts | 2 +- .../services/heartbeat/run-lifecycle.test.ts | 19 +- .../src/services/heartbeat/run-lifecycle.ts | 5 +- .../native-finalization-notifications.ts | 10 + .../native-pending-effect-notifications.ts | 8 - ...ive-response-presentation-notifications.ts | 14 + .../native-finalization-reconciler.ts | 5 +- .../native-interaction-bridge.ts | 4 +- .../native-pending-effect-work.ts | 10 +- .../native-response-presentation-work.ts | 117 +++ .../native-run-coordinator-store.ts | 10 +- .../native-runtime/native-run-finalizer.ts | 51 +- .../native-runtime/native-session-executor.ts | 10 +- .../native-workspace-export-recovery.ts | 4 +- .../native-workspace-export-retry.ts | 4 +- .../paperclip-control-plane-port.test.ts | 18 +- .../status-decision-committer.ts | 8 +- .../services/preserved-result-finalization.ts | 4 +- 53 files changed, 1960 insertions(+), 108 deletions(-) create mode 100644 doc/plans/2026-10-25-native-response-presentation-sleep-audit.md create mode 100644 packages/db/src/migrations/0412_tearful_squadron_supreme.sql rename packages/db/src/migrations/meta/{0407_snapshot.json => 0412_snapshot.json} (99%) create mode 100644 server/src/__tests__/committed-chat-response-authority.test.ts create mode 100644 server/src/services/native-finalization-notifications.ts delete mode 100644 server/src/services/native-pending-effect-notifications.ts create mode 100644 server/src/services/native-response-presentation-notifications.ts create mode 100644 server/src/services/native-runtime/native-response-presentation-work.ts diff --git a/doc/plans/2026-10-24-recurring-server-sleep-stack.md b/doc/plans/2026-10-24-recurring-server-sleep-stack.md index 14a2777304e..d800eeea39f 100644 --- a/doc/plans/2026-10-24-recurring-server-sleep-stack.md +++ b/doc/plans/2026-10-24-recurring-server-sleep-stack.md @@ -2715,3 +2715,114 @@ and all final-head PR gates remain open. No PR has been merged by this stack wor Final slice 47 source passes workspace typecheck/build and 158 worker/startup cases, including exact near-term concurrent deadlines. Remote publication and acceptance are the next gates; local success does not satisfy the full stack. + +## Slice 47 published; slice 48 response-owner audit + +Slice 47 is draft [#17690](https://github.com/paperclipai/paperclip/pull/17690), +head `56c3601ea605b333141e86e0a810ed790fb6ce48`, based on #17687. Its final local +checks passed; remote CI and Greptile are running. No merge or remote acceptance +is claimed yet. + +Slice 48 starts accepted-response presentation recovery, excluding governed-review +presentation. Its current helper conflates dormant/settled work and transient +failures as false; resolve that owner contract before removing its backstop. +The exact null-lease, digest, selected-answer, and publication-authority fences +must remain intact. The existing external-chat baseline passed 33 selected cases; +1,551 others were not selected. See the +[response-presentation audit](2026-10-25-native-response-presentation-sleep-audit.md). + +The old action-required check on #17576 is Superagent Security Scan. Its existing +voice-review findings and the older E2E failures remain outstanding. Work on the +rest of the original stack continues; none of these gates are waived. + +Slice 48's first owner-contract change now passes the same 33 real-PostgreSQL +response cases plus server typecheck. The strict recovery entry point propagates +operational errors; existing callers keep their logging/boolean contract. No +response worker has been wired and no response polling is removed yet. + +The Superagent action-required gate on #17576 was traced to review 5643793631, +inline finding 4371283330: repeated connection/health writes reload the full live +voice-session inventory. This is the already-recorded bounded-discovery issue, +not a missing credential or permission. It still needs a code/test repair and +fresh review; the gate must not be bypassed. + +## Slice 47 accepted + +[#17690](https://github.com/paperclipai/paperclip/pull/17690) at exact head +`56c3601ea605b333141e86e0a810ed790fb6ce48` has all 53 checks passing, Greptile 5/5 +with no actionable findings or review threads, and a mergeable parent. Its PR body +records those gates. It remains draft and unmerged. Evidence is retained in +`/tmp/sleep47-checks-accepted.json`, `/tmp/sleep47-final-gates.json`, +`/tmp/sleep47-review-accepted.json`, and `/tmp/sleep47-review-threads.json`. + +Slice 48 remains local and incomplete: owner error separation and the 33-case +baseline pass; candidate discovery, producer coverage, lifecycle wiring, and +periodic removal remain. Older unresolved stack gates and every other original +recurring-server duty still require completion. The active goal is not complete. + + +## Slice 48 local worker progress + +Accepted-response recovery now has a local scoped worker, locked retry/effect +fences, milestone-topic notifications, and live authority registration invalidation. +The final selected real-PostgreSQL/lifecycle run passes 45 cases, and server types +pass. An elapsed-deadline race was reproduced before repair. A generated concurrent +partial index plus expression statistics avoids selected-history scans in the +10,000-row fixture; fresh migration and replay pass. See the +[response audit](2026-10-25-native-response-presentation-sleep-audit.md) for exact +scope and preserved failures. + +The worker is not wired into startup and periodic response recovery remains. +Producer coverage, remaining variants/failure tests, large dormant-backlog cost, +full local/remote acceptance, and PR publication are still required. Master was +refreshed to `7703220d38`; the complete stack has not yet been reconciled with it. +No merge, complete migration, whole-instance sleep, or full-goal completion is +claimed. + +## Slice 48 producer and lifecycle checkpoint + +Response recovery now shares the finalization notification boundary, receives +run-scoped late evidence, and wakes when a paused agent resumes or retry admission +settles. The resume gap was reproduced before fixing it at lifecycle's transaction +boundary. The existing lifecycle suite, both response variants' supporting suites, +and the combined acceptance selection cover 175 distinct passing cases across +runs; server types pass. A real source-row lock verifies retry across drain. +Earlier harness/test failures and the 10,000-unpresented-row discovery measurements +are preserved in the response audit. No startup wiring, periodic removal, PR +publication, remote acceptance, or whole-instance sleep is claimed yet. + + +## Slice 48 startup integration, 2026-10-26 + +The response worker now registers under enabled maintenance and releases its +subscription during shutdown. Periodic recovery skips both its direct repair and +already-committed finalizer replay; fresh finalization and explicit recovery stay +active. The startup suite passes 66 cases, and two database regressions confirm +periodic calls defer to the new worker. Workspace types pass. Full local tests +remain in progress with embedded-database initialization failures; build and +remote checks remain pending. The stack goal is still incomplete. + + +## Slice 48 review repair, 2026-10-26 + +Draft #17696 is open and attached. Review identified company-wide response reloads +from routine run-status notifications; three real database regressions reproduced +it. Status/claim/release wakes now retain task or exact-run scope while task +lineage remains task-wide. All 34 heartbeat lifecycle and 63 response/authority/ +admission tests pass, as do server types and the workspace build. The full local +suite was stopped after repeated embedded PostgreSQL startup failures and remains +unproven. Remote failures at `79bf36cbf8` include an early startup-settlement test +assertion (repaired), a workspace staging timeout, and an E2E runner shutdown. +See the response audit for logs. Fresh-head CI/review, earlier stack findings, +master reconciliation, and all remaining original duties stay open. Nothing merged. + + +## Slice 48 full-history acceptance repair + +Fresh review of `c4beddad1e` is 5/5 with its scope finding resolved. Remote CI +exposed a second early startup assertion. Full local chat verification finished +1,595 passed / seven failed; all seven were new worker tests observing a transient +null deadline or unfinished startup batch. Their settlement predicate now includes +zero active work. All 18 affected cases pass against the full suite's retained +history. Fresh remote acceptance remains pending; these are recorded test repairs, +not a waiver of failed gates or a change to production batch behavior. diff --git a/doc/plans/2026-10-25-native-pending-effects-sleep-audit.md b/doc/plans/2026-10-25-native-pending-effects-sleep-audit.md index 83347984ffc..987e0dcc500 100644 --- a/doc/plans/2026-10-25-native-pending-effects-sleep-audit.md +++ b/doc/plans/2026-10-25-native-pending-effects-sleep-audit.md @@ -160,6 +160,12 @@ deadline repair (`/tmp/sleep47-deadlines-typecheck.log`, `/tmp/sleep47-deadlines-build.log`). The final worker/startup run now passes 158 cases, giving 683 distinct passing cases with the earlier unaffected consumers. +## Remote acceptance + +Published as [#17690](https://github.com/paperclipai/paperclip/pull/17690). +Exact head `56c3601ea605b333141e86e0a810ed790fb6ce48`: all 53 checks pass; +Greptile 5/5 with no actionable findings or review threads; mergeable parent. +The PR remains draft and unmerged. This accepts slice 47 only, not the full stack. ## Current migration411 integration and complete acceptance diff --git a/doc/plans/2026-10-25-native-response-presentation-sleep-audit.md b/doc/plans/2026-10-25-native-response-presentation-sleep-audit.md new file mode 100644 index 00000000000..05486e9c98b --- /dev/null +++ b/doc/plans/2026-10-25-native-response-presentation-sleep-audit.md @@ -0,0 +1,665 @@ +# Native accepted-response presentation recovery + +Status: worker, producers, and startup/periodic integration implemented; full local and remote acceptance pending. +Branch: `codex/sleep-48-native-response-presentation`. +Parent: [#17690](https://github.com/paperclipai/paperclip/pull/17690), +`56c3601ea605b333141e86e0a810ed790fb6ce48`. + +## Reason for work + +A native run can have an accepted, committed result but no saved public reply. +The finalization reconciler calls `repairCommittedNativeChatResponse` before +skipping superseded status decisions. A later task decision must not erase an +already accepted, still-authorized response. This work neither re-executes the +provider nor changes issue disposition. It is separate from governed-review +presentation, evidence reassessment, and result/workspace settlement. + +The current helper covers three variants of the same accepted-response duty: +a saved same-agent/monitor continuation summary, an internal task's exact final +assistant message, and an authorized external response-wake summary. Governed +review responses are explicitly excluded and retain their own stricter owner. +Do not merge the review response migration into this slice merely because both +helpers live in the finalizer. + +## Existing authority and mutation boundary + +`repairCommittedNativeChatResponse` locks issue, run, and coordinator in that +order. It requires native runtime; succeeded, failed, or timed_out with finishedAt; +a committed coordinator with result/assessment/decision and a null leaseOwner. +It refuses to replace a selected comment, including a deleted selected answer. +It checks the accepted result's digest using either the coordinator format or +the older authenticated control-plane format, retaining its bounded journal +source lookup. It excludes externalChatReviewPresentation decisions. + +A lease that expired but still has an owner is not permission to publish: the +helper requires actual release. Do not reuse the pending-effects lease rule. +The outer periodic candidate query additionally respects saved nextAttemptAt. +A future worker must preserve that fence where applicable and wake on release. + +Internal completion needs an exact final assistant event for the accepted turn, +current internal-write authorization, and permission to create a new completion +comment. Continuation summaries check provider-owned interactions. External +response-wake summaries require the live registered chat authority, exact durable +source provenance, and current presentation authorization. The chat owner validates +source edits/deletion, membership, endpoint state/generation, consumed failure +lanes, and ambiguous delivery. None of this policy belongs in a generic scheduler. + +The comment and presentationDecision are written in the same transaction, through +the existing comment/publication owner. Notification follows durable commit. +Recovery must not revive private provider narration, duplicate a target, publish +under another source, or reinterpret the saved result using new task settings. + +## Contract gap to resolve before retiring polling + +The helper currently returns false for several distinct outcomes: already settled, +structurally ineligible, authority unavailable/revoked, and transient SQL/lock or +materialization failure. It logs most errors then returns false. A worker cannot +use that boolean as a pending-work predicate: dropping false loses retryable +responses, while retrying every false rescans ineligible history indefinitely. + +Keep the current boolean caller contract where needed, but expose the smallest +owner-level result/error contract that distinguishes dormant history from concrete +retry debt. Do not translate all authorization failures into retry forever. The +registered authority lives in durable-chat-wakeup.ts and is installed by +chat-channels/failed-run-retry.ts. Missing registration and later registration need +an explicit lifecycle wake path, separate from committed database mutations. + +Audit and test the candidate predicate before introducing a worker. Startup scans +must filter selected answers, irrelevant result kinds, review-owned responses, +and other passive history. Full authorization stays in the locked owner; SQL +prefiltering is not a new publication authorization mechanism. Measure actual query +plans with substantial completed history before claiming bounded discovery. + +## Producer audit still required + +- Native coordinator decision commit, persisted-attention replay, and ownership + release; result/assessment/decision binding changes. +- Run terminalization and accepted-turn event persistence, including late journal + arrival and direct recovery writers. Avoid waking on every unrelated progress + event merely because it shares the journal. +- Presentation selection/materialization and existing-comment suppression. A + deleted selected answer remains settled rather than becoming repair debt. +- Issue conversation/context changes and skipIssueComment behavior. +- Exact chat source, action/admission, publication, endpoint generation/status, + principal/membership, continuation provenance, and ambiguous-delivery resolution. + Separate mutations that only revoke work from supported reactivation paths. +- Live chat-authority registration/replacement/shutdown; subscription ownership. +- Startup, first write during drain, retry during drain, resume, and shutdown. + +Use existing durable records, scoped transaction signals, and the shared scheduler. +No new response queue table or permissive publication fallback is justified. +Independent SQL/multi-process writes remain outside the existing local signal +contract unless explicitly integrated; preserve the documented limitation. + +## Baseline evidence + +On the unchanged parent source, a fresh external PostgreSQL selection passed +33 accepted-response cases in chat-channels.integration.test.ts. The remaining +1,551 cases in that file were not selected, not claimed passed. The selection +covers duplicate/concurrent repair, source/authority changes, digest changes, +private disposition, cross-company/run binding, live heartbeat fences, the older +port-bound digest, GitHub internal prose/attachments, failure-lane replacement, +provider-dispatch reauthorization, selected/deleted answers, and contention. +Log: `/tmp/sleep48-chat-baseline.log`. Database: +`paperclip_sleep48_chat_baseline_20261025_test`. + +The monitor-continuation and control-plane-port suites currently require embedded +PostgreSQL directly. Qualify a validated external-test path if needed before using +their results on this host; do not count skipped or initialization-failed suites. +No sleep or migrated-worker behavior is established by this baseline. + +## Initial owner-contract change + +`recoverCommittedNativeChatResponse` now exposes the existing transaction and +returns false for settled/ineligible/authorization-denied state, while propagating +operational failures. The existing `repairCommittedNativeChatResponse` wrapper +retains logging and false-on-error behavior for current callers. Authority checks, +lock order, target materialization, and commit signaling remain in the same owner. +The commit signal itself contains subscriber errors, so the wrapper does not hide +a previously propagated publication error. + +The same 33-case real-PostgreSQL selection passes after this change, with added +assertions that the new entry point rejects under actual source-action lock +contention and returns false for the existing authorization/binding fences. The +server typecheck passes (`/tmp/sleep48-contract-typecheck.log`). Worker discovery, +producer notifications, startup wiring, and periodic removal are not implemented. + +## Existing notification boundary to reuse + +The current `notifyChatRunMilestones` boundary already covers 102 production call +sites and fans out lineage/authority changes to native progress, inbound notices, +confirmations, and GitHub work. Its scope is company plus optional issue because +source changes can authorize descendants. Audit its coverage for this duty before +adding caller wrappers. It already reaches status-decision settlement and run +projection, but raw coordinator-store/session release and accepted-turn journal +writers need individual inspection. Reuse is not proof that all producers exist. + + +## Local worker and locked scheduling fences + +The local worker now discovers exact run/coordinator/result/decision bindings, +filters selected answers and structurally irrelevant results, and dispatches +through `recoverCommittedNativeChatResponse`. It does not own publication policy. +`onlyWhenDue` rechecks the saved retry and outstanding status effects while the +existing coordinator lock is held. Existing explicit callers retain their old +behavior. A future retry is retained even if it becomes due between the locked +check and the follow-up read; its next attempt is immediate rather than discarded. + +Operational errors remain retryable through the shared coordinator. A structurally +eligible but unauthorized row becomes dormant until an eligibility mutation or +live authority registration invalidates its scope. Selected comments, including +deleted selections, remain excluded. No provider rerun or new work table is added. + +The existing milestone notification boundary now includes the response topic. +Hints explicitly select company and issue IDs: structural TypeScript callers may +pass larger objects, so serializing the whole argument is not safe. The first +integration run exposed that mistake as circular-object serialization during +status materialization; it was corrected before the passing runs below. + +Two narrow lifecycle capabilities support the new consumer: + +- `registerScopedDeadlineWork.invalidate` marks reconstruction dirty before waking. + A plain wake is insufficient after a prior dormant result removed its entry. + Existing consumers keep the same registration and scheduling behavior. +- The committed-response authority owner exposes registration notifications. + Registering/replacing the actual chat service reactivates dormant responses. + The worker's `stop` removes this subscription; the delivery coordinator still + owns worker cancellation and awaiting in-flight work. Repeated unsubscription + cannot delete a later listener set. The authority remains scoped to the exact Db. + +No startup registration or periodic skip is wired yet. The old recovery path is +still present. Full producer coverage, internal/continuation variants, recovery +failure/drain tests, and repeated large-scope mutation cost remain acceptance work. +Current discovery pages by run ID but reconstructs all matching pages in one +load; its resource bound under a large unresolved/dormant backlog needs proof +before activation. Do not describe the worker as a completed migration yet. + +## Query-plan and migration evidence + +The actual discovery query was captured from the driver. On a disposable database +with 10,000 additional selected runs, results, assessments, decisions, and +coordinators, it initially scanned completed history (22.656 ms). A partial +unpresented-run index alone still chose the wrong join order (16.532 ms). +Expression statistics for the selected-comment predicate allowed PostgreSQL to +start from the one unpresented run (0.091 ms). + +Generated migration `0404_futuristic_earthquake.sql` creates the partial index +concurrently, adds idempotent expression statistics, and analyzes the table. +The existing top-level migration runner owns concurrent-index replay/repair. +Snapshot retention contains exactly 0396–0400. Fresh migration, journal replay, +direct SQL replay, ready/valid index flags, and statistics presence passed. + +Rechecking the actual migration's objects (after removing experimental objects) +used the driver's real global, company, and issue-scoped queries. Global discovery +found one candidate in 0.078 ms; history-only company/issue scopes returned none +in 0.024/0.019 ms without sequential scans. These are query-plan fixtures, not +latency or fleet-sleep claims. Large unselected ineligible histories and repeated +configuration signals still need separate qualification. + +Evidence: `/tmp/sleep48-query.json`, `/tmp/sleep48-query-plan.sql`, +`/tmp/sleep48-query-plan.log`, `/tmp/sleep48-query-statistics.log`, +`/tmp/sleep48-production-plan.log`, `/tmp/sleep48-migration-verify.log`. + +## Current verification and preserved failures + +The expanded final selection passed 45 cases in two files: 33 existing accepted- +response cases, ten new worker/owner cases, and two authority lifecycle cases. +The 1,551 unselected chat cases are not claimed passed. Server typecheck passed. +The worker cases establish startup recovery, selected-history quiet over former +poll periods, real chat-service authority restoration, outer commit/rollback, +first notification during drain, SQL-free drain waiting, resume, saved future +retry, pending/failed status-effect ordering, and an elapsed-deadline race. + +The elapsed-deadline test failed before its fix: the worker returned null even +though the locked owner had deferred the response. It now retains the due entry +and the next attempt publishes exactly once. Log: +`/tmp/sleep48-elapsed-deadline-before.log`. + +Earlier worker runs exposed the hint serialization defect (nine fixture failures), +then two test-observation mistakes. A timer created before fake-time installation +was not advanced by the fake clock; the test now installs time before registration. +A null next deadline can occur while a sweep is in flight; commit assertions now +wait for the persisted publication, zero tracked activity, and no next deadline +jointly. The original logs remain `/tmp/sleep48-worker-tests.log`, +`/tmp/sleep48-worker-tests-v2.log`, `/tmp/sleep48-drain-isolated.log`, and +`/tmp/sleep48-worker-tests-v3.log`. The corrected initial selection passed eleven +cases in `/tmp/sleep48-worker-tests-v4.log`; the final expanded result is +`/tmp/sleep48-acceptance-v2.log` with server types in +`/tmp/sleep48-acceptance-typecheck-v2.log`. + +No slice 48 PR, full workspace acceptance, remote CI/review, or provider sleep +qualification is claimed by these local results. The earlier stack gates and +remaining recurring-server duties are unchanged. + +## Producer integration and lifecycle evidence + +The existing pending-effect notification boundary is now named +`notifyNativeFinalizationWork` and wakes both consumers. Its existing coordinator +claim/retry, effect settlement, persisted-attention replay, retained-session release, +and recovery callers retain transaction ownership. No new wrappers are threaded +through those callers. Response discovery now accepts exact run and agent scopes. + +Both journal writers notify for accepted-result evidence and normalized completed +final assistant messages, after source/deduplication checks and before persistence. +Deltas, tool completions, and analysis messages do not activate response recovery. +The port's late accepted-result journal test and the internal final-message test +exercise these existing writers. Monitor continuation recovery is also exercised +through the worker, without another accepted result or provider execution. + +Agent availability was a real missing producer: the recovery owner rejects a +paused agent, but resuming that agent did not invalidate dormant responses. A +real resume regression failed before the fix. Lifecycle now exposes a narrow +`beforeStatusChange` integration callback, invoked inside its existing transaction +before the compatibility status write. The service supplies response policy only +for a transition from non-invokable to invokable, scoped to that agent. Lifecycle +still owns status transitions and the transaction; its existing resource-event +callback retains its semantics. With no response worker registered, this adds only +the existing notification protocol, not an independent worker or provider call. +The complete lifecycle suite exercises existing consumers of the new contract. + +Failed-chat retry admission also notifies response recovery when its source action +becomes processed. This can establish the ancestry proof after a very fast child +run has already finished. Five real admission scenarios verify that the response +consumer observes the committed action and the exact issue scope: commit, retry, +startup, stale claim, and rollback. An assignment-reactivation experiment instead +hit the existing prohibition on changing a channel-bound task's agent; that +unsupported fixture was removed without relaxing the guard. + +Validation across these runs covers 175 distinct cases: 59 response/authority and +retry-admission cases, 58 lifecycle cases, 21 monitor-continuation cases, and 37 +control-plane-port cases. Server typechecking passes. These are selected suites, +not a claim that the entire repository suite or remote PR checks passed. + +- `/tmp/sleep48-producers-rerun.log`: port 37/37, continuation 20/20, server types. +- `/tmp/sleep48-producer-acceptance.log`: lifecycle 58/58, expanded continuation + 21/21, response/authority 53 selected, server types. +- `/tmp/sleep48-acceptance-v5.log`: combined response/authority/admission 59 passed; + 1,546 unrelated cases not selected. Types in + `/tmp/sleep48-acceptance-typecheck-v5.log`. +- Real action-row contention now verifies retry retention through drain, zero + passive SQL while paused, and eventual single publication after resume. + +Failures remain recorded. The first external-port run had 36 passes and one +observer connection using the absent embedded database; the observer now uses the +validated external URL. The resume regression and invalid reassignment experiment +are in `/tmp/sleep48-eligibility-before.log`. A test-only observer initially omitted +its required retry interval and produced finite-deadline warnings; corrected in +`/tmp/sleep48-retry-producer-v2.log`. The combined v4 run exposed a global fault +injection targeting another retained fixture (13 transactions, not one). The v5 +run uses an actual lock on the tested response's source instead of assuming global +candidate order; all 59 cases pass with that stronger fault boundary. + +## Unpresented-backlog measurement + +A disposable clone of the earlier plan fixture has 10,008 structurally eligible, +unpresented candidates. The actual worker loader took 2,980 ms and 101 queries for +global reconstruction; 10,001 candidates in one company took 2,828 ms; the exact +run took 1.73 ms and one query. Agent reconstruction for that company's 10,001 +candidates took 2,826 ms. Serialized entries occupy about 2.05 MB. The drain gate +stopped discovery after two pages. Evidence: +`/tmp/sleep48-unpresented-backlog.mts` and `.log`. + +This measures discovery, not authorization or publication throughput: the cloned +rows do not establish valid independent publication authority. Each SQL page is +bounded to 100 rows, but total reconstruction memory/time still scales with the +unpresented backlog. Ordinary finalization and journal notifications are run-scoped; +agent resume is agent-scoped; actual authority/configuration changes may reconstruct +a larger scope. Health-only configuration signals do not invalidate this worker. +Do not describe total reconstruction as constant-cost or claim fleet qualification. + +Startup/shutdown integration, removal of this duty's periodic call, final producer +coverage review, full PR verification, and publication remain outstanding. The old +backstop is still present. Master was refreshed to `6bdcb70244`; the full stack +still needs reconciliation and its earlier failing/review gates remain open. + + +## Startup integration, 2026-10-26 + +The heartbeat facade now registers this worker with the existing task-drain gate. +Enabled maintenance awaits reconstruction during startup. Shutdown and listener +close release its live-authority subscription; coordinator shutdown retains +ownership of cancellation and in-flight work. + +The periodic orphan-recovery caller now passes `skipNativeResponsePresentation`. +The reconciler skips its direct repair call and passes the same option into +canonical finalizer replay. The finalizer skips only its already-committed response +repair in that mode. Fresh finalization still delivers immediately. Explicit +recovery and startup recovery keep their default behavior. Governed-review +presentation and other native finalization duties remain separate. + +All 66 startup tests pass, including disabled maintenance and close subscription +registration. Two real PostgreSQL regressions verify that both periodic entry +points leave the response unmaterialized, and the worker then creates exactly one +publication from the accepted result. Log: `/tmp/sleep48-wiring-tests.log`. +Workspace typecheck passes (`/tmp/sleep48-full-typecheck.log`). The full local suite +is still running and has reported embedded PostgreSQL initialization failures; +its final result and build status remain pending. No full-suite or remote check +acceptance is claimed at this checkpoint. + + +## Review repair and current verification, 2026-10-26 + +Published draft #17696 at `79bf36cbf8`. Greptile found a real precision gap: +`writeRunStatus` still sent a company-only milestone hint. Every unrelated run +status could reconstruct dormant response history. Three real-PostgreSQL cases +(native binding, legacy context binding, and taskless run) failed before repair +(`/tmp/sleep48-scope-before.log`). + +The milestone contract now accepts an optional run identity. Status producers +supply the task when bound and the exact run otherwise. The response fanout +prefers task scope over run scope, because lineage changes can authorize other +runs on that task. Known run claim/release and completion producers also preserve +an exact-run fallback. Existing company configuration invalidations remain broad. +No new durable rows or caller recovery wrappers are introduced. + +All 34 heartbeat lifecycle tests pass after repair. Two real worker regressions +retain a dormant eligible response, send unrelated task/run wakes, and observe +one actual candidate SQL query at the postgres.js debug boundary, without +rechecking the dormant row. The outer-commit/rollback cases additionally send a +different run identity within the same task and retain task-level recovery. +An initial test mistakenly counted Drizzle subquery builder construction as a +second SQL statement; driver-level observation corrected the measurement. + +The response/authority/admission selection now passes all 63 cases +(`/tmp/sleep48-scope-acceptance.log`). Server typechecking and the workspace build +pass (`/tmp/sleep48-scope-typecheck-v3.log`, `/tmp/sleep48-scope-build.log`). +The previous full workspace typecheck passed at `79bf36cbf8`. The full local +Vitest run was explicitly stopped after repeated embedded PostgreSQL bootstrap +failures; `/tmp/sleep48-full-tests.log` preserves those failures. It is not a +full-suite pass. Focused database acceptance uses the validated external +PostgreSQL path and isolated disposable test databases. + +Remote CI at `79bf36cbf8` also exposed a late-journal test that assumed worker +startup drains all candidates in one batch. It now waits for actual settlement, +as the shared worker deliberately processes fair batches. The workspaces-b job +hit a five-second sandbox staging test timeout; E2E shard five received a runner +shutdown signal. These failures remain recorded in `/tmp/sleep48-ci-chat3.log`, +`/tmp/sleep48-ci-workspaces-b.log`, and `/tmp/sleep48-ci-e2e5.log`. Fresh-head CI and +review are required after the repair; no failing gate is waived. + + +## Full-history test repair, 2026-10-26 + +At `c4beddad1e`, fresh Greptile review is 5/5 and the scope thread is resolved. +Remote CI passed 51 contexts but failed the chat shard and its aggregate gate: +a second startup assertion assumed `worker.ready` drained every fair batch. +The full local chat file finished with 1,595 passing and seven failing tests +(`/tmp/sleep48-chat-full.log`); all seven were in the newly added worker tests. + +The larger run exposed two related observation errors: startup can have another +batch scheduled, and `nextWakeAt() === null` can be transient while a batch runs. +The latter caused query instrumentation to count unfinished startup checks and +an outcome assertion to run before its response had been materialized. An early +future-deadline test failure then left its fixture's real deadline for later tests. + +Worker startup/settlement assertions now require both zero active work and the +expected next deadline. This changes test observation, not scheduler behavior or +batch limits. All 18 affected cases pass against the retained history database +from the full 1,602-case run (`/tmp/sleep48-settlement-history.log`), including +single-query unrelated wakes, future deadlines, and coordinator retry release. +The full-run failure remains recorded; fresh remote CI is still required. + + +## Current migration 0412 integration and complete acceptance + +The original response-presentation patch is integrated with the validated current +native-effect parent. The complete owned runtime/schema changes are preserved, +including fourteen files with combined upstream edits and 10,401 unchanged +unrelated upstream source files. The control-plane fixture conflict retains its +dedicated external database path and the current recovery-helper fallback; all +upstream conformance cases and owned assertions remain. Both audit histories +are retained. + +Only this slice's unlanded migration artifacts were deferred before integration. +`pnpm db:generate` generated `0412_tearful_squadron_supreme.sql`. The generated +index definition was checked against the original index before restoring all +three original statements byte for byte: concurrent idempotent index creation, +selection statistics, and ANALYZE. No original tuning statement is dropped. All +prior executable SQL, journal entries and retained snapshot bytes are unchanged. +Five snapshots 0408 through 0412 remain; the concurrent allowlist includes 0412. + +Source `9feec170ef8174011fc3c5d3d6d76f0a0f39fea0` passes **4171 distinct cases +in 29 complete files**, without skips: 2313 owner/producer/consumer cases in +221.96 seconds, 59 safe-replacement cases in 29.10 seconds, and all 1799 +assembled chat integration cases in 769.87 seconds. This includes the complete +GitHub workflow, accepted-response worker lifecycle, authenticated late evidence, +outer commit/rollback, contention, authority registration, agent resume, startup, +retry, drain/resume, and unrelated retained-history isolation. Per-file counts: + +| Complete file | Passing cases | +| --- | ---: | +| src/__tests__/chat-channels.integration.test.ts | 1799 | +| src/__tests__/committed-chat-response-authority.test.ts | 7 | +| src/__tests__/issue-recovery-actions.test.ts | 89 | +| src/__tests__/native-finalization-recovery.test.ts | 32 | +| src/__tests__/native-interaction-bridge.test.ts | 7 | +| src/__tests__/native-status-arbiter-corpus.test.ts | 94 | +| src/__tests__/native-workspace-sync.test.ts | 10 | +| src/__tests__/recovery-stale-issue-lock-sweep.test.ts | 40 | +| src/__tests__/server-startup-feedback-export.test.ts | 69 | +| src/__tests__/slack-conversation-lifecycle.test.ts | 41 | +| src/modules/run-dispatch/adapters/postgres.test.ts | 69 | +| src/modules/wake-queue/adapters/postgres.test.ts | 93 | +| src/modules/wake-queue/adapters/queued-comment-postgres.test.ts | 5 | +| src/services/chat-monitor-continuation.test.ts | 21 | +| src/services/execution-control-reconciliation.test.ts | 10 | +| src/services/explicit-native-continuation.test.ts | 335 | +| src/services/heartbeat/queue.test.ts | 32 | +| src/services/heartbeat/recovery.test.ts | 45 | +| src/services/heartbeat/run-control.test.ts | 51 | +| src/services/heartbeat/run-lifecycle.test.ts | 34 | +| src/services/heartbeat/upgrade-readiness.test.ts | 7 | +| src/services/legacy-execution-recovery.test.ts | 15 | +| src/services/native-runtime/native-restart-recovery.test.ts | 25 | +| src/services/native-runtime/native-run-finalizer-telemetry.test.ts | 29 | +| src/services/native-runtime/native-safe-replacement.test.ts | 59 | +| src/services/native-runtime/native-session-executor.test.ts | 968 | +| src/services/native-runtime/native-workspace-export-retry.test.ts | 80 | +| src/services/native-runtime/native-workspace-reconstruction.test.ts | 50 | +| src/services/native-runtime/paperclip-control-plane-port.test.ts | 55 | + +The complete 94-case status-authority corpus also passes again on the same +database in 16.68 seconds. It is not double-counted. Three retained PSC/OUT and +sentinel company rows remain unchanged, and each invocation uses distinct +UUID-derived fixture prefixes. Twenty-four separate fresh PostgreSQL 17 +databases migrate through 0412 before setup. Four embedded-only fixture imports +temporarily use their own external database on this IPC-exhausted host, then +restore byte for byte. All assertions and timing limits remain unchanged. This +is qualified external-database acceptance, not an unmodified embedded CI run. + +Actual database build, server types/build, module boundaries and exact-head +migration order pass. A populated 0411-to0412 upgrade preserves all 410 prior +migration receipts and all 38 companies, appends exactly one receipt, and adds +none on replay. The partial index is valid and ready on both fresh and upgraded +databases; the existing selection statistic is present. Earlier failed runs +above retain their historical scopes and explanations. The current complete +chat run passes, rather than substituting a focused retry for the full suite. + +Normal merge `1c019b48c98cd8273aa3b7e3cf95433a369b9531` adopts published native-effect +parent `5bb5ddacd3d1e6817de4825b283c6e92f85bd124` and actual unsafe-export ancestry. +All complete server, package, UI and test trees remain identical to the accepted +source. Publication remains held until #17690 lands and its actual squash +ancestry is reconciled. Fresh published-head CI and exact-head Greptile 5/5 +remain required. These checks do not establish deployed suspension, external +Cloud alarm delivery or fleet savings. + + +## Actual native-effect prerequisite ancestry + +Native-effect prerequisite #17690 merged as +`a8a038e659c50c1c5e3ce532d13072a29a5cf951`, reviewed at +`5bb5ddacd3d1e6817de4825b283c6e92f85bd124`. Its entire squash tree equals +the reviewed parent already adopted by this child. The normal ancestry merge +automatically restored the removed pending-effect notification module. The +whole-tree identity check rejected that composition; its deletion was restored +before publication. All accepted source, including the complete 0412 database +tree and all 29 test files, remains byte-identical. The following edit records +the ancestry proof. The 4171-case acceptance keeps its original source scope; +fresh published-head CI and exact-head Greptile 5/5 are required. + + +## 2026-10-29 review repair: routine hints cannot invalidate a company + +Superagent review of `a5648c92f4601fb72cec26b9ff2e99e40d502042` identified a +real P2: coalesced inbound messages used a company-only milestone hint, and +the new response fanout reconstructed every unpresented response in that +company. Pending-effect settlement also used a company-only finalization hint. +The current-head scan required action, so the branch was not merged. Greptile +rated that head 5/5 but conditioned readiness on the later #17702 scoped +pending-effect fix. That fix is included here; this slice no longer requires +its later dependent PR to be independently correct. + +The repair keeps routine milestones and finalization hints from entering the +response queue without an issue or run. Real coalescing names its already-known +active run. Pending-effect settlement names the affected issue, covering older +decisions on that issue. Retry promotion/cancellation retain their known run +when older context omits an issue. Existing pending-effect and chat consumers +keep their previous serialized hints; only response reconstruction is narrowed. + +Genuine company-wide authority changes remain supported through one explicit +`notifyChatCompanyAuthorityChange` boundary. Sixteen existing mutation hooks in +nine owning files use it: membership and board-claim restoration, Cloud actor +membership repair, GitHub/Slack identity and sponsor changes, endpoint policy, +resource enablement and connection configuration. They keep their original +change predicates and transaction owners. Two issue-scoped resource/publication +hooks remain issue-scoped. No transaction wrappers, queue tables, polling or +new persisted bookkeeping are introduced. Dynamic milestone producers carry +issue-bound interaction rows or typed recovery inputs. + +The ordinary coalescing regression first fails on the actual PostgreSQL writer: +its settled hint has only the company instead of the active run. Two receiver +regressions each issue three routine company hints against dormant response +history and observe **12 response-history SELECTs**. Both must observe zero. +A positive test restores suspended membership through the actual access owner +and delivers the waiting response once; it passes before and after the repair. +The unchanged four focused assertions pass after the fix. Assertion time limits +are unchanged. These tests measure real driver SQL and persisted producer work. + +Repair source `2fc8b0614aaf9ac7337045de4ced7325f95edc45` passes **1821 distinct +server cases**: 1496 in 13 complete suites in 110.63 seconds, and 325 selected +chat cases in 278.99 seconds. The chat selection includes the complete GitHub +workflow, accepted-response lifecycle, the new regressions, and real post-close +authority restoration. Another 1477 chat cases are outside the name pattern; +the complete 1802-case file is not claimed for this repair run. Counts: + +| File | Passing cases | +| --- | ---: | +| src/__tests__/access-service.test.ts | 18 | +| src/__tests__/auth-session-route.test.ts | 12 | +| src/__tests__/board-claim.test.ts | 3 | +| src/__tests__/chat-channels.integration.test.ts (focused) | 325 | +| src/__tests__/committed-chat-response-authority.test.ts | 7 | +| src/__tests__/native-finalization-recovery.test.ts | 32 | +| src/__tests__/native-status-arbiter-corpus.test.ts | 94 | +| src/__tests__/server-startup-feedback-export.test.ts | 69 | +| src/modules/run-dispatch/adapters/postgres.test.ts | 69 | +| src/modules/wake-queue/adapters/postgres.test.ts | 94 | +| src/modules/wake-queue/application/use-cases.test.ts | 67 | +| src/services/heartbeat/run-lifecycle.test.ts | 34 | +| src/services/native-runtime/native-run-finalizer-telemetry.test.ts | 29 | +| src/services/native-runtime/native-session-executor.test.ts | 968 | + +Eleven dedicated fresh PostgreSQL 17 databases migrate through unchanged 0412. +The wake-queue fixture's embedded-only import temporarily uses its own external +database and restores byte for byte. All 14 selected source files match their +pre-run hashes. Actual database build, server types/build, module boundaries +and full-SHA migration order pass. The earlier 4171-case full baseline keeps +its original pre-repair source scope; it is not represented as this repair run. + +Master independently advanced to `522c653d562a41ecad64561d35457cb99ecf6cd6` +with #18126's Pi archive-download retry fix. Normal merge +`2a535bc61259ef99387369e8b32498f774e5ba2c` preserves both complete binary patches. +All accepted server, database, UI, test and CLI trees remain identical; only +the three independently changed runner installer/manifest files are added. +All 13 installer tests pass in 200.18 milliseconds, including stalled-header +and stalled-body cancellation. Combined server types/build, boundaries and +migration order pass. No unrelated upstream source is discarded. + +Explicit global authority invalidation and startup still reconstruct their +company's owed candidates with existing 100-row discovery pages and fair +execution batches. Discovery retains linear time and memory; it is not a +constant-cost global operation. The prior 10,001-candidate profile remains +relevant to those genuine global obligations. This repair removes ordinary +inbound traffic's ability to request that global reconstruction and prevents +unscoped routine traffic from rebuilding dormant response history. It does not waive an actual review +failure or establish deployed sleep, Cloud wake delivery or fleet savings. +Fresh published-head CI, Greptile 5/5 and a clean security scan remain required. + + +A second publication guard detected master #18128 at +`e8b78cce5bb557b086bee9ca930eeaab1d09b244`. Normal merge +`06d55ca9f872f7e7794680335811b496942be034` retains its three test-only file changes. +Both complete owned and upstream binary patches are preserved. An isolated Git +index projects only the owned audit onto the original repair source to prove +this equality; it does not rewrite branch history or include master changes +as owned edits. All 14 repair test hashes and the entire package tree remain +unchanged. The changed HTTPS backfill case passes on a fresh 0412 PostgreSQL +fixture in 7.62 seconds; 178 other workspace cases are outside that focused +selection. Its embedded-only fixture import restores byte for byte. Combined +server types/build, boundaries and migration order pass again. The independent +browser fixture changes are preserved exactly; no local browser run is claimed. + +### Recorded presentation and structural eligibility repair + +The exact published head `ae75e21bd5aa5841b66f0d3f421181cdcfa9a3be` +received Greptile 5/5, a clean security scan, and passing CI. Its completed +Ponytail 4/5 verdict nevertheless required two correctness fixes. That actual +failing verdict was honored; no review timeout or numeric score waived it. +The review also found a missing fresh-commit skip guard, which is repaired here. + +A recorded non-create presentation decision (`none`, `reuse`, or another explicit +non-create value) now excludes both discovery and locked recovery. A string +selected comment ID still fences recovery, including deleted comments. The +`skipIssueComment` policy applies to all recovery branches. Missing legacy +presentation metadata and explicit `create` with a null comment ID remain owed +work and can recover once. No per-worker table or transaction wrapper is added. + +Discovery joins the owning company/issue and applies the cheap durable guards +before taking recovery locks: done results belong to succeeded non-conversation +runs; response waits have empty attention requests and usable summary/key; +monitor and same-agent continuations require succeeded runs and their accepted +reason codes. Digest integrity, current publication authority, provider-owned +interactions, final-message selection and other owner checks remain under the +existing transaction. These predicates narrow structural candidates; they do not +claim to establish authorization or replace late-evidence and authority-change +notifications. Fresh finalization now honors `skipNativeResponsePresentation` +as the already-committed path does. + +The final corrected regression source is unchanged between the red and green +runs. The published baseline fails eight behavioral assertions and passes the +three positive genuine-answer cases. Repair source +`892f36e5f66ba1af123034936a7160c64c3ab883` passes all eleven. Recorded none/reuse +produce no new comments or publications and keep their decision. Five families +of structurally ineligible retained rows produce zero issue/run/finalization +`FOR UPDATE` queries across three issue-scoped notifications each. Those selector +fixtures deliberately vary durable payload fields; integrity and authorization +remain separate recovery requirements. Genuine missing, legacy and create-null +presentations still publish once. A genuinely uncommitted accepted run proves +that periodic finalization leaves presentation to the worker. + +Earlier material failures remain in the verification record: an incomplete +conversation identity fixture, a duplicate variable declaration, a remaining +stale variable reference and its downstream retry contamination, a test that +accidentally used the matching continuation reason, and a missing fixture import. +The fixture was corrected to exercise the actual incompatible reason; assertions +and time limits were not weakened. The final red comparison restores the exact +published production files temporarily and verifies byte-exact restoration of +the repair afterward. These were explained local setup/implementation failures, +not inferred infrastructure faults. + +Repair source passes **1489 distinct cases**: 1139 across six complete consumer +suites in 81.02 seconds, plus 350 selected chat integration cases in 297.68s. +The remaining 1463 chat cases are outside the name pattern; the full 1813-case +file is not claimed for this repair. The eleven new regressions are included +in the selected count, not added again. The six complete files cover 968 native +session cases, 79 response-summary cases, 21 monitor-continuation cases, 29 native +finalizer cases, 32 finalization-recovery cases and 10 workspace-sync cases. + +No executable migration, schema, prior history, index/statistics statement or +snapshot changes in this repair. Five fresh PostgreSQL 17 fixtures migrate +through 0412; no embedded fixture adaptation is needed for these suites. All +selected test hashes match the captured source. Database build, server +types/build, module boundaries and full-SHA migration ordering pass. Actual host +sleep, external wake and fleet cost savings remain unqualified. A newly published +head still requires fresh exact-head Greptile 5/5, passing CI and no actionable +review findings before merge. diff --git a/packages/db/src/client.ts b/packages/db/src/client.ts index 6e562381f7c..55d1a1ca9a1 100644 --- a/packages/db/src/client.ts +++ b/packages/db/src/client.ts @@ -561,6 +561,7 @@ const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([ "0409_uneven_zuras.sql", "0410_freezing_kinsey_walden.sql", "0411_real_scrambler.sql", + "0412_tearful_squadron_supreme.sql", "0384_funny_adam_warlock.sql", diff --git a/packages/db/src/migrations/0412_tearful_squadron_supreme.sql b/packages/db/src/migrations/0412_tearful_squadron_supreme.sql new file mode 100644 index 00000000000..ca0c9e198e3 --- /dev/null +++ b/packages/db/src/migrations/0412_tearful_squadron_supreme.sql @@ -0,0 +1,3 @@ +CREATE INDEX CONCURRENTLY IF NOT EXISTS "heartbeat_runs_native_unpresented_response_idx" ON "heartbeat_runs" USING btree ("company_id","native_issue_id","id") WHERE "heartbeat_runs"."runtime_mode" = 'native' AND "heartbeat_runs"."status" in ('succeeded', 'failed', 'timed_out') AND "heartbeat_runs"."finished_at" is not null AND jsonb_typeof("heartbeat_runs"."result_json" #> '{presentationDecision,commentId}') is distinct from 'string';--> statement-breakpoint +CREATE STATISTICS IF NOT EXISTS "heartbeat_runs_native_response_selection_stats" ON (jsonb_typeof("result_json" #> '{presentationDecision,commentId}')) FROM "heartbeat_runs";--> statement-breakpoint +ANALYZE "heartbeat_runs"; diff --git a/packages/db/src/migrations/meta/0407_snapshot.json b/packages/db/src/migrations/meta/0412_snapshot.json similarity index 99% rename from packages/db/src/migrations/meta/0407_snapshot.json rename to packages/db/src/migrations/meta/0412_snapshot.json index 54cb791facf..698666748b6 100644 --- a/packages/db/src/migrations/meta/0407_snapshot.json +++ b/packages/db/src/migrations/meta/0412_snapshot.json @@ -1,6 +1,6 @@ { - "id": "856a147c-73dc-4908-a8d9-4f19cab7dbf6", - "prevId": "c34fcd91-0d57-4241-b8b4-399bc9f6dcb0", + "id": "3b8bb508-028d-4154-abdd-af870f1f3ca6", + "prevId": "ff56ee93-8cfc-465a-b26a-1c12d10773ff", "version": "7", "dialect": "postgresql", "tables": { @@ -2665,6 +2665,28 @@ } }, "indexes": { + "agent_instruction_copies_collection_idx": { + "name": "agent_instruction_copies_collection_idx", + "columns": [ + { + "expression": "run_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "NOT (coalesce(\"agent_instruction_working_copies\".\"receipt\", '{}'::jsonb) ? 'retainedByRunId') AND (\n (\"agent_instruction_working_copies\".\"attempts\" < 3 AND (\"agent_instruction_working_copies\".\"state\" IN ('prepared', 'pending_collection', 'unchanged_turn', 'warm_saved')\n OR (\"agent_instruction_working_copies\".\"state\" = 'preparing' AND \"agent_instruction_working_copies\".\"receipt\"->>'schema' = 'paperclip.agent-files.v1')))\n OR (\"agent_instruction_working_copies\".\"state\" = 'unavailable' AND \"agent_instruction_working_copies\".\"process_stopped_at\" IS NULL\n AND \"agent_instruction_working_copies\".\"location\" LIKE 'remote:%' AND \"agent_instruction_working_copies\".\"receipt\"->>'schema' = 'paperclip.agent-files.v1'))", + "concurrently": false, + "method": "btree", + "with": {} + }, "agent_instruction_copies_cleanup_idx": { "name": "agent_instruction_copies_cleanup_idx", "columns": [ @@ -33749,6 +33771,56 @@ } }, "indexes": { + "heartbeat_runs_native_unpresented_response_idx": { + "name": "heartbeat_runs_native_unpresented_response_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "native_issue_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"heartbeat_runs\".\"runtime_mode\" = 'native' AND \"heartbeat_runs\".\"status\" in ('succeeded', 'failed', 'timed_out') AND \"heartbeat_runs\".\"finished_at\" is not null AND jsonb_typeof(\"heartbeat_runs\".\"result_json\" #> '{presentationDecision,commentId}') is distinct from 'string'", + "concurrently": false, + "method": "btree", + "with": {} + }, + "heartbeat_runs_cancelled_workspace_preservation_idx": { + "name": "heartbeat_runs_cancelled_workspace_preservation_idx", + "columns": [ + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"heartbeat_runs\".\"runtime_mode\" = 'native' AND \"heartbeat_runs\".\"status\" = 'cancelled'\n AND \"heartbeat_runs\".\"runner_profile_json\"->'nativeWorkspaceSync'->>'state' = 'prepared'\n AND \"heartbeat_runs\".\"result_json\"->'nativeCancellation'->>'dispatchState' = 'acknowledged'\n AND (\"heartbeat_runs\".\"result_json\"->'workspaceRestoreRecovery'->>'schema' = 'paperclip.workspace-restore-recovery.v1'\n AND \"heartbeat_runs\".\"result_json\"->>'workspaceRestoreFailure' IN ('restore_permission_denied', 'restore_lock_timeout', 'restore_unsafe_archive', 'restore_failed')) IS NOT TRUE", + "concurrently": false, + "method": "btree", + "with": {} + }, "heartbeat_runs_retained_recovery_adoption_idx": { "name": "heartbeat_runs_retained_recovery_adoption_idx", "columns": [ @@ -43064,6 +43136,62 @@ } }, "indexes": { + "native_run_finalizations_pending_effect_owner_idx": { + "name": "native_run_finalizations_pending_effect_owner_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "issue_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "decision_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"native_run_finalizations\".\"decision_id\" is not null AND \"native_run_finalizations\".\"result_id\" is not null AND \"native_run_finalizations\".\"phase\" <> 'terminal_failure'", + "concurrently": false, + "method": "btree", + "with": {} + }, + "native_run_finalizations_unsafe_export_recovery_idx": { + "name": "native_run_finalizations_unsafe_export_recovery_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "issue_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "run_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"native_run_finalizations\".\"phase\" = 'terminal_failure' AND \"native_run_finalizations\".\"failure_code\" = 'native_workspace_sync_out_unsafe_archive'", + "concurrently": false, + "method": "btree", + "with": {} + }, "native_run_finalizations_control_deadline_idx": { "name": "native_run_finalizations_control_deadline_idx", "columns": [ @@ -52368,6 +52496,34 @@ } }, "indexes": { + "status_decision_effects_pending_delivery_idx": { + "name": "status_decision_effects_pending_delivery_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "issue_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "decision_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"status_decision_effects\".\"delivery_state\" in ('pending', 'failed')", + "concurrently": false, + "method": "btree", + "with": {} + }, "status_decision_effects_decision_ordinal_uq": { "name": "status_decision_effects_decision_ordinal_uq", "columns": [ diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index f02c1c52a93..76a38c88a5b 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -2871,6 +2871,13 @@ "when": 1793246172969, "tag": "0411_real_scrambler", "breakpoints": true + }, + { + "idx": 412, + "version": "7", + "when": 1793249495056, + "tag": "0412_tearful_squadron_supreme", + "breakpoints": true } ] } \ No newline at end of file diff --git a/packages/db/src/schema/heartbeat_runs.ts b/packages/db/src/schema/heartbeat_runs.ts index 99f06a955dc..9e215f825f0 100644 --- a/packages/db/src/schema/heartbeat_runs.ts +++ b/packages/db/src/schema/heartbeat_runs.ts @@ -111,6 +111,9 @@ export const heartbeatRuns = pgTable( updatedAt: timestamp("updated_at", { withTimezone: true }).notNull().defaultNow(), }, (table) => ({ + nativeUnpresentedResponseIdx: index("heartbeat_runs_native_unpresented_response_idx") + .on(table.companyId, table.nativeIssueId, table.id) + .where(sql`${table.runtimeMode} = 'native' AND ${table.status} in ('succeeded', 'failed', 'timed_out') AND ${table.finishedAt} is not null AND jsonb_typeof(${table.resultJson} #> '{presentationDecision,commentId}') is distinct from 'string'`), cancelledWorkspacePreservationIdx: index("heartbeat_runs_cancelled_workspace_preservation_idx").on(table.id, table.companyId) .where(sql`${table.runtimeMode} = 'native' AND ${table.status} = 'cancelled' AND ${table.runnerProfileJson}->'nativeWorkspaceSync'->>'state' = 'prepared' diff --git a/server/src/__tests__/chat-channels.integration.test.ts b/server/src/__tests__/chat-channels.integration.test.ts index 3c2bc210f31..c24d5eb5051 100644 --- a/server/src/__tests__/chat-channels.integration.test.ts +++ b/server/src/__tests__/chat-channels.integration.test.ts @@ -1,6 +1,8 @@ +import { notifyNativeFinalizationWork } from "../services/native-finalization-notifications.js"; +import { nativeResponsePresentationWork } from "../services/native-runtime/native-response-presentation-work.js"; +import { notifyChatRunMilestones } from "../services/chat-run-milestone-notifications.js"; import { registerGitHubReviewProjectionWork } from "../services/chat-channels/github/review-projection-work.js"; import { retainGitHubReviewReceipt } from "../services/chat-channels/github/task-preparation.js"; -import { notifyChatRunMilestones } from "../services/chat-run-milestone-notifications.js"; import { settleUnrecoverableExecutions } from "../services/execution-recovery-resolution.js"; import { recordRunPresentationDecision } from "../services/heartbeat/run-completion.js"; import { environmentService } from "../services/environments.js"; @@ -16,13 +18,13 @@ import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db"; import { claimTaskRetryInput, releaseUnadmittedTaskRetryInput } from "../services/chat-channels/task-retry-input.js"; import { restoreNativeChatReviewPresentationInTransaction } from "../services/native-runtime/native-chat-review-presentation.js"; import { notifyChatEndpointConfiguration, notifyChatCompanyConfiguration } from "../services/chat-channels/configuration-work-notifications.js"; +import { configureAgentLifecycle, createAgentLifecycle, processAgentLifecycle } from "../services/agent-lifecycle.js"; import { deleteCompany } from "../services/company-deletion.js"; import { adapterExecutionControls, createAdapterExecutionControl } from "../services/adapter-execution-control.js"; import { chatRunSourceMessageId } from "../services/chat-channels/run-origin.js"; import * as durableChatWakeup from "../services/durable-chat-wakeup.js"; import { conflict } from "../errors.js"; import * as githubTaskPreparation from "../services/chat-channels/github/task-preparation.js"; -import { configureAgentLifecycle, createAgentLifecycle } from "../services/agent-lifecycle.js"; import { budgetPolicies } from "@paperclipai/db"; import { budgetService } from "../services/budgets.js"; import { idleWorkSnapshot } from "../services/task-admission.js"; @@ -150,6 +152,8 @@ import { issues, nativeRunResults, nativeRunFinalizations, + statusDecisions, + statusDecisionEffects, environmentLeases, environments, principalPermissionGrants, @@ -267,6 +271,7 @@ import type { import { finalizeNativeRun, repairCommittedNativeChatResponse, + recoverCommittedNativeChatResponse, } from "../services/native-runtime/native-run-finalizer.js"; import { NativeRunCoordinatorStore } from "../services/native-runtime/native-run-coordinator-store.js"; import { reconcileNativeFinalizations } from "../services/native-runtime/native-finalization-reconciler.js"; @@ -66562,6 +66567,19 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { const input = { companyId: context.fixture.companyId, issueId: context.issue.id, agentId: context.fixture.assignedAgentId, failedRunId: context.runId, initiatedByUserId: "owner-user" }; const stage = () => db.transaction(tx => context.service.prepareFailedChatRunRetry(tx, input)); + const observedResponseAdmissions: string[] = []; + const responseHints: string[] = []; + await coordinator.register(DELIVERY_QUEUES.nativeResponsePresentation, { + retryMs: 30_000, + onSettled: hints => { responseHints.push(...hints ?? []); }, + run: async () => { + const admitted = await db.select({ id: chatActions.id }).from(chatActions).where(and( + eq(chatActions.companyId, context.fixture.companyId), eq(chatActions.kind, "failed_run_retry"), eq(chatActions.status, "processed"), + )); + observedResponseAdmissions.push(...admitted.map(row => row.id)); + }, + hasPending: async () => false, + }).ready; context.wakeup.mockClear(); try { let actionId: string; @@ -66607,6 +66625,8 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { expect(action?.result.attemptCount).toBe(mode === "retry" ? 2 : 1); expect(coordinator.nextWakeAt()).toBeNull(); }, { timeout: 10_000 }); + await vi.waitFor(() => expect(observedResponseAdmissions).toContain(actionId), { timeout: 5000 }); + expect(responseHints.map(hint => JSON.parse(hint))).toContainEqual({ companyId: context.fixture.companyId, issueId: context.issue.id }); expect(context.wakeup).toHaveBeenCalledTimes(mode === "retry" ? 2 : 1); const receipts = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, actionId)); expect(receipts).toHaveLength(1); @@ -74255,6 +74275,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { context: Awaited>, ) => Promise, summary?: (issueId: string) => string, + finalize = true, ) { const context = await failedChatRetryFixture( provider, @@ -74409,7 +74430,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { .from(nativeRunResults) .where(eq(nativeRunResults.runId, context.runId)); const accepted = { resultId: acceptedRow.id }; - await finalizeNativeRun({ + if (finalize) await finalizeNativeRun({ db, runId: context.runId, workspaceFinalizeStatus: "succeeded", @@ -74418,7 +74439,8 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { .select() .from(nativeRunFinalizations) .where(eq(nativeRunFinalizations.runId, context.runId)); - expect(coordinator.phase).toBe("committed"); + if (finalize) expect(coordinator.phase).toBe("committed"); + else expect(coordinator.phase).not.toBe("committed"); await db .update(heartbeatRuns) .set({ @@ -74616,6 +74638,11 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { }) : context.repair(), ).resolves.toBe(false); + await expect(recoverCommittedNativeChatResponse(db, { + companyId: mutation === "cross_company" ? randomUUID() : context.fixture.companyId, + issueId: context.issue.id, + runId: mutation === "cross_run" ? randomUUID() : context.runId, + })).resolves.toBe(false); await expect( db .select() @@ -75257,6 +75284,519 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { }, ); + it.each(["none", "reuse"] as const)("preserves a recorded %s presentation instead of recovering an unsolicited answer", async commentAction => { + const context = await committedChatResponseRecoveryFixture("telegram"); + const presentationDecision = { commentAction, commentId: null, reasonCodes: ["interaction_owns_response"] }; + try { + await db.update(heartbeatRuns).set({ resultJson: { presentationDecision } }).where(eq(heartbeatRuns.id, context.runId)); + await expect(recoverCommittedNativeChatResponse(db, { companyId: context.fixture.companyId, + issueId: context.issue.id, runId: context.runId })).resolves.toBe(false); + expect(await nativeResponsePresentationWork(db).load({ companyId: context.fixture.companyId, runId: context.runId })).toEqual([]); + expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, context.runId))).toEqual([]); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toEqual([]); + expect((await readHeartbeatRunsById(context.runId))[0].resultJson).toEqual({ presentationDecision }); + } finally { await context.service.shutdown(); } + }); + + it.each(["create", "legacy", "missing"] as const)("recovers a genuine unanswered %s presentation", async action => { + const context = await committedChatResponseRecoveryFixture("telegram"); + try { + await db.update(heartbeatRuns).set({ resultJson: action === "missing" ? {} : { + presentationDecision: { commentId: null, ...(action === "create" ? { commentAction: "create" } : {}) }, + } }).where(eq(heartbeatRuns.id, context.runId)); + const work = nativeResponsePresentationWork(db); + const entries = await work.load({ companyId: context.fixture.companyId, runId: context.runId }); + expect(entries).toHaveLength(1); + await expect(work.check(entries[0])).resolves.toBeNull(); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1); + await expect(context.repair()).resolves.toBe(false); + } finally { await context.service.shutdown(); } + }); + + it.each(["conversation_done", "attention_request", "same_agent_reason", "monitor_reason", "skip_comment"] as const)( + "does not lock durably ineligible %s history on repeated issue milestones", async kind => { + const context = await committedChatResponseRecoveryFixture("telegram"); + await context.service.shutdown(); + await db.update(heartbeatRuns).set({ status: "succeeded", + ...(kind === "skip_comment" ? { contextSnapshot: { skipIssueComment: true } } : {}) }).where(eq(heartbeatRuns.id, context.runId)); + const [accepted] = await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, context.accepted.resultId)); + const envelope = accepted.resultJson as { result: PrpStructuredRunResult; terminal: PrpTerminalState }; + if (kind === "conversation_done") { + envelope.result = { ...envelope.result, reportedWorkDisposition: "done" }; + envelope.terminal = { ...envelope.terminal, reportedWorkDisposition: "done" }; + await db.update(issues).set({ conversationAgentId: context.fixture.assignedAgentId, conversationUserId: "owner-user", conversationState: "active", assigneeUserId: null }) + .where(eq(issues.id, context.issue.id)); + } else if (kind === "attention_request") { + envelope.result = { ...envelope.result, attentionRequests: [{ kind: "review", summary: "Needs a decision", ownerClass: "human" }] }; + } else if (kind === "same_agent_reason" || kind === "monitor_reason") { + envelope.result = { ...envelope.result, continuation: { ...envelope.result.continuation, + kind: kind === "same_agent_reason" ? "same_agent" : "monitor" } as PrpStructuredRunResult["continuation"] }; + await db.update(statusDecisions).set({ reasonCode: "external_chat_response_waiting" }) + .where(eq(statusDecisions.id, context.coordinator.decisionId!)); + } + await db.update(nativeRunResults).set({ resultJson: envelope }).where(eq(nativeRunResults.id, accepted.id)); + const errors: unknown[] = []; + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) }); + const work = nativeResponsePresentationWork(db); + const worker = work.register(coordinator, () => true); + const debug = db.$client.options.debug; + const locks: string[] = []; + try { + await worker.ready; + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), { timeout: 10_000 }) + .toEqual({ active: 0, next: null }); + db.$client.options.debug = (_connection, query) => { + if (query.startsWith("select") && query.includes("for update") && /"(?:issues|heartbeat_runs|native_run_finalizations)"/.test(query)) locks.push(query); + }; + for (let i = 0; i < 3; i++) { + await db.transaction(tx => notifyChatRunMilestones(tx, { companyId: context.fixture.companyId, issueId: context.issue.id })); + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), { timeout: 5000 }) + .toEqual({ active: 0, next: null }); + } + expect(locks).toEqual([]); + expect(await work.load({ companyId: context.fixture.companyId, issueId: context.issue.id })).toEqual([]); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toEqual([]); + expect(errors).toEqual([]); + } finally { db.$client.options.debug = debug; worker.stop(); await coordinator.stop(); } + }); + + it("leaves fresh-commit accepted-response presentation to the worker when periodic replay skips it", async () => { + const context = await committedChatResponseRecoveryFixture("telegram", "coordinator", false, undefined, undefined, false); + try { + await finalizeNativeRun({ db, runId: context.runId, workspaceFinalizeStatus: "succeeded", + projectRunStatus: true, skipNativeResponsePresentation: true }); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toEqual([]); + expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, context.runId))).toEqual([]); + const work = nativeResponsePresentationWork(db); + const entries = await work.load({ companyId: context.fixture.companyId, runId: context.runId }); + expect(entries).toHaveLength(1); + await expect(work.check(entries[0])).resolves.toBeNull(); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1); + } finally { await context.service.shutdown(); } + }); + + it.each(["reconciler", "finalizer"])("leaves accepted-response presentation to its worker during periodic %s replay", async caller => { + const context = await committedChatResponseRecoveryFixture("telegram"); + try { + if (caller === "reconciler") { + await reconcileNativeFinalizations(db, [context.runId], { skipNativeResponsePresentation: true }); + } else { + await finalizeNativeRun({ db, runId: context.runId, workspaceFinalizeStatus: "succeeded", + projectRunStatus: true, skipNativeResponsePresentation: true }); + } + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0); + expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, context.runId))).toHaveLength(0); + const work = nativeResponsePresentationWork(db); + const entries = await work.load({ companyId: context.fixture.companyId, runId: context.runId }); + expect(entries).toHaveLength(1); + await expect(work.check(entries[0])).resolves.toBeNull(); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1); + await expect(context.repair()).resolves.toBe(false); + } finally { await context.service.shutdown(); } + }); + + it("recovers accepted-response work on startup and leaves selected history quiet", async () => { + const context = await committedChatResponseRecoveryFixture("telegram"); + const errors: unknown[] = []; + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) }); + const work = nativeResponsePresentationWork(db); + const worker = work.register(coordinator, () => true); + try { + await worker.ready; + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), + { timeout: 10_000 }).toEqual({ active: 0, next: null }); + const comments = await db.select().from(issueComments).where(eq(issueComments.createdByRunId, context.runId)); + expect(comments).toHaveLength(1); + expect(comments[0].body).toBe(context.result.summary); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1); + expect(await work.load({ companyId: context.fixture.companyId })).toEqual([]); + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true }); + const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction"); + try { + await vi.advanceTimersByTimeAsync(120_000); + expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); + } finally { reads.mockRestore(); writes.mockRestore(); } + expect(errors).toEqual([]); + } finally { worker.stop(); await coordinator.stop(); vi.useRealTimers(); await context.service.shutdown(); } + }); + + it("reactivates dormant accepted-response work when the live chat authority registers", async () => { + const context = await committedChatResponseRecoveryFixture("telegram"); + await context.service.shutdown(); + const errors: unknown[] = []; + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) }); + const worker = nativeResponsePresentationWork(db).register(coordinator, () => true); + let replacement: ReturnType | undefined; + try { + await worker.ready; + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), + { timeout: 10_000 }).toEqual({ active: 0, next: null }); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0); + replacement = createService(new FakeChatSdkRuntime(), fakeTelegramFetch() as typeof fetch); + await expect.poll(async () => (await readChatPublicationsByEndpointId(context.endpoint.id)).length, { timeout: 5000 }).toBe(1); + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), + { timeout: 10_000 }).toEqual({ active: 0, next: null }); + expect(errors).toEqual([]); + } finally { worker.stop(); await coordinator.stop(); await replacement?.service.shutdown(); } + }); + + it.each(["issue", "run"])("does not reload dormant accepted-response history for an unrelated %s wake", async scope => { + const context = await committedChatResponseRecoveryFixture("telegram"); + await context.service.shutdown(); + const errors: unknown[] = []; + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) }); + const work = nativeResponsePresentationWork(db); + const worker = work.register(coordinator, () => true); + try { + await worker.ready; + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), + { timeout: 10_000 }).toEqual({ active: 0, next: null }); + expect(await work.load({ companyId: context.fixture.companyId, runId: context.runId })).toHaveLength(1); + const previousDebug = db.$client.options.debug; + const queries: string[] = []; + db.$client.options.debug = (_connection, query) => { + if (query.startsWith("select") && query.includes("native_run_finalizations")) queries.push(query); + }; + try { + await db.transaction(tx => notifyChatRunMilestones(tx, { companyId: context.fixture.companyId, + ...(scope === "issue" ? { issueId: randomUUID() } : { runId: randomUUID() }) })); + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), + { timeout: 5000 }).toEqual({ active: 0, next: null }); + expect(queries).toHaveLength(1); + } finally { db.$client.options.debug = previousDebug; } + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0); + expect(errors).toEqual([]); + } finally { worker.stop(); await coordinator.stop(); } + }); + + it.each(["milestone", "finalization"] as const)("ignores repeated unscoped routine %s hints without rebuilding dormant accepted-response history", async source => { + const context = await committedChatResponseRecoveryFixture("telegram"); + await context.service.shutdown(); + const errors: unknown[] = []; + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) }); + const work = nativeResponsePresentationWork(db); + const worker = work.register(coordinator, () => true); + try { + await worker.ready; + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), { timeout: 10_000 }) + .toEqual({ active: 0, next: null }); + expect(await work.load({ companyId: context.fixture.companyId, runId: context.runId })).toHaveLength(1); + const debug = db.$client.options.debug; + const queries: string[] = []; + db.$client.options.debug = (_connection, query) => { + if (query.startsWith("select") && query.includes("native_run_finalizations")) queries.push(query); + }; + try { + for (let i = 0; i < 3; i++) { + await db.transaction(tx => source === "milestone" + ? notifyChatRunMilestones(tx, { companyId: context.fixture.companyId }) + : notifyNativeFinalizationWork(tx, { companyId: context.fixture.companyId })); + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), { timeout: 5000 }) + .toEqual({ active: 0, next: null }); + } + expect(queries).toEqual([]); + } finally { db.$client.options.debug = debug; } + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0); + expect(errors).toEqual([]); + } finally { worker.stop(); await coordinator.stop(); } + }); + + it("wakes dormant accepted-response history from the real company membership restoration owner", async () => { + const context = await committedChatResponseRecoveryFixture("telegram"); + const [membership] = await db.select().from(companyMemberships).where(and( + eq(companyMemberships.companyId, context.fixture.companyId), eq(companyMemberships.principalId, "owner-user"))); + await db.update(companyMemberships).set({ status: "suspended" }).where(eq(companyMemberships.id, membership.id)); + const errors: unknown[] = []; + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) }); + const worker = nativeResponsePresentationWork(db).register(coordinator, () => true); + try { + await worker.ready; + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), { timeout: 10_000 }) + .toEqual({ active: 0, next: null }); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0); + await accessService(db).updateMember(context.fixture.companyId, membership.id, { status: "active" }); + await expect.poll(async () => ({ count: (await readChatPublicationsByEndpointId(context.endpoint.id)).length, + active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), { timeout: 5000 }) + .toEqual({ count: 1, active: 0, next: null }); + expect(errors).toEqual([]); + } finally { worker.stop(); await coordinator.stop(); await context.service.shutdown(); } + }); + + it.each(["commit", "rollback"])("keeps accepted-response work behind the outer %s boundary", async outcome => { + const context = await committedChatResponseRecoveryFixture("telegram"); + await db.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, context.runId)); + const errors: unknown[] = []; + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) }); + const worker = nativeResponsePresentationWork(db).register(coordinator, () => true); + try { + await worker.ready; + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), + { timeout: 10_000 }).toEqual({ active: 0, next: null }); + const transaction = db.transaction(async tx => { + await tx.transaction(async nested => { + await notifyChatRunMilestones(nested, { companyId: context.fixture.companyId, issueId: context.issue.id, runId: randomUUID() }); + await nested.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, context.runId)); + }); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0); + expect(idleWorkSnapshot().active).toBeGreaterThan(0); + if (outcome === "rollback") throw new Error("response fixture rollback"); + }); + if (outcome === "rollback") await expect(transaction).rejects.toThrow("response fixture rollback"); + else await transaction; + await expect.poll(async () => ({ + publications: (await readChatPublicationsByEndpointId(context.endpoint.id)).length, + active: idleWorkSnapshot().active, + next: coordinator.nextWakeAt(), + }), { timeout: 5000 }).toEqual({ publications: outcome === "commit" ? 1 : 0, active: 0, next: null }); + expect(errors).toEqual([]); + } finally { worker.stop(); await coordinator.stop(); await context.service.shutdown(); } + }); + + it("retains the first accepted-response wake through idle drain without passive SQL", async () => { + const context = await committedChatResponseRecoveryFixture("telegram"); + await db.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, context.runId)); + let canRun = true; + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true }); + const errors: unknown[] = []; + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => canRun, canReconcile: () => true, onError: error => errors.push(error) }); + const worker = nativeResponsePresentationWork(db).register(coordinator, () => canRun); + try { + await worker.ready; + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), + { timeout: 10_000 }).toEqual({ active: 0, next: null }); + canRun = false; + await db.transaction(async tx => { + await notifyChatRunMilestones(tx, { companyId: context.fixture.companyId, issueId: context.issue.id }); + await tx.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, context.runId)); + }); + await expect.poll(() => idleWorkSnapshot().active, { timeout: 5000 }).toBe(0); + expect(coordinator.nextWakeAt()).not.toBeNull(); + const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction"); + try { + await vi.advanceTimersByTimeAsync(120_000); + expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); + } finally { reads.mockRestore(); writes.mockRestore(); } + canRun = true; worker.wake(); + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(1000); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1); + expect(coordinator.nextWakeAt()).toBeNull(); + }, { timeout: 5000 }); + expect(errors).toEqual([]); + } finally { worker.stop(); await coordinator.stop(); vi.useRealTimers(); await context.service.shutdown(); } + }); + + it("retains a failed accepted-response attempt across drain and retries after resume", async () => { + const context = await committedChatResponseRecoveryFixture("telegram"); + let release!: () => void, observed!: () => void; + const hold = new Promise(resolve => { release = resolve; }); + const locked = new Promise(resolve => { observed = resolve; }); + const holder = db.transaction(async tx => { + await tx.select().from(chatActions).where(eq(chatActions.id, context.action.id)).for("update"); + observed(); + await hold; + }); + await locked; + let canRun = true; + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true }); + const errors: unknown[] = []; + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => canRun, canReconcile: () => true, onError: error => errors.push(error) }); + const worker = nativeResponsePresentationWork(db).register(coordinator, () => canRun); + try { + await worker.ready; + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(1000); + expect(errors).toHaveLength(1); + expect(idleWorkSnapshot().active).toBe(0); + }, { timeout: 5000 }); + expect(coordinator.nextWakeAt()).toBeGreaterThan(Date.now()); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0); + canRun = false; + const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction"); + try { + await vi.advanceTimersByTimeAsync(120_000); + expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); + } finally { reads.mockRestore(); writes.mockRestore(); } + release(); await holder; + canRun = true; worker.wake(); + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(1000); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1); + expect(idleWorkSnapshot().active).toBe(0); + expect(coordinator.nextWakeAt()).toBeNull(); + }, { timeout: 5000 }); + expect(errors).toHaveLength(1); + } finally { release(); await holder; worker.stop(); await coordinator.stop(); vi.useRealTimers(); await context.service.shutdown(); } + }); + + it("reconstructs a future accepted-response deadline without early publication", async () => { + const context = await committedChatResponseRecoveryFixture("telegram"); + const at = new Date(Date.now() + 60_000); + await db.update(nativeRunFinalizations).set({ nextAttemptAt: at }).where(eq(nativeRunFinalizations.runId, context.runId)); + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true }); + const errors: unknown[] = []; + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) }); + const worker = nativeResponsePresentationWork(db).register(coordinator, () => true); + try { + await worker.ready; + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), + { timeout: 10_000 }).toEqual({ active: 0, next: at.getTime() }); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0); + await vi.advanceTimersByTimeAsync(Math.max(0, at.getTime() - Date.now())); + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(1000); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1); + expect(coordinator.nextWakeAt()).toBeNull(); + }, { timeout: 5000 }); + expect(errors).toEqual([]); + } finally { worker.stop(); await coordinator.stop(); vi.useRealTimers(); await context.service.shutdown(); } + }); + + it("retains an accepted-response deadline that elapses during the locked recovery check", async () => { + const context = await committedChatResponseRecoveryFixture("telegram"); + const work = nativeResponsePresentationWork(db); + const [entry] = await work.load({ companyId: context.fixture.companyId }); + expect(entry?.id).toBe(context.runId); + vi.useFakeTimers({ toFake: ["Date"] }); + const due = new Date(Date.now() + 1000); + const transaction = db.transaction.bind(db); + const intercept = vi.spyOn(db, "transaction").mockImplementationOnce((async (callback, options) => { + await transaction(async tx => { + await tx.update(nativeRunFinalizations).set({ nextAttemptAt: due }).where(eq(nativeRunFinalizations.runId, context.runId)); + }); + const result = await transaction(callback, options); + vi.setSystemTime(due.getTime() + 1); + return result; + }) as Db["transaction"]); + try { + const deferred = await work.check(entry); + intercept.mockRestore(); + expect(deferred).toEqual({ ...entry, dueAt: due.getTime() }); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0); + await expect(work.check(deferred!)).resolves.toBeNull(); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1); + } finally { intercept.mockRestore(); vi.useRealTimers(); await context.service.shutdown(); } + }); + + it("wakes dormant accepted-response recovery after agent resume", async () => { + const context = await committedChatResponseRecoveryFixture("telegram"); + await db.update(agents).set({ status: "paused", lifecycleState: "paused", lifecycleHolds: ["manual"], lifecycleOperation: null }) + .where(eq(agents.id, context.binding.agentId)); + const errors: unknown[] = []; + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) }); + const work = nativeResponsePresentationWork(db); + const worker = work.register(coordinator, () => true); + const lifecycle = configureAgentLifecycle(db, { + requiredPluginIds: async () => [], runHost: async () => "complete", runPlugin: async () => "complete", + }); + try { + await worker.ready; + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), + { timeout: 10_000 }).toEqual({ active: 0, next: null }); + expect(await work.load({ companyId: context.fixture.companyId, agentId: randomUUID() })).toEqual([]); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0); + await createAgentLifecycle(db).resumeAgent(context.binding.agentId); + await processAgentLifecycle(db, context.binding.agentId); + await expect.poll(async () => (await db.select().from(agents).where(eq(agents.id, context.binding.agentId)))[0].status).toBe("idle"); + await expect.poll(async () => (await readChatPublicationsByEndpointId(context.endpoint.id)).length, { timeout: 5000 }).toBe(1); + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), + { timeout: 10_000 }).toEqual({ active: 0, next: null }); + expect(errors).toEqual([]); + } finally { await lifecycle.stop(); worker.stop(); await coordinator.stop(); await context.service.shutdown(); } + }); + + it("wakes accepted-response recovery from the real coordinator retry release", async () => { + const context = await committedChatResponseRecoveryFixture("telegram"); + const store = new NativeRunCoordinatorStore(db, context.binding); + await store.claimFinalization({ leaseOwner: "response-recovery-fixture", leaseTtlMs: 1000 }); + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true }); + const errors: unknown[] = []; + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) }); + const worker = nativeResponsePresentationWork(db).register(coordinator, () => true); + try { + await worker.ready; + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), + { timeout: 10_000 }).toEqual({ active: 0, next: null }); + await vi.advanceTimersByTimeAsync(2000); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0); + expect(coordinator.nextWakeAt()).toBeNull(); + await store.markFinalizationRetry({ leaseOwner: "response-recovery-fixture", failureCode: "response_fixture_retry", retryAfterMs: 1000 }); + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(1000); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1); + expect(idleWorkSnapshot().active).toBe(0); + expect(coordinator.nextWakeAt()).toBeNull(); + }, { timeout: 5000 }); + expect(errors).toEqual([]); + } finally { worker.stop(); await coordinator.stop(); vi.useRealTimers(); await context.service.shutdown(); } + }); + + it("wakes accepted-response recovery when its authenticated accepted-result journal arrives late", async () => { + const context = await committedChatResponseRecoveryFixture("telegram", "port"); + await db.delete(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.runId, context.runId), eq(heartbeatRunEvents.eventType, "run.result.accepted"))); + const errors: unknown[] = []; + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) }); + const worker = nativeResponsePresentationWork(db).register(coordinator, () => true); + try { + await worker.ready; + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), + { timeout: 10_000 }).toEqual({ active: 0, next: null }); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0); + const port = new PaperclipControlPlanePort(db, { + companyId: context.binding.companyId, issueId: context.binding.issueId, + runId: context.runId, agentId: context.binding.agentId, sessionId: context.binding.normalizedSessionId, + completionContractId: context.binding.completionContractId, + completionContractSha256: context.binding.completionContractSha256, + sourceInstanceId: context.binding.runnerSourceInstanceId, controlPlaneSourceInstanceId: `control-${context.runId}`, + }); + await port.appendEvent({ + schema: "paperclip.prp.event.v1", sourceEventId: `accepted-${context.runId}`, sourceSeq: 1, + sourceInstanceId: `control-${context.runId}`, sourceKind: "control_plane", + runId: context.runId, normalizedSessionId: context.binding.normalizedSessionId, + turnId: `turn-${context.runId}`, eventType: "run.result.accepted", schemaVersion: 1, + priority: 0, emittedAt: new Date().toISOString(), payload: { result: context.result }, + }); + await expect.poll(async () => ({ + publications: (await readChatPublicationsByEndpointId(context.endpoint.id)).length, + active: idleWorkSnapshot().active, next: coordinator.nextWakeAt(), + }), { timeout: 5000 }).toEqual({ publications: 1, active: 0, next: null }); + expect(errors).toEqual([]); + } finally { worker.stop(); await coordinator.stop(); await context.service.shutdown(); } + }); + + it("defers scheduled accepted-response recovery until the persisted retry is due", async () => { + const context = await committedChatResponseRecoveryFixture("telegram"); + const input = { companyId: context.fixture.companyId, issueId: context.issue.id, runId: context.runId, onlyWhenDue: true }; + try { + await db.update(nativeRunFinalizations).set({ nextAttemptAt: new Date(Date.now() + 60_000) }) + .where(eq(nativeRunFinalizations.runId, context.runId)); + await expect(recoverCommittedNativeChatResponse(db, input)).resolves.toBe(false); + expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, context.runId))).toHaveLength(0); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0); + await db.update(nativeRunFinalizations).set({ nextAttemptAt: new Date(0) }) + .where(eq(nativeRunFinalizations.runId, context.runId)); + await expect(recoverCommittedNativeChatResponse(db, input)).resolves.toBe(true); + expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, context.runId))).toHaveLength(1); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1); + } finally { await context.service.shutdown(); } + }); + + it.each(["pending", "failed"])("defers scheduled accepted-response recovery behind %s status effects", async deliveryState => { + const context = await committedChatResponseRecoveryFixture("telegram"); + const input = { companyId: context.fixture.companyId, issueId: context.issue.id, runId: context.runId, onlyWhenDue: true }; + try { + const [effect] = await db.select().from(statusDecisionEffects).where(eq(statusDecisionEffects.decisionId, context.coordinator.decisionId!)); + expect(effect).toBeDefined(); + await db.update(statusDecisionEffects).set({ deliveryState }).where(eq(statusDecisionEffects.id, effect.id)); + await expect(recoverCommittedNativeChatResponse(db, input)).resolves.toBe(false); + expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, context.runId))).toHaveLength(0); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0); + await db.update(statusDecisionEffects).set({ deliveryState: "delivered" }).where(eq(statusDecisionEffects.id, effect.id)); + await expect(recoverCommittedNativeChatResponse(db, input)).resolves.toBe(true); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1); + } finally { await context.service.shutdown(); } + }); + it("leaves a contended accepted response retryable without reclassifying or partially publishing it", async () => { const context = await committedChatResponseRecoveryFixture("telegram"); let release!: () => void; @@ -75279,6 +75819,9 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { try { await locked; await expect(context.repair()).resolves.toBe(false); + await expect(recoverCommittedNativeChatResponse(db, { + companyId: context.fixture.companyId, issueId: context.issue.id, runId: context.runId, + })).rejects.toThrow(); await expect( db .select() diff --git a/server/src/__tests__/committed-chat-response-authority.test.ts b/server/src/__tests__/committed-chat-response-authority.test.ts new file mode 100644 index 00000000000..a09b8847c1b --- /dev/null +++ b/server/src/__tests__/committed-chat-response-authority.test.ts @@ -0,0 +1,55 @@ +import { isNativeResponseEvidence } from "../services/native-response-presentation-notifications.js"; +import { describe, expect, it, vi } from "vitest"; +import type { Db } from "@paperclipai/db"; +import { authorizeCommittedChatResponse, onCommittedChatResponseAuthorityRegistered, registerCommittedChatResponseAuthority } from "../services/durable-chat-wakeup.js"; + +describe("committed response authority lifecycle", () => { + it("reactivates only its database and unsubscribes without removing a replacement listener", () => { + const db = {} as Db, otherDb = {} as Db; + const first = vi.fn(), second = vi.fn(), other = vi.fn(); + const disposeFirst = onCommittedChatResponseAuthorityRegistered(db, first); + const disposeOther = onCommittedChatResponseAuthorityRegistered(otherDb, other); + const removeAuthority = registerCommittedChatResponseAuthority(db, async () => {}); + expect(first).toHaveBeenCalledTimes(1); + expect(other).not.toHaveBeenCalled(); + disposeFirst(); + const disposeSecond = onCommittedChatResponseAuthorityRegistered(db, second); + disposeFirst(); + const removeReplacement = registerCommittedChatResponseAuthority(db, async () => {}); + expect(first).toHaveBeenCalledTimes(1); + expect(second).toHaveBeenCalledTimes(1); + removeAuthority(); + expect(second).toHaveBeenCalledTimes(1); + disposeSecond(); disposeOther(); removeReplacement(); + const removeLast = registerCommittedChatResponseAuthority(db, async () => {}); + expect(second).toHaveBeenCalledTimes(1); + removeLast(); + }); + + it("retains the replacement authority when the old service shuts down", async () => { + const db = {} as Db; + const first = vi.fn(async () => {}), second = vi.fn(async () => {}); + const removeFirst = registerCommittedChatResponseAuthority(db, first); + const removeSecond = registerCommittedChatResponseAuthority(db, second); + try { + removeFirst(); + const input = { companyId: "company", issueId: "issue", agentId: "agent", runId: "run", resultId: "result" }; + await authorizeCommittedChatResponse(db, db, input); + expect(first).not.toHaveBeenCalled(); + expect(second).toHaveBeenCalledExactlyOnceWith(db, input); + } finally { removeSecond(); } + }); +}); + + +describe("accepted response evidence selection", () => { + it.each([ + { type: "item.delta", kind: "agentMessage", channel: "final", expected: false }, + { type: "item.completed", kind: "agentMessage", channel: "analysis", expected: false }, + { type: "item.completed", kind: "commandExecution", channel: "final", expected: false }, + { type: "item.completed", kind: "agentMessage", channel: "final", expected: true }, + { type: "run.result.accepted", kind: "agentMessage", channel: "final", expected: true }, + ])("signals only response evidence: $type/$kind/$channel", ({ type, kind, channel, expected }) => { + expect(isNativeResponseEvidence(type, { payload: { kind, channel, text: "Saved evidence" } })).toBe(expected); + }); +}); diff --git a/server/src/__tests__/server-startup-feedback-export.test.ts b/server/src/__tests__/server-startup-feedback-export.test.ts index 62bc485a0c2..8777ebc7858 100644 --- a/server/src/__tests__/server-startup-feedback-export.test.ts +++ b/server/src/__tests__/server-startup-feedback-export.test.ts @@ -116,6 +116,7 @@ const { reapOrphanedRuns: vi.fn(async () => ({ reaped: 0, runIds: [] })), registerScheduledRetryCancellationWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), registerStoppedInstructionCollectionWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), + registerNativeResponsePresentationWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve(), stop: vi.fn() })), registerNativePendingEffectWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), registerUnsafeWorkspaceExportWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), registerAutomaticCompletionReviewWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), @@ -558,6 +559,17 @@ describe("startServer feedback export wiring", () => { process.env.BETTER_AUTH_SECRET = "test-secret"; }); + it.each([false, true])("registers response recovery and its shutdown subscription only when maintenance is enabled (%s)", async enabled => { + loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled })); + await startServer(); await settleStartupHousekeeping(); + expect(heartbeatServiceMock.registerNativeResponsePresentationWork).toHaveBeenCalledTimes(enabled ? 1 : 0); + if (enabled) { + expect(heartbeatServiceMock.registerNativeResponsePresentationWork.mock.calls[0][1]()).toBe(true); + const worker = heartbeatServiceMock.registerNativeResponsePresentationWork.mock.results[0].value; + expect(fakeServer.on).toHaveBeenCalledWith("close", worker.stop); + } + }); + it.each([false, true])("registers native pending effects only when maintenance is enabled (%s)", async enabled => { loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled })); await startServer(); await settleStartupHousekeeping(); @@ -1005,7 +1017,7 @@ describe("startServer feedback export wiring", () => { intervalCallback?.(); await new Promise((resolve) => setImmediate(resolve)); expect(heartbeatServiceMock.sweepStaleIssueLocks).toHaveBeenCalledTimes(2); - expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true }); + expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true }); expect(heartbeatServiceMock.reconcileCostAccounting).toHaveBeenCalledTimes(1); expect(retiredDetector).not.toHaveBeenCalled(); } finally { diff --git a/server/src/board-claim.ts b/server/src/board-claim.ts index 275dfdf0180..a5a2cbdf20e 100644 --- a/server/src/board-claim.ts +++ b/server/src/board-claim.ts @@ -4,7 +4,7 @@ import type { Db } from "@paperclipai/db"; import { companies, companyMemberships, instanceUserRoles } from "@paperclipai/db"; import type { DeploymentMode } from "@paperclipai/shared"; import { ensureHumanRoleDefaultGrants } from "./services/principal-access-compatibility.js"; -import { notifyChatRunMilestones } from "./services/chat-run-milestone-notifications.js"; +import { notifyChatCompanyAuthorityChange } from "./services/chat-run-milestone-notifications.js"; const LOCAL_BOARD_USER_ID = "local-board"; const CLAIM_TTL_MS = 1000 * 60 * 60 * 24; @@ -125,7 +125,7 @@ export async function claimBoardOwnership( .then((rows) => rows[0] ?? null); if (!existing) { - await notifyChatRunMilestones(tx, { companyId: company.id }); + await notifyChatCompanyAuthorityChange(tx, { companyId: company.id }); await tx.insert(companyMemberships).values({ companyId: company.id, principalType: "user", @@ -137,7 +137,7 @@ export async function claimBoardOwnership( } if (existing.status !== "active") { - await notifyChatRunMilestones(tx, { companyId: company.id }); + await notifyChatCompanyAuthorityChange(tx, { companyId: company.id }); await tx .update(companyMemberships) .set({ status: "active", membershipRole: "owner", updatedAt: new Date() }) diff --git a/server/src/index.ts b/server/src/index.ts index 719ea5ee7c6..86a7b9da72c 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1403,6 +1403,7 @@ async function startServerWithDatabaseTeardown( hasPending: () => caseHandoffDeliveries.hasPending(), }); let unregisterTerminalWorkspaceProbe: (() => void) | undefined; + let unsubscribeNativeResponsePresentation: (() => void) | undefined; let unsubscribeNativeWorkspaceSyncRetention: (() => void) | undefined; const fastResponses = app.locals.fastResponses as ReturnType; const fastResponseWorker = deliveryWork.register(DELIVERY_QUEUES.fastResponse, { @@ -1824,6 +1825,13 @@ async function startServerWithDatabaseTeardown( () => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready; if (config.heartbeatSchedulerEnabled) await heartbeat.registerInstructionDirectoryCleanupWork(deliveryWork, () => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready; + if (config.heartbeatSchedulerEnabled) { + const worker = heartbeat.registerNativeResponsePresentationWork(deliveryWork, + () => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()); + unsubscribeNativeResponsePresentation = worker.stop; + server.on("close", worker.stop); + await worker.ready; + } if (config.heartbeatSchedulerEnabled) await heartbeat.registerNativePendingEffectWork(deliveryWork, () => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready; if (config.heartbeatSchedulerEnabled) await heartbeat.registerUnsafeWorkspaceExportWork(deliveryWork, @@ -1964,7 +1972,7 @@ async function startServerWithDatabaseTeardown( // Periodically reap orphaned runs (5-min staleness threshold) and make sure // persisted queued work is still being driven forward. runPeriodicRecovery(() => heartbeat - .reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true }) + .reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true }) .then(() => heartbeat.promoteDueScheduledRetries()) .then(async (promotion) => { await heartbeat.resumeQueuedRuns(); @@ -2160,6 +2168,7 @@ async function startServerWithDatabaseTeardown( await skillVersionRetention.dispose(); await skillSourceSync.stop(); unsubscribeChatCompletions(); + unsubscribeNativeResponsePresentation?.(); await deliveryWork.stop(); await workspaceRecoveryWork.stop(); unregisterTerminalWorkspaceProbe?.(); diff --git a/server/src/middleware/auth.ts b/server/src/middleware/auth.ts index aa9e3c13a37..f9bd7446947 100644 --- a/server/src/middleware/auth.ts +++ b/server/src/middleware/auth.ts @@ -30,7 +30,7 @@ import { captureRunIdentity } from "../services/run-identity.js"; import { boardAuthService } from "../services/board-auth.js"; import { retryIdempotentDatabaseOperation } from "../database-retry.js"; import { beginIdleTrackedWork } from "../services/task-admission.js"; -import { notifyChatRunMilestones } from "../services/chat-run-milestone-notifications.js"; +import { notifyChatCompanyAuthorityChange } from "../services/chat-run-milestone-notifications.js"; export { isTransientDbConnectionError, @@ -665,7 +665,7 @@ async function resolveCloudTenantActorOnce( // Serialize authority restoration and its notice while retaining the // guarded upsert that leaves unchanged memberships untouched. if (!existing || existing.status !== "active" || existing.membershipRole === "viewer") { - await notifyChatRunMilestones(tx, { companyId }); + await notifyChatCompanyAuthorityChange(tx, { companyId }); } return tx .insert(companyMemberships) diff --git a/server/src/modules/run-dispatch/adapters/postgres.ts b/server/src/modules/run-dispatch/adapters/postgres.ts index 8531240df68..4c313517ef3 100644 --- a/server/src/modules/run-dispatch/adapters/postgres.ts +++ b/server/src/modules/run-dispatch/adapters/postgres.ts @@ -673,7 +673,7 @@ export function createPostgresRunDispatchAdapter( await notifyInstructionCollection(tx, { companyId: input.companyId, runId: input.runId }); await notifyCancelledWorkspacePreservation(tx, { companyId: input.companyId, runId: input.runId }); await notifyScheduledRetryCancellation(tx, { companyId: input.companyId, runId: input.runId }); - await notifyChatRunMilestones(tx, { companyId: input.companyId, issueId: issueId ?? undefined }); + await notifyChatRunMilestones(tx, { companyId: input.companyId, issueId: issueId ?? undefined, runId: input.runId }); const [row] = await tx .update(heartbeatRuns) .set({ @@ -719,7 +719,7 @@ export function createPostgresRunDispatchAdapter( await notifyCancelledWorkspacePreservation(tx, { companyId: input.companyId, runId: input.runId }); await notifyScheduledRetryCancellation(tx, { companyId: input.companyId, runId: input.runId }); await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: input.companyId, issueId: input.issueId ?? undefined }); - await notifyChatRunMilestones(tx, { companyId: input.companyId, issueId: input.issueId ?? undefined }); + await notifyChatRunMilestones(tx, { companyId: input.companyId, issueId: input.issueId ?? undefined, runId: input.runId }); const [row] = await tx .update(heartbeatRuns) .set({ diff --git a/server/src/modules/wake-queue/adapters/postgres.test.ts b/server/src/modules/wake-queue/adapters/postgres.test.ts index 83d5bb6ee44..240776e2278 100644 --- a/server/src/modules/wake-queue/adapters/postgres.test.ts +++ b/server/src/modules/wake-queue/adapters/postgres.test.ts @@ -1,3 +1,4 @@ +import { DELIVERY_QUEUES } from "../../../services/delivery-work-notifications.js"; import { instanceSettingsService } from "../../../services/instance-settings.js"; import { randomBytes, randomUUID } from "node:crypto"; import { eq } from "drizzle-orm"; @@ -16,6 +17,7 @@ import { issueRecoveryActions, issues, nativeRunFinalizations, + subscribeDatabaseWork, } from "@paperclipai/db"; import { getEmbeddedPostgresTestSupport, @@ -1222,6 +1224,30 @@ describeEmbeddedPostgres("wake-queue postgres adapter", () => { expect(wakeRow?.payload).not.toHaveProperty("foo"); }); + it("scopes a real coalesced-input response wake to its active run after commit", async () => { + const companyId = await seedCompany(); + const agentId = await seedAgent({ companyId }); + const runId = await seedRun({ companyId, agentId, status: "running" }); + const hints: Array> = []; + const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.nativeResponsePresentation, (event, values) => { + if (event === "settled") for (const value of values ?? []) hints.push(JSON.parse(value)); + }); + try { + await db.transaction(async tx => { + await createWakeAdmissionWriter().coalesceIntoActiveExecutionRun(createAdmissionTransactionScope(companyId, tx as unknown as Db), { + companyId, activeExecutionRunId: runId, mergedContextSnapshot: { commentId: "fresh-input" }, + agentId, source: "on_demand", triggerDetail: null, payload: null, + requestedByActorType: "user", requestedByActorId: null, idempotencyKey: null, + }); + expect(hints).toEqual([]); + }); + expect(hints).toEqual([{ companyId, runId }]); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0].contextSnapshot) + .toMatchObject({ commentId: "fresh-input" }); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.runId, runId))).toHaveLength(1); + } finally { unsubscribe(); } + }); + // Review test (b), the coalesce write: a foreign company affects no row // on the active execution run either. it("refuses to coalesce into a run for a company that does not own it, and leaves the run and the wake table untouched", async () => { diff --git a/server/src/modules/wake-queue/adapters/postgres.ts b/server/src/modules/wake-queue/adapters/postgres.ts index 7b3b45f1a39..0c80fd6f5d9 100644 --- a/server/src/modules/wake-queue/adapters/postgres.ts +++ b/server/src/modules/wake-queue/adapters/postgres.ts @@ -956,7 +956,7 @@ export function createWakeAdmissionWriter(): WakeAdmissionWriter { async coalesceIntoActiveExecutionRun(scope, input) { const tx = requireAdmissionTx(scope, input.companyId); const now = new Date(); - await notifyChatRunMilestones(tx, { companyId: input.companyId }); + await notifyChatRunMilestones(tx, { companyId: input.companyId, runId: input.activeExecutionRunId }); const mergedRun = await tx .update(heartbeatRuns) .set({ contextSnapshot: input.mergedContextSnapshot, updatedAt: now }) diff --git a/server/src/services/access.ts b/server/src/services/access.ts index c76608597cc..64f57cb1f1a 100644 --- a/server/src/services/access.ts +++ b/server/src/services/access.ts @@ -1,5 +1,5 @@ import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js"; -import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js"; +import { notifyChatCompanyAuthorityChange } from "./chat-run-milestone-notifications.js"; import { notifyVoiceChange } from "./voice/voice-recovery-notifications.js"; import { notifyChatEndpointWork } from "./chat-work-notifications.js"; import { notifyToolConnectionHealth } from "./tool-connection-health-notifications.js"; @@ -558,7 +558,7 @@ export function accessService(db: Db) { if (existing.principalType === "user" && nextStatus === "active" && nextMembershipRole !== "viewer" && (existing.status !== "active" || existing.membershipRole === "viewer")) { - await notifyChatRunMilestones(tx, { companyId }); + await notifyChatCompanyAuthorityChange(tx, { companyId }); } if ( @@ -894,7 +894,7 @@ export function accessService(db: Db) { const existingMembership = existingByCompany.get(companyId); if (existingMembership) { if (existingMembership.status !== "active") { - if (existingMembership.membershipRole !== "viewer") await notifyChatRunMilestones(tx, { companyId }); + if (existingMembership.membershipRole !== "viewer") await notifyChatCompanyAuthorityChange(tx, { companyId }); await tx .update(companyMemberships) .set({ @@ -906,7 +906,7 @@ export function accessService(db: Db) { } continue; } - await notifyChatRunMilestones(tx, { companyId }); + await notifyChatCompanyAuthorityChange(tx, { companyId }); await tx.insert(companyMemberships).values({ companyId, principalType: "user", @@ -936,7 +936,7 @@ export function accessService(db: Db) { if (existing.status !== status || existing.membershipRole !== membershipRole) { if (principalType === "user" && status === "active" && membershipRole !== "viewer" && (existing.status !== "active" || existing.membershipRole === "viewer")) { - await notifyChatRunMilestones(tx, { companyId }); + await notifyChatCompanyAuthorityChange(tx, { companyId }); } const updated = await tx .update(companyMemberships) @@ -950,7 +950,7 @@ export function accessService(db: Db) { } if (principalType === "user" && status === "active" && membershipRole !== "viewer") { - await notifyChatRunMilestones(tx, { companyId }); + await notifyChatCompanyAuthorityChange(tx, { companyId }); } return tx .insert(companyMemberships) @@ -1146,7 +1146,7 @@ export function accessService(db: Db) { if (existing.principalType === "user" && nextStatus === "active" && nextMembershipRole !== "viewer" && (existing.status !== "active" || existing.membershipRole === "viewer")) { - await notifyChatRunMilestones(tx, { companyId }); + await notifyChatCompanyAuthorityChange(tx, { companyId }); } if ( diff --git a/server/src/services/agent-lifecycle.ts b/server/src/services/agent-lifecycle.ts index 164e94420db..041ae077db4 100644 --- a/server/src/services/agent-lifecycle.ts +++ b/server/src/services/agent-lifecycle.ts @@ -1,4 +1,5 @@ import { isAgentStatusInvokable } from "@paperclipai/shared"; +import { notifyNativeResponsePresentation } from "./native-response-presentation-notifications.js"; import { notifyChatInboundWakeNotices } from "./chat-inbound-wake-notice-notifications.js"; import { logger } from "../middleware/logger.js"; import { companies, type Db } from "@paperclipai/db"; @@ -35,8 +36,9 @@ export function createAgentLifecycleEffects(_hooks: BudgetServiceHooks = {}): li return true; }, notifyWork: tx => notifyDeliveryWork(tx, DELIVERY_QUEUES.agentLifecycle), - async beforeStatusChange(tx, companyId, _agentId, before, after) { + async beforeStatusChange(tx, companyId, agentId, before, after) { if (!isAgentStatusInvokable(before) && isAgentStatusInvokable(after)) { + await notifyNativeResponsePresentation(tx, { companyId, agentId }); await notifyChatInboundWakeNotices(tx, { companyId }); } }, diff --git a/server/src/services/chat-channels/endpoints.ts b/server/src/services/chat-channels/endpoints.ts index 86b788cea74..0dc9f07e9f7 100644 --- a/server/src/services/chat-channels/endpoints.ts +++ b/server/src/services/chat-channels/endpoints.ts @@ -1,4 +1,4 @@ -import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js"; +import { notifyChatCompanyAuthorityChange } from "../chat-run-milestone-notifications.js"; import { chatConnectionTransport } from "./provider-catalog.js"; import type { ConnectionManagementContext } from "../connection-management.js"; import { slackSetupState } from "./slack/setup.js"; @@ -458,7 +458,7 @@ function providerSetupState( if ((input.allowDirectMessages === true && !existing.endpoint.allowDirectMessages) || (input.allowGroupChats === true && !existing.endpoint.allowGroupChats) || (input.allowUnlinkedPeople === true && !existing.endpoint.allowUnlinkedPeople)) { - await notifyChatRunMilestones(tx, { companyId: existing.endpoint.companyId }); + await notifyChatCompanyAuthorityChange(tx, { companyId: existing.endpoint.companyId }); } if (input.executionDefaults !== undefined) { await assertChatExecutionDefaultsAccess(db, { diff --git a/server/src/services/chat-channels/failed-run-retry.ts b/server/src/services/chat-channels/failed-run-retry.ts index 8dd6ace33a9..c324a6f3c34 100644 --- a/server/src/services/chat-channels/failed-run-retry.ts +++ b/server/src/services/chat-channels/failed-run-retry.ts @@ -1,3 +1,4 @@ +import { notifyNativeResponsePresentation } from "../native-response-presentation-notifications.js"; import { notifyChatInboundWakeNotices } from "../chat-inbound-wake-notice-notifications.js"; import { createFailedRetryNotices } from "./failed-retry-notices.js"; import { safeChatWakeAuthorityReason, type ChatWakeAuthorityReason } from "./inbound/wakeup-authorization-error.js"; @@ -1784,6 +1785,9 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti const settle = async (status: string, result: Record) => { const [current] = await db.transaction(async (tx) => { await notifyChatActionWork(tx, "failed_run_retry"); + if (status === "processed") await notifyNativeResponsePresentation(tx, { + companyId: claimed.companyId, issueId: String(claimed.payload.issueId), + }); return tx .update(chatActions) .set({ status, result, updatedAt: new Date() }) diff --git a/server/src/services/chat-channels/identity-links.ts b/server/src/services/chat-channels/identity-links.ts index abbd2340c94..e1e1b0b2712 100644 --- a/server/src/services/chat-channels/identity-links.ts +++ b/server/src/services/chat-channels/identity-links.ts @@ -1,4 +1,4 @@ -import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js"; +import { notifyChatCompanyAuthorityChange } from "../chat-run-milestone-notifications.js"; import { githubIdentityFields } from "./github/identity-links.js"; import { createSlackIdentityLinks, type SlackIdentityEffectStager } from "./slack/identity-links.js"; import { createHash, randomBytes } from "node:crypto"; @@ -320,7 +320,7 @@ export function createChatIdentityLinks(db: Db, options: ChatIdentityLinkOptions }, ); } - await notifyChatRunMilestones(tx, { companyId: currentLink.companyId }); + await notifyChatCompanyAuthorityChange(tx, { companyId: currentLink.companyId }); const confirmedLink = await tx .update(chatIdentityLinks) .set({ diff --git a/server/src/services/chat-channels/resource-reconciliation.ts b/server/src/services/chat-channels/resource-reconciliation.ts index 5cf5f570266..7e0cc6b5ed8 100644 --- a/server/src/services/chat-channels/resource-reconciliation.ts +++ b/server/src/services/chat-channels/resource-reconciliation.ts @@ -1,4 +1,4 @@ -import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js"; +import { notifyChatRunMilestones, notifyChatCompanyAuthorityChange } from "../chat-run-milestone-notifications.js"; import { notifyChatEndpointConfiguration } from "./configuration-work-notifications.js"; import { notifyChatConnectionHealth } from "./connection-health-notifications.js"; import type { ConnectionManagementContext } from "../connection-management.js"; @@ -287,7 +287,7 @@ export function createChatResourceReconciliation(db: Db, options: ResourceReconc after: { enabled: finalEnabled.get(row.id)!, ...(JSON.stringify(row.executionDefaults) !== JSON.stringify(finalDefaults.get(row.id)) ? { executionDefaults: finalDefaults.get(row.id) } : {}) }, })); if (changes.some(change => !change.before.enabled && change.after.enabled)) { - await notifyChatRunMilestones(tx, { companyId: endpoint.companyId }); + await notifyChatCompanyAuthorityChange(tx, { companyId: endpoint.companyId }); } if (bulk) { // One atomic database update, independent of pagination/search and diff --git a/server/src/services/chat-channels/slack/setup-services.ts b/server/src/services/chat-channels/slack/setup-services.ts index 45f8d3bc59c..7463d3bc6e2 100644 --- a/server/src/services/chat-channels/slack/setup-services.ts +++ b/server/src/services/chat-channels/slack/setup-services.ts @@ -1,4 +1,4 @@ -import { notifyChatRunMilestones } from "../../chat-run-milestone-notifications.js"; +import { notifyChatCompanyAuthorityChange } from "../../chat-run-milestone-notifications.js"; import { chatIdentityLinks, chatExternalPrincipals, companyMemberships, toolConnections, type Db } from "@paperclipai/db"; import { and, eq, ne, sql } from "drizzle-orm"; import { conflict, forbidden, notFound } from "../../../errors.js"; @@ -47,7 +47,7 @@ export function createSlackSetupServices(db: Db, options: SetupServicesOptions) const links = await tx.select().from(chatIdentityLinks).where(and(eq(chatIdentityLinks.companyId, record.endpoint.companyId), eq(chatIdentityLinks.principalId, principal.id))).for("update"); if (links.some(link => link.status === "linked" && link.paperclipUserId !== actor.userId) || links.some(link => link.endpointId === endpointId && link.status === "revoked")) throw conflict("This Slack account has an existing link", { code: "chat_identity_link_conflict" }); if (links.some(link => link.endpointId === endpointId && link.status === "linked")) return; - await notifyChatRunMilestones(tx, { companyId: record.endpoint.companyId }); + await notifyChatCompanyAuthorityChange(tx, { companyId: record.endpoint.companyId }); await tx.insert(chatIdentityLinks).values({ companyId: record.endpoint.companyId, endpointId, principalId: principal.id, paperclipUserId: actor.userId, status: "linked", confirmedAt: new Date() }) .onConflictDoUpdate({ target: [chatIdentityLinks.endpointId, chatIdentityLinks.principalId], set: { diff --git a/server/src/services/chat-github-management.ts b/server/src/services/chat-github-management.ts index 3f1bc34a6b4..8c6c66ff9fb 100644 --- a/server/src/services/chat-github-management.ts +++ b/server/src/services/chat-github-management.ts @@ -24,7 +24,7 @@ import { import { HttpError, badRequest, conflict, forbidden, notFound, unprocessable } from "../errors.js"; import { toolAccessService } from "./tool-access.js"; import { logActivity } from "./activity-log.js"; -import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js"; +import { notifyChatCompanyAuthorityChange } from "./chat-run-milestone-notifications.js"; import { githubBotRequest } from "./chat-github-client.js"; import { environmentService } from "./environments.js"; import { instanceSettingsService } from "./instance-settings.js"; @@ -165,7 +165,7 @@ export function githubChatManagementService(db: Db, fetchImpl = fetch) { const allowUnlinkedPeople = config.people.some((p) => p.kind === "guest"); if ((!currentEndpoint.allowUnlinkedPeople && allowUnlinkedPeople) || (allowUnlinkedPeople && currentEndpoint.sponsorUserId !== config.responsibleUserId)) { - await notifyChatRunMilestones(tx, { companyId: bot.companyId }); + await notifyChatCompanyAuthorityChange(tx, { companyId: bot.companyId }); } const values = { companyId: bot.companyId, @@ -335,7 +335,7 @@ export function githubChatManagementService(db: Db, fetchImpl = fetch) { updatedAt: new Date(), }; if (prior?.status !== "linked") { - await notifyChatRunMilestones(tx, { companyId: bot.companyId }); + await notifyChatCompanyAuthorityChange(tx, { companyId: bot.companyId }); } await tx .insert(chatIdentityLinks) diff --git a/server/src/services/chat-monitor-continuation.test.ts b/server/src/services/chat-monitor-continuation.test.ts index f8e277d587b..26163e02d9a 100644 --- a/server/src/services/chat-monitor-continuation.test.ts +++ b/server/src/services/chat-monitor-continuation.test.ts @@ -1,5 +1,7 @@ +import { nativeResponsePresentationWork } from "./native-runtime/native-response-presentation-work.js"; +import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js"; import { randomBytes, randomUUID } from "node:crypto"; -import { and, eq } from "drizzle-orm"; +import { and, eq, sql } from "drizzle-orm"; import { afterAll, beforeAll, describe, expect, it } from "vitest"; import { agents, agentWakeupRequests, authUsers, chatConversations, chatEndpoints, chatMessageLinks, @@ -16,14 +18,18 @@ import { createHeartbeatQueueTestDependencies } from "../__tests__/helpers/heart import { createHeartbeatQueue } from "./heartbeat/queue.js"; import { instanceSettingsService } from "./instance-settings.js"; -const support = await getEmbeddedPostgresTestSupport(); +const external = resolveExternalTestDatabaseUrl("PAPERCLIP_MONITOR_CONTINUATION_TEST_DATABASE_URL"); +const support = external ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport(); if (!support.supported) console.warn(`Monitor publication database unavailable: ${support.reason}`); (support.supported ? describe : describe.skip)("monitor conversation continuation", () => { - let temporary: Awaited>; + let temporary: Awaited> | undefined; let db: ReturnType; beforeAll(async () => { - temporary = await startEmbeddedPostgresTestDatabase("paperclip-monitor-publications-"); - db = createDb(temporary.connectionString); + if (external) db = createDb(external); + else { + temporary = await startEmbeddedPostgresTestDatabase("paperclip-monitor-publications-"); + db = createDb(temporary.connectionString); + } await db.insert(authUsers).values({ id: "requester", name: "Requester", email: "requester@example.test", createdAt: new Date(), updatedAt: new Date() }); }); afterAll(async () => { await temporary?.cleanup(); }); @@ -235,7 +241,7 @@ if (!support.supported) console.warn(`Monitor publication database unavailable: expect(await f.bindings()).toEqual([]); }); - it("recovers the accepted continuation explanation after interrupted finalization without another provider result", async () => { + it.each([false, true])("recovers the accepted continuation explanation without another provider result (missing presentation: %s)", async missingPresentation => { const f = await fixture(); const executionPayload = randomBytes(128 * 1024).toString("base64"); await db.update(heartbeatRuns).set({ @@ -281,6 +287,20 @@ if (!support.supported) console.warn(`Monitor publication database unavailable: expect(settledRun.stdoutExcerpt).toBe(executionPayload); const priorComments = await db.select({ body: issueComments.body }).from(issueComments).where(eq(issueComments.createdByRunId, f.source.id)); expect(priorComments).toEqual([{ body: summary }]); + expect(await nativeResponsePresentationWork(db).load({ companyId: f.company.id, runId: f.source.id })).toEqual([]); + if (missingPresentation) { + await db.transaction(async tx => { + await tx.delete(chatPublications).where(eq(chatPublications.issueId, f.issue.id)); + await tx.delete(issueComments).where(eq(issueComments.createdByRunId, f.source.id)); + await tx.update(heartbeatRuns).set({ resultJson: sql`${heartbeatRuns.resultJson} - 'presentationDecision'` }) + .where(eq(heartbeatRuns.id, f.source.id)); + }); + const work = nativeResponsePresentationWork(db); + const entries = await work.load({ companyId: f.company.id, runId: f.source.id }); + expect(entries).toHaveLength(1); + await expect(work.check(entries[0])).resolves.toBeNull(); + expect(await work.load({ companyId: f.company.id, runId: f.source.id })).toEqual([]); + } expect(await repair()).toBe(false); expect(await repair()).toBe(false); const comments = await db.select().from(issueComments).where(eq(issueComments.createdByRunId, f.source.id)); diff --git a/server/src/services/chat-run-milestone-notifications.ts b/server/src/services/chat-run-milestone-notifications.ts index 8ab9636b6af..17316dbc678 100644 --- a/server/src/services/chat-run-milestone-notifications.ts +++ b/server/src/services/chat-run-milestone-notifications.ts @@ -1,3 +1,4 @@ +import { notifyNativeResponsePresentation } from "./native-response-presentation-notifications.js"; import { notifyChatResponseAvailability } from "./chat-response-availability.js"; import { notifyChatConfirmationWork } from "./chat-confirmation-notifications.js"; import { notifyChatInboundWakeNotices } from "./chat-inbound-wake-notice-notifications.js"; @@ -6,9 +7,13 @@ import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notificatio /** Lineage changes can authorize descendants, so invalidate the issue rather * than just the run whose row changed. Configuration may invalidate a company. */ -export type ChatRunMilestoneScope = { companyId: string; issueId?: string }; +export type ChatRunMilestoneScope = { companyId: string; issueId?: string; runId?: string }; export async function notifyChatRunMilestones(tx: object, scope: ChatRunMilestoneScope) { + if (scope.issueId || scope.runId) { + await notifyNativeResponsePresentation(tx, { companyId: scope.companyId, issueId: scope.issueId, + runId: scope.issueId ? undefined : scope.runId }); + } await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatGitHubReviewChecks, JSON.stringify({ companyId: scope.companyId })); await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatGitHubTerminalResponses, JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId })); await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatRunMilestones, JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId })); @@ -21,3 +26,8 @@ export async function notifyChatRunMilestones(tx: object, scope: ChatRunMileston // Native presentation and run ownership can unblock a parked acknowledgement. await notifyChatResponseAvailability(tx, scope); } + +export async function notifyChatCompanyAuthorityChange(tx: object, scope: { companyId: string }) { + await notifyChatRunMilestones(tx, scope); + await notifyNativeResponsePresentation(tx, scope); +} diff --git a/server/src/services/chat-work-notifications.ts b/server/src/services/chat-work-notifications.ts index 9b878913e8d..433f75de4a6 100644 --- a/server/src/services/chat-work-notifications.ts +++ b/server/src/services/chat-work-notifications.ts @@ -1,5 +1,5 @@ import { notifyChatResponseAvailability } from "./chat-response-availability.js"; -import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js"; +import { notifyChatRunMilestones, notifyChatCompanyAuthorityChange } from "./chat-run-milestone-notifications.js"; import { notifyChatConnectionConfiguration } from "./chat-channels/configuration-work-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; @@ -47,7 +47,7 @@ export async function notifyChatVerificationWork(tx: object): Promise { export async function notifyChatEndpointWork(tx: object, connectionId: string, companyId: string): Promise { // Configuration/status transitions can restore post-control publication // authority. Ordinary health checks use the narrower configuration signals. - await notifyChatRunMilestones(tx, { companyId }); + await notifyChatCompanyAuthorityChange(tx, { companyId }); await notifyChatDeliveryWork(tx); await notifyChatPublicationWork(tx); for (const queue of Object.values(CHAT_ACTION_QUEUES)) await notifyDeliveryWork(tx, queue); diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index aa160fed3dc..413c91f3b76 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -2,6 +2,7 @@ import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db"; /** Topics for existing durable queues; register intent before their writes. */ export const DELIVERY_QUEUES = { + nativeResponsePresentation: "native-response-presentation", nativePendingEffects: "native-pending-effects", unsafeWorkspaceExportRecovery: "unsafe-workspace-export-recovery", automaticCompletionReviews: "automatic-completion-reviews", diff --git a/server/src/services/durable-chat-wakeup.ts b/server/src/services/durable-chat-wakeup.ts index 65a150f8a26..2870eac88ba 100644 --- a/server/src/services/durable-chat-wakeup.ts +++ b/server/src/services/durable-chat-wakeup.ts @@ -21,6 +21,21 @@ const committedChatResponseAuthorities = new WeakMap< CommittedChatResponseAuthority >(); +const committedChatResponseAuthorityListeners = new WeakMap void>>(); + +export function onCommittedChatResponseAuthorityRegistered(db: Db, listener: () => void): () => void { + let listeners = committedChatResponseAuthorityListeners.get(db); + if (!listeners) { + listeners = new Set(); + committedChatResponseAuthorityListeners.set(db, listeners); + } + listeners.add(listener); + return () => { + listeners.delete(listener); + if (listeners.size === 0 && committedChatResponseAuthorityListeners.get(db) === listeners) committedChatResponseAuthorityListeners.delete(db); + }; +} + export class CommittedChatResponseAuthorizationError extends HttpError { constructor() { super( @@ -41,6 +56,7 @@ export function registerCommittedChatResponseAuthority( authority: CommittedChatResponseAuthority, ): () => void { committedChatResponseAuthorities.set(db, authority); + for (const listener of committedChatResponseAuthorityListeners.get(db) ?? []) listener(); return () => { if (committedChatResponseAuthorities.get(db) === authority) { committedChatResponseAuthorities.delete(db); diff --git a/server/src/services/execution-control-reconciliation.ts b/server/src/services/execution-control-reconciliation.ts index 7c0f0c4eb18..3bbe612e612 100644 --- a/server/src/services/execution-control-reconciliation.ts +++ b/server/src/services/execution-control-reconciliation.ts @@ -1,4 +1,4 @@ -import { notifyNativePendingEffects } from "./native-pending-effect-notifications.js"; +import { notifyNativeFinalizationWork } from "./native-finalization-notifications.js"; import { notifyUnsafeWorkspaceExportRunChange } from "./unsafe-workspace-export-notifications.js"; import { notifyCancelledWorkspacePreservation } from "./cancelled-workspace-preservation-notifications.js"; import { notifyInstructionCollection } from "./instruction-collection-notifications.js"; @@ -147,7 +147,7 @@ export async function reconcileAbandonedExecutionControl( } // A persisted result belongs to the existing finalizer, not a replacement provider. if (coordinator?.resultId) { - await notifyNativePendingEffects(tx, { companyId: run.companyId, runId: run.id }); + await notifyNativeFinalizationWork(tx, { companyId: run.companyId, runId: run.id }); await tx .update(nativeRunFinalizations) .set({ diff --git a/server/src/services/heartbeat-run-events.ts b/server/src/services/heartbeat-run-events.ts index 53e0bd26206..b6b1b694a49 100644 --- a/server/src/services/heartbeat-run-events.ts +++ b/server/src/services/heartbeat-run-events.ts @@ -1,3 +1,4 @@ +import { isNativeResponseEvidence, notifyNativeResponsePresentation } from "./native-response-presentation-notifications.js"; import { isSafeNativeChatProgressEventType } from "./safe-native-chat-progress.js"; import { notifyChatNativeProgress } from "./chat-native-progress-notifications.js"; import { boundRunIssueIdSql } from "./run-task-binding.js"; @@ -72,7 +73,8 @@ export async function appendHeartbeatRunEvent( input: AppendHeartbeatRunEventInput, ): Promise { const projectsProgress = isSafeNativeChatProgressEventType(input.eventType); - if (!input.nativeSource && !input.retryExhaustion && !projectsProgress) { + const projectsResponse = isNativeResponseEvidence(input.eventType, input.payload?.prpEvent); + if (!input.nativeSource && !input.retryExhaustion && !projectsProgress && !projectsResponse) { const allocated = db.$with("allocated_run_event", {}).as(sql` update ${heartbeatRuns} set next_event_seq = ${heartbeatRuns.nextEventSeq} + 1, @@ -101,7 +103,8 @@ export async function appendHeartbeatRunEvent( .select({ companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, // Only progress needs the task binding. Other native/recovery events // must not read a potentially large context snapshot to append a log. - ...(projectsProgress ? { runtimeMode: heartbeatRuns.runtimeMode, issueId: boundRunIssueIdSql() } : {}) }) + ...(projectsProgress || projectsResponse ? { runtimeMode: heartbeatRuns.runtimeMode } : {}), + ...(projectsProgress ? { issueId: boundRunIssueIdSql() } : {}) }) .from(heartbeatRuns) .where(eq(heartbeatRuns.id, input.runId)) .for("update") @@ -175,6 +178,9 @@ export async function appendHeartbeatRunEvent( if (projectsProgress && run.runtimeMode === "native" && run.issueId) { await notifyChatNativeProgress(tx, { companyId: run.companyId, issueId: run.issueId }); } + if (projectsResponse && run.runtimeMode === "native") { + await notifyNativeResponsePresentation(tx, { companyId: run.companyId, runId: input.runId }); + } const seq = await allocateHeartbeatRunEventSeq( tx as unknown as Db, input.runId, diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 7d4630a41f2..141c1d6d7d9 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -1,3 +1,4 @@ +import { registerNativeResponsePresentationWork } from "./native-runtime/native-response-presentation-work.js"; import { registerNativePendingEffectWork } from "./native-runtime/native-pending-effect-work.js"; import { registerUnsafeWorkspaceExportWork } from "./native-runtime/native-workspace-export-recovery.js"; import { registerAutomaticCompletionReviewWork } from "./native-runtime/automatic-completion-reviews.js"; @@ -4089,6 +4090,8 @@ export function heartbeatService( triggerIssueMonitor, reportRunActivity: clearDetachedRunWarning, + registerNativeResponsePresentationWork: (coordinator: Parameters[1], canRun: () => boolean) => + registerNativeResponsePresentationWork(db, coordinator, () => canRun() && readTaskDrain(new Date()) === null), registerNativePendingEffectWork: (coordinator: Parameters[1], canRun: () => boolean) => registerNativePendingEffectWork(db, coordinator, () => canRun() && readTaskDrain(new Date()) === null), registerUnsafeWorkspaceExportWork: (coordinator: Parameters[1], canRun: () => boolean) => diff --git a/server/src/services/heartbeat/queue.ts b/server/src/services/heartbeat/queue.ts index 4d93f0fade8..2be0fd457f7 100644 --- a/server/src/services/heartbeat/queue.ts +++ b/server/src/services/heartbeat/queue.ts @@ -976,7 +976,8 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende .where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.status, "queued"))) .for("update"); if (!currentRun) return null; - await notifyChatRunMilestones(tx, { companyId: run.companyId }); + await notifyChatRunMilestones(tx, { companyId: run.companyId, runId: run.id, + issueId: run.nativeIssueId ?? run.issueId ?? readNonEmptyString(parseObject(run.contextSnapshot).issueId) ?? undefined }); await notifyInstructionCollection(tx, { companyId: run.companyId, runId: run.id }); await notifyCancelledWorkspacePreservation(tx, { companyId: run.companyId, runId: run.id }); await notifyUnsafeWorkspaceExportRunChange(tx, run); @@ -1625,7 +1626,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende )).for("update"); if (!lockedRun || lockedRun.status !== "queued") return null; await withExternalAdmissionGuard(claimTx, run.companyId, run.agentId, () => assertNoExternalOverlap(claimTx, run.companyId, run.agentId)); - await notifyChatRunMilestones(claimTx, { companyId: run.companyId, issueId: issueId ?? undefined }); + await notifyChatRunMilestones(claimTx, { companyId: run.companyId, runId: run.id, issueId: issueId ?? undefined }); const claimedRun = await claimTx.update(heartbeatRuns).set(claimValues).where(and( eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.status, "queued"), )).returning().then((rows) => rows[0] ?? null); @@ -1726,7 +1727,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende const [candidate] = await tx.select({ companyId: heartbeatRuns.companyId, contextSnapshot: heartbeatRuns.contextSnapshot }) .from(heartbeatRuns).where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.status, "running"))).for("update"); if (!candidate) return; - await notifyChatRunMilestones(tx, { companyId: candidate.companyId, + await notifyChatRunMilestones(tx, { companyId: candidate.companyId, runId, issueId: readNonEmptyString(parseObject(candidate.contextSnapshot).issueId) ?? undefined }); const released = await tx .update(heartbeatRuns) diff --git a/server/src/services/heartbeat/recovery.ts b/server/src/services/heartbeat/recovery.ts index 2e1b8675580..a6fbf3432d7 100644 --- a/server/src/services/heartbeat/recovery.ts +++ b/server/src/services/heartbeat/recovery.ts @@ -1947,7 +1947,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD let settledProcessesDiscovered = false; - async function reapOrphanedRuns(opts?: { staleThresholdMs?: number; skipScheduledRetryCancellation?: boolean; skipCapturedInstructionCommit?: boolean; skipInstructionDirectoryCleanup?: boolean; skipStoppedInstructionCollection?: boolean; skipCancelledWorkspacePreservation?: boolean; skipObsoletePolicyReviews?: boolean; skipAutomaticCompletionReviews?: boolean; skipUnsafeWorkspaceExports?: boolean; skipNativePendingEffects?: boolean }) { + async function reapOrphanedRuns(opts?: { staleThresholdMs?: number; skipScheduledRetryCancellation?: boolean; skipCapturedInstructionCommit?: boolean; skipInstructionDirectoryCleanup?: boolean; skipStoppedInstructionCollection?: boolean; skipCancelledWorkspacePreservation?: boolean; skipObsoletePolicyReviews?: boolean; skipAutomaticCompletionReviews?: boolean; skipUnsafeWorkspaceExports?: boolean; skipNativePendingEffects?: boolean; skipNativeResponsePresentation?: boolean }) { if (!opts?.skipCancelledWorkspacePreservation) await preserveCancelledNativeWorkspaces(db); const staleThresholdMs = opts?.staleThresholdMs ?? 0; const now = new Date(); @@ -1969,6 +1969,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD skipAutomaticCompletionReviews: opts?.skipAutomaticCompletionReviews, skipUnsafeWorkspaceExports: opts?.skipUnsafeWorkspaceExports, skipNativePendingEffects: opts?.skipNativePendingEffects, + skipNativeResponsePresentation: opts?.skipNativeResponsePresentation, environmentRuntime, pluginWorkerManager: dependencies.pluginWorkerManager, onWorkspaceSettled: settleRecoveredNativeWorkspace, diff --git a/server/src/services/heartbeat/run-completion.ts b/server/src/services/heartbeat/run-completion.ts index 1e5e13b858c..39bb4a509ab 100644 --- a/server/src/services/heartbeat/run-completion.ts +++ b/server/src/services/heartbeat/run-completion.ts @@ -351,7 +351,7 @@ export async function recordRunPresentationDecision(db: Db, input: { companyId: string; runId: string; issueId?: string; decision: RunPresentationDecision; }) { await db.transaction(async tx => { - await notifyChatRunMilestones(tx, { companyId: input.companyId, issueId: input.issueId }); + await notifyChatRunMilestones(tx, { companyId: input.companyId, runId: input.runId, issueId: input.issueId }); await tx.update(heartbeatRuns).set({ resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${JSON.stringify({ presentationDecision: input.decision })}::jsonb`, updatedAt: new Date(), diff --git a/server/src/services/heartbeat/run-lifecycle.test.ts b/server/src/services/heartbeat/run-lifecycle.test.ts index 1360bafe0c3..6d862394ed2 100644 --- a/server/src/services/heartbeat/run-lifecycle.test.ts +++ b/server/src/services/heartbeat/run-lifecycle.test.ts @@ -1,7 +1,7 @@ import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js"; import { randomBytes } from "node:crypto"; import type { Db } from "@paperclipai/db"; -import { agents, agentWakeupRequests, companies, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRunEvents, heartbeatRuns } from "@paperclipai/db"; +import { agents, agentWakeupRequests, companies, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRunEvents, heartbeatRuns, issues, subscribeDatabaseWork } from "@paperclipai/db"; import { eq, sql } from "drizzle-orm"; import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; @@ -241,6 +241,23 @@ describe.skipIf(!support.supported)("heartbeat lifecycle database wiring", () => } finally { unsubscribe(); } }); + it.each(["native", "legacy", "unbound"] as const)("keeps %s run-status response recovery scoped to its task or run", async binding => { + const { run } = await fixture(); + const [issue] = await db.insert(issues).values({ companyId: run.companyId, title: "Scope owner" }).returning(); + await db.update(heartbeatRuns).set(binding === "native" ? { nativeIssueId: issue.id } + : binding === "legacy" ? { contextSnapshot: { issueId: issue.id } } : { contextSnapshot: {} }).where(eq(heartbeatRuns.id, run.id)); + const hints: unknown[] = []; + const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.nativeResponsePresentation, (event, scopes) => { + if (event === "settled") for (const hint of scopes ?? []) hints.push(JSON.parse(hint)); + }); + try { + await createHeartbeatLifecycle(db, callbacks(db)).setRunStatusIfRunning(run.id, "succeeded", { finishedAt: new Date() }); + expect(hints).toEqual([binding === "unbound" ? { companyId: run.companyId, runId: run.id } + : { companyId: run.companyId, issueId: issue.id }]); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0].status).toBe("succeeded"); + } finally { unsubscribe(); } + }); + it("does not request accounting for an unchanged running status", async () => { const { run } = await fixture(); const settled = vi.fn(); diff --git a/server/src/services/heartbeat/run-lifecycle.ts b/server/src/services/heartbeat/run-lifecycle.ts index 00f664d4769..0904b58a42f 100644 --- a/server/src/services/heartbeat/run-lifecycle.ts +++ b/server/src/services/heartbeat/run-lifecycle.ts @@ -1017,7 +1017,7 @@ export function createHeartbeatLifecycle(db: Db, dependencies: HeartbeatLifecycl return recordHeartbeatRunRuntimeProgress(currentRun, update, issueId); } - async function writeRunStatus(status: string, previous: Pick | null, + async function writeRunStatus(status: string, previous: Pick | null, write: (tx: Db) => Promise) { const companyId = previous?.companyId; // Register on the outer transaction before the terminal write. An unknown @@ -1031,7 +1031,8 @@ export function createHeartbeatLifecycle(db: Db, dependencies: HeartbeatLifecycl if (previous) await notifyCancelledWorkspacePreservation(tx, { companyId: previous.companyId, runId: previous.id }); if (isHeartbeatRunTerminalStatus(status)) await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, companyId); await notifyExecutionRunWork(tx); - if (companyId) await notifyChatRunMilestones(tx, { companyId }); + if (previous) await notifyChatRunMilestones(tx, { companyId: previous.companyId, runId: previous.id, + issueId: previous.nativeIssueId ?? previous.issueId ?? readNonEmptyString(parseObject(previous.contextSnapshot).issueId) ?? undefined }); return write(tx); }); } diff --git a/server/src/services/native-finalization-notifications.ts b/server/src/services/native-finalization-notifications.ts new file mode 100644 index 00000000000..f14f1e79db8 --- /dev/null +++ b/server/src/services/native-finalization-notifications.ts @@ -0,0 +1,10 @@ +import { notifyNativeResponsePresentation } from "./native-response-presentation-notifications.js"; +import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; + +export type NativeFinalizationWorkScope = { companyId: string; runId?: string; issueId?: string }; + +export async function notifyNativeFinalizationWork(transaction: object, scope: NativeFinalizationWorkScope) { + if (scope.issueId || scope.runId) await notifyNativeResponsePresentation(transaction, scope); + await notifyDeliveryWork(transaction, DELIVERY_QUEUES.nativePendingEffects, + JSON.stringify({ companyId: scope.companyId, runId: scope.runId })); +} diff --git a/server/src/services/native-pending-effect-notifications.ts b/server/src/services/native-pending-effect-notifications.ts deleted file mode 100644 index e9c28d79782..00000000000 --- a/server/src/services/native-pending-effect-notifications.ts +++ /dev/null @@ -1,8 +0,0 @@ -import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; - -export type NativePendingEffectScope = { companyId: string; runId?: string }; - -export async function notifyNativePendingEffects(transaction: object, scope: NativePendingEffectScope) { - await notifyDeliveryWork(transaction, DELIVERY_QUEUES.nativePendingEffects, - JSON.stringify({ companyId: scope.companyId, runId: scope.runId })); -} diff --git a/server/src/services/native-response-presentation-notifications.ts b/server/src/services/native-response-presentation-notifications.ts new file mode 100644 index 00000000000..8be14daae95 --- /dev/null +++ b/server/src/services/native-response-presentation-notifications.ts @@ -0,0 +1,14 @@ +import { readCompletedAssistantMessageCandidate } from "./heartbeat-run-summary.js"; +import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; + +export type NativeResponsePresentationScope = { companyId: string; issueId?: string; runId?: string; agentId?: string }; + +export async function notifyNativeResponsePresentation(tx: object, scope: NativeResponsePresentationScope) { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.nativeResponsePresentation, + JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId, runId: scope.runId, agentId: scope.agentId })); +} + +export function isNativeResponseEvidence(eventType: string, prpEvent: unknown): boolean { + return eventType === "run.result.accepted" || eventType === "item.completed" && + readCompletedAssistantMessageCandidate({ seq: 0, prpEvent })?.channel === "final"; +} diff --git a/server/src/services/native-runtime/native-finalization-reconciler.ts b/server/src/services/native-runtime/native-finalization-reconciler.ts index 4797142552e..5fc5567ea2e 100644 --- a/server/src/services/native-runtime/native-finalization-reconciler.ts +++ b/server/src/services/native-runtime/native-finalization-reconciler.ts @@ -644,6 +644,7 @@ type NativeReconciliationOptions = { skipAutomaticCompletionReviews?: boolean; skipUnsafeWorkspaceExports?: boolean; skipNativePendingEffects?: boolean; + skipNativeResponsePresentation?: boolean; pluginWorkerManager?: WorkspaceWorkers; environmentRuntime?: EnvironmentRuntimeService; onWorkspaceSettled?: (input: { @@ -861,7 +862,7 @@ async function reconcileNativeFinalizationsPass( )).limit(1).then((entries) => entries[0] ?? null) : null; const currentDecisionJson = record(currentDecision?.decisionJson); - if (row.coordinatorPhase === "committed") { + if (row.coordinatorPhase === "committed" && !options.skipNativeResponsePresentation) { // A later decision can supersede this run's task status, but cannot // erase its accepted, still-authorized response. Repair presentation // before the status-only early return below, without rerunning work. @@ -1123,6 +1124,7 @@ async function reconcileNativeFinalizationsPass( runId: row.runId, workspaceFinalizeStatus, projectRunStatus: true, + skipNativeResponsePresentation: options.skipNativeResponsePresentation, preserveProviderAttempt: Boolean( readNativeWorkspaceSyncReference( row.nativeWorkspaceSync, @@ -1181,6 +1183,7 @@ async function reconcileNativeFinalizationsPass( runId: row.runId, workspaceFinalizeStatus: barrier.status as "succeeded" | "failed", projectRunStatus: true, + skipNativeResponsePresentation: options.skipNativeResponsePresentation, preserveProviderAttempt: Boolean( readNativeWorkspaceSyncReference( row.nativeWorkspaceSync, diff --git a/server/src/services/native-runtime/native-interaction-bridge.ts b/server/src/services/native-runtime/native-interaction-bridge.ts index 680ab867033..4243bcfa6a3 100644 --- a/server/src/services/native-runtime/native-interaction-bridge.ts +++ b/server/src/services/native-runtime/native-interaction-bridge.ts @@ -1,4 +1,4 @@ -import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js"; +import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js"; import type { Db } from "@paperclipai/db"; import type { NativeInteractionResponseEnvelope } from "../../vendor/paperclip-runner/index.js"; @@ -633,7 +633,7 @@ export async function routePersistedNativeResultAttention(input: { const delegation = effects.find((effect) => effect.kind === "create_delegated_issue"); const finalizationError = effects.find((effect) => effect.kind === "record_finalization_error"); await input.db.transaction(async tx => { - await notifyNativePendingEffects(tx, { companyId: binding.companyId, runId: input.runId }); + await notifyNativeFinalizationWork(tx, { companyId: binding.companyId, runId: input.runId }); await tx.update(nativeRunFinalizations).set({ phase: finalizationError ? "retryable_failure" : "committed", assessmentId: existingAssessment.id, diff --git a/server/src/services/native-runtime/native-pending-effect-work.ts b/server/src/services/native-runtime/native-pending-effect-work.ts index 5893f2eadf3..3f276a0b636 100644 --- a/server/src/services/native-runtime/native-pending-effect-work.ts +++ b/server/src/services/native-runtime/native-pending-effect-work.ts @@ -2,7 +2,7 @@ import { and, asc, eq, exists, gt, inArray, isNotNull, ne } from "drizzle-orm"; import { heartbeatRuns, issues, nativeRunFinalizations, statusDecisionEffects, type Db } from "@paperclipai/db"; import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js"; import { DELIVERY_QUEUES } from "../delivery-work-notifications.js"; -import type { NativePendingEffectScope } from "../native-pending-effect-notifications.js"; +import type { NativeFinalizationWorkScope } from "../native-finalization-notifications.js"; import { registerScopedDeadlineWork } from "../scoped-deadline-work.js"; import { dispatchPendingNativeStatusEffects, nativePendingEffectDueAt } from "./status-decision-committer.js"; @@ -15,7 +15,7 @@ export function pendingNativeStatusEffectsCondition(db: Db) { ))); } -function findPendingEffects(db: Db, scope?: NativePendingEffectScope, after?: string) { +function findPendingEffects(db: Db, scope?: NativeFinalizationWorkScope, after?: string) { return db.select({ id: nativeRunFinalizations.runId, companyId: nativeRunFinalizations.companyId, issueId: nativeRunFinalizations.issueId, decisionId: nativeRunFinalizations.decisionId, @@ -36,7 +36,7 @@ function findPendingEffects(db: Db, scope?: NativePendingEffectScope, after?: st export function registerNativePendingEffectWork(db: Db, coordinator: ReturnType, canRun: () => boolean) { type Entry = { id: string; companyId: string; dueAt: number }; - async function load(scope?: NativePendingEffectScope, signal?: AbortSignal): Promise { + async function load(scope?: NativeFinalizationWorkScope, signal?: AbortSignal): Promise { const entries: Entry[] = []; let after: string | undefined; while (canRun() && !signal?.aborted) { @@ -49,8 +49,8 @@ export function registerNativePendingEffectWork(db: Db, coordinator: ReturnType< } return registerScopedDeadlineWork(coordinator, { queue: DELIVERY_QUEUES.nativePendingEffects, batchSize: 25, - decodeScope: (hint: string): NativePendingEffectScope | undefined => hint === "*" ? undefined : JSON.parse(hint), - matches: (entry: Entry, scope?: NativePendingEffectScope) => !scope || entry.companyId === scope.companyId && (!scope.runId || entry.id === scope.runId), + decodeScope: (hint: string): NativeFinalizationWorkScope | undefined => hint === "*" ? undefined : JSON.parse(hint), + matches: (entry: Entry, scope?: NativeFinalizationWorkScope) => !scope || entry.companyId === scope.companyId && (!scope.runId || entry.id === scope.runId), load, check: async (entry: Entry, signal?: AbortSignal) => { if (!canRun() || signal?.aborted) throw new Error("Native pending effect recovery stopped"); diff --git a/server/src/services/native-runtime/native-response-presentation-work.ts b/server/src/services/native-runtime/native-response-presentation-work.ts new file mode 100644 index 00000000000..6f545e2c923 --- /dev/null +++ b/server/src/services/native-runtime/native-response-presentation-work.ts @@ -0,0 +1,117 @@ +import { and, asc, eq, gt, inArray, isNotNull, isNull, notExists, or, sql } from "drizzle-orm"; +import { heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults, statusDecisions, statusDecisionEffects, type Db } from "@paperclipai/db"; +import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js"; +import { DELIVERY_QUEUES } from "../delivery-work-notifications.js"; +import type { NativeResponsePresentationScope } from "../native-response-presentation-notifications.js"; +import { onCommittedChatResponseAuthorityRegistered } from "../durable-chat-wakeup.js"; +import { registerScopedDeadlineWork } from "../scoped-deadline-work.js"; +import { recoverCommittedNativeChatResponse } from "./native-run-finalizer.js"; + +export function registerNativeResponsePresentationWork(db: Db, + coordinator: ReturnType, canRun: () => boolean) { + return nativeResponsePresentationWork(db).register(coordinator, canRun); +} + +type Entry = { id: string; companyId: string; issueId: string; agentId: string; dueAt: number }; + +export function nativeResponsePresentationWork(db: Db) { + function candidates(scope?: NativeResponsePresentationScope, after?: string, runId?: string) { + return db.select({ id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, + issueId: nativeRunFinalizations.issueId, agentId: heartbeatRuns.agentId, nextAttemptAt: nativeRunFinalizations.nextAttemptAt, + }).from(nativeRunFinalizations) + .innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, nativeRunFinalizations.runId), + eq(heartbeatRuns.companyId, nativeRunFinalizations.companyId), eq(heartbeatRuns.nativeIssueId, nativeRunFinalizations.issueId))) + .innerJoin(issues, and(eq(issues.id, nativeRunFinalizations.issueId), eq(issues.companyId, nativeRunFinalizations.companyId))) + .innerJoin(nativeRunResults, and(eq(nativeRunResults.id, nativeRunFinalizations.resultId), + eq(nativeRunResults.companyId, nativeRunFinalizations.companyId), eq(nativeRunResults.issueId, nativeRunFinalizations.issueId), + eq(nativeRunResults.runId, heartbeatRuns.id), eq(nativeRunResults.completionContractId, heartbeatRuns.completionContractId))) + .innerJoin(statusDecisions, and(eq(statusDecisions.id, nativeRunFinalizations.decisionId), + eq(statusDecisions.companyId, nativeRunFinalizations.companyId), eq(statusDecisions.issueId, nativeRunFinalizations.issueId), + eq(statusDecisions.runId, heartbeatRuns.id), eq(statusDecisions.assessmentId, nativeRunFinalizations.assessmentId))) + .where(and( + eq(heartbeatRuns.runtimeMode, "native"), inArray(heartbeatRuns.status, ["succeeded", "failed", "timed_out"]), + isNotNull(heartbeatRuns.finishedAt), eq(nativeRunFinalizations.phase, "committed"), isNull(nativeRunFinalizations.leaseOwner), + eq(nativeRunResults.schemaStatus, "accepted"), + sql`jsonb_typeof(${heartbeatRuns.resultJson} #> '{presentationDecision,commentId}') is distinct from 'string'`, + sql`coalesce(${heartbeatRuns.resultJson} #> '{presentationDecision,commentAction}', '"create"'::jsonb) = '"create"'::jsonb`, + sql`${heartbeatRuns.contextSnapshot}->'skipIssueComment' is distinct from 'true'::jsonb`, + sql`coalesce(${statusDecisions.decisionJson}->'externalChatReviewPresentation', 'null'::jsonb) in ('null'::jsonb, 'false'::jsonb, '0'::jsonb, '""'::jsonb)`, + sql`${nativeRunResults.resultJson} #>> '{result,schema}' = 'paperclip.run_result.v1'`, + sql`${nativeRunResults.resultJson} #>> '{terminal,runTerminalState}' = 'succeeded'`, + sql`${nativeRunResults.resultJson} #>> '{terminal,turnTerminalState}' = 'completed'`, + or( + and(eq(heartbeatRuns.status, "succeeded"), + sql`(coalesce(${issues.conversationAgentId}::text, '') = '' or coalesce(${issues.conversationUserId}, '') = '')`, + sql`${nativeRunResults.resultJson} #>> '{result,reportedWorkDisposition}' = 'done'`, + sql`${nativeRunResults.resultJson} #>> '{terminal,reportedWorkDisposition}' = 'done'`), + and(sql`${nativeRunResults.resultJson} #>> '{result,reportedWorkDisposition}' = 'yielded'`, + sql`${nativeRunResults.resultJson} #>> '{terminal,reportedWorkDisposition}' = 'yielded'`, + sql`jsonb_typeof(${nativeRunResults.resultJson} #> '{result,summary}') = 'string' + and ${nativeRunResults.resultJson} #>> '{result,summary}' ~ '[^[:space:]]'`, + sql`jsonb_typeof(${nativeRunResults.resultJson} #> '{result,continuation,idempotencyKey}') = 'string' + and ${nativeRunResults.resultJson} #>> '{result,continuation,idempotencyKey}' ~ '[^[:space:]]'`, + or( + and(sql`${nativeRunResults.resultJson} #>> '{result,continuation,kind}' = 'response_wake'`, + sql`${nativeRunResults.resultJson} #> '{result,attentionRequests}' = '[]'::jsonb`, + sql`jsonb_typeof(${nativeRunResults.resultJson} #> '{result,continuation,summary}') = 'string' + and ${nativeRunResults.resultJson} #>> '{result,continuation,summary}' ~ '[^[:space:]]'`), + and(eq(heartbeatRuns.status, "succeeded"), or( + and(sql`${nativeRunResults.resultJson} #>> '{result,continuation,kind}' = 'same_agent'`, + eq(statusDecisions.reasonCode, "live_continuation_registered")), + and(sql`${nativeRunResults.resultJson} #>> '{result,continuation,kind}' = 'monitor'`, + inArray(statusDecisions.reasonCode, ["scheduled_monitor_waiting", "scheduled_monitor_dispatched", "scheduled_monitor_preserved"])), + )), + )), + ), + notExists(db.select({ id: statusDecisionEffects.id }).from(statusDecisionEffects).where(and( + eq(statusDecisionEffects.companyId, nativeRunFinalizations.companyId), + eq(statusDecisionEffects.issueId, nativeRunFinalizations.issueId), + eq(statusDecisionEffects.decisionId, nativeRunFinalizations.decisionId), + inArray(statusDecisionEffects.deliveryState, ["pending", "failed"]), + ))), + scope ? and(eq(heartbeatRuns.companyId, scope.companyId), scope.issueId ? eq(nativeRunFinalizations.issueId, scope.issueId) : undefined) : undefined, + scope?.runId ? eq(heartbeatRuns.id, scope.runId) : undefined, + scope?.agentId ? eq(heartbeatRuns.agentId, scope.agentId) : undefined, + after ? gt(heartbeatRuns.id, after) : undefined, runId ? eq(heartbeatRuns.id, runId) : undefined, + )).orderBy(asc(heartbeatRuns.id)).limit(100); + } + async function load(scope?: NativeResponsePresentationScope, signal?: AbortSignal, canRun = () => true): Promise { + const entries: Entry[] = []; + let after: string | undefined; + while (canRun() && !signal?.aborted) { + const rows = await candidates(scope, after); + for (const row of rows) entries.push({ id: row.id, companyId: row.companyId, issueId: row.issueId, agentId: row.agentId, dueAt: row.nextAttemptAt?.getTime() ?? 0 }); + if (rows.length < 100) return entries; + after = rows[rows.length - 1].id; + } + throw new Error("Native response presentation discovery stopped"); + } + async function check(entry: Entry, signal?: AbortSignal): Promise { + if (signal?.aborted) throw new Error("Native response presentation recovery stopped"); + const scope = { companyId: entry.companyId, issueId: entry.issueId }; + const [current] = await candidates(scope, undefined, entry.id); + if (!current) return null; + if (current.nextAttemptAt && current.nextAttemptAt.getTime() > Date.now()) return { ...entry, dueAt: current.nextAttemptAt.getTime() }; + if (signal?.aborted) throw new Error("Native response presentation recovery stopped"); + const recoveryStartedAt = Date.now(); + await recoverCommittedNativeChatResponse(db, { ...scope, runId: entry.id, onlyWhenDue: true }); + const [remaining] = await candidates(scope, undefined, entry.id); + return remaining?.nextAttemptAt && remaining.nextAttemptAt.getTime() > recoveryStartedAt + ? { ...entry, dueAt: remaining.nextAttemptAt.getTime() } : null; + } + function register(coordinator: ReturnType, canRun: () => boolean) { + const worker = registerScopedDeadlineWork(coordinator, { + queue: DELIVERY_QUEUES.nativeResponsePresentation, + decodeScope: (hint: string): NativeResponsePresentationScope | undefined => hint === "*" ? undefined : JSON.parse(hint), + matches: (entry: Entry, scope?: NativeResponsePresentationScope) => !scope || entry.companyId === scope.companyId && (!scope.issueId || entry.issueId === scope.issueId) && (!scope.runId || entry.id === scope.runId) && (!scope.agentId || entry.agentId === scope.agentId), + load: (scope, signal) => load(scope, signal, canRun), + check: (entry, signal) => { + if (!canRun()) throw new Error("Native response presentation recovery paused"); + return check(entry, signal); + }, + }, canRun); + const unsubscribe = onCommittedChatResponseAuthorityRegistered(db, () => worker.invalidate()); + return { ...worker, stop: unsubscribe }; + } + return { load, check, register }; +} diff --git a/server/src/services/native-runtime/native-run-coordinator-store.ts b/server/src/services/native-runtime/native-run-coordinator-store.ts index 593f5c0b4b2..1f911a854b3 100644 --- a/server/src/services/native-runtime/native-run-coordinator-store.ts +++ b/server/src/services/native-runtime/native-run-coordinator-store.ts @@ -1,4 +1,5 @@ -import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js"; +import { isNativeResponseEvidence, notifyNativeResponsePresentation } from "../native-response-presentation-notifications.js"; +import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js"; import { isSafeNativeChatProgressEventType } from "../safe-native-chat-progress.js"; import { notifyChatNativeProgress } from "../chat-native-progress-notifications.js"; @@ -297,6 +298,9 @@ export class NativeRunCoordinatorStore { if (isSafeNativeChatProgressEventType(event.eventType)) { await notifyChatNativeProgress(tx, { companyId: this.#binding.companyId, issueId: this.#binding.issueId }); } + if (isNativeResponseEvidence(event.eventType, canonicalPayload)) { + await notifyNativeResponsePresentation(tx, this.#binding); + } const cursor = run.nextEventSeq; const [inserted] = await tx .insert(heartbeatRunEvents) @@ -523,7 +527,7 @@ export class NativeRunCoordinatorStore { ) { throw new Error("native_finalization_lease_conflict"); } - if (row.decisionId) await notifyNativePendingEffects(tx, this.#binding); + if (row.decisionId) await notifyNativeFinalizationWork(tx, this.#binding); if ( row.leaseOwner === input.leaseOwner && row.leaseExpiresAt && @@ -566,7 +570,7 @@ export class NativeRunCoordinatorStore { throw new Error("native_finalization_retry_invalid"); } const [updated] = await this.#db.transaction(async tx => { - await notifyNativePendingEffects(tx, this.#binding); + await notifyNativeFinalizationWork(tx, this.#binding); if (input.failureCode === "native_workspace_sync_out_unsafe_archive") { await notifyUnsafeWorkspaceExportRecovery(tx, this.#binding); } diff --git a/server/src/services/native-runtime/native-run-finalizer.ts b/server/src/services/native-runtime/native-run-finalizer.ts index f6b93c9699f..9df2211a496 100644 --- a/server/src/services/native-runtime/native-run-finalizer.ts +++ b/server/src/services/native-runtime/native-run-finalizer.ts @@ -1,4 +1,4 @@ -import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js"; +import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js"; import { notifyUnsafeWorkspaceExportRunChange } from "../unsafe-workspace-export-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js"; import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js"; @@ -39,6 +39,7 @@ import { nativeRunFinalizations, nativeRunResults, statusDecisions, + statusDecisionEffects, workAssessments, workspaceOperations, } from "@paperclipai/db"; @@ -215,7 +216,7 @@ async function claimCoordinator(input: { coordinator.leaseOwner !== leaseOwner ) throw new Error("native_finalization_lease_busy"); - if (coordinator.decisionId) await notifyNativePendingEffects(tx, coordinator); + if (coordinator.decisionId) await notifyNativeFinalizationWork(tx, coordinator); const [updated] = await tx .update(nativeRunFinalizations) .set({ @@ -383,7 +384,7 @@ async function recordRetryableFailure(input: { ? "native_workspace_sync_out_retry_exhausted" : "native_finalization_retry_exhausted" : input.failureCode; - await notifyNativePendingEffects(tx, { companyId: input.run.companyId, runId: input.run.id }); + await notifyNativeFinalizationWork(tx, { companyId: input.run.companyId, runId: input.run.id }); if (phase === "terminal_failure") await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, input.run.companyId); if (phase === "terminal_failure" && failureCode === "native_workspace_sync_out_unsafe_archive") { await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: input.run.companyId, issueId: input.coordinator.issueId }); @@ -848,6 +849,21 @@ async function acceptedResponseDigestMatches( export async function repairCommittedNativeChatResponse( db: Db, input: { companyId: string; issueId: string; runId: string }, +): Promise { + try { + return await recoverCommittedNativeChatResponse(db, input); + } catch (error) { + logger.warn( + { runId: input.runId, error }, + "Committed chat response is awaiting presentation retry", + ); + return false; + } +} + +export async function recoverCommittedNativeChatResponse( + db: Db, + input: { companyId: string; issueId: string; runId: string; onlyWhenDue?: boolean }, ): Promise { let agentId: string | null = null; let materialized = false; @@ -909,9 +925,11 @@ export async function repairCommittedNativeChatResponse( return false; // An existing selected answer, including a subsequently deleted one, must // never be replaced or resurrected by a background presentation repair. + const recordedPresentation = record(record(run.resultJson).presentationDecision); if ( - typeof record(record(run.resultJson).presentationDecision).commentId === - "string" + typeof recordedPresentation.commentId === "string" || + (recordedPresentation.commentAction !== undefined && recordedPresentation.commentAction !== "create") || + record(run.contextSnapshot).skipIssueComment === true ) return false; const [coordinator] = await tx @@ -933,6 +951,16 @@ export async function repairCommittedNativeChatResponse( !coordinator.assessmentId ) return false; + if (input.onlyWhenDue) { + if (coordinator.nextAttemptAt && coordinator.nextAttemptAt.getTime() > Date.now()) return false; + const [pendingEffect] = await tx.select({ id: statusDecisionEffects.id }).from(statusDecisionEffects).where(and( + eq(statusDecisionEffects.companyId, input.companyId), + eq(statusDecisionEffects.issueId, input.issueId), + eq(statusDecisionEffects.decisionId, coordinator.decisionId), + inArray(statusDecisionEffects.deliveryState, ["pending", "failed"]), + )).limit(1); + if (pendingEffect) return false; + } const [accepted] = await tx .select() .from(nativeRunResults) @@ -1123,13 +1151,7 @@ export async function repairCommittedNativeChatResponse( }); } catch (error) { if (error instanceof CommittedChatResponseAuthorizationError) return false; - // Keep transient failures visible and retryable without reclassifying the - // committed run or acknowledging a publication that never committed. - logger.warn( - { runId: input.runId, error }, - "Committed chat response is awaiting presentation retry", - ); - return false; + throw error; } if (materialized && agentId) publishChatPublicationCommitSignal({ @@ -1167,6 +1189,7 @@ export async function finalizeNativeRun(input: { workspaceFinalizeStatus: "succeeded" | "failed"; /** Reconciliation owns terminal run projection; the live heartbeat does it afterward. */ projectRunStatus?: boolean; + skipNativeResponsePresentation?: boolean; /** Workspace-only replay must not consume the provider recovery budget. */ preserveProviderAttempt?: boolean; failpoint?: NativeStatusCommitFailpoint; @@ -1220,7 +1243,7 @@ export async function finalizeNativeRun(input: { await projectCommittedRun({ db: input.db, run: admittedRun, coordinator }); if (input.projectRunStatus && !presentationAlreadyMaterialized) await materializeCommittedReviewResponse(input.db, input.runId); - if (input.projectRunStatus) + if (input.projectRunStatus && !input.skipNativeResponsePresentation) await repairCommittedNativeChatResponse(input.db, { companyId: admittedRun.companyId, issueId: coordinator.issueId, @@ -1721,7 +1744,7 @@ export async function finalizeNativeRun(input: { } if (input.projectRunStatus) await materializeCommittedReviewResponse(input.db, input.runId); - if (input.projectRunStatus && finalizationPhase === "committed") + if (input.projectRunStatus && finalizationPhase === "committed" && !input.skipNativeResponsePresentation) await repairCommittedNativeChatResponse(input.db, { companyId: run.companyId, issueId: coordinator.issueId, diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index c854d1a2e45..4e73bfcfa45 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -1,4 +1,4 @@ -import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js"; +import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js"; import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js"; import { notifyInstructionCollection } from "../instruction-collection-notifications.js"; @@ -3898,7 +3898,7 @@ export async function reconcileRetainedNativeSessionCleanup( renameSync(owned.root, archive); } renameSync(copy, owned.root); - await notifyNativePendingEffects(tx, { companyId: owned.run.companyId, runId: owned.run.id }); + await notifyNativeFinalizationWork(tx, { companyId: owned.run.companyId, runId: owned.run.id }); await tx .update(nativeRunFinalizations) .set({ @@ -3956,7 +3956,7 @@ export async function reconcileRetainedNativeSessionCleanup( .limit(1) .then((rows) => rows[0]); if (!current || current.leaseOwner !== leaseOwner) return; - await notifyNativePendingEffects(tx, { companyId: owned.run.companyId, runId: owned.run.id }); + await notifyNativeFinalizationWork(tx, { companyId: owned.run.companyId, runId: owned.run.id }); await tx .update(nativeRunFinalizations) .set({ @@ -9700,7 +9700,7 @@ async function executePaperclipNativeSessionWithinScope( ) .for("update"); await notifyExecutionRunWork(tx); - await notifyNativePendingEffects(tx, input.execution.binding); + await notifyNativeFinalizationWork(tx, input.execution.binding); const updated = await tx .update(nativeRunFinalizations) .set({ @@ -10036,7 +10036,7 @@ async function executePaperclipNativeSessionWithinScope( } const releaseNow = new Date(); const released = await timeController("coordinator_release", () => input.db.transaction(async tx => { - await notifyNativePendingEffects(tx, input.execution.binding); + await notifyNativeFinalizationWork(tx, input.execution.binding); return tx.update(nativeRunFinalizations) .set({ leaseOwner: null, diff --git a/server/src/services/native-runtime/native-workspace-export-recovery.ts b/server/src/services/native-runtime/native-workspace-export-recovery.ts index 8974e68a6e6..c638f3b3e94 100644 --- a/server/src/services/native-runtime/native-workspace-export-recovery.ts +++ b/server/src/services/native-runtime/native-workspace-export-recovery.ts @@ -1,4 +1,4 @@ -import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js"; +import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js"; import { alias } from "drizzle-orm/pg-core"; import { notifyUnsafeWorkspaceExportRecovery, type UnsafeWorkspaceExportScope } from "../unsafe-workspace-export-notifications.js"; import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js"; @@ -101,7 +101,7 @@ async function recoverCandidate(db: Db, candidate: { runId: string; companyId: s phase: "workspace_finalize", status: "succeeded", exitCode: 0, finishedAt: now, metadata: { owningService: "native_workspace_finalizer", workspaceSync: { omitted: true, reason: "restore_unsafe_archive", legacy: true } }, }); - await notifyNativePendingEffects(tx, { companyId: run.companyId, runId: run.id }); + await notifyNativeFinalizationWork(tx, { companyId: run.companyId, runId: run.id }); await tx.update(nativeRunFinalizations).set({ phase: "result_accepted", failureCode: null, nextAttemptAt: null, failureDetail: { workspaceFinalizeAttempt: 0, legacyUnsafeExportOmitted: { resultId: result.id, recoveredAt: now.toISOString() } }, updatedAt: now, }).where(eq(nativeRunFinalizations.runId, run.id)); diff --git a/server/src/services/native-runtime/native-workspace-export-retry.ts b/server/src/services/native-runtime/native-workspace-export-retry.ts index 69485cb4d74..99e286a17ff 100644 --- a/server/src/services/native-runtime/native-workspace-export-retry.ts +++ b/server/src/services/native-runtime/native-workspace-export-retry.ts @@ -1,4 +1,4 @@ -import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js"; +import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js"; import { notifyEnvironmentCleanup } from "../environment-cleanup-notifications.js"; import { randomUUID } from "node:crypto"; import type { Environment, EnvironmentLease } from "@paperclipai/shared"; @@ -135,7 +135,7 @@ export async function retryNativeWorkspaceExport(input: { const now = new Date(); const retry = { ...receipt, requestId: randomUUID(), actorId: input.actorId, repairNote: input.repairNote, requestedAt: now.toISOString(), stoppedProvider: initial.lease.metadata?.remoteExecutionTermination }; - await notifyNativePendingEffects(tx, { companyId: current.run.companyId, runId: current.run.id }); + await notifyNativeFinalizationWork(tx, { companyId: current.run.companyId, runId: current.run.id }); await tx.update(nativeRunFinalizations).set({ phase: "result_accepted", failureCode: null, nextAttemptAt: null, failureDetail: { ...current.coordinator.failureDetail, workspaceFinalizeAttempt: 0, workspaceExportRetry: retry }, updatedAt: now }).where(eq(nativeRunFinalizations.runId, current.run.id)); await tx.update(heartbeatRuns).set({ status: "running", finishedAt: null, error: null, errorCode: null, diff --git a/server/src/services/native-runtime/paperclip-control-plane-port.test.ts b/server/src/services/native-runtime/paperclip-control-plane-port.test.ts index 57d49cff0aa..b6bec7a4214 100644 --- a/server/src/services/native-runtime/paperclip-control-plane-port.test.ts +++ b/server/src/services/native-runtime/paperclip-control-plane-port.test.ts @@ -1,3 +1,5 @@ +import { nativeResponsePresentationWork } from "./native-response-presentation-work.js"; +import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js"; import { afterAll, beforeAll, describe, expect, it } from "vitest"; import { asc, eq, sql } from "drizzle-orm"; import { generateKeyPairSync, randomUUID } from "node:crypto"; @@ -65,6 +67,8 @@ import { subscribeCompanyLiveEvents, } from "../live-events.js"; +const external = resolveExternalTestDatabaseUrl("PAPERCLIP_CONTROL_PLANE_TEST_DATABASE_URL"); + describe("PaperclipControlPlanePort conformance", () => { let temporary: Awaited> | null = null; let db: ReturnType; @@ -87,8 +91,11 @@ describe("PaperclipControlPlanePort conformance", () => { const governanceRunnerId = "00000000-0000-4000-8000-000000000021"; beforeAll(async () => { - temporary = await startRecoveryPostgres("paperclip-native-port-"); - db = createDb(temporary.connectionString); + if (external) db = createDb(external); + else { + temporary = await startRecoveryPostgres("paperclip-native-port-"); + db = createDb(temporary.connectionString); + } const identity = CONTROL_PLANE_CONFORMANCE_OPEN.identity; await db.insert(companies).values({ id: identity.companyId, name: "Phase 6", issuePrefix: "P6C" }); await db.insert(agents).values({ @@ -645,7 +652,7 @@ describe("PaperclipControlPlanePort conformance", () => { completionContractSha256: localContractSha, contextSnapshot: { issueId }, }); - const observerDb = createDb(temporary!.connectionString); + const observerDb = createDb(external ?? temporary!.connectionString); const observed: Array> = []; const committedCallbacks: string[] = []; let visibilityCheck: Promise> | null = null; @@ -1096,7 +1103,10 @@ describe("PaperclipControlPlanePort conformance", () => { .where(eq(heartbeatRuns.id, taskRunId)); expect(await repair()).toBe(false); await db.update(heartbeatRuns).set({ contextSnapshot: {} }).where(eq(heartbeatRuns.id, taskRunId)); - expect(await repair()).toBe(true); + const work = nativeResponsePresentationWork(db); + const [entry] = await work.load({ companyId: identity.companyId, runId: taskRunId }); + expect(entry?.id).toBe(taskRunId); + expect(await work.check(entry)).toBeNull(); expect(await repair()).toBe(false); expect(await db.select({ body: issueComments.body }).from(issueComments) .where(eq(issueComments.createdByRunId, taskRunId)).orderBy(asc(issueComments.createdAt))) diff --git a/server/src/services/native-runtime/status-decision-committer.ts b/server/src/services/native-runtime/status-decision-committer.ts index 751734d70fb..04696bc1c85 100644 --- a/server/src/services/native-runtime/status-decision-committer.ts +++ b/server/src/services/native-runtime/status-decision-committer.ts @@ -1,4 +1,4 @@ -import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js"; +import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js"; import { notifyAutomaticCompletionReviews } from "../automatic-completion-review-notifications.js"; import { notifyObsoletePolicyReviews } from "../obsolete-policy-review-notifications.js"; @@ -154,7 +154,7 @@ export async function dispatchPendingNativeStatusEffects(input: { targetId: effect.targetId, }); } - await notifyNativePendingEffects(tx, input); + await notifyNativeFinalizationWork(tx, input); await notifyChatRunMilestones(tx, input); await tx .update(statusDecisionEffects) @@ -1241,7 +1241,7 @@ async function materializeDecisionEffect(input: { ) .for("update"); if (pending.length === 0) throw new Error("native_pending_effect_missing"); - await notifyNativePendingEffects(input.tx, { companyId: input.companyId }); + await notifyNativeFinalizationWork(input.tx, { companyId: input.companyId, issueId: input.issue.id }); await input.tx .update(statusDecisionEffects) .set({ @@ -1931,7 +1931,7 @@ export async function commitNativeStatusDecision(input: { effect => effect.kind === "record_finalization_error", ); await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, input.companyId); - await notifyNativePendingEffects(tx, input); + await notifyNativeFinalizationWork(tx, input); await tx .update(nativeRunFinalizations) .set({ diff --git a/server/src/services/preserved-result-finalization.ts b/server/src/services/preserved-result-finalization.ts index 083e84fefe6..69c47de6ac5 100644 --- a/server/src/services/preserved-result-finalization.ts +++ b/server/src/services/preserved-result-finalization.ts @@ -1,4 +1,4 @@ -import { notifyNativePendingEffects } from "./native-pending-effect-notifications.js"; +import { notifyNativeFinalizationWork } from "./native-finalization-notifications.js"; import { and, desc, eq, gte, ne, or, sql } from "drizzle-orm"; import { agentWakeupRequests, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db"; import { readSystemRecovery } from "./system-recovery-state.js"; @@ -46,7 +46,7 @@ export async function resumePreservedResultFinalization(db: Db, companyId: strin const generationId = intent?.id ?? recovery.generationId; if (coordinator.recoveryHistory.some(entry => entry.kind === "preserved_result_finalization" && entry.generationId === generationId)) return false; - await notifyNativePendingEffects(tx, { companyId, runId }); + await notifyNativeFinalizationWork(tx, { companyId, runId }); await tx.update(nativeRunFinalizations).set({ phase: coordinator.assessmentId ? "arbitrating" : "workspace_finalizing", nextAttemptAt: new Date(), failureCode: null, failureDetail: null, updatedAt: new Date(),