diff --git a/doc/plans/2026-10-24-recurring-server-sleep-stack.md b/doc/plans/2026-10-24-recurring-server-sleep-stack.md index 14a2777304e..d800eeea39f 100644 --- a/doc/plans/2026-10-24-recurring-server-sleep-stack.md +++ b/doc/plans/2026-10-24-recurring-server-sleep-stack.md @@ -2715,3 +2715,114 @@ and all final-head PR gates remain open. No PR has been merged by this stack wor Final slice 47 source passes workspace typecheck/build and 158 worker/startup cases, including exact near-term concurrent deadlines. Remote publication and acceptance are the next gates; local success does not satisfy the full stack. + +## Slice 47 published; slice 48 response-owner audit + +Slice 47 is draft [#17690](https://github.com/paperclipai/paperclip/pull/17690), +head `56c3601ea605b333141e86e0a810ed790fb6ce48`, based on #17687. Its final local +checks passed; remote CI and Greptile are running. No merge or remote acceptance +is claimed yet. + +Slice 48 starts accepted-response presentation recovery, excluding governed-review +presentation. Its current helper conflates dormant/settled work and transient +failures as false; resolve that owner contract before removing its backstop. +The exact null-lease, digest, selected-answer, and publication-authority fences +must remain intact. The existing external-chat baseline passed 33 selected cases; +1,551 others were not selected. See the +[response-presentation audit](2026-10-25-native-response-presentation-sleep-audit.md). + +The old action-required check on #17576 is Superagent Security Scan. Its existing +voice-review findings and the older E2E failures remain outstanding. Work on the +rest of the original stack continues; none of these gates are waived. + +Slice 48's first owner-contract change now passes the same 33 real-PostgreSQL +response cases plus server typecheck. The strict recovery entry point propagates +operational errors; existing callers keep their logging/boolean contract. No +response worker has been wired and no response polling is removed yet. + +The Superagent action-required gate on #17576 was traced to review 5643793631, +inline finding 4371283330: repeated connection/health writes reload the full live +voice-session inventory. This is the already-recorded bounded-discovery issue, +not a missing credential or permission. It still needs a code/test repair and +fresh review; the gate must not be bypassed. + +## Slice 47 accepted + +[#17690](https://github.com/paperclipai/paperclip/pull/17690) at exact head +`56c3601ea605b333141e86e0a810ed790fb6ce48` has all 53 checks passing, Greptile 5/5 +with no actionable findings or review threads, and a mergeable parent. Its PR body +records those gates. It remains draft and unmerged. Evidence is retained in +`/tmp/sleep47-checks-accepted.json`, `/tmp/sleep47-final-gates.json`, +`/tmp/sleep47-review-accepted.json`, and `/tmp/sleep47-review-threads.json`. + +Slice 48 remains local and incomplete: owner error separation and the 33-case +baseline pass; candidate discovery, producer coverage, lifecycle wiring, and +periodic removal remain. Older unresolved stack gates and every other original +recurring-server duty still require completion. The active goal is not complete. + + +## Slice 48 local worker progress + +Accepted-response recovery now has a local scoped worker, locked retry/effect +fences, milestone-topic notifications, and live authority registration invalidation. +The final selected real-PostgreSQL/lifecycle run passes 45 cases, and server types +pass. An elapsed-deadline race was reproduced before repair. A generated concurrent +partial index plus expression statistics avoids selected-history scans in the +10,000-row fixture; fresh migration and replay pass. See the +[response audit](2026-10-25-native-response-presentation-sleep-audit.md) for exact +scope and preserved failures. + +The worker is not wired into startup and periodic response recovery remains. +Producer coverage, remaining variants/failure tests, large dormant-backlog cost, +full local/remote acceptance, and PR publication are still required. Master was +refreshed to `7703220d38`; the complete stack has not yet been reconciled with it. +No merge, complete migration, whole-instance sleep, or full-goal completion is +claimed. + +## Slice 48 producer and lifecycle checkpoint + +Response recovery now shares the finalization notification boundary, receives +run-scoped late evidence, and wakes when a paused agent resumes or retry admission +settles. The resume gap was reproduced before fixing it at lifecycle's transaction +boundary. The existing lifecycle suite, both response variants' supporting suites, +and the combined acceptance selection cover 175 distinct passing cases across +runs; server types pass. A real source-row lock verifies retry across drain. +Earlier harness/test failures and the 10,000-unpresented-row discovery measurements +are preserved in the response audit. No startup wiring, periodic removal, PR +publication, remote acceptance, or whole-instance sleep is claimed yet. + + +## Slice 48 startup integration, 2026-10-26 + +The response worker now registers under enabled maintenance and releases its +subscription during shutdown. Periodic recovery skips both its direct repair and +already-committed finalizer replay; fresh finalization and explicit recovery stay +active. The startup suite passes 66 cases, and two database regressions confirm +periodic calls defer to the new worker. Workspace types pass. Full local tests +remain in progress with embedded-database initialization failures; build and +remote checks remain pending. The stack goal is still incomplete. + + +## Slice 48 review repair, 2026-10-26 + +Draft #17696 is open and attached. Review identified company-wide response reloads +from routine run-status notifications; three real database regressions reproduced +it. Status/claim/release wakes now retain task or exact-run scope while task +lineage remains task-wide. All 34 heartbeat lifecycle and 63 response/authority/ +admission tests pass, as do server types and the workspace build. The full local +suite was stopped after repeated embedded PostgreSQL startup failures and remains +unproven. Remote failures at `79bf36cbf8` include an early startup-settlement test +assertion (repaired), a workspace staging timeout, and an E2E runner shutdown. +See the response audit for logs. Fresh-head CI/review, earlier stack findings, +master reconciliation, and all remaining original duties stay open. Nothing merged. + + +## Slice 48 full-history acceptance repair + +Fresh review of `c4beddad1e` is 5/5 with its scope finding resolved. Remote CI +exposed a second early startup assertion. Full local chat verification finished +1,595 passed / seven failed; all seven were new worker tests observing a transient +null deadline or unfinished startup batch. Their settlement predicate now includes +zero active work. All 18 affected cases pass against the full suite's retained +history. Fresh remote acceptance remains pending; these are recorded test repairs, +not a waiver of failed gates or a change to production batch behavior. diff --git a/doc/plans/2026-10-25-native-pending-effects-sleep-audit.md b/doc/plans/2026-10-25-native-pending-effects-sleep-audit.md index 83347984ffc..987e0dcc500 100644 --- a/doc/plans/2026-10-25-native-pending-effects-sleep-audit.md +++ b/doc/plans/2026-10-25-native-pending-effects-sleep-audit.md @@ -160,6 +160,12 @@ deadline repair (`/tmp/sleep47-deadlines-typecheck.log`, `/tmp/sleep47-deadlines-build.log`). The final worker/startup run now passes 158 cases, giving 683 distinct passing cases with the earlier unaffected consumers. +## Remote acceptance + +Published as [#17690](https://github.com/paperclipai/paperclip/pull/17690). +Exact head `56c3601ea605b333141e86e0a810ed790fb6ce48`: all 53 checks pass; +Greptile 5/5 with no actionable findings or review threads; mergeable parent. +The PR remains draft and unmerged. This accepts slice 47 only, not the full stack. ## Current migration411 integration and complete acceptance diff --git a/doc/plans/2026-10-25-native-response-presentation-sleep-audit.md b/doc/plans/2026-10-25-native-response-presentation-sleep-audit.md new file mode 100644 index 00000000000..05486e9c98b --- /dev/null +++ b/doc/plans/2026-10-25-native-response-presentation-sleep-audit.md @@ -0,0 +1,665 @@ +# Native accepted-response presentation recovery + +Status: worker, producers, and startup/periodic integration implemented; full local and remote acceptance pending. +Branch: `codex/sleep-48-native-response-presentation`. +Parent: [#17690](https://github.com/paperclipai/paperclip/pull/17690), +`56c3601ea605b333141e86e0a810ed790fb6ce48`. + +## Reason for work + +A native run can have an accepted, committed result but no saved public reply. +The finalization reconciler calls `repairCommittedNativeChatResponse` before +skipping superseded status decisions. A later task decision must not erase an +already accepted, still-authorized response. This work neither re-executes the +provider nor changes issue disposition. It is separate from governed-review +presentation, evidence reassessment, and result/workspace settlement. + +The current helper covers three variants of the same accepted-response duty: +a saved same-agent/monitor continuation summary, an internal task's exact final +assistant message, and an authorized external response-wake summary. Governed +review responses are explicitly excluded and retain their own stricter owner. +Do not merge the review response migration into this slice merely because both +helpers live in the finalizer. + +## Existing authority and mutation boundary + +`repairCommittedNativeChatResponse` locks issue, run, and coordinator in that +order. It requires native runtime; succeeded, failed, or timed_out with finishedAt; +a committed coordinator with result/assessment/decision and a null leaseOwner. +It refuses to replace a selected comment, including a deleted selected answer. +It checks the accepted result's digest using either the coordinator format or +the older authenticated control-plane format, retaining its bounded journal +source lookup. It excludes externalChatReviewPresentation decisions. + +A lease that expired but still has an owner is not permission to publish: the +helper requires actual release. Do not reuse the pending-effects lease rule. +The outer periodic candidate query additionally respects saved nextAttemptAt. +A future worker must preserve that fence where applicable and wake on release. + +Internal completion needs an exact final assistant event for the accepted turn, +current internal-write authorization, and permission to create a new completion +comment. Continuation summaries check provider-owned interactions. External +response-wake summaries require the live registered chat authority, exact durable +source provenance, and current presentation authorization. The chat owner validates +source edits/deletion, membership, endpoint state/generation, consumed failure +lanes, and ambiguous delivery. None of this policy belongs in a generic scheduler. + +The comment and presentationDecision are written in the same transaction, through +the existing comment/publication owner. Notification follows durable commit. +Recovery must not revive private provider narration, duplicate a target, publish +under another source, or reinterpret the saved result using new task settings. + +## Contract gap to resolve before retiring polling + +The helper currently returns false for several distinct outcomes: already settled, +structurally ineligible, authority unavailable/revoked, and transient SQL/lock or +materialization failure. It logs most errors then returns false. A worker cannot +use that boolean as a pending-work predicate: dropping false loses retryable +responses, while retrying every false rescans ineligible history indefinitely. + +Keep the current boolean caller contract where needed, but expose the smallest +owner-level result/error contract that distinguishes dormant history from concrete +retry debt. Do not translate all authorization failures into retry forever. The +registered authority lives in durable-chat-wakeup.ts and is installed by +chat-channels/failed-run-retry.ts. Missing registration and later registration need +an explicit lifecycle wake path, separate from committed database mutations. + +Audit and test the candidate predicate before introducing a worker. Startup scans +must filter selected answers, irrelevant result kinds, review-owned responses, +and other passive history. Full authorization stays in the locked owner; SQL +prefiltering is not a new publication authorization mechanism. Measure actual query +plans with substantial completed history before claiming bounded discovery. + +## Producer audit still required + +- Native coordinator decision commit, persisted-attention replay, and ownership + release; result/assessment/decision binding changes. +- Run terminalization and accepted-turn event persistence, including late journal + arrival and direct recovery writers. Avoid waking on every unrelated progress + event merely because it shares the journal. +- Presentation selection/materialization and existing-comment suppression. A + deleted selected answer remains settled rather than becoming repair debt. +- Issue conversation/context changes and skipIssueComment behavior. +- Exact chat source, action/admission, publication, endpoint generation/status, + principal/membership, continuation provenance, and ambiguous-delivery resolution. + Separate mutations that only revoke work from supported reactivation paths. +- Live chat-authority registration/replacement/shutdown; subscription ownership. +- Startup, first write during drain, retry during drain, resume, and shutdown. + +Use existing durable records, scoped transaction signals, and the shared scheduler. +No new response queue table or permissive publication fallback is justified. +Independent SQL/multi-process writes remain outside the existing local signal +contract unless explicitly integrated; preserve the documented limitation. + +## Baseline evidence + +On the unchanged parent source, a fresh external PostgreSQL selection passed +33 accepted-response cases in chat-channels.integration.test.ts. The remaining +1,551 cases in that file were not selected, not claimed passed. The selection +covers duplicate/concurrent repair, source/authority changes, digest changes, +private disposition, cross-company/run binding, live heartbeat fences, the older +port-bound digest, GitHub internal prose/attachments, failure-lane replacement, +provider-dispatch reauthorization, selected/deleted answers, and contention. +Log: `/tmp/sleep48-chat-baseline.log`. Database: +`paperclip_sleep48_chat_baseline_20261025_test`. + +The monitor-continuation and control-plane-port suites currently require embedded +PostgreSQL directly. Qualify a validated external-test path if needed before using +their results on this host; do not count skipped or initialization-failed suites. +No sleep or migrated-worker behavior is established by this baseline. + +## Initial owner-contract change + +`recoverCommittedNativeChatResponse` now exposes the existing transaction and +returns false for settled/ineligible/authorization-denied state, while propagating +operational failures. The existing `repairCommittedNativeChatResponse` wrapper +retains logging and false-on-error behavior for current callers. Authority checks, +lock order, target materialization, and commit signaling remain in the same owner. +The commit signal itself contains subscriber errors, so the wrapper does not hide +a previously propagated publication error. + +The same 33-case real-PostgreSQL selection passes after this change, with added +assertions that the new entry point rejects under actual source-action lock +contention and returns false for the existing authorization/binding fences. The +server typecheck passes (`/tmp/sleep48-contract-typecheck.log`). Worker discovery, +producer notifications, startup wiring, and periodic removal are not implemented. + +## Existing notification boundary to reuse + +The current `notifyChatRunMilestones` boundary already covers 102 production call +sites and fans out lineage/authority changes to native progress, inbound notices, +confirmations, and GitHub work. Its scope is company plus optional issue because +source changes can authorize descendants. Audit its coverage for this duty before +adding caller wrappers. It already reaches status-decision settlement and run +projection, but raw coordinator-store/session release and accepted-turn journal +writers need individual inspection. Reuse is not proof that all producers exist. + + +## Local worker and locked scheduling fences + +The local worker now discovers exact run/coordinator/result/decision bindings, +filters selected answers and structurally irrelevant results, and dispatches +through `recoverCommittedNativeChatResponse`. It does not own publication policy. +`onlyWhenDue` rechecks the saved retry and outstanding status effects while the +existing coordinator lock is held. Existing explicit callers retain their old +behavior. A future retry is retained even if it becomes due between the locked +check and the follow-up read; its next attempt is immediate rather than discarded. + +Operational errors remain retryable through the shared coordinator. A structurally +eligible but unauthorized row becomes dormant until an eligibility mutation or +live authority registration invalidates its scope. Selected comments, including +deleted selections, remain excluded. No provider rerun or new work table is added. + +The existing milestone notification boundary now includes the response topic. +Hints explicitly select company and issue IDs: structural TypeScript callers may +pass larger objects, so serializing the whole argument is not safe. The first +integration run exposed that mistake as circular-object serialization during +status materialization; it was corrected before the passing runs below. + +Two narrow lifecycle capabilities support the new consumer: + +- `registerScopedDeadlineWork.invalidate` marks reconstruction dirty before waking. + A plain wake is insufficient after a prior dormant result removed its entry. + Existing consumers keep the same registration and scheduling behavior. +- The committed-response authority owner exposes registration notifications. + Registering/replacing the actual chat service reactivates dormant responses. + The worker's `stop` removes this subscription; the delivery coordinator still + owns worker cancellation and awaiting in-flight work. Repeated unsubscription + cannot delete a later listener set. The authority remains scoped to the exact Db. + +No startup registration or periodic skip is wired yet. The old recovery path is +still present. Full producer coverage, internal/continuation variants, recovery +failure/drain tests, and repeated large-scope mutation cost remain acceptance work. +Current discovery pages by run ID but reconstructs all matching pages in one +load; its resource bound under a large unresolved/dormant backlog needs proof +before activation. Do not describe the worker as a completed migration yet. + +## Query-plan and migration evidence + +The actual discovery query was captured from the driver. On a disposable database +with 10,000 additional selected runs, results, assessments, decisions, and +coordinators, it initially scanned completed history (22.656 ms). A partial +unpresented-run index alone still chose the wrong join order (16.532 ms). +Expression statistics for the selected-comment predicate allowed PostgreSQL to +start from the one unpresented run (0.091 ms). + +Generated migration `0404_futuristic_earthquake.sql` creates the partial index +concurrently, adds idempotent expression statistics, and analyzes the table. +The existing top-level migration runner owns concurrent-index replay/repair. +Snapshot retention contains exactly 0396–0400. Fresh migration, journal replay, +direct SQL replay, ready/valid index flags, and statistics presence passed. + +Rechecking the actual migration's objects (after removing experimental objects) +used the driver's real global, company, and issue-scoped queries. Global discovery +found one candidate in 0.078 ms; history-only company/issue scopes returned none +in 0.024/0.019 ms without sequential scans. These are query-plan fixtures, not +latency or fleet-sleep claims. Large unselected ineligible histories and repeated +configuration signals still need separate qualification. + +Evidence: `/tmp/sleep48-query.json`, `/tmp/sleep48-query-plan.sql`, +`/tmp/sleep48-query-plan.log`, `/tmp/sleep48-query-statistics.log`, +`/tmp/sleep48-production-plan.log`, `/tmp/sleep48-migration-verify.log`. + +## Current verification and preserved failures + +The expanded final selection passed 45 cases in two files: 33 existing accepted- +response cases, ten new worker/owner cases, and two authority lifecycle cases. +The 1,551 unselected chat cases are not claimed passed. Server typecheck passed. +The worker cases establish startup recovery, selected-history quiet over former +poll periods, real chat-service authority restoration, outer commit/rollback, +first notification during drain, SQL-free drain waiting, resume, saved future +retry, pending/failed status-effect ordering, and an elapsed-deadline race. + +The elapsed-deadline test failed before its fix: the worker returned null even +though the locked owner had deferred the response. It now retains the due entry +and the next attempt publishes exactly once. Log: +`/tmp/sleep48-elapsed-deadline-before.log`. + +Earlier worker runs exposed the hint serialization defect (nine fixture failures), +then two test-observation mistakes. A timer created before fake-time installation +was not advanced by the fake clock; the test now installs time before registration. +A null next deadline can occur while a sweep is in flight; commit assertions now +wait for the persisted publication, zero tracked activity, and no next deadline +jointly. The original logs remain `/tmp/sleep48-worker-tests.log`, +`/tmp/sleep48-worker-tests-v2.log`, `/tmp/sleep48-drain-isolated.log`, and +`/tmp/sleep48-worker-tests-v3.log`. The corrected initial selection passed eleven +cases in `/tmp/sleep48-worker-tests-v4.log`; the final expanded result is +`/tmp/sleep48-acceptance-v2.log` with server types in +`/tmp/sleep48-acceptance-typecheck-v2.log`. + +No slice 48 PR, full workspace acceptance, remote CI/review, or provider sleep +qualification is claimed by these local results. The earlier stack gates and +remaining recurring-server duties are unchanged. + +## Producer integration and lifecycle evidence + +The existing pending-effect notification boundary is now named +`notifyNativeFinalizationWork` and wakes both consumers. Its existing coordinator +claim/retry, effect settlement, persisted-attention replay, retained-session release, +and recovery callers retain transaction ownership. No new wrappers are threaded +through those callers. Response discovery now accepts exact run and agent scopes. + +Both journal writers notify for accepted-result evidence and normalized completed +final assistant messages, after source/deduplication checks and before persistence. +Deltas, tool completions, and analysis messages do not activate response recovery. +The port's late accepted-result journal test and the internal final-message test +exercise these existing writers. Monitor continuation recovery is also exercised +through the worker, without another accepted result or provider execution. + +Agent availability was a real missing producer: the recovery owner rejects a +paused agent, but resuming that agent did not invalidate dormant responses. A +real resume regression failed before the fix. Lifecycle now exposes a narrow +`beforeStatusChange` integration callback, invoked inside its existing transaction +before the compatibility status write. The service supplies response policy only +for a transition from non-invokable to invokable, scoped to that agent. Lifecycle +still owns status transitions and the transaction; its existing resource-event +callback retains its semantics. With no response worker registered, this adds only +the existing notification protocol, not an independent worker or provider call. +The complete lifecycle suite exercises existing consumers of the new contract. + +Failed-chat retry admission also notifies response recovery when its source action +becomes processed. This can establish the ancestry proof after a very fast child +run has already finished. Five real admission scenarios verify that the response +consumer observes the committed action and the exact issue scope: commit, retry, +startup, stale claim, and rollback. An assignment-reactivation experiment instead +hit the existing prohibition on changing a channel-bound task's agent; that +unsupported fixture was removed without relaxing the guard. + +Validation across these runs covers 175 distinct cases: 59 response/authority and +retry-admission cases, 58 lifecycle cases, 21 monitor-continuation cases, and 37 +control-plane-port cases. Server typechecking passes. These are selected suites, +not a claim that the entire repository suite or remote PR checks passed. + +- `/tmp/sleep48-producers-rerun.log`: port 37/37, continuation 20/20, server types. +- `/tmp/sleep48-producer-acceptance.log`: lifecycle 58/58, expanded continuation + 21/21, response/authority 53 selected, server types. +- `/tmp/sleep48-acceptance-v5.log`: combined response/authority/admission 59 passed; + 1,546 unrelated cases not selected. Types in + `/tmp/sleep48-acceptance-typecheck-v5.log`. +- Real action-row contention now verifies retry retention through drain, zero + passive SQL while paused, and eventual single publication after resume. + +Failures remain recorded. The first external-port run had 36 passes and one +observer connection using the absent embedded database; the observer now uses the +validated external URL. The resume regression and invalid reassignment experiment +are in `/tmp/sleep48-eligibility-before.log`. A test-only observer initially omitted +its required retry interval and produced finite-deadline warnings; corrected in +`/tmp/sleep48-retry-producer-v2.log`. The combined v4 run exposed a global fault +injection targeting another retained fixture (13 transactions, not one). The v5 +run uses an actual lock on the tested response's source instead of assuming global +candidate order; all 59 cases pass with that stronger fault boundary. + +## Unpresented-backlog measurement + +A disposable clone of the earlier plan fixture has 10,008 structurally eligible, +unpresented candidates. The actual worker loader took 2,980 ms and 101 queries for +global reconstruction; 10,001 candidates in one company took 2,828 ms; the exact +run took 1.73 ms and one query. Agent reconstruction for that company's 10,001 +candidates took 2,826 ms. Serialized entries occupy about 2.05 MB. The drain gate +stopped discovery after two pages. Evidence: +`/tmp/sleep48-unpresented-backlog.mts` and `.log`. + +This measures discovery, not authorization or publication throughput: the cloned +rows do not establish valid independent publication authority. Each SQL page is +bounded to 100 rows, but total reconstruction memory/time still scales with the +unpresented backlog. Ordinary finalization and journal notifications are run-scoped; +agent resume is agent-scoped; actual authority/configuration changes may reconstruct +a larger scope. Health-only configuration signals do not invalidate this worker. +Do not describe total reconstruction as constant-cost or claim fleet qualification. + +Startup/shutdown integration, removal of this duty's periodic call, final producer +coverage review, full PR verification, and publication remain outstanding. The old +backstop is still present. Master was refreshed to `6bdcb70244`; the full stack +still needs reconciliation and its earlier failing/review gates remain open. + + +## Startup integration, 2026-10-26 + +The heartbeat facade now registers this worker with the existing task-drain gate. +Enabled maintenance awaits reconstruction during startup. Shutdown and listener +close release its live-authority subscription; coordinator shutdown retains +ownership of cancellation and in-flight work. + +The periodic orphan-recovery caller now passes `skipNativeResponsePresentation`. +The reconciler skips its direct repair call and passes the same option into +canonical finalizer replay. The finalizer skips only its already-committed response +repair in that mode. Fresh finalization still delivers immediately. Explicit +recovery and startup recovery keep their default behavior. Governed-review +presentation and other native finalization duties remain separate. + +All 66 startup tests pass, including disabled maintenance and close subscription +registration. Two real PostgreSQL regressions verify that both periodic entry +points leave the response unmaterialized, and the worker then creates exactly one +publication from the accepted result. Log: `/tmp/sleep48-wiring-tests.log`. +Workspace typecheck passes (`/tmp/sleep48-full-typecheck.log`). The full local suite +is still running and has reported embedded PostgreSQL initialization failures; +its final result and build status remain pending. No full-suite or remote check +acceptance is claimed at this checkpoint. + + +## Review repair and current verification, 2026-10-26 + +Published draft #17696 at `79bf36cbf8`. Greptile found a real precision gap: +`writeRunStatus` still sent a company-only milestone hint. Every unrelated run +status could reconstruct dormant response history. Three real-PostgreSQL cases +(native binding, legacy context binding, and taskless run) failed before repair +(`/tmp/sleep48-scope-before.log`). + +The milestone contract now accepts an optional run identity. Status producers +supply the task when bound and the exact run otherwise. The response fanout +prefers task scope over run scope, because lineage changes can authorize other +runs on that task. Known run claim/release and completion producers also preserve +an exact-run fallback. Existing company configuration invalidations remain broad. +No new durable rows or caller recovery wrappers are introduced. + +All 34 heartbeat lifecycle tests pass after repair. Two real worker regressions +retain a dormant eligible response, send unrelated task/run wakes, and observe +one actual candidate SQL query at the postgres.js debug boundary, without +rechecking the dormant row. The outer-commit/rollback cases additionally send a +different run identity within the same task and retain task-level recovery. +An initial test mistakenly counted Drizzle subquery builder construction as a +second SQL statement; driver-level observation corrected the measurement. + +The response/authority/admission selection now passes all 63 cases +(`/tmp/sleep48-scope-acceptance.log`). Server typechecking and the workspace build +pass (`/tmp/sleep48-scope-typecheck-v3.log`, `/tmp/sleep48-scope-build.log`). +The previous full workspace typecheck passed at `79bf36cbf8`. The full local +Vitest run was explicitly stopped after repeated embedded PostgreSQL bootstrap +failures; `/tmp/sleep48-full-tests.log` preserves those failures. It is not a +full-suite pass. Focused database acceptance uses the validated external +PostgreSQL path and isolated disposable test databases. + +Remote CI at `79bf36cbf8` also exposed a late-journal test that assumed worker +startup drains all candidates in one batch. It now waits for actual settlement, +as the shared worker deliberately processes fair batches. The workspaces-b job +hit a five-second sandbox staging test timeout; E2E shard five received a runner +shutdown signal. These failures remain recorded in `/tmp/sleep48-ci-chat3.log`, +`/tmp/sleep48-ci-workspaces-b.log`, and `/tmp/sleep48-ci-e2e5.log`. Fresh-head CI and +review are required after the repair; no failing gate is waived. + + +## Full-history test repair, 2026-10-26 + +At `c4beddad1e`, fresh Greptile review is 5/5 and the scope thread is resolved. +Remote CI passed 51 contexts but failed the chat shard and its aggregate gate: +a second startup assertion assumed `worker.ready` drained every fair batch. +The full local chat file finished with 1,595 passing and seven failing tests +(`/tmp/sleep48-chat-full.log`); all seven were in the newly added worker tests. + +The larger run exposed two related observation errors: startup can have another +batch scheduled, and `nextWakeAt() === null` can be transient while a batch runs. +The latter caused query instrumentation to count unfinished startup checks and +an outcome assertion to run before its response had been materialized. An early +future-deadline test failure then left its fixture's real deadline for later tests. + +Worker startup/settlement assertions now require both zero active work and the +expected next deadline. This changes test observation, not scheduler behavior or +batch limits. All 18 affected cases pass against the retained history database +from the full 1,602-case run (`/tmp/sleep48-settlement-history.log`), including +single-query unrelated wakes, future deadlines, and coordinator retry release. +The full-run failure remains recorded; fresh remote CI is still required. + + +## Current migration 0412 integration and complete acceptance + +The original response-presentation patch is integrated with the validated current +native-effect parent. The complete owned runtime/schema changes are preserved, +including fourteen files with combined upstream edits and 10,401 unchanged +unrelated upstream source files. The control-plane fixture conflict retains its +dedicated external database path and the current recovery-helper fallback; all +upstream conformance cases and owned assertions remain. Both audit histories +are retained. + +Only this slice's unlanded migration artifacts were deferred before integration. +`pnpm db:generate` generated `0412_tearful_squadron_supreme.sql`. The generated +index definition was checked against the original index before restoring all +three original statements byte for byte: concurrent idempotent index creation, +selection statistics, and ANALYZE. No original tuning statement is dropped. All +prior executable SQL, journal entries and retained snapshot bytes are unchanged. +Five snapshots 0408 through 0412 remain; the concurrent allowlist includes 0412. + +Source `9feec170ef8174011fc3c5d3d6d76f0a0f39fea0` passes **4171 distinct cases +in 29 complete files**, without skips: 2313 owner/producer/consumer cases in +221.96 seconds, 59 safe-replacement cases in 29.10 seconds, and all 1799 +assembled chat integration cases in 769.87 seconds. This includes the complete +GitHub workflow, accepted-response worker lifecycle, authenticated late evidence, +outer commit/rollback, contention, authority registration, agent resume, startup, +retry, drain/resume, and unrelated retained-history isolation. Per-file counts: + +| Complete file | Passing cases | +| --- | ---: | +| src/__tests__/chat-channels.integration.test.ts | 1799 | +| src/__tests__/committed-chat-response-authority.test.ts | 7 | +| src/__tests__/issue-recovery-actions.test.ts | 89 | +| src/__tests__/native-finalization-recovery.test.ts | 32 | +| src/__tests__/native-interaction-bridge.test.ts | 7 | +| src/__tests__/native-status-arbiter-corpus.test.ts | 94 | +| src/__tests__/native-workspace-sync.test.ts | 10 | +| src/__tests__/recovery-stale-issue-lock-sweep.test.ts | 40 | +| src/__tests__/server-startup-feedback-export.test.ts | 69 | +| src/__tests__/slack-conversation-lifecycle.test.ts | 41 | +| src/modules/run-dispatch/adapters/postgres.test.ts | 69 | +| src/modules/wake-queue/adapters/postgres.test.ts | 93 | +| src/modules/wake-queue/adapters/queued-comment-postgres.test.ts | 5 | +| src/services/chat-monitor-continuation.test.ts | 21 | +| src/services/execution-control-reconciliation.test.ts | 10 | +| src/services/explicit-native-continuation.test.ts | 335 | +| src/services/heartbeat/queue.test.ts | 32 | +| src/services/heartbeat/recovery.test.ts | 45 | +| src/services/heartbeat/run-control.test.ts | 51 | +| src/services/heartbeat/run-lifecycle.test.ts | 34 | +| src/services/heartbeat/upgrade-readiness.test.ts | 7 | +| src/services/legacy-execution-recovery.test.ts | 15 | +| src/services/native-runtime/native-restart-recovery.test.ts | 25 | +| src/services/native-runtime/native-run-finalizer-telemetry.test.ts | 29 | +| src/services/native-runtime/native-safe-replacement.test.ts | 59 | +| src/services/native-runtime/native-session-executor.test.ts | 968 | +| src/services/native-runtime/native-workspace-export-retry.test.ts | 80 | +| src/services/native-runtime/native-workspace-reconstruction.test.ts | 50 | +| src/services/native-runtime/paperclip-control-plane-port.test.ts | 55 | + +The complete 94-case status-authority corpus also passes again on the same +database in 16.68 seconds. It is not double-counted. Three retained PSC/OUT and +sentinel company rows remain unchanged, and each invocation uses distinct +UUID-derived fixture prefixes. Twenty-four separate fresh PostgreSQL 17 +databases migrate through 0412 before setup. Four embedded-only fixture imports +temporarily use their own external database on this IPC-exhausted host, then +restore byte for byte. All assertions and timing limits remain unchanged. This +is qualified external-database acceptance, not an unmodified embedded CI run. + +Actual database build, server types/build, module boundaries and exact-head +migration order pass. A populated 0411-to0412 upgrade preserves all 410 prior +migration receipts and all 38 companies, appends exactly one receipt, and adds +none on replay. The partial index is valid and ready on both fresh and upgraded +databases; the existing selection statistic is present. Earlier failed runs +above retain their historical scopes and explanations. The current complete +chat run passes, rather than substituting a focused retry for the full suite. + +Normal merge `1c019b48c98cd8273aa3b7e3cf95433a369b9531` adopts published native-effect +parent `5bb5ddacd3d1e6817de4825b283c6e92f85bd124` and actual unsafe-export ancestry. +All complete server, package, UI and test trees remain identical to the accepted +source. Publication remains held until #17690 lands and its actual squash +ancestry is reconciled. Fresh published-head CI and exact-head Greptile 5/5 +remain required. These checks do not establish deployed suspension, external +Cloud alarm delivery or fleet savings. + + +## Actual native-effect prerequisite ancestry + +Native-effect prerequisite #17690 merged as +`a8a038e659c50c1c5e3ce532d13072a29a5cf951`, reviewed at +`5bb5ddacd3d1e6817de4825b283c6e92f85bd124`. Its entire squash tree equals +the reviewed parent already adopted by this child. The normal ancestry merge +automatically restored the removed pending-effect notification module. The +whole-tree identity check rejected that composition; its deletion was restored +before publication. All accepted source, including the complete 0412 database +tree and all 29 test files, remains byte-identical. The following edit records +the ancestry proof. The 4171-case acceptance keeps its original source scope; +fresh published-head CI and exact-head Greptile 5/5 are required. + + +## 2026-10-29 review repair: routine hints cannot invalidate a company + +Superagent review of `a5648c92f4601fb72cec26b9ff2e99e40d502042` identified a +real P2: coalesced inbound messages used a company-only milestone hint, and +the new response fanout reconstructed every unpresented response in that +company. Pending-effect settlement also used a company-only finalization hint. +The current-head scan required action, so the branch was not merged. Greptile +rated that head 5/5 but conditioned readiness on the later #17702 scoped +pending-effect fix. That fix is included here; this slice no longer requires +its later dependent PR to be independently correct. + +The repair keeps routine milestones and finalization hints from entering the +response queue without an issue or run. Real coalescing names its already-known +active run. Pending-effect settlement names the affected issue, covering older +decisions on that issue. Retry promotion/cancellation retain their known run +when older context omits an issue. Existing pending-effect and chat consumers +keep their previous serialized hints; only response reconstruction is narrowed. + +Genuine company-wide authority changes remain supported through one explicit +`notifyChatCompanyAuthorityChange` boundary. Sixteen existing mutation hooks in +nine owning files use it: membership and board-claim restoration, Cloud actor +membership repair, GitHub/Slack identity and sponsor changes, endpoint policy, +resource enablement and connection configuration. They keep their original +change predicates and transaction owners. Two issue-scoped resource/publication +hooks remain issue-scoped. No transaction wrappers, queue tables, polling or +new persisted bookkeeping are introduced. Dynamic milestone producers carry +issue-bound interaction rows or typed recovery inputs. + +The ordinary coalescing regression first fails on the actual PostgreSQL writer: +its settled hint has only the company instead of the active run. Two receiver +regressions each issue three routine company hints against dormant response +history and observe **12 response-history SELECTs**. Both must observe zero. +A positive test restores suspended membership through the actual access owner +and delivers the waiting response once; it passes before and after the repair. +The unchanged four focused assertions pass after the fix. Assertion time limits +are unchanged. These tests measure real driver SQL and persisted producer work. + +Repair source `2fc8b0614aaf9ac7337045de4ced7325f95edc45` passes **1821 distinct +server cases**: 1496 in 13 complete suites in 110.63 seconds, and 325 selected +chat cases in 278.99 seconds. The chat selection includes the complete GitHub +workflow, accepted-response lifecycle, the new regressions, and real post-close +authority restoration. Another 1477 chat cases are outside the name pattern; +the complete 1802-case file is not claimed for this repair run. Counts: + +| File | Passing cases | +| --- | ---: | +| src/__tests__/access-service.test.ts | 18 | +| src/__tests__/auth-session-route.test.ts | 12 | +| src/__tests__/board-claim.test.ts | 3 | +| src/__tests__/chat-channels.integration.test.ts (focused) | 325 | +| src/__tests__/committed-chat-response-authority.test.ts | 7 | +| src/__tests__/native-finalization-recovery.test.ts | 32 | +| src/__tests__/native-status-arbiter-corpus.test.ts | 94 | +| src/__tests__/server-startup-feedback-export.test.ts | 69 | +| src/modules/run-dispatch/adapters/postgres.test.ts | 69 | +| src/modules/wake-queue/adapters/postgres.test.ts | 94 | +| src/modules/wake-queue/application/use-cases.test.ts | 67 | +| src/services/heartbeat/run-lifecycle.test.ts | 34 | +| src/services/native-runtime/native-run-finalizer-telemetry.test.ts | 29 | +| src/services/native-runtime/native-session-executor.test.ts | 968 | + +Eleven dedicated fresh PostgreSQL 17 databases migrate through unchanged 0412. +The wake-queue fixture's embedded-only import temporarily uses its own external +database and restores byte for byte. All 14 selected source files match their +pre-run hashes. Actual database build, server types/build, module boundaries +and full-SHA migration order pass. The earlier 4171-case full baseline keeps +its original pre-repair source scope; it is not represented as this repair run. + +Master independently advanced to `522c653d562a41ecad64561d35457cb99ecf6cd6` +with #18126's Pi archive-download retry fix. Normal merge +`2a535bc61259ef99387369e8b32498f774e5ba2c` preserves both complete binary patches. +All accepted server, database, UI, test and CLI trees remain identical; only +the three independently changed runner installer/manifest files are added. +All 13 installer tests pass in 200.18 milliseconds, including stalled-header +and stalled-body cancellation. Combined server types/build, boundaries and +migration order pass. No unrelated upstream source is discarded. + +Explicit global authority invalidation and startup still reconstruct their +company's owed candidates with existing 100-row discovery pages and fair +execution batches. Discovery retains linear time and memory; it is not a +constant-cost global operation. The prior 10,001-candidate profile remains +relevant to those genuine global obligations. This repair removes ordinary +inbound traffic's ability to request that global reconstruction and prevents +unscoped routine traffic from rebuilding dormant response history. It does not waive an actual review +failure or establish deployed sleep, Cloud wake delivery or fleet savings. +Fresh published-head CI, Greptile 5/5 and a clean security scan remain required. + + +A second publication guard detected master #18128 at +`e8b78cce5bb557b086bee9ca930eeaab1d09b244`. Normal merge +`06d55ca9f872f7e7794680335811b496942be034` retains its three test-only file changes. +Both complete owned and upstream binary patches are preserved. An isolated Git +index projects only the owned audit onto the original repair source to prove +this equality; it does not rewrite branch history or include master changes +as owned edits. All 14 repair test hashes and the entire package tree remain +unchanged. The changed HTTPS backfill case passes on a fresh 0412 PostgreSQL +fixture in 7.62 seconds; 178 other workspace cases are outside that focused +selection. Its embedded-only fixture import restores byte for byte. Combined +server types/build, boundaries and migration order pass again. The independent +browser fixture changes are preserved exactly; no local browser run is claimed. + +### Recorded presentation and structural eligibility repair + +The exact published head `ae75e21bd5aa5841b66f0d3f421181cdcfa9a3be` +received Greptile 5/5, a clean security scan, and passing CI. Its completed +Ponytail 4/5 verdict nevertheless required two correctness fixes. That actual +failing verdict was honored; no review timeout or numeric score waived it. +The review also found a missing fresh-commit skip guard, which is repaired here. + +A recorded non-create presentation decision (`none`, `reuse`, or another explicit +non-create value) now excludes both discovery and locked recovery. A string +selected comment ID still fences recovery, including deleted comments. The +`skipIssueComment` policy applies to all recovery branches. Missing legacy +presentation metadata and explicit `create` with a null comment ID remain owed +work and can recover once. No per-worker table or transaction wrapper is added. + +Discovery joins the owning company/issue and applies the cheap durable guards +before taking recovery locks: done results belong to succeeded non-conversation +runs; response waits have empty attention requests and usable summary/key; +monitor and same-agent continuations require succeeded runs and their accepted +reason codes. Digest integrity, current publication authority, provider-owned +interactions, final-message selection and other owner checks remain under the +existing transaction. These predicates narrow structural candidates; they do not +claim to establish authorization or replace late-evidence and authority-change +notifications. Fresh finalization now honors `skipNativeResponsePresentation` +as the already-committed path does. + +The final corrected regression source is unchanged between the red and green +runs. The published baseline fails eight behavioral assertions and passes the +three positive genuine-answer cases. Repair source +`892f36e5f66ba1af123034936a7160c64c3ab883` passes all eleven. Recorded none/reuse +produce no new comments or publications and keep their decision. Five families +of structurally ineligible retained rows produce zero issue/run/finalization +`FOR UPDATE` queries across three issue-scoped notifications each. Those selector +fixtures deliberately vary durable payload fields; integrity and authorization +remain separate recovery requirements. Genuine missing, legacy and create-null +presentations still publish once. A genuinely uncommitted accepted run proves +that periodic finalization leaves presentation to the worker. + +Earlier material failures remain in the verification record: an incomplete +conversation identity fixture, a duplicate variable declaration, a remaining +stale variable reference and its downstream retry contamination, a test that +accidentally used the matching continuation reason, and a missing fixture import. +The fixture was corrected to exercise the actual incompatible reason; assertions +and time limits were not weakened. The final red comparison restores the exact +published production files temporarily and verifies byte-exact restoration of +the repair afterward. These were explained local setup/implementation failures, +not inferred infrastructure faults. + +Repair source passes **1489 distinct cases**: 1139 across six complete consumer +suites in 81.02 seconds, plus 350 selected chat integration cases in 297.68s. +The remaining 1463 chat cases are outside the name pattern; the full 1813-case +file is not claimed for this repair. The eleven new regressions are included +in the selected count, not added again. The six complete files cover 968 native +session cases, 79 response-summary cases, 21 monitor-continuation cases, 29 native +finalizer cases, 32 finalization-recovery cases and 10 workspace-sync cases. + +No executable migration, schema, prior history, index/statistics statement or +snapshot changes in this repair. Five fresh PostgreSQL 17 fixtures migrate +through 0412; no embedded fixture adaptation is needed for these suites. All +selected test hashes match the captured source. Database build, server +types/build, module boundaries and full-SHA migration ordering pass. Actual host +sleep, external wake and fleet cost savings remain unqualified. A newly published +head still requires fresh exact-head Greptile 5/5, passing CI and no actionable +review findings before merge. diff --git a/packages/db/src/client.ts b/packages/db/src/client.ts index 6e562381f7c..55d1a1ca9a1 100644 --- a/packages/db/src/client.ts +++ b/packages/db/src/client.ts @@ -561,6 +561,7 @@ const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([ "0409_uneven_zuras.sql", "0410_freezing_kinsey_walden.sql", "0411_real_scrambler.sql", + "0412_tearful_squadron_supreme.sql", "0384_funny_adam_warlock.sql", diff --git a/packages/db/src/migrations/0412_tearful_squadron_supreme.sql b/packages/db/src/migrations/0412_tearful_squadron_supreme.sql new file mode 100644 index 00000000000..ca0c9e198e3 --- /dev/null +++ b/packages/db/src/migrations/0412_tearful_squadron_supreme.sql @@ -0,0 +1,3 @@ +CREATE INDEX CONCURRENTLY IF NOT EXISTS "heartbeat_runs_native_unpresented_response_idx" ON "heartbeat_runs" USING btree ("company_id","native_issue_id","id") WHERE "heartbeat_runs"."runtime_mode" = 'native' AND "heartbeat_runs"."status" in ('succeeded', 'failed', 'timed_out') AND "heartbeat_runs"."finished_at" is not null AND jsonb_typeof("heartbeat_runs"."result_json" #> '{presentationDecision,commentId}') is distinct from 'string';--> statement-breakpoint +CREATE STATISTICS IF NOT EXISTS "heartbeat_runs_native_response_selection_stats" ON (jsonb_typeof("result_json" #> '{presentationDecision,commentId}')) FROM "heartbeat_runs";--> statement-breakpoint +ANALYZE "heartbeat_runs"; diff --git a/packages/db/src/migrations/meta/0407_snapshot.json b/packages/db/src/migrations/meta/0412_snapshot.json similarity index 99% rename from packages/db/src/migrations/meta/0407_snapshot.json rename to packages/db/src/migrations/meta/0412_snapshot.json index 54cb791facf..698666748b6 100644 --- a/packages/db/src/migrations/meta/0407_snapshot.json +++ b/packages/db/src/migrations/meta/0412_snapshot.json @@ -1,6 +1,6 @@ { - "id": "856a147c-73dc-4908-a8d9-4f19cab7dbf6", - "prevId": "c34fcd91-0d57-4241-b8b4-399bc9f6dcb0", + "id": "3b8bb508-028d-4154-abdd-af870f1f3ca6", + "prevId": "ff56ee93-8cfc-465a-b26a-1c12d10773ff", "version": "7", "dialect": "postgresql", "tables": { @@ -2665,6 +2665,28 @@ } }, "indexes": { + "agent_instruction_copies_collection_idx": { + "name": "agent_instruction_copies_collection_idx", + "columns": [ + { + "expression": "run_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "NOT (coalesce(\"agent_instruction_working_copies\".\"receipt\", '{}'::jsonb) ? 'retainedByRunId') AND (\n (\"agent_instruction_working_copies\".\"attempts\" < 3 AND (\"agent_instruction_working_copies\".\"state\" IN ('prepared', 'pending_collection', 'unchanged_turn', 'warm_saved')\n OR (\"agent_instruction_working_copies\".\"state\" = 'preparing' AND \"agent_instruction_working_copies\".\"receipt\"->>'schema' = 'paperclip.agent-files.v1')))\n OR (\"agent_instruction_working_copies\".\"state\" = 'unavailable' AND \"agent_instruction_working_copies\".\"process_stopped_at\" IS NULL\n AND \"agent_instruction_working_copies\".\"location\" LIKE 'remote:%' AND \"agent_instruction_working_copies\".\"receipt\"->>'schema' = 'paperclip.agent-files.v1'))", + "concurrently": false, + "method": "btree", + "with": {} + }, "agent_instruction_copies_cleanup_idx": { "name": "agent_instruction_copies_cleanup_idx", "columns": [ @@ -33749,6 +33771,56 @@ } }, "indexes": { + "heartbeat_runs_native_unpresented_response_idx": { + "name": "heartbeat_runs_native_unpresented_response_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "native_issue_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"heartbeat_runs\".\"runtime_mode\" = 'native' AND \"heartbeat_runs\".\"status\" in ('succeeded', 'failed', 'timed_out') AND \"heartbeat_runs\".\"finished_at\" is not null AND jsonb_typeof(\"heartbeat_runs\".\"result_json\" #> '{presentationDecision,commentId}') is distinct from 'string'", + "concurrently": false, + "method": "btree", + "with": {} + }, + "heartbeat_runs_cancelled_workspace_preservation_idx": { + "name": "heartbeat_runs_cancelled_workspace_preservation_idx", + "columns": [ + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"heartbeat_runs\".\"runtime_mode\" = 'native' AND \"heartbeat_runs\".\"status\" = 'cancelled'\n AND \"heartbeat_runs\".\"runner_profile_json\"->'nativeWorkspaceSync'->>'state' = 'prepared'\n AND \"heartbeat_runs\".\"result_json\"->'nativeCancellation'->>'dispatchState' = 'acknowledged'\n AND (\"heartbeat_runs\".\"result_json\"->'workspaceRestoreRecovery'->>'schema' = 'paperclip.workspace-restore-recovery.v1'\n AND \"heartbeat_runs\".\"result_json\"->>'workspaceRestoreFailure' IN ('restore_permission_denied', 'restore_lock_timeout', 'restore_unsafe_archive', 'restore_failed')) IS NOT TRUE", + "concurrently": false, + "method": "btree", + "with": {} + }, "heartbeat_runs_retained_recovery_adoption_idx": { "name": "heartbeat_runs_retained_recovery_adoption_idx", "columns": [ @@ -43064,6 +43136,62 @@ } }, "indexes": { + "native_run_finalizations_pending_effect_owner_idx": { + "name": "native_run_finalizations_pending_effect_owner_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "issue_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "decision_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"native_run_finalizations\".\"decision_id\" is not null AND \"native_run_finalizations\".\"result_id\" is not null AND \"native_run_finalizations\".\"phase\" <> 'terminal_failure'", + "concurrently": false, + "method": "btree", + "with": {} + }, + "native_run_finalizations_unsafe_export_recovery_idx": { + "name": "native_run_finalizations_unsafe_export_recovery_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "issue_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "run_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"native_run_finalizations\".\"phase\" = 'terminal_failure' AND \"native_run_finalizations\".\"failure_code\" = 'native_workspace_sync_out_unsafe_archive'", + "concurrently": false, + "method": "btree", + "with": {} + }, "native_run_finalizations_control_deadline_idx": { "name": "native_run_finalizations_control_deadline_idx", "columns": [ @@ -52368,6 +52496,34 @@ } }, "indexes": { + "status_decision_effects_pending_delivery_idx": { + "name": "status_decision_effects_pending_delivery_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "issue_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "decision_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"status_decision_effects\".\"delivery_state\" in ('pending', 'failed')", + "concurrently": false, + "method": "btree", + "with": {} + }, "status_decision_effects_decision_ordinal_uq": { "name": "status_decision_effects_decision_ordinal_uq", "columns": [ diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index f02c1c52a93..76a38c88a5b 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -2871,6 +2871,13 @@ "when": 1793246172969, "tag": "0411_real_scrambler", "breakpoints": true + }, + { + "idx": 412, + "version": "7", + "when": 1793249495056, + "tag": "0412_tearful_squadron_supreme", + "breakpoints": true } ] } \ No newline at end of file diff --git a/packages/db/src/schema/heartbeat_runs.ts b/packages/db/src/schema/heartbeat_runs.ts index 99f06a955dc..9e215f825f0 100644 --- a/packages/db/src/schema/heartbeat_runs.ts +++ b/packages/db/src/schema/heartbeat_runs.ts @@ -111,6 +111,9 @@ export const heartbeatRuns = pgTable( updatedAt: timestamp("updated_at", { withTimezone: true }).notNull().defaultNow(), }, (table) => ({ + nativeUnpresentedResponseIdx: index("heartbeat_runs_native_unpresented_response_idx") + .on(table.companyId, table.nativeIssueId, table.id) + .where(sql`${table.runtimeMode} = 'native' AND ${table.status} in ('succeeded', 'failed', 'timed_out') AND ${table.finishedAt} is not null AND jsonb_typeof(${table.resultJson} #> '{presentationDecision,commentId}') is distinct from 'string'`), cancelledWorkspacePreservationIdx: index("heartbeat_runs_cancelled_workspace_preservation_idx").on(table.id, table.companyId) .where(sql`${table.runtimeMode} = 'native' AND ${table.status} = 'cancelled' AND ${table.runnerProfileJson}->'nativeWorkspaceSync'->>'state' = 'prepared' diff --git a/server/src/__tests__/chat-channels.integration.test.ts b/server/src/__tests__/chat-channels.integration.test.ts index 3c2bc210f31..c24d5eb5051 100644 --- a/server/src/__tests__/chat-channels.integration.test.ts +++ b/server/src/__tests__/chat-channels.integration.test.ts @@ -1,6 +1,8 @@ +import { notifyNativeFinalizationWork } from "../services/native-finalization-notifications.js"; +import { nativeResponsePresentationWork } from "../services/native-runtime/native-response-presentation-work.js"; +import { notifyChatRunMilestones } from "../services/chat-run-milestone-notifications.js"; import { registerGitHubReviewProjectionWork } from "../services/chat-channels/github/review-projection-work.js"; import { retainGitHubReviewReceipt } from "../services/chat-channels/github/task-preparation.js"; -import { notifyChatRunMilestones } from "../services/chat-run-milestone-notifications.js"; import { settleUnrecoverableExecutions } from "../services/execution-recovery-resolution.js"; import { recordRunPresentationDecision } from "../services/heartbeat/run-completion.js"; import { environmentService } from "../services/environments.js"; @@ -16,13 +18,13 @@ import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db"; import { claimTaskRetryInput, releaseUnadmittedTaskRetryInput } from "../services/chat-channels/task-retry-input.js"; import { restoreNativeChatReviewPresentationInTransaction } from "../services/native-runtime/native-chat-review-presentation.js"; import { notifyChatEndpointConfiguration, notifyChatCompanyConfiguration } from "../services/chat-channels/configuration-work-notifications.js"; +import { configureAgentLifecycle, createAgentLifecycle, processAgentLifecycle } from "../services/agent-lifecycle.js"; import { deleteCompany } from "../services/company-deletion.js"; import { adapterExecutionControls, createAdapterExecutionControl } from "../services/adapter-execution-control.js"; import { chatRunSourceMessageId } from "../services/chat-channels/run-origin.js"; import * as durableChatWakeup from "../services/durable-chat-wakeup.js"; import { conflict } from "../errors.js"; import * as githubTaskPreparation from "../services/chat-channels/github/task-preparation.js"; -import { configureAgentLifecycle, createAgentLifecycle } from "../services/agent-lifecycle.js"; import { budgetPolicies } from "@paperclipai/db"; import { budgetService } from "../services/budgets.js"; import { idleWorkSnapshot } from "../services/task-admission.js"; @@ -150,6 +152,8 @@ import { issues, nativeRunResults, nativeRunFinalizations, + statusDecisions, + statusDecisionEffects, environmentLeases, environments, principalPermissionGrants, @@ -267,6 +271,7 @@ import type { import { finalizeNativeRun, repairCommittedNativeChatResponse, + recoverCommittedNativeChatResponse, } from "../services/native-runtime/native-run-finalizer.js"; import { NativeRunCoordinatorStore } from "../services/native-runtime/native-run-coordinator-store.js"; import { reconcileNativeFinalizations } from "../services/native-runtime/native-finalization-reconciler.js"; @@ -66562,6 +66567,19 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { const input = { companyId: context.fixture.companyId, issueId: context.issue.id, agentId: context.fixture.assignedAgentId, failedRunId: context.runId, initiatedByUserId: "owner-user" }; const stage = () => db.transaction(tx => context.service.prepareFailedChatRunRetry(tx, input)); + const observedResponseAdmissions: string[] = []; + const responseHints: string[] = []; + await coordinator.register(DELIVERY_QUEUES.nativeResponsePresentation, { + retryMs: 30_000, + onSettled: hints => { responseHints.push(...hints ?? []); }, + run: async () => { + const admitted = await db.select({ id: chatActions.id }).from(chatActions).where(and( + eq(chatActions.companyId, context.fixture.companyId), eq(chatActions.kind, "failed_run_retry"), eq(chatActions.status, "processed"), + )); + observedResponseAdmissions.push(...admitted.map(row => row.id)); + }, + hasPending: async () => false, + }).ready; context.wakeup.mockClear(); try { let actionId: string; @@ -66607,6 +66625,8 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { expect(action?.result.attemptCount).toBe(mode === "retry" ? 2 : 1); expect(coordinator.nextWakeAt()).toBeNull(); }, { timeout: 10_000 }); + await vi.waitFor(() => expect(observedResponseAdmissions).toContain(actionId), { timeout: 5000 }); + expect(responseHints.map(hint => JSON.parse(hint))).toContainEqual({ companyId: context.fixture.companyId, issueId: context.issue.id }); expect(context.wakeup).toHaveBeenCalledTimes(mode === "retry" ? 2 : 1); const receipts = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, actionId)); expect(receipts).toHaveLength(1); @@ -74255,6 +74275,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { context: Awaited>, ) => Promise, summary?: (issueId: string) => string, + finalize = true, ) { const context = await failedChatRetryFixture( provider, @@ -74409,7 +74430,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { .from(nativeRunResults) .where(eq(nativeRunResults.runId, context.runId)); const accepted = { resultId: acceptedRow.id }; - await finalizeNativeRun({ + if (finalize) await finalizeNativeRun({ db, runId: context.runId, workspaceFinalizeStatus: "succeeded", @@ -74418,7 +74439,8 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { .select() .from(nativeRunFinalizations) .where(eq(nativeRunFinalizations.runId, context.runId)); - expect(coordinator.phase).toBe("committed"); + if (finalize) expect(coordinator.phase).toBe("committed"); + else expect(coordinator.phase).not.toBe("committed"); await db .update(heartbeatRuns) .set({ @@ -74616,6 +74638,11 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { }) : context.repair(), ).resolves.toBe(false); + await expect(recoverCommittedNativeChatResponse(db, { + companyId: mutation === "cross_company" ? randomUUID() : context.fixture.companyId, + issueId: context.issue.id, + runId: mutation === "cross_run" ? randomUUID() : context.runId, + })).resolves.toBe(false); await expect( db .select() @@ -75257,6 +75284,519 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { }, ); + it.each(["none", "reuse"] as const)("preserves a recorded %s presentation instead of recovering an unsolicited answer", async commentAction => { + const context = await committedChatResponseRecoveryFixture("telegram"); + const presentationDecision = { commentAction, commentId: null, reasonCodes: ["interaction_owns_response"] }; + try { + await db.update(heartbeatRuns).set({ resultJson: { presentationDecision } }).where(eq(heartbeatRuns.id, context.runId)); + await expect(recoverCommittedNativeChatResponse(db, { companyId: context.fixture.companyId, + issueId: context.issue.id, runId: context.runId })).resolves.toBe(false); + expect(await nativeResponsePresentationWork(db).load({ companyId: context.fixture.companyId, runId: context.runId })).toEqual([]); + expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, context.runId))).toEqual([]); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toEqual([]); + expect((await readHeartbeatRunsById(context.runId))[0].resultJson).toEqual({ presentationDecision }); + } finally { await context.service.shutdown(); } + }); + + it.each(["create", "legacy", "missing"] as const)("recovers a genuine unanswered %s presentation", async action => { + const context = await committedChatResponseRecoveryFixture("telegram"); + try { + await db.update(heartbeatRuns).set({ resultJson: action === "missing" ? {} : { + presentationDecision: { commentId: null, ...(action === "create" ? { commentAction: "create" } : {}) }, + } }).where(eq(heartbeatRuns.id, context.runId)); + const work = nativeResponsePresentationWork(db); + const entries = await work.load({ companyId: context.fixture.companyId, runId: context.runId }); + expect(entries).toHaveLength(1); + await expect(work.check(entries[0])).resolves.toBeNull(); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1); + await expect(context.repair()).resolves.toBe(false); + } finally { await context.service.shutdown(); } + }); + + it.each(["conversation_done", "attention_request", "same_agent_reason", "monitor_reason", "skip_comment"] as const)( + "does not lock durably ineligible %s history on repeated issue milestones", async kind => { + const context = await committedChatResponseRecoveryFixture("telegram"); + await context.service.shutdown(); + await db.update(heartbeatRuns).set({ status: "succeeded", + ...(kind === "skip_comment" ? { contextSnapshot: { skipIssueComment: true } } : {}) }).where(eq(heartbeatRuns.id, context.runId)); + const [accepted] = await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, context.accepted.resultId)); + const envelope = accepted.resultJson as { result: PrpStructuredRunResult; terminal: PrpTerminalState }; + if (kind === "conversation_done") { + envelope.result = { ...envelope.result, reportedWorkDisposition: "done" }; + envelope.terminal = { ...envelope.terminal, reportedWorkDisposition: "done" }; + await db.update(issues).set({ conversationAgentId: context.fixture.assignedAgentId, conversationUserId: "owner-user", conversationState: "active", assigneeUserId: null }) + .where(eq(issues.id, context.issue.id)); + } else if (kind === "attention_request") { + envelope.result = { ...envelope.result, attentionRequests: [{ kind: "review", summary: "Needs a decision", ownerClass: "human" }] }; + } else if (kind === "same_agent_reason" || kind === "monitor_reason") { + envelope.result = { ...envelope.result, continuation: { ...envelope.result.continuation, + kind: kind === "same_agent_reason" ? "same_agent" : "monitor" } as PrpStructuredRunResult["continuation"] }; + await db.update(statusDecisions).set({ reasonCode: "external_chat_response_waiting" }) + .where(eq(statusDecisions.id, context.coordinator.decisionId!)); + } + await db.update(nativeRunResults).set({ resultJson: envelope }).where(eq(nativeRunResults.id, accepted.id)); + const errors: unknown[] = []; + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) }); + const work = nativeResponsePresentationWork(db); + const worker = work.register(coordinator, () => true); + const debug = db.$client.options.debug; + const locks: string[] = []; + try { + await worker.ready; + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), { timeout: 10_000 }) + .toEqual({ active: 0, next: null }); + db.$client.options.debug = (_connection, query) => { + if (query.startsWith("select") && query.includes("for update") && /"(?:issues|heartbeat_runs|native_run_finalizations)"/.test(query)) locks.push(query); + }; + for (let i = 0; i < 3; i++) { + await db.transaction(tx => notifyChatRunMilestones(tx, { companyId: context.fixture.companyId, issueId: context.issue.id })); + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), { timeout: 5000 }) + .toEqual({ active: 0, next: null }); + } + expect(locks).toEqual([]); + expect(await work.load({ companyId: context.fixture.companyId, issueId: context.issue.id })).toEqual([]); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toEqual([]); + expect(errors).toEqual([]); + } finally { db.$client.options.debug = debug; worker.stop(); await coordinator.stop(); } + }); + + it("leaves fresh-commit accepted-response presentation to the worker when periodic replay skips it", async () => { + const context = await committedChatResponseRecoveryFixture("telegram", "coordinator", false, undefined, undefined, false); + try { + await finalizeNativeRun({ db, runId: context.runId, workspaceFinalizeStatus: "succeeded", + projectRunStatus: true, skipNativeResponsePresentation: true }); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toEqual([]); + expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, context.runId))).toEqual([]); + const work = nativeResponsePresentationWork(db); + const entries = await work.load({ companyId: context.fixture.companyId, runId: context.runId }); + expect(entries).toHaveLength(1); + await expect(work.check(entries[0])).resolves.toBeNull(); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1); + } finally { await context.service.shutdown(); } + }); + + it.each(["reconciler", "finalizer"])("leaves accepted-response presentation to its worker during periodic %s replay", async caller => { + const context = await committedChatResponseRecoveryFixture("telegram"); + try { + if (caller === "reconciler") { + await reconcileNativeFinalizations(db, [context.runId], { skipNativeResponsePresentation: true }); + } else { + await finalizeNativeRun({ db, runId: context.runId, workspaceFinalizeStatus: "succeeded", + projectRunStatus: true, skipNativeResponsePresentation: true }); + } + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0); + expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, context.runId))).toHaveLength(0); + const work = nativeResponsePresentationWork(db); + const entries = await work.load({ companyId: context.fixture.companyId, runId: context.runId }); + expect(entries).toHaveLength(1); + await expect(work.check(entries[0])).resolves.toBeNull(); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1); + await expect(context.repair()).resolves.toBe(false); + } finally { await context.service.shutdown(); } + }); + + it("recovers accepted-response work on startup and leaves selected history quiet", async () => { + const context = await committedChatResponseRecoveryFixture("telegram"); + const errors: unknown[] = []; + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) }); + const work = nativeResponsePresentationWork(db); + const worker = work.register(coordinator, () => true); + try { + await worker.ready; + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), + { timeout: 10_000 }).toEqual({ active: 0, next: null }); + const comments = await db.select().from(issueComments).where(eq(issueComments.createdByRunId, context.runId)); + expect(comments).toHaveLength(1); + expect(comments[0].body).toBe(context.result.summary); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1); + expect(await work.load({ companyId: context.fixture.companyId })).toEqual([]); + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true }); + const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction"); + try { + await vi.advanceTimersByTimeAsync(120_000); + expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); + } finally { reads.mockRestore(); writes.mockRestore(); } + expect(errors).toEqual([]); + } finally { worker.stop(); await coordinator.stop(); vi.useRealTimers(); await context.service.shutdown(); } + }); + + it("reactivates dormant accepted-response work when the live chat authority registers", async () => { + const context = await committedChatResponseRecoveryFixture("telegram"); + await context.service.shutdown(); + const errors: unknown[] = []; + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) }); + const worker = nativeResponsePresentationWork(db).register(coordinator, () => true); + let replacement: ReturnType | undefined; + try { + await worker.ready; + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), + { timeout: 10_000 }).toEqual({ active: 0, next: null }); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0); + replacement = createService(new FakeChatSdkRuntime(), fakeTelegramFetch() as typeof fetch); + await expect.poll(async () => (await readChatPublicationsByEndpointId(context.endpoint.id)).length, { timeout: 5000 }).toBe(1); + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), + { timeout: 10_000 }).toEqual({ active: 0, next: null }); + expect(errors).toEqual([]); + } finally { worker.stop(); await coordinator.stop(); await replacement?.service.shutdown(); } + }); + + it.each(["issue", "run"])("does not reload dormant accepted-response history for an unrelated %s wake", async scope => { + const context = await committedChatResponseRecoveryFixture("telegram"); + await context.service.shutdown(); + const errors: unknown[] = []; + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) }); + const work = nativeResponsePresentationWork(db); + const worker = work.register(coordinator, () => true); + try { + await worker.ready; + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), + { timeout: 10_000 }).toEqual({ active: 0, next: null }); + expect(await work.load({ companyId: context.fixture.companyId, runId: context.runId })).toHaveLength(1); + const previousDebug = db.$client.options.debug; + const queries: string[] = []; + db.$client.options.debug = (_connection, query) => { + if (query.startsWith("select") && query.includes("native_run_finalizations")) queries.push(query); + }; + try { + await db.transaction(tx => notifyChatRunMilestones(tx, { companyId: context.fixture.companyId, + ...(scope === "issue" ? { issueId: randomUUID() } : { runId: randomUUID() }) })); + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), + { timeout: 5000 }).toEqual({ active: 0, next: null }); + expect(queries).toHaveLength(1); + } finally { db.$client.options.debug = previousDebug; } + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0); + expect(errors).toEqual([]); + } finally { worker.stop(); await coordinator.stop(); } + }); + + it.each(["milestone", "finalization"] as const)("ignores repeated unscoped routine %s hints without rebuilding dormant accepted-response history", async source => { + const context = await committedChatResponseRecoveryFixture("telegram"); + await context.service.shutdown(); + const errors: unknown[] = []; + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) }); + const work = nativeResponsePresentationWork(db); + const worker = work.register(coordinator, () => true); + try { + await worker.ready; + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), { timeout: 10_000 }) + .toEqual({ active: 0, next: null }); + expect(await work.load({ companyId: context.fixture.companyId, runId: context.runId })).toHaveLength(1); + const debug = db.$client.options.debug; + const queries: string[] = []; + db.$client.options.debug = (_connection, query) => { + if (query.startsWith("select") && query.includes("native_run_finalizations")) queries.push(query); + }; + try { + for (let i = 0; i < 3; i++) { + await db.transaction(tx => source === "milestone" + ? notifyChatRunMilestones(tx, { companyId: context.fixture.companyId }) + : notifyNativeFinalizationWork(tx, { companyId: context.fixture.companyId })); + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), { timeout: 5000 }) + .toEqual({ active: 0, next: null }); + } + expect(queries).toEqual([]); + } finally { db.$client.options.debug = debug; } + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0); + expect(errors).toEqual([]); + } finally { worker.stop(); await coordinator.stop(); } + }); + + it("wakes dormant accepted-response history from the real company membership restoration owner", async () => { + const context = await committedChatResponseRecoveryFixture("telegram"); + const [membership] = await db.select().from(companyMemberships).where(and( + eq(companyMemberships.companyId, context.fixture.companyId), eq(companyMemberships.principalId, "owner-user"))); + await db.update(companyMemberships).set({ status: "suspended" }).where(eq(companyMemberships.id, membership.id)); + const errors: unknown[] = []; + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) }); + const worker = nativeResponsePresentationWork(db).register(coordinator, () => true); + try { + await worker.ready; + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), { timeout: 10_000 }) + .toEqual({ active: 0, next: null }); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0); + await accessService(db).updateMember(context.fixture.companyId, membership.id, { status: "active" }); + await expect.poll(async () => ({ count: (await readChatPublicationsByEndpointId(context.endpoint.id)).length, + active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), { timeout: 5000 }) + .toEqual({ count: 1, active: 0, next: null }); + expect(errors).toEqual([]); + } finally { worker.stop(); await coordinator.stop(); await context.service.shutdown(); } + }); + + it.each(["commit", "rollback"])("keeps accepted-response work behind the outer %s boundary", async outcome => { + const context = await committedChatResponseRecoveryFixture("telegram"); + await db.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, context.runId)); + const errors: unknown[] = []; + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) }); + const worker = nativeResponsePresentationWork(db).register(coordinator, () => true); + try { + await worker.ready; + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), + { timeout: 10_000 }).toEqual({ active: 0, next: null }); + const transaction = db.transaction(async tx => { + await tx.transaction(async nested => { + await notifyChatRunMilestones(nested, { companyId: context.fixture.companyId, issueId: context.issue.id, runId: randomUUID() }); + await nested.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, context.runId)); + }); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0); + expect(idleWorkSnapshot().active).toBeGreaterThan(0); + if (outcome === "rollback") throw new Error("response fixture rollback"); + }); + if (outcome === "rollback") await expect(transaction).rejects.toThrow("response fixture rollback"); + else await transaction; + await expect.poll(async () => ({ + publications: (await readChatPublicationsByEndpointId(context.endpoint.id)).length, + active: idleWorkSnapshot().active, + next: coordinator.nextWakeAt(), + }), { timeout: 5000 }).toEqual({ publications: outcome === "commit" ? 1 : 0, active: 0, next: null }); + expect(errors).toEqual([]); + } finally { worker.stop(); await coordinator.stop(); await context.service.shutdown(); } + }); + + it("retains the first accepted-response wake through idle drain without passive SQL", async () => { + const context = await committedChatResponseRecoveryFixture("telegram"); + await db.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, context.runId)); + let canRun = true; + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true }); + const errors: unknown[] = []; + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => canRun, canReconcile: () => true, onError: error => errors.push(error) }); + const worker = nativeResponsePresentationWork(db).register(coordinator, () => canRun); + try { + await worker.ready; + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), + { timeout: 10_000 }).toEqual({ active: 0, next: null }); + canRun = false; + await db.transaction(async tx => { + await notifyChatRunMilestones(tx, { companyId: context.fixture.companyId, issueId: context.issue.id }); + await tx.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, context.runId)); + }); + await expect.poll(() => idleWorkSnapshot().active, { timeout: 5000 }).toBe(0); + expect(coordinator.nextWakeAt()).not.toBeNull(); + const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction"); + try { + await vi.advanceTimersByTimeAsync(120_000); + expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); + } finally { reads.mockRestore(); writes.mockRestore(); } + canRun = true; worker.wake(); + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(1000); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1); + expect(coordinator.nextWakeAt()).toBeNull(); + }, { timeout: 5000 }); + expect(errors).toEqual([]); + } finally { worker.stop(); await coordinator.stop(); vi.useRealTimers(); await context.service.shutdown(); } + }); + + it("retains a failed accepted-response attempt across drain and retries after resume", async () => { + const context = await committedChatResponseRecoveryFixture("telegram"); + let release!: () => void, observed!: () => void; + const hold = new Promise(resolve => { release = resolve; }); + const locked = new Promise(resolve => { observed = resolve; }); + const holder = db.transaction(async tx => { + await tx.select().from(chatActions).where(eq(chatActions.id, context.action.id)).for("update"); + observed(); + await hold; + }); + await locked; + let canRun = true; + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true }); + const errors: unknown[] = []; + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => canRun, canReconcile: () => true, onError: error => errors.push(error) }); + const worker = nativeResponsePresentationWork(db).register(coordinator, () => canRun); + try { + await worker.ready; + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(1000); + expect(errors).toHaveLength(1); + expect(idleWorkSnapshot().active).toBe(0); + }, { timeout: 5000 }); + expect(coordinator.nextWakeAt()).toBeGreaterThan(Date.now()); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0); + canRun = false; + const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction"); + try { + await vi.advanceTimersByTimeAsync(120_000); + expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); + } finally { reads.mockRestore(); writes.mockRestore(); } + release(); await holder; + canRun = true; worker.wake(); + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(1000); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1); + expect(idleWorkSnapshot().active).toBe(0); + expect(coordinator.nextWakeAt()).toBeNull(); + }, { timeout: 5000 }); + expect(errors).toHaveLength(1); + } finally { release(); await holder; worker.stop(); await coordinator.stop(); vi.useRealTimers(); await context.service.shutdown(); } + }); + + it("reconstructs a future accepted-response deadline without early publication", async () => { + const context = await committedChatResponseRecoveryFixture("telegram"); + const at = new Date(Date.now() + 60_000); + await db.update(nativeRunFinalizations).set({ nextAttemptAt: at }).where(eq(nativeRunFinalizations.runId, context.runId)); + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true }); + const errors: unknown[] = []; + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) }); + const worker = nativeResponsePresentationWork(db).register(coordinator, () => true); + try { + await worker.ready; + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), + { timeout: 10_000 }).toEqual({ active: 0, next: at.getTime() }); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0); + await vi.advanceTimersByTimeAsync(Math.max(0, at.getTime() - Date.now())); + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(1000); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1); + expect(coordinator.nextWakeAt()).toBeNull(); + }, { timeout: 5000 }); + expect(errors).toEqual([]); + } finally { worker.stop(); await coordinator.stop(); vi.useRealTimers(); await context.service.shutdown(); } + }); + + it("retains an accepted-response deadline that elapses during the locked recovery check", async () => { + const context = await committedChatResponseRecoveryFixture("telegram"); + const work = nativeResponsePresentationWork(db); + const [entry] = await work.load({ companyId: context.fixture.companyId }); + expect(entry?.id).toBe(context.runId); + vi.useFakeTimers({ toFake: ["Date"] }); + const due = new Date(Date.now() + 1000); + const transaction = db.transaction.bind(db); + const intercept = vi.spyOn(db, "transaction").mockImplementationOnce((async (callback, options) => { + await transaction(async tx => { + await tx.update(nativeRunFinalizations).set({ nextAttemptAt: due }).where(eq(nativeRunFinalizations.runId, context.runId)); + }); + const result = await transaction(callback, options); + vi.setSystemTime(due.getTime() + 1); + return result; + }) as Db["transaction"]); + try { + const deferred = await work.check(entry); + intercept.mockRestore(); + expect(deferred).toEqual({ ...entry, dueAt: due.getTime() }); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0); + await expect(work.check(deferred!)).resolves.toBeNull(); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1); + } finally { intercept.mockRestore(); vi.useRealTimers(); await context.service.shutdown(); } + }); + + it("wakes dormant accepted-response recovery after agent resume", async () => { + const context = await committedChatResponseRecoveryFixture("telegram"); + await db.update(agents).set({ status: "paused", lifecycleState: "paused", lifecycleHolds: ["manual"], lifecycleOperation: null }) + .where(eq(agents.id, context.binding.agentId)); + const errors: unknown[] = []; + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) }); + const work = nativeResponsePresentationWork(db); + const worker = work.register(coordinator, () => true); + const lifecycle = configureAgentLifecycle(db, { + requiredPluginIds: async () => [], runHost: async () => "complete", runPlugin: async () => "complete", + }); + try { + await worker.ready; + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), + { timeout: 10_000 }).toEqual({ active: 0, next: null }); + expect(await work.load({ companyId: context.fixture.companyId, agentId: randomUUID() })).toEqual([]); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0); + await createAgentLifecycle(db).resumeAgent(context.binding.agentId); + await processAgentLifecycle(db, context.binding.agentId); + await expect.poll(async () => (await db.select().from(agents).where(eq(agents.id, context.binding.agentId)))[0].status).toBe("idle"); + await expect.poll(async () => (await readChatPublicationsByEndpointId(context.endpoint.id)).length, { timeout: 5000 }).toBe(1); + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), + { timeout: 10_000 }).toEqual({ active: 0, next: null }); + expect(errors).toEqual([]); + } finally { await lifecycle.stop(); worker.stop(); await coordinator.stop(); await context.service.shutdown(); } + }); + + it("wakes accepted-response recovery from the real coordinator retry release", async () => { + const context = await committedChatResponseRecoveryFixture("telegram"); + const store = new NativeRunCoordinatorStore(db, context.binding); + await store.claimFinalization({ leaseOwner: "response-recovery-fixture", leaseTtlMs: 1000 }); + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true }); + const errors: unknown[] = []; + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) }); + const worker = nativeResponsePresentationWork(db).register(coordinator, () => true); + try { + await worker.ready; + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), + { timeout: 10_000 }).toEqual({ active: 0, next: null }); + await vi.advanceTimersByTimeAsync(2000); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0); + expect(coordinator.nextWakeAt()).toBeNull(); + await store.markFinalizationRetry({ leaseOwner: "response-recovery-fixture", failureCode: "response_fixture_retry", retryAfterMs: 1000 }); + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(1000); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1); + expect(idleWorkSnapshot().active).toBe(0); + expect(coordinator.nextWakeAt()).toBeNull(); + }, { timeout: 5000 }); + expect(errors).toEqual([]); + } finally { worker.stop(); await coordinator.stop(); vi.useRealTimers(); await context.service.shutdown(); } + }); + + it("wakes accepted-response recovery when its authenticated accepted-result journal arrives late", async () => { + const context = await committedChatResponseRecoveryFixture("telegram", "port"); + await db.delete(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.runId, context.runId), eq(heartbeatRunEvents.eventType, "run.result.accepted"))); + const errors: unknown[] = []; + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) }); + const worker = nativeResponsePresentationWork(db).register(coordinator, () => true); + try { + await worker.ready; + await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), + { timeout: 10_000 }).toEqual({ active: 0, next: null }); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0); + const port = new PaperclipControlPlanePort(db, { + companyId: context.binding.companyId, issueId: context.binding.issueId, + runId: context.runId, agentId: context.binding.agentId, sessionId: context.binding.normalizedSessionId, + completionContractId: context.binding.completionContractId, + completionContractSha256: context.binding.completionContractSha256, + sourceInstanceId: context.binding.runnerSourceInstanceId, controlPlaneSourceInstanceId: `control-${context.runId}`, + }); + await port.appendEvent({ + schema: "paperclip.prp.event.v1", sourceEventId: `accepted-${context.runId}`, sourceSeq: 1, + sourceInstanceId: `control-${context.runId}`, sourceKind: "control_plane", + runId: context.runId, normalizedSessionId: context.binding.normalizedSessionId, + turnId: `turn-${context.runId}`, eventType: "run.result.accepted", schemaVersion: 1, + priority: 0, emittedAt: new Date().toISOString(), payload: { result: context.result }, + }); + await expect.poll(async () => ({ + publications: (await readChatPublicationsByEndpointId(context.endpoint.id)).length, + active: idleWorkSnapshot().active, next: coordinator.nextWakeAt(), + }), { timeout: 5000 }).toEqual({ publications: 1, active: 0, next: null }); + expect(errors).toEqual([]); + } finally { worker.stop(); await coordinator.stop(); await context.service.shutdown(); } + }); + + it("defers scheduled accepted-response recovery until the persisted retry is due", async () => { + const context = await committedChatResponseRecoveryFixture("telegram"); + const input = { companyId: context.fixture.companyId, issueId: context.issue.id, runId: context.runId, onlyWhenDue: true }; + try { + await db.update(nativeRunFinalizations).set({ nextAttemptAt: new Date(Date.now() + 60_000) }) + .where(eq(nativeRunFinalizations.runId, context.runId)); + await expect(recoverCommittedNativeChatResponse(db, input)).resolves.toBe(false); + expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, context.runId))).toHaveLength(0); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0); + await db.update(nativeRunFinalizations).set({ nextAttemptAt: new Date(0) }) + .where(eq(nativeRunFinalizations.runId, context.runId)); + await expect(recoverCommittedNativeChatResponse(db, input)).resolves.toBe(true); + expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, context.runId))).toHaveLength(1); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1); + } finally { await context.service.shutdown(); } + }); + + it.each(["pending", "failed"])("defers scheduled accepted-response recovery behind %s status effects", async deliveryState => { + const context = await committedChatResponseRecoveryFixture("telegram"); + const input = { companyId: context.fixture.companyId, issueId: context.issue.id, runId: context.runId, onlyWhenDue: true }; + try { + const [effect] = await db.select().from(statusDecisionEffects).where(eq(statusDecisionEffects.decisionId, context.coordinator.decisionId!)); + expect(effect).toBeDefined(); + await db.update(statusDecisionEffects).set({ deliveryState }).where(eq(statusDecisionEffects.id, effect.id)); + await expect(recoverCommittedNativeChatResponse(db, input)).resolves.toBe(false); + expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, context.runId))).toHaveLength(0); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0); + await db.update(statusDecisionEffects).set({ deliveryState: "delivered" }).where(eq(statusDecisionEffects.id, effect.id)); + await expect(recoverCommittedNativeChatResponse(db, input)).resolves.toBe(true); + expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1); + } finally { await context.service.shutdown(); } + }); + it("leaves a contended accepted response retryable without reclassifying or partially publishing it", async () => { const context = await committedChatResponseRecoveryFixture("telegram"); let release!: () => void; @@ -75279,6 +75819,9 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { try { await locked; await expect(context.repair()).resolves.toBe(false); + await expect(recoverCommittedNativeChatResponse(db, { + companyId: context.fixture.companyId, issueId: context.issue.id, runId: context.runId, + })).rejects.toThrow(); await expect( db .select() diff --git a/server/src/__tests__/committed-chat-response-authority.test.ts b/server/src/__tests__/committed-chat-response-authority.test.ts new file mode 100644 index 00000000000..a09b8847c1b --- /dev/null +++ b/server/src/__tests__/committed-chat-response-authority.test.ts @@ -0,0 +1,55 @@ +import { isNativeResponseEvidence } from "../services/native-response-presentation-notifications.js"; +import { describe, expect, it, vi } from "vitest"; +import type { Db } from "@paperclipai/db"; +import { authorizeCommittedChatResponse, onCommittedChatResponseAuthorityRegistered, registerCommittedChatResponseAuthority } from "../services/durable-chat-wakeup.js"; + +describe("committed response authority lifecycle", () => { + it("reactivates only its database and unsubscribes without removing a replacement listener", () => { + const db = {} as Db, otherDb = {} as Db; + const first = vi.fn(), second = vi.fn(), other = vi.fn(); + const disposeFirst = onCommittedChatResponseAuthorityRegistered(db, first); + const disposeOther = onCommittedChatResponseAuthorityRegistered(otherDb, other); + const removeAuthority = registerCommittedChatResponseAuthority(db, async () => {}); + expect(first).toHaveBeenCalledTimes(1); + expect(other).not.toHaveBeenCalled(); + disposeFirst(); + const disposeSecond = onCommittedChatResponseAuthorityRegistered(db, second); + disposeFirst(); + const removeReplacement = registerCommittedChatResponseAuthority(db, async () => {}); + expect(first).toHaveBeenCalledTimes(1); + expect(second).toHaveBeenCalledTimes(1); + removeAuthority(); + expect(second).toHaveBeenCalledTimes(1); + disposeSecond(); disposeOther(); removeReplacement(); + const removeLast = registerCommittedChatResponseAuthority(db, async () => {}); + expect(second).toHaveBeenCalledTimes(1); + removeLast(); + }); + + it("retains the replacement authority when the old service shuts down", async () => { + const db = {} as Db; + const first = vi.fn(async () => {}), second = vi.fn(async () => {}); + const removeFirst = registerCommittedChatResponseAuthority(db, first); + const removeSecond = registerCommittedChatResponseAuthority(db, second); + try { + removeFirst(); + const input = { companyId: "company", issueId: "issue", agentId: "agent", runId: "run", resultId: "result" }; + await authorizeCommittedChatResponse(db, db, input); + expect(first).not.toHaveBeenCalled(); + expect(second).toHaveBeenCalledExactlyOnceWith(db, input); + } finally { removeSecond(); } + }); +}); + + +describe("accepted response evidence selection", () => { + it.each([ + { type: "item.delta", kind: "agentMessage", channel: "final", expected: false }, + { type: "item.completed", kind: "agentMessage", channel: "analysis", expected: false }, + { type: "item.completed", kind: "commandExecution", channel: "final", expected: false }, + { type: "item.completed", kind: "agentMessage", channel: "final", expected: true }, + { type: "run.result.accepted", kind: "agentMessage", channel: "final", expected: true }, + ])("signals only response evidence: $type/$kind/$channel", ({ type, kind, channel, expected }) => { + expect(isNativeResponseEvidence(type, { payload: { kind, channel, text: "Saved evidence" } })).toBe(expected); + }); +}); diff --git a/server/src/__tests__/server-startup-feedback-export.test.ts b/server/src/__tests__/server-startup-feedback-export.test.ts index 62bc485a0c2..8777ebc7858 100644 --- a/server/src/__tests__/server-startup-feedback-export.test.ts +++ b/server/src/__tests__/server-startup-feedback-export.test.ts @@ -116,6 +116,7 @@ const { reapOrphanedRuns: vi.fn(async () => ({ reaped: 0, runIds: [] })), registerScheduledRetryCancellationWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), registerStoppedInstructionCollectionWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), + registerNativeResponsePresentationWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve(), stop: vi.fn() })), registerNativePendingEffectWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), registerUnsafeWorkspaceExportWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), registerAutomaticCompletionReviewWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), @@ -558,6 +559,17 @@ describe("startServer feedback export wiring", () => { process.env.BETTER_AUTH_SECRET = "test-secret"; }); + it.each([false, true])("registers response recovery and its shutdown subscription only when maintenance is enabled (%s)", async enabled => { + loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled })); + await startServer(); await settleStartupHousekeeping(); + expect(heartbeatServiceMock.registerNativeResponsePresentationWork).toHaveBeenCalledTimes(enabled ? 1 : 0); + if (enabled) { + expect(heartbeatServiceMock.registerNativeResponsePresentationWork.mock.calls[0][1]()).toBe(true); + const worker = heartbeatServiceMock.registerNativeResponsePresentationWork.mock.results[0].value; + expect(fakeServer.on).toHaveBeenCalledWith("close", worker.stop); + } + }); + it.each([false, true])("registers native pending effects only when maintenance is enabled (%s)", async enabled => { loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled })); await startServer(); await settleStartupHousekeeping(); @@ -1005,7 +1017,7 @@ describe("startServer feedback export wiring", () => { intervalCallback?.(); await new Promise((resolve) => setImmediate(resolve)); expect(heartbeatServiceMock.sweepStaleIssueLocks).toHaveBeenCalledTimes(2); - expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true }); + expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true }); expect(heartbeatServiceMock.reconcileCostAccounting).toHaveBeenCalledTimes(1); expect(retiredDetector).not.toHaveBeenCalled(); } finally { diff --git a/server/src/board-claim.ts b/server/src/board-claim.ts index 275dfdf0180..a5a2cbdf20e 100644 --- a/server/src/board-claim.ts +++ b/server/src/board-claim.ts @@ -4,7 +4,7 @@ import type { Db } from "@paperclipai/db"; import { companies, companyMemberships, instanceUserRoles } from "@paperclipai/db"; import type { DeploymentMode } from "@paperclipai/shared"; import { ensureHumanRoleDefaultGrants } from "./services/principal-access-compatibility.js"; -import { notifyChatRunMilestones } from "./services/chat-run-milestone-notifications.js"; +import { notifyChatCompanyAuthorityChange } from "./services/chat-run-milestone-notifications.js"; const LOCAL_BOARD_USER_ID = "local-board"; const CLAIM_TTL_MS = 1000 * 60 * 60 * 24; @@ -125,7 +125,7 @@ export async function claimBoardOwnership( .then((rows) => rows[0] ?? null); if (!existing) { - await notifyChatRunMilestones(tx, { companyId: company.id }); + await notifyChatCompanyAuthorityChange(tx, { companyId: company.id }); await tx.insert(companyMemberships).values({ companyId: company.id, principalType: "user", @@ -137,7 +137,7 @@ export async function claimBoardOwnership( } if (existing.status !== "active") { - await notifyChatRunMilestones(tx, { companyId: company.id }); + await notifyChatCompanyAuthorityChange(tx, { companyId: company.id }); await tx .update(companyMemberships) .set({ status: "active", membershipRole: "owner", updatedAt: new Date() }) diff --git a/server/src/index.ts b/server/src/index.ts index 719ea5ee7c6..86a7b9da72c 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1403,6 +1403,7 @@ async function startServerWithDatabaseTeardown( hasPending: () => caseHandoffDeliveries.hasPending(), }); let unregisterTerminalWorkspaceProbe: (() => void) | undefined; + let unsubscribeNativeResponsePresentation: (() => void) | undefined; let unsubscribeNativeWorkspaceSyncRetention: (() => void) | undefined; const fastResponses = app.locals.fastResponses as ReturnType; const fastResponseWorker = deliveryWork.register(DELIVERY_QUEUES.fastResponse, { @@ -1824,6 +1825,13 @@ async function startServerWithDatabaseTeardown( () => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready; if (config.heartbeatSchedulerEnabled) await heartbeat.registerInstructionDirectoryCleanupWork(deliveryWork, () => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready; + if (config.heartbeatSchedulerEnabled) { + const worker = heartbeat.registerNativeResponsePresentationWork(deliveryWork, + () => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()); + unsubscribeNativeResponsePresentation = worker.stop; + server.on("close", worker.stop); + await worker.ready; + } if (config.heartbeatSchedulerEnabled) await heartbeat.registerNativePendingEffectWork(deliveryWork, () => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready; if (config.heartbeatSchedulerEnabled) await heartbeat.registerUnsafeWorkspaceExportWork(deliveryWork, @@ -1964,7 +1972,7 @@ async function startServerWithDatabaseTeardown( // Periodically reap orphaned runs (5-min staleness threshold) and make sure // persisted queued work is still being driven forward. runPeriodicRecovery(() => heartbeat - .reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true }) + .reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true }) .then(() => heartbeat.promoteDueScheduledRetries()) .then(async (promotion) => { await heartbeat.resumeQueuedRuns(); @@ -2160,6 +2168,7 @@ async function startServerWithDatabaseTeardown( await skillVersionRetention.dispose(); await skillSourceSync.stop(); unsubscribeChatCompletions(); + unsubscribeNativeResponsePresentation?.(); await deliveryWork.stop(); await workspaceRecoveryWork.stop(); unregisterTerminalWorkspaceProbe?.(); diff --git a/server/src/middleware/auth.ts b/server/src/middleware/auth.ts index aa9e3c13a37..f9bd7446947 100644 --- a/server/src/middleware/auth.ts +++ b/server/src/middleware/auth.ts @@ -30,7 +30,7 @@ import { captureRunIdentity } from "../services/run-identity.js"; import { boardAuthService } from "../services/board-auth.js"; import { retryIdempotentDatabaseOperation } from "../database-retry.js"; import { beginIdleTrackedWork } from "../services/task-admission.js"; -import { notifyChatRunMilestones } from "../services/chat-run-milestone-notifications.js"; +import { notifyChatCompanyAuthorityChange } from "../services/chat-run-milestone-notifications.js"; export { isTransientDbConnectionError, @@ -665,7 +665,7 @@ async function resolveCloudTenantActorOnce( // Serialize authority restoration and its notice while retaining the // guarded upsert that leaves unchanged memberships untouched. if (!existing || existing.status !== "active" || existing.membershipRole === "viewer") { - await notifyChatRunMilestones(tx, { companyId }); + await notifyChatCompanyAuthorityChange(tx, { companyId }); } return tx .insert(companyMemberships) diff --git a/server/src/modules/run-dispatch/adapters/postgres.ts b/server/src/modules/run-dispatch/adapters/postgres.ts index 8531240df68..4c313517ef3 100644 --- a/server/src/modules/run-dispatch/adapters/postgres.ts +++ b/server/src/modules/run-dispatch/adapters/postgres.ts @@ -673,7 +673,7 @@ export function createPostgresRunDispatchAdapter( await notifyInstructionCollection(tx, { companyId: input.companyId, runId: input.runId }); await notifyCancelledWorkspacePreservation(tx, { companyId: input.companyId, runId: input.runId }); await notifyScheduledRetryCancellation(tx, { companyId: input.companyId, runId: input.runId }); - await notifyChatRunMilestones(tx, { companyId: input.companyId, issueId: issueId ?? undefined }); + await notifyChatRunMilestones(tx, { companyId: input.companyId, issueId: issueId ?? undefined, runId: input.runId }); const [row] = await tx .update(heartbeatRuns) .set({ @@ -719,7 +719,7 @@ export function createPostgresRunDispatchAdapter( await notifyCancelledWorkspacePreservation(tx, { companyId: input.companyId, runId: input.runId }); await notifyScheduledRetryCancellation(tx, { companyId: input.companyId, runId: input.runId }); await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: input.companyId, issueId: input.issueId ?? undefined }); - await notifyChatRunMilestones(tx, { companyId: input.companyId, issueId: input.issueId ?? undefined }); + await notifyChatRunMilestones(tx, { companyId: input.companyId, issueId: input.issueId ?? undefined, runId: input.runId }); const [row] = await tx .update(heartbeatRuns) .set({ diff --git a/server/src/modules/wake-queue/adapters/postgres.test.ts b/server/src/modules/wake-queue/adapters/postgres.test.ts index 83d5bb6ee44..240776e2278 100644 --- a/server/src/modules/wake-queue/adapters/postgres.test.ts +++ b/server/src/modules/wake-queue/adapters/postgres.test.ts @@ -1,3 +1,4 @@ +import { DELIVERY_QUEUES } from "../../../services/delivery-work-notifications.js"; import { instanceSettingsService } from "../../../services/instance-settings.js"; import { randomBytes, randomUUID } from "node:crypto"; import { eq } from "drizzle-orm"; @@ -16,6 +17,7 @@ import { issueRecoveryActions, issues, nativeRunFinalizations, + subscribeDatabaseWork, } from "@paperclipai/db"; import { getEmbeddedPostgresTestSupport, @@ -1222,6 +1224,30 @@ describeEmbeddedPostgres("wake-queue postgres adapter", () => { expect(wakeRow?.payload).not.toHaveProperty("foo"); }); + it("scopes a real coalesced-input response wake to its active run after commit", async () => { + const companyId = await seedCompany(); + const agentId = await seedAgent({ companyId }); + const runId = await seedRun({ companyId, agentId, status: "running" }); + const hints: Array> = []; + const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.nativeResponsePresentation, (event, values) => { + if (event === "settled") for (const value of values ?? []) hints.push(JSON.parse(value)); + }); + try { + await db.transaction(async tx => { + await createWakeAdmissionWriter().coalesceIntoActiveExecutionRun(createAdmissionTransactionScope(companyId, tx as unknown as Db), { + companyId, activeExecutionRunId: runId, mergedContextSnapshot: { commentId: "fresh-input" }, + agentId, source: "on_demand", triggerDetail: null, payload: null, + requestedByActorType: "user", requestedByActorId: null, idempotencyKey: null, + }); + expect(hints).toEqual([]); + }); + expect(hints).toEqual([{ companyId, runId }]); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0].contextSnapshot) + .toMatchObject({ commentId: "fresh-input" }); + expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.runId, runId))).toHaveLength(1); + } finally { unsubscribe(); } + }); + // Review test (b), the coalesce write: a foreign company affects no row // on the active execution run either. it("refuses to coalesce into a run for a company that does not own it, and leaves the run and the wake table untouched", async () => { diff --git a/server/src/modules/wake-queue/adapters/postgres.ts b/server/src/modules/wake-queue/adapters/postgres.ts index 7b3b45f1a39..0c80fd6f5d9 100644 --- a/server/src/modules/wake-queue/adapters/postgres.ts +++ b/server/src/modules/wake-queue/adapters/postgres.ts @@ -956,7 +956,7 @@ export function createWakeAdmissionWriter(): WakeAdmissionWriter { async coalesceIntoActiveExecutionRun(scope, input) { const tx = requireAdmissionTx(scope, input.companyId); const now = new Date(); - await notifyChatRunMilestones(tx, { companyId: input.companyId }); + await notifyChatRunMilestones(tx, { companyId: input.companyId, runId: input.activeExecutionRunId }); const mergedRun = await tx .update(heartbeatRuns) .set({ contextSnapshot: input.mergedContextSnapshot, updatedAt: now }) diff --git a/server/src/services/access.ts b/server/src/services/access.ts index c76608597cc..64f57cb1f1a 100644 --- a/server/src/services/access.ts +++ b/server/src/services/access.ts @@ -1,5 +1,5 @@ import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js"; -import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js"; +import { notifyChatCompanyAuthorityChange } from "./chat-run-milestone-notifications.js"; import { notifyVoiceChange } from "./voice/voice-recovery-notifications.js"; import { notifyChatEndpointWork } from "./chat-work-notifications.js"; import { notifyToolConnectionHealth } from "./tool-connection-health-notifications.js"; @@ -558,7 +558,7 @@ export function accessService(db: Db) { if (existing.principalType === "user" && nextStatus === "active" && nextMembershipRole !== "viewer" && (existing.status !== "active" || existing.membershipRole === "viewer")) { - await notifyChatRunMilestones(tx, { companyId }); + await notifyChatCompanyAuthorityChange(tx, { companyId }); } if ( @@ -894,7 +894,7 @@ export function accessService(db: Db) { const existingMembership = existingByCompany.get(companyId); if (existingMembership) { if (existingMembership.status !== "active") { - if (existingMembership.membershipRole !== "viewer") await notifyChatRunMilestones(tx, { companyId }); + if (existingMembership.membershipRole !== "viewer") await notifyChatCompanyAuthorityChange(tx, { companyId }); await tx .update(companyMemberships) .set({ @@ -906,7 +906,7 @@ export function accessService(db: Db) { } continue; } - await notifyChatRunMilestones(tx, { companyId }); + await notifyChatCompanyAuthorityChange(tx, { companyId }); await tx.insert(companyMemberships).values({ companyId, principalType: "user", @@ -936,7 +936,7 @@ export function accessService(db: Db) { if (existing.status !== status || existing.membershipRole !== membershipRole) { if (principalType === "user" && status === "active" && membershipRole !== "viewer" && (existing.status !== "active" || existing.membershipRole === "viewer")) { - await notifyChatRunMilestones(tx, { companyId }); + await notifyChatCompanyAuthorityChange(tx, { companyId }); } const updated = await tx .update(companyMemberships) @@ -950,7 +950,7 @@ export function accessService(db: Db) { } if (principalType === "user" && status === "active" && membershipRole !== "viewer") { - await notifyChatRunMilestones(tx, { companyId }); + await notifyChatCompanyAuthorityChange(tx, { companyId }); } return tx .insert(companyMemberships) @@ -1146,7 +1146,7 @@ export function accessService(db: Db) { if (existing.principalType === "user" && nextStatus === "active" && nextMembershipRole !== "viewer" && (existing.status !== "active" || existing.membershipRole === "viewer")) { - await notifyChatRunMilestones(tx, { companyId }); + await notifyChatCompanyAuthorityChange(tx, { companyId }); } if ( diff --git a/server/src/services/agent-lifecycle.ts b/server/src/services/agent-lifecycle.ts index 164e94420db..041ae077db4 100644 --- a/server/src/services/agent-lifecycle.ts +++ b/server/src/services/agent-lifecycle.ts @@ -1,4 +1,5 @@ import { isAgentStatusInvokable } from "@paperclipai/shared"; +import { notifyNativeResponsePresentation } from "./native-response-presentation-notifications.js"; import { notifyChatInboundWakeNotices } from "./chat-inbound-wake-notice-notifications.js"; import { logger } from "../middleware/logger.js"; import { companies, type Db } from "@paperclipai/db"; @@ -35,8 +36,9 @@ export function createAgentLifecycleEffects(_hooks: BudgetServiceHooks = {}): li return true; }, notifyWork: tx => notifyDeliveryWork(tx, DELIVERY_QUEUES.agentLifecycle), - async beforeStatusChange(tx, companyId, _agentId, before, after) { + async beforeStatusChange(tx, companyId, agentId, before, after) { if (!isAgentStatusInvokable(before) && isAgentStatusInvokable(after)) { + await notifyNativeResponsePresentation(tx, { companyId, agentId }); await notifyChatInboundWakeNotices(tx, { companyId }); } }, diff --git a/server/src/services/chat-channels/endpoints.ts b/server/src/services/chat-channels/endpoints.ts index 86b788cea74..0dc9f07e9f7 100644 --- a/server/src/services/chat-channels/endpoints.ts +++ b/server/src/services/chat-channels/endpoints.ts @@ -1,4 +1,4 @@ -import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js"; +import { notifyChatCompanyAuthorityChange } from "../chat-run-milestone-notifications.js"; import { chatConnectionTransport } from "./provider-catalog.js"; import type { ConnectionManagementContext } from "../connection-management.js"; import { slackSetupState } from "./slack/setup.js"; @@ -458,7 +458,7 @@ function providerSetupState( if ((input.allowDirectMessages === true && !existing.endpoint.allowDirectMessages) || (input.allowGroupChats === true && !existing.endpoint.allowGroupChats) || (input.allowUnlinkedPeople === true && !existing.endpoint.allowUnlinkedPeople)) { - await notifyChatRunMilestones(tx, { companyId: existing.endpoint.companyId }); + await notifyChatCompanyAuthorityChange(tx, { companyId: existing.endpoint.companyId }); } if (input.executionDefaults !== undefined) { await assertChatExecutionDefaultsAccess(db, { diff --git a/server/src/services/chat-channels/failed-run-retry.ts b/server/src/services/chat-channels/failed-run-retry.ts index 8dd6ace33a9..c324a6f3c34 100644 --- a/server/src/services/chat-channels/failed-run-retry.ts +++ b/server/src/services/chat-channels/failed-run-retry.ts @@ -1,3 +1,4 @@ +import { notifyNativeResponsePresentation } from "../native-response-presentation-notifications.js"; import { notifyChatInboundWakeNotices } from "../chat-inbound-wake-notice-notifications.js"; import { createFailedRetryNotices } from "./failed-retry-notices.js"; import { safeChatWakeAuthorityReason, type ChatWakeAuthorityReason } from "./inbound/wakeup-authorization-error.js"; @@ -1784,6 +1785,9 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti const settle = async (status: string, result: Record) => { const [current] = await db.transaction(async (tx) => { await notifyChatActionWork(tx, "failed_run_retry"); + if (status === "processed") await notifyNativeResponsePresentation(tx, { + companyId: claimed.companyId, issueId: String(claimed.payload.issueId), + }); return tx .update(chatActions) .set({ status, result, updatedAt: new Date() }) diff --git a/server/src/services/chat-channels/identity-links.ts b/server/src/services/chat-channels/identity-links.ts index abbd2340c94..e1e1b0b2712 100644 --- a/server/src/services/chat-channels/identity-links.ts +++ b/server/src/services/chat-channels/identity-links.ts @@ -1,4 +1,4 @@ -import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js"; +import { notifyChatCompanyAuthorityChange } from "../chat-run-milestone-notifications.js"; import { githubIdentityFields } from "./github/identity-links.js"; import { createSlackIdentityLinks, type SlackIdentityEffectStager } from "./slack/identity-links.js"; import { createHash, randomBytes } from "node:crypto"; @@ -320,7 +320,7 @@ export function createChatIdentityLinks(db: Db, options: ChatIdentityLinkOptions }, ); } - await notifyChatRunMilestones(tx, { companyId: currentLink.companyId }); + await notifyChatCompanyAuthorityChange(tx, { companyId: currentLink.companyId }); const confirmedLink = await tx .update(chatIdentityLinks) .set({ diff --git a/server/src/services/chat-channels/resource-reconciliation.ts b/server/src/services/chat-channels/resource-reconciliation.ts index 5cf5f570266..7e0cc6b5ed8 100644 --- a/server/src/services/chat-channels/resource-reconciliation.ts +++ b/server/src/services/chat-channels/resource-reconciliation.ts @@ -1,4 +1,4 @@ -import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js"; +import { notifyChatRunMilestones, notifyChatCompanyAuthorityChange } from "../chat-run-milestone-notifications.js"; import { notifyChatEndpointConfiguration } from "./configuration-work-notifications.js"; import { notifyChatConnectionHealth } from "./connection-health-notifications.js"; import type { ConnectionManagementContext } from "../connection-management.js"; @@ -287,7 +287,7 @@ export function createChatResourceReconciliation(db: Db, options: ResourceReconc after: { enabled: finalEnabled.get(row.id)!, ...(JSON.stringify(row.executionDefaults) !== JSON.stringify(finalDefaults.get(row.id)) ? { executionDefaults: finalDefaults.get(row.id) } : {}) }, })); if (changes.some(change => !change.before.enabled && change.after.enabled)) { - await notifyChatRunMilestones(tx, { companyId: endpoint.companyId }); + await notifyChatCompanyAuthorityChange(tx, { companyId: endpoint.companyId }); } if (bulk) { // One atomic database update, independent of pagination/search and diff --git a/server/src/services/chat-channels/slack/setup-services.ts b/server/src/services/chat-channels/slack/setup-services.ts index 45f8d3bc59c..7463d3bc6e2 100644 --- a/server/src/services/chat-channels/slack/setup-services.ts +++ b/server/src/services/chat-channels/slack/setup-services.ts @@ -1,4 +1,4 @@ -import { notifyChatRunMilestones } from "../../chat-run-milestone-notifications.js"; +import { notifyChatCompanyAuthorityChange } from "../../chat-run-milestone-notifications.js"; import { chatIdentityLinks, chatExternalPrincipals, companyMemberships, toolConnections, type Db } from "@paperclipai/db"; import { and, eq, ne, sql } from "drizzle-orm"; import { conflict, forbidden, notFound } from "../../../errors.js"; @@ -47,7 +47,7 @@ export function createSlackSetupServices(db: Db, options: SetupServicesOptions) const links = await tx.select().from(chatIdentityLinks).where(and(eq(chatIdentityLinks.companyId, record.endpoint.companyId), eq(chatIdentityLinks.principalId, principal.id))).for("update"); if (links.some(link => link.status === "linked" && link.paperclipUserId !== actor.userId) || links.some(link => link.endpointId === endpointId && link.status === "revoked")) throw conflict("This Slack account has an existing link", { code: "chat_identity_link_conflict" }); if (links.some(link => link.endpointId === endpointId && link.status === "linked")) return; - await notifyChatRunMilestones(tx, { companyId: record.endpoint.companyId }); + await notifyChatCompanyAuthorityChange(tx, { companyId: record.endpoint.companyId }); await tx.insert(chatIdentityLinks).values({ companyId: record.endpoint.companyId, endpointId, principalId: principal.id, paperclipUserId: actor.userId, status: "linked", confirmedAt: new Date() }) .onConflictDoUpdate({ target: [chatIdentityLinks.endpointId, chatIdentityLinks.principalId], set: { diff --git a/server/src/services/chat-github-management.ts b/server/src/services/chat-github-management.ts index 3f1bc34a6b4..8c6c66ff9fb 100644 --- a/server/src/services/chat-github-management.ts +++ b/server/src/services/chat-github-management.ts @@ -24,7 +24,7 @@ import { import { HttpError, badRequest, conflict, forbidden, notFound, unprocessable } from "../errors.js"; import { toolAccessService } from "./tool-access.js"; import { logActivity } from "./activity-log.js"; -import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js"; +import { notifyChatCompanyAuthorityChange } from "./chat-run-milestone-notifications.js"; import { githubBotRequest } from "./chat-github-client.js"; import { environmentService } from "./environments.js"; import { instanceSettingsService } from "./instance-settings.js"; @@ -165,7 +165,7 @@ export function githubChatManagementService(db: Db, fetchImpl = fetch) { const allowUnlinkedPeople = config.people.some((p) => p.kind === "guest"); if ((!currentEndpoint.allowUnlinkedPeople && allowUnlinkedPeople) || (allowUnlinkedPeople && currentEndpoint.sponsorUserId !== config.responsibleUserId)) { - await notifyChatRunMilestones(tx, { companyId: bot.companyId }); + await notifyChatCompanyAuthorityChange(tx, { companyId: bot.companyId }); } const values = { companyId: bot.companyId, @@ -335,7 +335,7 @@ export function githubChatManagementService(db: Db, fetchImpl = fetch) { updatedAt: new Date(), }; if (prior?.status !== "linked") { - await notifyChatRunMilestones(tx, { companyId: bot.companyId }); + await notifyChatCompanyAuthorityChange(tx, { companyId: bot.companyId }); } await tx .insert(chatIdentityLinks) diff --git a/server/src/services/chat-monitor-continuation.test.ts b/server/src/services/chat-monitor-continuation.test.ts index f8e277d587b..26163e02d9a 100644 --- a/server/src/services/chat-monitor-continuation.test.ts +++ b/server/src/services/chat-monitor-continuation.test.ts @@ -1,5 +1,7 @@ +import { nativeResponsePresentationWork } from "./native-runtime/native-response-presentation-work.js"; +import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js"; import { randomBytes, randomUUID } from "node:crypto"; -import { and, eq } from "drizzle-orm"; +import { and, eq, sql } from "drizzle-orm"; import { afterAll, beforeAll, describe, expect, it } from "vitest"; import { agents, agentWakeupRequests, authUsers, chatConversations, chatEndpoints, chatMessageLinks, @@ -16,14 +18,18 @@ import { createHeartbeatQueueTestDependencies } from "../__tests__/helpers/heart import { createHeartbeatQueue } from "./heartbeat/queue.js"; import { instanceSettingsService } from "./instance-settings.js"; -const support = await getEmbeddedPostgresTestSupport(); +const external = resolveExternalTestDatabaseUrl("PAPERCLIP_MONITOR_CONTINUATION_TEST_DATABASE_URL"); +const support = external ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport(); if (!support.supported) console.warn(`Monitor publication database unavailable: ${support.reason}`); (support.supported ? describe : describe.skip)("monitor conversation continuation", () => { - let temporary: Awaited>; + let temporary: Awaited> | undefined; let db: ReturnType; beforeAll(async () => { - temporary = await startEmbeddedPostgresTestDatabase("paperclip-monitor-publications-"); - db = createDb(temporary.connectionString); + if (external) db = createDb(external); + else { + temporary = await startEmbeddedPostgresTestDatabase("paperclip-monitor-publications-"); + db = createDb(temporary.connectionString); + } await db.insert(authUsers).values({ id: "requester", name: "Requester", email: "requester@example.test", createdAt: new Date(), updatedAt: new Date() }); }); afterAll(async () => { await temporary?.cleanup(); }); @@ -235,7 +241,7 @@ if (!support.supported) console.warn(`Monitor publication database unavailable: expect(await f.bindings()).toEqual([]); }); - it("recovers the accepted continuation explanation after interrupted finalization without another provider result", async () => { + it.each([false, true])("recovers the accepted continuation explanation without another provider result (missing presentation: %s)", async missingPresentation => { const f = await fixture(); const executionPayload = randomBytes(128 * 1024).toString("base64"); await db.update(heartbeatRuns).set({ @@ -281,6 +287,20 @@ if (!support.supported) console.warn(`Monitor publication database unavailable: expect(settledRun.stdoutExcerpt).toBe(executionPayload); const priorComments = await db.select({ body: issueComments.body }).from(issueComments).where(eq(issueComments.createdByRunId, f.source.id)); expect(priorComments).toEqual([{ body: summary }]); + expect(await nativeResponsePresentationWork(db).load({ companyId: f.company.id, runId: f.source.id })).toEqual([]); + if (missingPresentation) { + await db.transaction(async tx => { + await tx.delete(chatPublications).where(eq(chatPublications.issueId, f.issue.id)); + await tx.delete(issueComments).where(eq(issueComments.createdByRunId, f.source.id)); + await tx.update(heartbeatRuns).set({ resultJson: sql`${heartbeatRuns.resultJson} - 'presentationDecision'` }) + .where(eq(heartbeatRuns.id, f.source.id)); + }); + const work = nativeResponsePresentationWork(db); + const entries = await work.load({ companyId: f.company.id, runId: f.source.id }); + expect(entries).toHaveLength(1); + await expect(work.check(entries[0])).resolves.toBeNull(); + expect(await work.load({ companyId: f.company.id, runId: f.source.id })).toEqual([]); + } expect(await repair()).toBe(false); expect(await repair()).toBe(false); const comments = await db.select().from(issueComments).where(eq(issueComments.createdByRunId, f.source.id)); diff --git a/server/src/services/chat-run-milestone-notifications.ts b/server/src/services/chat-run-milestone-notifications.ts index 8ab9636b6af..17316dbc678 100644 --- a/server/src/services/chat-run-milestone-notifications.ts +++ b/server/src/services/chat-run-milestone-notifications.ts @@ -1,3 +1,4 @@ +import { notifyNativeResponsePresentation } from "./native-response-presentation-notifications.js"; import { notifyChatResponseAvailability } from "./chat-response-availability.js"; import { notifyChatConfirmationWork } from "./chat-confirmation-notifications.js"; import { notifyChatInboundWakeNotices } from "./chat-inbound-wake-notice-notifications.js"; @@ -6,9 +7,13 @@ import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notificatio /** Lineage changes can authorize descendants, so invalidate the issue rather * than just the run whose row changed. Configuration may invalidate a company. */ -export type ChatRunMilestoneScope = { companyId: string; issueId?: string }; +export type ChatRunMilestoneScope = { companyId: string; issueId?: string; runId?: string }; export async function notifyChatRunMilestones(tx: object, scope: ChatRunMilestoneScope) { + if (scope.issueId || scope.runId) { + await notifyNativeResponsePresentation(tx, { companyId: scope.companyId, issueId: scope.issueId, + runId: scope.issueId ? undefined : scope.runId }); + } await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatGitHubReviewChecks, JSON.stringify({ companyId: scope.companyId })); await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatGitHubTerminalResponses, JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId })); await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatRunMilestones, JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId })); @@ -21,3 +26,8 @@ export async function notifyChatRunMilestones(tx: object, scope: ChatRunMileston // Native presentation and run ownership can unblock a parked acknowledgement. await notifyChatResponseAvailability(tx, scope); } + +export async function notifyChatCompanyAuthorityChange(tx: object, scope: { companyId: string }) { + await notifyChatRunMilestones(tx, scope); + await notifyNativeResponsePresentation(tx, scope); +} diff --git a/server/src/services/chat-work-notifications.ts b/server/src/services/chat-work-notifications.ts index 9b878913e8d..433f75de4a6 100644 --- a/server/src/services/chat-work-notifications.ts +++ b/server/src/services/chat-work-notifications.ts @@ -1,5 +1,5 @@ import { notifyChatResponseAvailability } from "./chat-response-availability.js"; -import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js"; +import { notifyChatRunMilestones, notifyChatCompanyAuthorityChange } from "./chat-run-milestone-notifications.js"; import { notifyChatConnectionConfiguration } from "./chat-channels/configuration-work-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; @@ -47,7 +47,7 @@ export async function notifyChatVerificationWork(tx: object): Promise { export async function notifyChatEndpointWork(tx: object, connectionId: string, companyId: string): Promise { // Configuration/status transitions can restore post-control publication // authority. Ordinary health checks use the narrower configuration signals. - await notifyChatRunMilestones(tx, { companyId }); + await notifyChatCompanyAuthorityChange(tx, { companyId }); await notifyChatDeliveryWork(tx); await notifyChatPublicationWork(tx); for (const queue of Object.values(CHAT_ACTION_QUEUES)) await notifyDeliveryWork(tx, queue); diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index aa160fed3dc..413c91f3b76 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -2,6 +2,7 @@ import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db"; /** Topics for existing durable queues; register intent before their writes. */ export const DELIVERY_QUEUES = { + nativeResponsePresentation: "native-response-presentation", nativePendingEffects: "native-pending-effects", unsafeWorkspaceExportRecovery: "unsafe-workspace-export-recovery", automaticCompletionReviews: "automatic-completion-reviews", diff --git a/server/src/services/durable-chat-wakeup.ts b/server/src/services/durable-chat-wakeup.ts index 65a150f8a26..2870eac88ba 100644 --- a/server/src/services/durable-chat-wakeup.ts +++ b/server/src/services/durable-chat-wakeup.ts @@ -21,6 +21,21 @@ const committedChatResponseAuthorities = new WeakMap< CommittedChatResponseAuthority >(); +const committedChatResponseAuthorityListeners = new WeakMap void>>(); + +export function onCommittedChatResponseAuthorityRegistered(db: Db, listener: () => void): () => void { + let listeners = committedChatResponseAuthorityListeners.get(db); + if (!listeners) { + listeners = new Set(); + committedChatResponseAuthorityListeners.set(db, listeners); + } + listeners.add(listener); + return () => { + listeners.delete(listener); + if (listeners.size === 0 && committedChatResponseAuthorityListeners.get(db) === listeners) committedChatResponseAuthorityListeners.delete(db); + }; +} + export class CommittedChatResponseAuthorizationError extends HttpError { constructor() { super( @@ -41,6 +56,7 @@ export function registerCommittedChatResponseAuthority( authority: CommittedChatResponseAuthority, ): () => void { committedChatResponseAuthorities.set(db, authority); + for (const listener of committedChatResponseAuthorityListeners.get(db) ?? []) listener(); return () => { if (committedChatResponseAuthorities.get(db) === authority) { committedChatResponseAuthorities.delete(db); diff --git a/server/src/services/execution-control-reconciliation.ts b/server/src/services/execution-control-reconciliation.ts index 7c0f0c4eb18..3bbe612e612 100644 --- a/server/src/services/execution-control-reconciliation.ts +++ b/server/src/services/execution-control-reconciliation.ts @@ -1,4 +1,4 @@ -import { notifyNativePendingEffects } from "./native-pending-effect-notifications.js"; +import { notifyNativeFinalizationWork } from "./native-finalization-notifications.js"; import { notifyUnsafeWorkspaceExportRunChange } from "./unsafe-workspace-export-notifications.js"; import { notifyCancelledWorkspacePreservation } from "./cancelled-workspace-preservation-notifications.js"; import { notifyInstructionCollection } from "./instruction-collection-notifications.js"; @@ -147,7 +147,7 @@ export async function reconcileAbandonedExecutionControl( } // A persisted result belongs to the existing finalizer, not a replacement provider. if (coordinator?.resultId) { - await notifyNativePendingEffects(tx, { companyId: run.companyId, runId: run.id }); + await notifyNativeFinalizationWork(tx, { companyId: run.companyId, runId: run.id }); await tx .update(nativeRunFinalizations) .set({ diff --git a/server/src/services/heartbeat-run-events.ts b/server/src/services/heartbeat-run-events.ts index 53e0bd26206..b6b1b694a49 100644 --- a/server/src/services/heartbeat-run-events.ts +++ b/server/src/services/heartbeat-run-events.ts @@ -1,3 +1,4 @@ +import { isNativeResponseEvidence, notifyNativeResponsePresentation } from "./native-response-presentation-notifications.js"; import { isSafeNativeChatProgressEventType } from "./safe-native-chat-progress.js"; import { notifyChatNativeProgress } from "./chat-native-progress-notifications.js"; import { boundRunIssueIdSql } from "./run-task-binding.js"; @@ -72,7 +73,8 @@ export async function appendHeartbeatRunEvent( input: AppendHeartbeatRunEventInput, ): Promise { const projectsProgress = isSafeNativeChatProgressEventType(input.eventType); - if (!input.nativeSource && !input.retryExhaustion && !projectsProgress) { + const projectsResponse = isNativeResponseEvidence(input.eventType, input.payload?.prpEvent); + if (!input.nativeSource && !input.retryExhaustion && !projectsProgress && !projectsResponse) { const allocated = db.$with("allocated_run_event", {}).as(sql` update ${heartbeatRuns} set next_event_seq = ${heartbeatRuns.nextEventSeq} + 1, @@ -101,7 +103,8 @@ export async function appendHeartbeatRunEvent( .select({ companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, // Only progress needs the task binding. Other native/recovery events // must not read a potentially large context snapshot to append a log. - ...(projectsProgress ? { runtimeMode: heartbeatRuns.runtimeMode, issueId: boundRunIssueIdSql() } : {}) }) + ...(projectsProgress || projectsResponse ? { runtimeMode: heartbeatRuns.runtimeMode } : {}), + ...(projectsProgress ? { issueId: boundRunIssueIdSql() } : {}) }) .from(heartbeatRuns) .where(eq(heartbeatRuns.id, input.runId)) .for("update") @@ -175,6 +178,9 @@ export async function appendHeartbeatRunEvent( if (projectsProgress && run.runtimeMode === "native" && run.issueId) { await notifyChatNativeProgress(tx, { companyId: run.companyId, issueId: run.issueId }); } + if (projectsResponse && run.runtimeMode === "native") { + await notifyNativeResponsePresentation(tx, { companyId: run.companyId, runId: input.runId }); + } const seq = await allocateHeartbeatRunEventSeq( tx as unknown as Db, input.runId, diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 7d4630a41f2..141c1d6d7d9 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -1,3 +1,4 @@ +import { registerNativeResponsePresentationWork } from "./native-runtime/native-response-presentation-work.js"; import { registerNativePendingEffectWork } from "./native-runtime/native-pending-effect-work.js"; import { registerUnsafeWorkspaceExportWork } from "./native-runtime/native-workspace-export-recovery.js"; import { registerAutomaticCompletionReviewWork } from "./native-runtime/automatic-completion-reviews.js"; @@ -4089,6 +4090,8 @@ export function heartbeatService( triggerIssueMonitor, reportRunActivity: clearDetachedRunWarning, + registerNativeResponsePresentationWork: (coordinator: Parameters[1], canRun: () => boolean) => + registerNativeResponsePresentationWork(db, coordinator, () => canRun() && readTaskDrain(new Date()) === null), registerNativePendingEffectWork: (coordinator: Parameters[1], canRun: () => boolean) => registerNativePendingEffectWork(db, coordinator, () => canRun() && readTaskDrain(new Date()) === null), registerUnsafeWorkspaceExportWork: (coordinator: Parameters[1], canRun: () => boolean) => diff --git a/server/src/services/heartbeat/queue.ts b/server/src/services/heartbeat/queue.ts index 4d93f0fade8..2be0fd457f7 100644 --- a/server/src/services/heartbeat/queue.ts +++ b/server/src/services/heartbeat/queue.ts @@ -976,7 +976,8 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende .where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.status, "queued"))) .for("update"); if (!currentRun) return null; - await notifyChatRunMilestones(tx, { companyId: run.companyId }); + await notifyChatRunMilestones(tx, { companyId: run.companyId, runId: run.id, + issueId: run.nativeIssueId ?? run.issueId ?? readNonEmptyString(parseObject(run.contextSnapshot).issueId) ?? undefined }); await notifyInstructionCollection(tx, { companyId: run.companyId, runId: run.id }); await notifyCancelledWorkspacePreservation(tx, { companyId: run.companyId, runId: run.id }); await notifyUnsafeWorkspaceExportRunChange(tx, run); @@ -1625,7 +1626,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende )).for("update"); if (!lockedRun || lockedRun.status !== "queued") return null; await withExternalAdmissionGuard(claimTx, run.companyId, run.agentId, () => assertNoExternalOverlap(claimTx, run.companyId, run.agentId)); - await notifyChatRunMilestones(claimTx, { companyId: run.companyId, issueId: issueId ?? undefined }); + await notifyChatRunMilestones(claimTx, { companyId: run.companyId, runId: run.id, issueId: issueId ?? undefined }); const claimedRun = await claimTx.update(heartbeatRuns).set(claimValues).where(and( eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.status, "queued"), )).returning().then((rows) => rows[0] ?? null); @@ -1726,7 +1727,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende const [candidate] = await tx.select({ companyId: heartbeatRuns.companyId, contextSnapshot: heartbeatRuns.contextSnapshot }) .from(heartbeatRuns).where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.status, "running"))).for("update"); if (!candidate) return; - await notifyChatRunMilestones(tx, { companyId: candidate.companyId, + await notifyChatRunMilestones(tx, { companyId: candidate.companyId, runId, issueId: readNonEmptyString(parseObject(candidate.contextSnapshot).issueId) ?? undefined }); const released = await tx .update(heartbeatRuns) diff --git a/server/src/services/heartbeat/recovery.ts b/server/src/services/heartbeat/recovery.ts index 2e1b8675580..a6fbf3432d7 100644 --- a/server/src/services/heartbeat/recovery.ts +++ b/server/src/services/heartbeat/recovery.ts @@ -1947,7 +1947,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD let settledProcessesDiscovered = false; - async function reapOrphanedRuns(opts?: { staleThresholdMs?: number; skipScheduledRetryCancellation?: boolean; skipCapturedInstructionCommit?: boolean; skipInstructionDirectoryCleanup?: boolean; skipStoppedInstructionCollection?: boolean; skipCancelledWorkspacePreservation?: boolean; skipObsoletePolicyReviews?: boolean; skipAutomaticCompletionReviews?: boolean; skipUnsafeWorkspaceExports?: boolean; skipNativePendingEffects?: boolean }) { + async function reapOrphanedRuns(opts?: { staleThresholdMs?: number; skipScheduledRetryCancellation?: boolean; skipCapturedInstructionCommit?: boolean; skipInstructionDirectoryCleanup?: boolean; skipStoppedInstructionCollection?: boolean; skipCancelledWorkspacePreservation?: boolean; skipObsoletePolicyReviews?: boolean; skipAutomaticCompletionReviews?: boolean; skipUnsafeWorkspaceExports?: boolean; skipNativePendingEffects?: boolean; skipNativeResponsePresentation?: boolean }) { if (!opts?.skipCancelledWorkspacePreservation) await preserveCancelledNativeWorkspaces(db); const staleThresholdMs = opts?.staleThresholdMs ?? 0; const now = new Date(); @@ -1969,6 +1969,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD skipAutomaticCompletionReviews: opts?.skipAutomaticCompletionReviews, skipUnsafeWorkspaceExports: opts?.skipUnsafeWorkspaceExports, skipNativePendingEffects: opts?.skipNativePendingEffects, + skipNativeResponsePresentation: opts?.skipNativeResponsePresentation, environmentRuntime, pluginWorkerManager: dependencies.pluginWorkerManager, onWorkspaceSettled: settleRecoveredNativeWorkspace, diff --git a/server/src/services/heartbeat/run-completion.ts b/server/src/services/heartbeat/run-completion.ts index 1e5e13b858c..39bb4a509ab 100644 --- a/server/src/services/heartbeat/run-completion.ts +++ b/server/src/services/heartbeat/run-completion.ts @@ -351,7 +351,7 @@ export async function recordRunPresentationDecision(db: Db, input: { companyId: string; runId: string; issueId?: string; decision: RunPresentationDecision; }) { await db.transaction(async tx => { - await notifyChatRunMilestones(tx, { companyId: input.companyId, issueId: input.issueId }); + await notifyChatRunMilestones(tx, { companyId: input.companyId, runId: input.runId, issueId: input.issueId }); await tx.update(heartbeatRuns).set({ resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${JSON.stringify({ presentationDecision: input.decision })}::jsonb`, updatedAt: new Date(), diff --git a/server/src/services/heartbeat/run-lifecycle.test.ts b/server/src/services/heartbeat/run-lifecycle.test.ts index 1360bafe0c3..6d862394ed2 100644 --- a/server/src/services/heartbeat/run-lifecycle.test.ts +++ b/server/src/services/heartbeat/run-lifecycle.test.ts @@ -1,7 +1,7 @@ import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js"; import { randomBytes } from "node:crypto"; import type { Db } from "@paperclipai/db"; -import { agents, agentWakeupRequests, companies, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRunEvents, heartbeatRuns } from "@paperclipai/db"; +import { agents, agentWakeupRequests, companies, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRunEvents, heartbeatRuns, issues, subscribeDatabaseWork } from "@paperclipai/db"; import { eq, sql } from "drizzle-orm"; import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; @@ -241,6 +241,23 @@ describe.skipIf(!support.supported)("heartbeat lifecycle database wiring", () => } finally { unsubscribe(); } }); + it.each(["native", "legacy", "unbound"] as const)("keeps %s run-status response recovery scoped to its task or run", async binding => { + const { run } = await fixture(); + const [issue] = await db.insert(issues).values({ companyId: run.companyId, title: "Scope owner" }).returning(); + await db.update(heartbeatRuns).set(binding === "native" ? { nativeIssueId: issue.id } + : binding === "legacy" ? { contextSnapshot: { issueId: issue.id } } : { contextSnapshot: {} }).where(eq(heartbeatRuns.id, run.id)); + const hints: unknown[] = []; + const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.nativeResponsePresentation, (event, scopes) => { + if (event === "settled") for (const hint of scopes ?? []) hints.push(JSON.parse(hint)); + }); + try { + await createHeartbeatLifecycle(db, callbacks(db)).setRunStatusIfRunning(run.id, "succeeded", { finishedAt: new Date() }); + expect(hints).toEqual([binding === "unbound" ? { companyId: run.companyId, runId: run.id } + : { companyId: run.companyId, issueId: issue.id }]); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0].status).toBe("succeeded"); + } finally { unsubscribe(); } + }); + it("does not request accounting for an unchanged running status", async () => { const { run } = await fixture(); const settled = vi.fn(); diff --git a/server/src/services/heartbeat/run-lifecycle.ts b/server/src/services/heartbeat/run-lifecycle.ts index 00f664d4769..0904b58a42f 100644 --- a/server/src/services/heartbeat/run-lifecycle.ts +++ b/server/src/services/heartbeat/run-lifecycle.ts @@ -1017,7 +1017,7 @@ export function createHeartbeatLifecycle(db: Db, dependencies: HeartbeatLifecycl return recordHeartbeatRunRuntimeProgress(currentRun, update, issueId); } - async function writeRunStatus(status: string, previous: Pick | null, + async function writeRunStatus(status: string, previous: Pick | null, write: (tx: Db) => Promise) { const companyId = previous?.companyId; // Register on the outer transaction before the terminal write. An unknown @@ -1031,7 +1031,8 @@ export function createHeartbeatLifecycle(db: Db, dependencies: HeartbeatLifecycl if (previous) await notifyCancelledWorkspacePreservation(tx, { companyId: previous.companyId, runId: previous.id }); if (isHeartbeatRunTerminalStatus(status)) await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, companyId); await notifyExecutionRunWork(tx); - if (companyId) await notifyChatRunMilestones(tx, { companyId }); + if (previous) await notifyChatRunMilestones(tx, { companyId: previous.companyId, runId: previous.id, + issueId: previous.nativeIssueId ?? previous.issueId ?? readNonEmptyString(parseObject(previous.contextSnapshot).issueId) ?? undefined }); return write(tx); }); } diff --git a/server/src/services/native-finalization-notifications.ts b/server/src/services/native-finalization-notifications.ts new file mode 100644 index 00000000000..f14f1e79db8 --- /dev/null +++ b/server/src/services/native-finalization-notifications.ts @@ -0,0 +1,10 @@ +import { notifyNativeResponsePresentation } from "./native-response-presentation-notifications.js"; +import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; + +export type NativeFinalizationWorkScope = { companyId: string; runId?: string; issueId?: string }; + +export async function notifyNativeFinalizationWork(transaction: object, scope: NativeFinalizationWorkScope) { + if (scope.issueId || scope.runId) await notifyNativeResponsePresentation(transaction, scope); + await notifyDeliveryWork(transaction, DELIVERY_QUEUES.nativePendingEffects, + JSON.stringify({ companyId: scope.companyId, runId: scope.runId })); +} diff --git a/server/src/services/native-pending-effect-notifications.ts b/server/src/services/native-pending-effect-notifications.ts deleted file mode 100644 index e9c28d79782..00000000000 --- a/server/src/services/native-pending-effect-notifications.ts +++ /dev/null @@ -1,8 +0,0 @@ -import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; - -export type NativePendingEffectScope = { companyId: string; runId?: string }; - -export async function notifyNativePendingEffects(transaction: object, scope: NativePendingEffectScope) { - await notifyDeliveryWork(transaction, DELIVERY_QUEUES.nativePendingEffects, - JSON.stringify({ companyId: scope.companyId, runId: scope.runId })); -} diff --git a/server/src/services/native-response-presentation-notifications.ts b/server/src/services/native-response-presentation-notifications.ts new file mode 100644 index 00000000000..8be14daae95 --- /dev/null +++ b/server/src/services/native-response-presentation-notifications.ts @@ -0,0 +1,14 @@ +import { readCompletedAssistantMessageCandidate } from "./heartbeat-run-summary.js"; +import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; + +export type NativeResponsePresentationScope = { companyId: string; issueId?: string; runId?: string; agentId?: string }; + +export async function notifyNativeResponsePresentation(tx: object, scope: NativeResponsePresentationScope) { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.nativeResponsePresentation, + JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId, runId: scope.runId, agentId: scope.agentId })); +} + +export function isNativeResponseEvidence(eventType: string, prpEvent: unknown): boolean { + return eventType === "run.result.accepted" || eventType === "item.completed" && + readCompletedAssistantMessageCandidate({ seq: 0, prpEvent })?.channel === "final"; +} diff --git a/server/src/services/native-runtime/native-finalization-reconciler.ts b/server/src/services/native-runtime/native-finalization-reconciler.ts index 4797142552e..5fc5567ea2e 100644 --- a/server/src/services/native-runtime/native-finalization-reconciler.ts +++ b/server/src/services/native-runtime/native-finalization-reconciler.ts @@ -644,6 +644,7 @@ type NativeReconciliationOptions = { skipAutomaticCompletionReviews?: boolean; skipUnsafeWorkspaceExports?: boolean; skipNativePendingEffects?: boolean; + skipNativeResponsePresentation?: boolean; pluginWorkerManager?: WorkspaceWorkers; environmentRuntime?: EnvironmentRuntimeService; onWorkspaceSettled?: (input: { @@ -861,7 +862,7 @@ async function reconcileNativeFinalizationsPass( )).limit(1).then((entries) => entries[0] ?? null) : null; const currentDecisionJson = record(currentDecision?.decisionJson); - if (row.coordinatorPhase === "committed") { + if (row.coordinatorPhase === "committed" && !options.skipNativeResponsePresentation) { // A later decision can supersede this run's task status, but cannot // erase its accepted, still-authorized response. Repair presentation // before the status-only early return below, without rerunning work. @@ -1123,6 +1124,7 @@ async function reconcileNativeFinalizationsPass( runId: row.runId, workspaceFinalizeStatus, projectRunStatus: true, + skipNativeResponsePresentation: options.skipNativeResponsePresentation, preserveProviderAttempt: Boolean( readNativeWorkspaceSyncReference( row.nativeWorkspaceSync, @@ -1181,6 +1183,7 @@ async function reconcileNativeFinalizationsPass( runId: row.runId, workspaceFinalizeStatus: barrier.status as "succeeded" | "failed", projectRunStatus: true, + skipNativeResponsePresentation: options.skipNativeResponsePresentation, preserveProviderAttempt: Boolean( readNativeWorkspaceSyncReference( row.nativeWorkspaceSync, diff --git a/server/src/services/native-runtime/native-interaction-bridge.ts b/server/src/services/native-runtime/native-interaction-bridge.ts index 680ab867033..4243bcfa6a3 100644 --- a/server/src/services/native-runtime/native-interaction-bridge.ts +++ b/server/src/services/native-runtime/native-interaction-bridge.ts @@ -1,4 +1,4 @@ -import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js"; +import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js"; import type { Db } from "@paperclipai/db"; import type { NativeInteractionResponseEnvelope } from "../../vendor/paperclip-runner/index.js"; @@ -633,7 +633,7 @@ export async function routePersistedNativeResultAttention(input: { const delegation = effects.find((effect) => effect.kind === "create_delegated_issue"); const finalizationError = effects.find((effect) => effect.kind === "record_finalization_error"); await input.db.transaction(async tx => { - await notifyNativePendingEffects(tx, { companyId: binding.companyId, runId: input.runId }); + await notifyNativeFinalizationWork(tx, { companyId: binding.companyId, runId: input.runId }); await tx.update(nativeRunFinalizations).set({ phase: finalizationError ? "retryable_failure" : "committed", assessmentId: existingAssessment.id, diff --git a/server/src/services/native-runtime/native-pending-effect-work.ts b/server/src/services/native-runtime/native-pending-effect-work.ts index 5893f2eadf3..3f276a0b636 100644 --- a/server/src/services/native-runtime/native-pending-effect-work.ts +++ b/server/src/services/native-runtime/native-pending-effect-work.ts @@ -2,7 +2,7 @@ import { and, asc, eq, exists, gt, inArray, isNotNull, ne } from "drizzle-orm"; import { heartbeatRuns, issues, nativeRunFinalizations, statusDecisionEffects, type Db } from "@paperclipai/db"; import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js"; import { DELIVERY_QUEUES } from "../delivery-work-notifications.js"; -import type { NativePendingEffectScope } from "../native-pending-effect-notifications.js"; +import type { NativeFinalizationWorkScope } from "../native-finalization-notifications.js"; import { registerScopedDeadlineWork } from "../scoped-deadline-work.js"; import { dispatchPendingNativeStatusEffects, nativePendingEffectDueAt } from "./status-decision-committer.js"; @@ -15,7 +15,7 @@ export function pendingNativeStatusEffectsCondition(db: Db) { ))); } -function findPendingEffects(db: Db, scope?: NativePendingEffectScope, after?: string) { +function findPendingEffects(db: Db, scope?: NativeFinalizationWorkScope, after?: string) { return db.select({ id: nativeRunFinalizations.runId, companyId: nativeRunFinalizations.companyId, issueId: nativeRunFinalizations.issueId, decisionId: nativeRunFinalizations.decisionId, @@ -36,7 +36,7 @@ function findPendingEffects(db: Db, scope?: NativePendingEffectScope, after?: st export function registerNativePendingEffectWork(db: Db, coordinator: ReturnType, canRun: () => boolean) { type Entry = { id: string; companyId: string; dueAt: number }; - async function load(scope?: NativePendingEffectScope, signal?: AbortSignal): Promise { + async function load(scope?: NativeFinalizationWorkScope, signal?: AbortSignal): Promise { const entries: Entry[] = []; let after: string | undefined; while (canRun() && !signal?.aborted) { @@ -49,8 +49,8 @@ export function registerNativePendingEffectWork(db: Db, coordinator: ReturnType< } return registerScopedDeadlineWork(coordinator, { queue: DELIVERY_QUEUES.nativePendingEffects, batchSize: 25, - decodeScope: (hint: string): NativePendingEffectScope | undefined => hint === "*" ? undefined : JSON.parse(hint), - matches: (entry: Entry, scope?: NativePendingEffectScope) => !scope || entry.companyId === scope.companyId && (!scope.runId || entry.id === scope.runId), + decodeScope: (hint: string): NativeFinalizationWorkScope | undefined => hint === "*" ? undefined : JSON.parse(hint), + matches: (entry: Entry, scope?: NativeFinalizationWorkScope) => !scope || entry.companyId === scope.companyId && (!scope.runId || entry.id === scope.runId), load, check: async (entry: Entry, signal?: AbortSignal) => { if (!canRun() || signal?.aborted) throw new Error("Native pending effect recovery stopped"); diff --git a/server/src/services/native-runtime/native-response-presentation-work.ts b/server/src/services/native-runtime/native-response-presentation-work.ts new file mode 100644 index 00000000000..6f545e2c923 --- /dev/null +++ b/server/src/services/native-runtime/native-response-presentation-work.ts @@ -0,0 +1,117 @@ +import { and, asc, eq, gt, inArray, isNotNull, isNull, notExists, or, sql } from "drizzle-orm"; +import { heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults, statusDecisions, statusDecisionEffects, type Db } from "@paperclipai/db"; +import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js"; +import { DELIVERY_QUEUES } from "../delivery-work-notifications.js"; +import type { NativeResponsePresentationScope } from "../native-response-presentation-notifications.js"; +import { onCommittedChatResponseAuthorityRegistered } from "../durable-chat-wakeup.js"; +import { registerScopedDeadlineWork } from "../scoped-deadline-work.js"; +import { recoverCommittedNativeChatResponse } from "./native-run-finalizer.js"; + +export function registerNativeResponsePresentationWork(db: Db, + coordinator: ReturnType, canRun: () => boolean) { + return nativeResponsePresentationWork(db).register(coordinator, canRun); +} + +type Entry = { id: string; companyId: string; issueId: string; agentId: string; dueAt: number }; + +export function nativeResponsePresentationWork(db: Db) { + function candidates(scope?: NativeResponsePresentationScope, after?: string, runId?: string) { + return db.select({ id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, + issueId: nativeRunFinalizations.issueId, agentId: heartbeatRuns.agentId, nextAttemptAt: nativeRunFinalizations.nextAttemptAt, + }).from(nativeRunFinalizations) + .innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, nativeRunFinalizations.runId), + eq(heartbeatRuns.companyId, nativeRunFinalizations.companyId), eq(heartbeatRuns.nativeIssueId, nativeRunFinalizations.issueId))) + .innerJoin(issues, and(eq(issues.id, nativeRunFinalizations.issueId), eq(issues.companyId, nativeRunFinalizations.companyId))) + .innerJoin(nativeRunResults, and(eq(nativeRunResults.id, nativeRunFinalizations.resultId), + eq(nativeRunResults.companyId, nativeRunFinalizations.companyId), eq(nativeRunResults.issueId, nativeRunFinalizations.issueId), + eq(nativeRunResults.runId, heartbeatRuns.id), eq(nativeRunResults.completionContractId, heartbeatRuns.completionContractId))) + .innerJoin(statusDecisions, and(eq(statusDecisions.id, nativeRunFinalizations.decisionId), + eq(statusDecisions.companyId, nativeRunFinalizations.companyId), eq(statusDecisions.issueId, nativeRunFinalizations.issueId), + eq(statusDecisions.runId, heartbeatRuns.id), eq(statusDecisions.assessmentId, nativeRunFinalizations.assessmentId))) + .where(and( + eq(heartbeatRuns.runtimeMode, "native"), inArray(heartbeatRuns.status, ["succeeded", "failed", "timed_out"]), + isNotNull(heartbeatRuns.finishedAt), eq(nativeRunFinalizations.phase, "committed"), isNull(nativeRunFinalizations.leaseOwner), + eq(nativeRunResults.schemaStatus, "accepted"), + sql`jsonb_typeof(${heartbeatRuns.resultJson} #> '{presentationDecision,commentId}') is distinct from 'string'`, + sql`coalesce(${heartbeatRuns.resultJson} #> '{presentationDecision,commentAction}', '"create"'::jsonb) = '"create"'::jsonb`, + sql`${heartbeatRuns.contextSnapshot}->'skipIssueComment' is distinct from 'true'::jsonb`, + sql`coalesce(${statusDecisions.decisionJson}->'externalChatReviewPresentation', 'null'::jsonb) in ('null'::jsonb, 'false'::jsonb, '0'::jsonb, '""'::jsonb)`, + sql`${nativeRunResults.resultJson} #>> '{result,schema}' = 'paperclip.run_result.v1'`, + sql`${nativeRunResults.resultJson} #>> '{terminal,runTerminalState}' = 'succeeded'`, + sql`${nativeRunResults.resultJson} #>> '{terminal,turnTerminalState}' = 'completed'`, + or( + and(eq(heartbeatRuns.status, "succeeded"), + sql`(coalesce(${issues.conversationAgentId}::text, '') = '' or coalesce(${issues.conversationUserId}, '') = '')`, + sql`${nativeRunResults.resultJson} #>> '{result,reportedWorkDisposition}' = 'done'`, + sql`${nativeRunResults.resultJson} #>> '{terminal,reportedWorkDisposition}' = 'done'`), + and(sql`${nativeRunResults.resultJson} #>> '{result,reportedWorkDisposition}' = 'yielded'`, + sql`${nativeRunResults.resultJson} #>> '{terminal,reportedWorkDisposition}' = 'yielded'`, + sql`jsonb_typeof(${nativeRunResults.resultJson} #> '{result,summary}') = 'string' + and ${nativeRunResults.resultJson} #>> '{result,summary}' ~ '[^[:space:]]'`, + sql`jsonb_typeof(${nativeRunResults.resultJson} #> '{result,continuation,idempotencyKey}') = 'string' + and ${nativeRunResults.resultJson} #>> '{result,continuation,idempotencyKey}' ~ '[^[:space:]]'`, + or( + and(sql`${nativeRunResults.resultJson} #>> '{result,continuation,kind}' = 'response_wake'`, + sql`${nativeRunResults.resultJson} #> '{result,attentionRequests}' = '[]'::jsonb`, + sql`jsonb_typeof(${nativeRunResults.resultJson} #> '{result,continuation,summary}') = 'string' + and ${nativeRunResults.resultJson} #>> '{result,continuation,summary}' ~ '[^[:space:]]'`), + and(eq(heartbeatRuns.status, "succeeded"), or( + and(sql`${nativeRunResults.resultJson} #>> '{result,continuation,kind}' = 'same_agent'`, + eq(statusDecisions.reasonCode, "live_continuation_registered")), + and(sql`${nativeRunResults.resultJson} #>> '{result,continuation,kind}' = 'monitor'`, + inArray(statusDecisions.reasonCode, ["scheduled_monitor_waiting", "scheduled_monitor_dispatched", "scheduled_monitor_preserved"])), + )), + )), + ), + notExists(db.select({ id: statusDecisionEffects.id }).from(statusDecisionEffects).where(and( + eq(statusDecisionEffects.companyId, nativeRunFinalizations.companyId), + eq(statusDecisionEffects.issueId, nativeRunFinalizations.issueId), + eq(statusDecisionEffects.decisionId, nativeRunFinalizations.decisionId), + inArray(statusDecisionEffects.deliveryState, ["pending", "failed"]), + ))), + scope ? and(eq(heartbeatRuns.companyId, scope.companyId), scope.issueId ? eq(nativeRunFinalizations.issueId, scope.issueId) : undefined) : undefined, + scope?.runId ? eq(heartbeatRuns.id, scope.runId) : undefined, + scope?.agentId ? eq(heartbeatRuns.agentId, scope.agentId) : undefined, + after ? gt(heartbeatRuns.id, after) : undefined, runId ? eq(heartbeatRuns.id, runId) : undefined, + )).orderBy(asc(heartbeatRuns.id)).limit(100); + } + async function load(scope?: NativeResponsePresentationScope, signal?: AbortSignal, canRun = () => true): Promise { + const entries: Entry[] = []; + let after: string | undefined; + while (canRun() && !signal?.aborted) { + const rows = await candidates(scope, after); + for (const row of rows) entries.push({ id: row.id, companyId: row.companyId, issueId: row.issueId, agentId: row.agentId, dueAt: row.nextAttemptAt?.getTime() ?? 0 }); + if (rows.length < 100) return entries; + after = rows[rows.length - 1].id; + } + throw new Error("Native response presentation discovery stopped"); + } + async function check(entry: Entry, signal?: AbortSignal): Promise { + if (signal?.aborted) throw new Error("Native response presentation recovery stopped"); + const scope = { companyId: entry.companyId, issueId: entry.issueId }; + const [current] = await candidates(scope, undefined, entry.id); + if (!current) return null; + if (current.nextAttemptAt && current.nextAttemptAt.getTime() > Date.now()) return { ...entry, dueAt: current.nextAttemptAt.getTime() }; + if (signal?.aborted) throw new Error("Native response presentation recovery stopped"); + const recoveryStartedAt = Date.now(); + await recoverCommittedNativeChatResponse(db, { ...scope, runId: entry.id, onlyWhenDue: true }); + const [remaining] = await candidates(scope, undefined, entry.id); + return remaining?.nextAttemptAt && remaining.nextAttemptAt.getTime() > recoveryStartedAt + ? { ...entry, dueAt: remaining.nextAttemptAt.getTime() } : null; + } + function register(coordinator: ReturnType, canRun: () => boolean) { + const worker = registerScopedDeadlineWork(coordinator, { + queue: DELIVERY_QUEUES.nativeResponsePresentation, + decodeScope: (hint: string): NativeResponsePresentationScope | undefined => hint === "*" ? undefined : JSON.parse(hint), + matches: (entry: Entry, scope?: NativeResponsePresentationScope) => !scope || entry.companyId === scope.companyId && (!scope.issueId || entry.issueId === scope.issueId) && (!scope.runId || entry.id === scope.runId) && (!scope.agentId || entry.agentId === scope.agentId), + load: (scope, signal) => load(scope, signal, canRun), + check: (entry, signal) => { + if (!canRun()) throw new Error("Native response presentation recovery paused"); + return check(entry, signal); + }, + }, canRun); + const unsubscribe = onCommittedChatResponseAuthorityRegistered(db, () => worker.invalidate()); + return { ...worker, stop: unsubscribe }; + } + return { load, check, register }; +} diff --git a/server/src/services/native-runtime/native-run-coordinator-store.ts b/server/src/services/native-runtime/native-run-coordinator-store.ts index 593f5c0b4b2..1f911a854b3 100644 --- a/server/src/services/native-runtime/native-run-coordinator-store.ts +++ b/server/src/services/native-runtime/native-run-coordinator-store.ts @@ -1,4 +1,5 @@ -import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js"; +import { isNativeResponseEvidence, notifyNativeResponsePresentation } from "../native-response-presentation-notifications.js"; +import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js"; import { isSafeNativeChatProgressEventType } from "../safe-native-chat-progress.js"; import { notifyChatNativeProgress } from "../chat-native-progress-notifications.js"; @@ -297,6 +298,9 @@ export class NativeRunCoordinatorStore { if (isSafeNativeChatProgressEventType(event.eventType)) { await notifyChatNativeProgress(tx, { companyId: this.#binding.companyId, issueId: this.#binding.issueId }); } + if (isNativeResponseEvidence(event.eventType, canonicalPayload)) { + await notifyNativeResponsePresentation(tx, this.#binding); + } const cursor = run.nextEventSeq; const [inserted] = await tx .insert(heartbeatRunEvents) @@ -523,7 +527,7 @@ export class NativeRunCoordinatorStore { ) { throw new Error("native_finalization_lease_conflict"); } - if (row.decisionId) await notifyNativePendingEffects(tx, this.#binding); + if (row.decisionId) await notifyNativeFinalizationWork(tx, this.#binding); if ( row.leaseOwner === input.leaseOwner && row.leaseExpiresAt && @@ -566,7 +570,7 @@ export class NativeRunCoordinatorStore { throw new Error("native_finalization_retry_invalid"); } const [updated] = await this.#db.transaction(async tx => { - await notifyNativePendingEffects(tx, this.#binding); + await notifyNativeFinalizationWork(tx, this.#binding); if (input.failureCode === "native_workspace_sync_out_unsafe_archive") { await notifyUnsafeWorkspaceExportRecovery(tx, this.#binding); } diff --git a/server/src/services/native-runtime/native-run-finalizer.ts b/server/src/services/native-runtime/native-run-finalizer.ts index f6b93c9699f..9df2211a496 100644 --- a/server/src/services/native-runtime/native-run-finalizer.ts +++ b/server/src/services/native-runtime/native-run-finalizer.ts @@ -1,4 +1,4 @@ -import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js"; +import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js"; import { notifyUnsafeWorkspaceExportRunChange } from "../unsafe-workspace-export-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js"; import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js"; @@ -39,6 +39,7 @@ import { nativeRunFinalizations, nativeRunResults, statusDecisions, + statusDecisionEffects, workAssessments, workspaceOperations, } from "@paperclipai/db"; @@ -215,7 +216,7 @@ async function claimCoordinator(input: { coordinator.leaseOwner !== leaseOwner ) throw new Error("native_finalization_lease_busy"); - if (coordinator.decisionId) await notifyNativePendingEffects(tx, coordinator); + if (coordinator.decisionId) await notifyNativeFinalizationWork(tx, coordinator); const [updated] = await tx .update(nativeRunFinalizations) .set({ @@ -383,7 +384,7 @@ async function recordRetryableFailure(input: { ? "native_workspace_sync_out_retry_exhausted" : "native_finalization_retry_exhausted" : input.failureCode; - await notifyNativePendingEffects(tx, { companyId: input.run.companyId, runId: input.run.id }); + await notifyNativeFinalizationWork(tx, { companyId: input.run.companyId, runId: input.run.id }); if (phase === "terminal_failure") await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, input.run.companyId); if (phase === "terminal_failure" && failureCode === "native_workspace_sync_out_unsafe_archive") { await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: input.run.companyId, issueId: input.coordinator.issueId }); @@ -848,6 +849,21 @@ async function acceptedResponseDigestMatches( export async function repairCommittedNativeChatResponse( db: Db, input: { companyId: string; issueId: string; runId: string }, +): Promise { + try { + return await recoverCommittedNativeChatResponse(db, input); + } catch (error) { + logger.warn( + { runId: input.runId, error }, + "Committed chat response is awaiting presentation retry", + ); + return false; + } +} + +export async function recoverCommittedNativeChatResponse( + db: Db, + input: { companyId: string; issueId: string; runId: string; onlyWhenDue?: boolean }, ): Promise { let agentId: string | null = null; let materialized = false; @@ -909,9 +925,11 @@ export async function repairCommittedNativeChatResponse( return false; // An existing selected answer, including a subsequently deleted one, must // never be replaced or resurrected by a background presentation repair. + const recordedPresentation = record(record(run.resultJson).presentationDecision); if ( - typeof record(record(run.resultJson).presentationDecision).commentId === - "string" + typeof recordedPresentation.commentId === "string" || + (recordedPresentation.commentAction !== undefined && recordedPresentation.commentAction !== "create") || + record(run.contextSnapshot).skipIssueComment === true ) return false; const [coordinator] = await tx @@ -933,6 +951,16 @@ export async function repairCommittedNativeChatResponse( !coordinator.assessmentId ) return false; + if (input.onlyWhenDue) { + if (coordinator.nextAttemptAt && coordinator.nextAttemptAt.getTime() > Date.now()) return false; + const [pendingEffect] = await tx.select({ id: statusDecisionEffects.id }).from(statusDecisionEffects).where(and( + eq(statusDecisionEffects.companyId, input.companyId), + eq(statusDecisionEffects.issueId, input.issueId), + eq(statusDecisionEffects.decisionId, coordinator.decisionId), + inArray(statusDecisionEffects.deliveryState, ["pending", "failed"]), + )).limit(1); + if (pendingEffect) return false; + } const [accepted] = await tx .select() .from(nativeRunResults) @@ -1123,13 +1151,7 @@ export async function repairCommittedNativeChatResponse( }); } catch (error) { if (error instanceof CommittedChatResponseAuthorizationError) return false; - // Keep transient failures visible and retryable without reclassifying the - // committed run or acknowledging a publication that never committed. - logger.warn( - { runId: input.runId, error }, - "Committed chat response is awaiting presentation retry", - ); - return false; + throw error; } if (materialized && agentId) publishChatPublicationCommitSignal({ @@ -1167,6 +1189,7 @@ export async function finalizeNativeRun(input: { workspaceFinalizeStatus: "succeeded" | "failed"; /** Reconciliation owns terminal run projection; the live heartbeat does it afterward. */ projectRunStatus?: boolean; + skipNativeResponsePresentation?: boolean; /** Workspace-only replay must not consume the provider recovery budget. */ preserveProviderAttempt?: boolean; failpoint?: NativeStatusCommitFailpoint; @@ -1220,7 +1243,7 @@ export async function finalizeNativeRun(input: { await projectCommittedRun({ db: input.db, run: admittedRun, coordinator }); if (input.projectRunStatus && !presentationAlreadyMaterialized) await materializeCommittedReviewResponse(input.db, input.runId); - if (input.projectRunStatus) + if (input.projectRunStatus && !input.skipNativeResponsePresentation) await repairCommittedNativeChatResponse(input.db, { companyId: admittedRun.companyId, issueId: coordinator.issueId, @@ -1721,7 +1744,7 @@ export async function finalizeNativeRun(input: { } if (input.projectRunStatus) await materializeCommittedReviewResponse(input.db, input.runId); - if (input.projectRunStatus && finalizationPhase === "committed") + if (input.projectRunStatus && finalizationPhase === "committed" && !input.skipNativeResponsePresentation) await repairCommittedNativeChatResponse(input.db, { companyId: run.companyId, issueId: coordinator.issueId, diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index c854d1a2e45..4e73bfcfa45 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -1,4 +1,4 @@ -import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js"; +import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js"; import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js"; import { notifyInstructionCollection } from "../instruction-collection-notifications.js"; @@ -3898,7 +3898,7 @@ export async function reconcileRetainedNativeSessionCleanup( renameSync(owned.root, archive); } renameSync(copy, owned.root); - await notifyNativePendingEffects(tx, { companyId: owned.run.companyId, runId: owned.run.id }); + await notifyNativeFinalizationWork(tx, { companyId: owned.run.companyId, runId: owned.run.id }); await tx .update(nativeRunFinalizations) .set({ @@ -3956,7 +3956,7 @@ export async function reconcileRetainedNativeSessionCleanup( .limit(1) .then((rows) => rows[0]); if (!current || current.leaseOwner !== leaseOwner) return; - await notifyNativePendingEffects(tx, { companyId: owned.run.companyId, runId: owned.run.id }); + await notifyNativeFinalizationWork(tx, { companyId: owned.run.companyId, runId: owned.run.id }); await tx .update(nativeRunFinalizations) .set({ @@ -9700,7 +9700,7 @@ async function executePaperclipNativeSessionWithinScope( ) .for("update"); await notifyExecutionRunWork(tx); - await notifyNativePendingEffects(tx, input.execution.binding); + await notifyNativeFinalizationWork(tx, input.execution.binding); const updated = await tx .update(nativeRunFinalizations) .set({ @@ -10036,7 +10036,7 @@ async function executePaperclipNativeSessionWithinScope( } const releaseNow = new Date(); const released = await timeController("coordinator_release", () => input.db.transaction(async tx => { - await notifyNativePendingEffects(tx, input.execution.binding); + await notifyNativeFinalizationWork(tx, input.execution.binding); return tx.update(nativeRunFinalizations) .set({ leaseOwner: null, diff --git a/server/src/services/native-runtime/native-workspace-export-recovery.ts b/server/src/services/native-runtime/native-workspace-export-recovery.ts index 8974e68a6e6..c638f3b3e94 100644 --- a/server/src/services/native-runtime/native-workspace-export-recovery.ts +++ b/server/src/services/native-runtime/native-workspace-export-recovery.ts @@ -1,4 +1,4 @@ -import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js"; +import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js"; import { alias } from "drizzle-orm/pg-core"; import { notifyUnsafeWorkspaceExportRecovery, type UnsafeWorkspaceExportScope } from "../unsafe-workspace-export-notifications.js"; import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js"; @@ -101,7 +101,7 @@ async function recoverCandidate(db: Db, candidate: { runId: string; companyId: s phase: "workspace_finalize", status: "succeeded", exitCode: 0, finishedAt: now, metadata: { owningService: "native_workspace_finalizer", workspaceSync: { omitted: true, reason: "restore_unsafe_archive", legacy: true } }, }); - await notifyNativePendingEffects(tx, { companyId: run.companyId, runId: run.id }); + await notifyNativeFinalizationWork(tx, { companyId: run.companyId, runId: run.id }); await tx.update(nativeRunFinalizations).set({ phase: "result_accepted", failureCode: null, nextAttemptAt: null, failureDetail: { workspaceFinalizeAttempt: 0, legacyUnsafeExportOmitted: { resultId: result.id, recoveredAt: now.toISOString() } }, updatedAt: now, }).where(eq(nativeRunFinalizations.runId, run.id)); diff --git a/server/src/services/native-runtime/native-workspace-export-retry.ts b/server/src/services/native-runtime/native-workspace-export-retry.ts index 69485cb4d74..99e286a17ff 100644 --- a/server/src/services/native-runtime/native-workspace-export-retry.ts +++ b/server/src/services/native-runtime/native-workspace-export-retry.ts @@ -1,4 +1,4 @@ -import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js"; +import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js"; import { notifyEnvironmentCleanup } from "../environment-cleanup-notifications.js"; import { randomUUID } from "node:crypto"; import type { Environment, EnvironmentLease } from "@paperclipai/shared"; @@ -135,7 +135,7 @@ export async function retryNativeWorkspaceExport(input: { const now = new Date(); const retry = { ...receipt, requestId: randomUUID(), actorId: input.actorId, repairNote: input.repairNote, requestedAt: now.toISOString(), stoppedProvider: initial.lease.metadata?.remoteExecutionTermination }; - await notifyNativePendingEffects(tx, { companyId: current.run.companyId, runId: current.run.id }); + await notifyNativeFinalizationWork(tx, { companyId: current.run.companyId, runId: current.run.id }); await tx.update(nativeRunFinalizations).set({ phase: "result_accepted", failureCode: null, nextAttemptAt: null, failureDetail: { ...current.coordinator.failureDetail, workspaceFinalizeAttempt: 0, workspaceExportRetry: retry }, updatedAt: now }).where(eq(nativeRunFinalizations.runId, current.run.id)); await tx.update(heartbeatRuns).set({ status: "running", finishedAt: null, error: null, errorCode: null, diff --git a/server/src/services/native-runtime/paperclip-control-plane-port.test.ts b/server/src/services/native-runtime/paperclip-control-plane-port.test.ts index 57d49cff0aa..b6bec7a4214 100644 --- a/server/src/services/native-runtime/paperclip-control-plane-port.test.ts +++ b/server/src/services/native-runtime/paperclip-control-plane-port.test.ts @@ -1,3 +1,5 @@ +import { nativeResponsePresentationWork } from "./native-response-presentation-work.js"; +import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js"; import { afterAll, beforeAll, describe, expect, it } from "vitest"; import { asc, eq, sql } from "drizzle-orm"; import { generateKeyPairSync, randomUUID } from "node:crypto"; @@ -65,6 +67,8 @@ import { subscribeCompanyLiveEvents, } from "../live-events.js"; +const external = resolveExternalTestDatabaseUrl("PAPERCLIP_CONTROL_PLANE_TEST_DATABASE_URL"); + describe("PaperclipControlPlanePort conformance", () => { let temporary: Awaited> | null = null; let db: ReturnType; @@ -87,8 +91,11 @@ describe("PaperclipControlPlanePort conformance", () => { const governanceRunnerId = "00000000-0000-4000-8000-000000000021"; beforeAll(async () => { - temporary = await startRecoveryPostgres("paperclip-native-port-"); - db = createDb(temporary.connectionString); + if (external) db = createDb(external); + else { + temporary = await startRecoveryPostgres("paperclip-native-port-"); + db = createDb(temporary.connectionString); + } const identity = CONTROL_PLANE_CONFORMANCE_OPEN.identity; await db.insert(companies).values({ id: identity.companyId, name: "Phase 6", issuePrefix: "P6C" }); await db.insert(agents).values({ @@ -645,7 +652,7 @@ describe("PaperclipControlPlanePort conformance", () => { completionContractSha256: localContractSha, contextSnapshot: { issueId }, }); - const observerDb = createDb(temporary!.connectionString); + const observerDb = createDb(external ?? temporary!.connectionString); const observed: Array> = []; const committedCallbacks: string[] = []; let visibilityCheck: Promise> | null = null; @@ -1096,7 +1103,10 @@ describe("PaperclipControlPlanePort conformance", () => { .where(eq(heartbeatRuns.id, taskRunId)); expect(await repair()).toBe(false); await db.update(heartbeatRuns).set({ contextSnapshot: {} }).where(eq(heartbeatRuns.id, taskRunId)); - expect(await repair()).toBe(true); + const work = nativeResponsePresentationWork(db); + const [entry] = await work.load({ companyId: identity.companyId, runId: taskRunId }); + expect(entry?.id).toBe(taskRunId); + expect(await work.check(entry)).toBeNull(); expect(await repair()).toBe(false); expect(await db.select({ body: issueComments.body }).from(issueComments) .where(eq(issueComments.createdByRunId, taskRunId)).orderBy(asc(issueComments.createdAt))) diff --git a/server/src/services/native-runtime/status-decision-committer.ts b/server/src/services/native-runtime/status-decision-committer.ts index 751734d70fb..04696bc1c85 100644 --- a/server/src/services/native-runtime/status-decision-committer.ts +++ b/server/src/services/native-runtime/status-decision-committer.ts @@ -1,4 +1,4 @@ -import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js"; +import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js"; import { notifyAutomaticCompletionReviews } from "../automatic-completion-review-notifications.js"; import { notifyObsoletePolicyReviews } from "../obsolete-policy-review-notifications.js"; @@ -154,7 +154,7 @@ export async function dispatchPendingNativeStatusEffects(input: { targetId: effect.targetId, }); } - await notifyNativePendingEffects(tx, input); + await notifyNativeFinalizationWork(tx, input); await notifyChatRunMilestones(tx, input); await tx .update(statusDecisionEffects) @@ -1241,7 +1241,7 @@ async function materializeDecisionEffect(input: { ) .for("update"); if (pending.length === 0) throw new Error("native_pending_effect_missing"); - await notifyNativePendingEffects(input.tx, { companyId: input.companyId }); + await notifyNativeFinalizationWork(input.tx, { companyId: input.companyId, issueId: input.issue.id }); await input.tx .update(statusDecisionEffects) .set({ @@ -1931,7 +1931,7 @@ export async function commitNativeStatusDecision(input: { effect => effect.kind === "record_finalization_error", ); await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, input.companyId); - await notifyNativePendingEffects(tx, input); + await notifyNativeFinalizationWork(tx, input); await tx .update(nativeRunFinalizations) .set({ diff --git a/server/src/services/preserved-result-finalization.ts b/server/src/services/preserved-result-finalization.ts index 083e84fefe6..69c47de6ac5 100644 --- a/server/src/services/preserved-result-finalization.ts +++ b/server/src/services/preserved-result-finalization.ts @@ -1,4 +1,4 @@ -import { notifyNativePendingEffects } from "./native-pending-effect-notifications.js"; +import { notifyNativeFinalizationWork } from "./native-finalization-notifications.js"; import { and, desc, eq, gte, ne, or, sql } from "drizzle-orm"; import { agentWakeupRequests, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db"; import { readSystemRecovery } from "./system-recovery-state.js"; @@ -46,7 +46,7 @@ export async function resumePreservedResultFinalization(db: Db, companyId: strin const generationId = intent?.id ?? recovery.generationId; if (coordinator.recoveryHistory.some(entry => entry.kind === "preserved_result_finalization" && entry.generationId === generationId)) return false; - await notifyNativePendingEffects(tx, { companyId, runId }); + await notifyNativeFinalizationWork(tx, { companyId, runId }); await tx.update(nativeRunFinalizations).set({ phase: coordinator.assessmentId ? "arbitrating" : "workspace_finalizing", nextAttemptAt: new Date(), failureCode: null, failureDetail: null, updatedAt: new Date(),