From df6156c2e37f542a69f393dde31e357b82b7cac1 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 29 Oct 2026 03:40:01 -0500 Subject: [PATCH] fix(server): wake native status reassessment from committed changes (#17709) ## Thinking Path > - Paperclip manages agents, tasks and their durable native results. > - Changed evidence, task authority or review retirement can require a new status assessment. > - Periodic orphan recovery repeatedly reads history to find these changes. > - This change gives reassessment an owner driven by committed changes and saved deadlines. > - Existing result, assessment, decision and task rows remain durable truth. > - The status committer still verifies current sources and authority before writing. > - Settled or insufficient evidence becomes quiet until a supported producer changes it. ## Linked Issues or Issue Description Refs #17702 (merged), #17679 and #17680. This is slice 50, now based on master. It owns native status reassessment; workspace settlement, retained sessions and other orphan duties remain separate. **What happened?** Periodic recovery repeatedly inspects completed native results. Removing that scan without complete writer coverage or current authority checks could strand new evidence or commit an old result over newer work. **Expected behavior** Reassess from existing durable facts after relevant outer transactions commit. Retain real retry/lease deadlines and restart/drain recovery. Preserve current authority, quality-review and supersession checks. Make no recurring SQL when work is settled or dormant. **Steps to reproduce** Complete a native result awaiting evidence or review, leave it idle, then approve a real work product or retire the selected automatic review. Its saved result should advance once when authorized, or record the existing nonterminal business gate, without a periodic scan. ## What Changed - Extract reassessment and its pure status policy from finalization recovery into their owning modules. Add a scoped worker using the existing delivery coordinator and scheduler; no queue table or migration is added. - Retain the three durable discovery triggers: changed work-product history, authoritative task status/version, or retirement of the selected automatic completion review. Preserve selected-source, ownership, effects, retry and lease fences under the existing committer. - Notify from actual task/work-product, claim release, approval, interaction, OAuth, event evidence and workspace-settlement owners before their transaction writes. Keep provider calls outside short transactions and nested writes behind outer commit. - Scope ordinary milestone/finalization/coalesced-input wakes to an issue/run. Keep genuine company authority restoration explicit. Real Photon and coalescer regressions cover all three response/status queues. - Move the current parent's required quality policy into the owner and apply it to every reassessment that could complete a task. An unmet review records the existing in-progress decision and next action, avoiding an operational retry loop. Preserve conversation/session reset locks and the workspace omission-barrier signal. - Skip reassessment in periodic recovery while preserving explicit/startup recovery and the historical supersession guard before independent workspace work. Document the source contract, producer inventory, failures and qualifications in the linked audit. ## Verification - Broad source `9fddb3f0b1157df32b64fb1f22e9d6e9624470e8` passes **2731 distinct cases in 28 complete files**, with no skips: 2458 producer/owner/authority/workspace/quality cases in 394.43 seconds and all 273 governed-response cases in 53.74 seconds. Nineteen dedicated fresh PostgreSQL 17 databases migrate through 0413; eighteen embedded-only invocations preserve fresh-per-call isolation through the existing recovery helper. All 22 temporary imports restore exactly and all 28 hashes match. - Final runtime source `5ea3b0fd016bf7e63ba82a7044825a46a0cd2082` passes **1211 cases in five complete affected files** in 108.23 seconds after the narrowly reproduced evidence-trigger fix: all 43 worker, 145 corpus, 32 finalization-recovery, 968 native-session and 23 quality-review cases. Four fresh databases plus the quality helper's own fresh fixture use PostgreSQL 17. Its one adapted import restores exactly; all five hashes match. The other 27 broad-run test files are unchanged. These qualifications are source-scoped and not cumulatively double-counted. - Required/optional real regressions reproduce retry loops on retired-review and evidence-approval paths before repair; both become SQL quiet afterward with unchanged assertions/time limits. Controlled removal of only ordinary scope guards reproduces two coarse-hint SQL failures; exact guard restoration passes. Real producer, outer transaction, claim, authority, 103-item backlog, retry and drain cases pass. Photon transport is mocked; callbacks, locks and observed SQL are real. - Normal merges preserve 12661 parent/unrelated entries, 35 owned entries and 21 automatic shared results, with five original conflicts reviewed explicitly. The parent's actual squash reconciliation preserves the entire held child tree. The final quality fix changes only its owner and tests. Fourteen added comments are removed with identical nontrivia tokens; no directives are removed and the final comment check is empty. - Database build, server types/build, boundaries and exact-head migration order pass. The database tree is identical to merged #17702: no new SQL, queue table or snapshot is generated. Earlier benchmarks, host limitations, failed assertions and their repairs remain in `doc/plans/2026-10-26-native-status-reassessment-sleep-audit.md` and `doc/idle-native-status-reassessment.md`. - This qualifies targeted external PostgreSQL behavior, not embedded bootstrap on the semaphore-exhausted host, a full local repository suite, deployed sleep or measured fleet savings. Fresh exact-head CI and review remain required. ## Risks - Notifications retain the process-local supported-writer contract. Independent SQL writers and uncoordinated replicas are not covered by a new transport. - Reassessment competes with current authority changes. Existing row/source locks and retries remain required; arbitrary missing identities have no row to lock, so this is not a general predicate-lock guarantee. - Discovery pages 100 rows but retains outstanding candidates in memory. It remains proportional to backlog size, and cancellation occurs between queries. - Other orphan/workspace/session duties and Cloud sleep eligibility remain unchanged. Missing supported producer coverage could strand dormant work, so the producer audit and real callback/transaction tests remain part of acceptance. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, shell execution and PostgreSQL tests. The runtime does not expose a more specific serving snapshot or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted scope is stated above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip Co-authored-by: Codex --- doc/idle-native-status-reassessment.md | 100 ++ ...2026-10-24-recurring-server-sleep-stack.md | 176 +++ ...-native-status-reassessment-sleep-audit.md | 1084 +++++++++++++++++ .../heartbeat-run-event-sequencing.test.ts | 29 +- server/src/__tests__/issues-service.test.ts | 10 +- .../low-trust-red-team-routes.test.ts | 11 +- .../native-status-arbiter-corpus.test.ts | 529 +++++++- .../native-status-reassessment-work.test.ts | 640 ++++++++++ .../photon/channel.integration.test.ts | 6 +- .../server-startup-feedback-export.test.ts | 10 +- .../src/__tests__/tool-access-service.test.ts | 40 + ...orkspace-operations-reconciliation.test.ts | 9 +- server/src/index.ts | 4 +- .../active-run-watchdog/adapters/postgres.ts | 2 + .../wake-queue/adapters/postgres.test.ts | 2 +- .../modules/wake-queue/adapters/postgres.ts | 2 + .../adapters/queued-comment-postgres.ts | 2 + server/src/routes/issues.ts | 2 + server/src/services/access.ts | 2 + .../services/approval-work-notifications.ts | 4 +- .../chat-run-milestone-notifications.ts | 6 +- .../services/delivery-work-notifications.ts | 1 + .../execution-control-reconciliation.ts | 2 + .../services/execution-recovery-resolution.ts | 2 + server/src/services/heartbeat-run-events.ts | 10 +- server/src/services/heartbeat.ts | 3 + server/src/services/heartbeat/queue.ts | 4 + server/src/services/heartbeat/recovery.ts | 3 +- .../src/services/issue-thread-interactions.ts | 43 +- server/src/services/issue-tree-control.ts | 2 + server/src/services/issues.ts | 42 +- .../src/services/legacy-execution-recovery.ts | 4 + .../native-finalization-notifications.ts | 2 + .../native-review-state-notifications.ts | 7 + .../automatic-completion-reviews.ts | 2 + .../chat-attachment-reuse.test.ts | 7 +- .../native-runtime/completion-contracts.ts | 23 +- .../native-runtime/evidence-classifier.ts | 60 +- .../native-finalization-reconciler.ts | 338 +---- .../native-runtime/native-governance.ts | 18 +- .../native-reconciliation-status.ts | 113 ++ .../native-runtime/native-restart-recovery.ts | 2 + .../native-runtime/native-safe-replacement.ts | 2 + .../native-runtime/native-session-executor.ts | 2 + .../native-status-reassessment-work.ts | 117 ++ .../native-status-reassessment.ts | 266 ++++ .../native-workspace-export-recovery.ts | 4 +- ...native-workspace-finalization-ownership.ts | 16 +- ...e-workspace-finalization-recovery-state.ts | 6 +- .../native-workspace-finalizer.test.ts | 38 +- .../native-workspace-owner-recovery.ts | 6 +- .../runner-task-attachments.test.ts | 11 +- .../status-decision-committer.ts | 78 +- .../native-runtime/work-assessments.ts | 34 +- ...ative-status-reassessment-notifications.ts | 8 + server/src/services/recovery/service.ts | 2 + .../services/satisfied-connection-intents.ts | 4 +- .../tool-access/oauth/authorization-start.ts | 123 +- .../tool-access/oauth/cloud-callback.ts | 4 +- .../tool-access/oauth/direct-callback.ts | 4 +- .../src/services/tool-access/oauth/state.ts | 4 +- server/src/services/work-products.ts | 10 + server/src/services/workspace-operations.ts | 21 +- .../src/services/workspace-recovery.test.ts | 30 +- server/src/services/workspace-recovery.ts | 2 + 65 files changed, 3639 insertions(+), 511 deletions(-) create mode 100644 doc/idle-native-status-reassessment.md create mode 100644 doc/plans/2026-10-26-native-status-reassessment-sleep-audit.md create mode 100644 server/src/__tests__/native-status-reassessment-work.test.ts create mode 100644 server/src/services/native-review-state-notifications.ts create mode 100644 server/src/services/native-runtime/native-reconciliation-status.ts create mode 100644 server/src/services/native-runtime/native-status-reassessment-work.ts create mode 100644 server/src/services/native-runtime/native-status-reassessment.ts create mode 100644 server/src/services/native-status-reassessment-notifications.ts diff --git a/doc/idle-native-status-reassessment.md b/doc/idle-native-status-reassessment.md new file mode 100644 index 00000000000..b9875615653 --- /dev/null +++ b/doc/idle-native-status-reassessment.md @@ -0,0 +1,100 @@ +# Native status reassessment without idle polling + +Native status reassessment uses committed changes and the shared delivery +scheduler. It no longer runs on every periodic orphan-recovery pass. + +## What creates work + +Existing native result, assessment, decision, task and review rows remain the +source of truth. No new queue table or database migration is required. + +| Change | Existing producer boundary | +| --- | --- | +| Work-product creation, update, import, removal, attachment or trusted promotion | Work-product/task services and the promotion transaction | +| Task status/version, assignment, execution state or blocker configuration | Task mutation transaction | +| Direct claim release, terminal/stale lock cleanup, queue release, recovery or tree restoration | Existing task-scoped claim mutation transaction | +| Automatic review retirement | Automatic completion review owner | +| Approval/link or interaction outcome, including OAuth card creation/reopening | Approval/review-state notification boundaries | +| Accepted/rejected control-plane run evidence | Run-event append transaction | +| Workspace barrier or physical ownership settlement | Workspace operation, finalizer and verified recovery owners | +| Coordinator claim/retry, effect settlement or run projection | Native finalization and run-milestone boundaries | + +These signals request inspection. They do not grant permission to complete a +task. Discovery retains the three existing conditions: changed work-product +history, changed authoritative task status/version, or a retired selected +completion review. The owner still classifies the selected result and appends +assessments and decisions through the existing status committer. + +A settled worker makes no recurring queries. Insufficient evidence is dormant. +Real contention or operational failure keeps a retry. A saved retry or lease +expiry keeps its actual future deadline. Startup reconstructs candidates in +100-row pages; the coordinator processes bounded batches and retains changes +through drain/resume and concurrent dispatch. Shutdown uses its existing owner. + +## Authority and separate responsibilities + +Scheduled commit checks the selected source, current task/run, retry and lease, +pending effects and native process ownership inside the status transaction. +Retired-review completion also checks the latest contract, workspace barrier, +physical workspace ownership, governance and dependencies. Evidence is +reclassified under source locks. Contention defers instead of blocking an idle +worker indefinitely or committing stale completion. + +The legacy reconciler retains explicit recovery and a one-time startup pass. +Its periodic call skips reassessment but preserves the superseded historical +source guard before independent workspace recovery. Workspace/result settlement, +retained session cleanup, native wake delivery and generic orphan recovery are +separate duties; their recurring scans have not been removed by this slice. + +Case-handoff delivery and lease changes do not improve evidence: every persisted +non-cancelled state already counts as accepted. Cancellation is negative evidence +and is fenced by the source lock at scheduled commit. Creation generates the +handoff identity inside its transaction before returning it. No reassessment wake +is added to every lease renewal. Target-agent status is also not a classification +trigger: attention validation checks existence, not whether an agent is invokable. +Creating an agent does not satisfy result criteria or verification. Missing +arbitrary target identities have no row to lock; this change does not provide a +general predicate-lock guarantee for independent identity insertion. + +Pending child completion is a guard and a fresh parent-turn obligation. It is not +silently consumed by completing the old result. New contracts supersede the old +result's eligibility; creating one does not revive that old result. Direct SQL +writers and independent replicas do not receive process-local notifications. +The supported producer model remains the active server and integrated transaction +owners; this change does not add a distributed notification transport. + +## Evidence and limits + +Current qualification is recorded in the [audit](plans/2026-10-26-native-status-reassessment-sleep-audit.md). +The broad 9fddb source passes 2731 cases in 28 complete files. The final 5ea3 +runtime source, after the evidence-trigger quality fix, passes 1211 cases in five +complete affected files. Their overlapping counts are not added together. Real +producer/transaction coverage includes first writes, rollback/outer commit, +authority races, saved deadlines, drain/resume, claim release and retained history. +The final worker suite includes all 43 cases. + +Required quality review remains mandatory on both retired-review and evidence +reassessment. An unmet review produces its existing nonterminal decision and +review recovery action, rather than repeatedly attempting completion as an +operational retry. Real required/optional regressions and controlled coarse-hint +negatives fail before repair and pass afterward without changing time limits. + +Historical discovery measurements in the audit retain their original revision: +10,000 selected-history tasks return no company candidates in about 35 ms; +10,000 real candidates require 101 queries and about 1.56 seconds to discover. +This is discovery cost, not complete backlog execution. The current worker still +retains candidates in memory proportional to backlog size. Its permanent +103-item lifecycle case verifies settlement across batches and SQL quiet. +Cancellation is checked between pages, not during an already-running SQL call. + +Database build, server typecheck/build, module boundaries and migration-order +checks pass. The host's embedded PostgreSQL bootstrap remains limited by +exhausted System V semaphores; acceptance uses fresh owned PostgreSQL 17 databases +and preserves embedded-only fresh-per-call isolation through the existing recovery +helper. Adapted test imports restore exactly; assertions are unchanged. No full +local repository-suite or live external-provider qualification is claimed. + +Remote exact-head CI and review still gate merging. The +[stack ledger](plans/2026-10-24-recurring-server-sleep-stack.md) tracks other duties. +Worker quiet does not establish database suspension, Cloud alarm coverage, host +sleep or fleet cost savings. diff --git a/doc/plans/2026-10-24-recurring-server-sleep-stack.md b/doc/plans/2026-10-24-recurring-server-sleep-stack.md index 3b5b3b9f45b..ff918b88dfc 100644 --- a/doc/plans/2026-10-24-recurring-server-sleep-stack.md +++ b/doc/plans/2026-10-24-recurring-server-sleep-stack.md @@ -2876,3 +2876,179 @@ between pages on drain. Fresh migration and interrupted/concurrent replay pass. The audit preserves the initial query, migration and test-clock failures. Full local tests are running; remote acceptance, older stack findings, master reconciliation and every remaining original duty are still open. Nothing merged. + + +## Slice 49 publication; slice 50 owner audit + +Draft #17702 at `be6ca6151b` is open on #17696. Greptile reviewed that head +at 5/5 with zero unresolved threads. CI remains in progress. The full local +suite was stopped after repeated embedded PostgreSQL bootstrap failures and +many skipped DB suites. One isolated native-session-executor failure reproduces +in startEmbeddedPostgresTestDatabase before recovery behavior runs; logs are +/tmp/sleep49-full-tests.log and /tmp/sleep49-full-failure-isolated.log. The +255-case real-PostgreSQL owner/startup selection, workspace types/build and +migration replay remain passing. This does not make full local acceptance green. + +Slice 50 audits native status reassessment: changed evidence, authoritative task +state and retired automatic reviews all feed the existing append-only assessment +and decision owner. Its producer/guard inventory is in the dated reassessment +audit. No implementation or periodic removal is claimed for slice 50 yet. Earlier +stack gates, master reconciliation and the full remaining scope remain open. + + +Slice 49 remote acceptance: #17702 at +`be6ca6151bb04e25f691a830734ddb5856701d27` now has all 53 checks passing, +Greptile 5/5 at that exact head and zero unresolved review threads. It remains +draft and unmerged. /tmp/sleep49-final-gates.json, /tmp/sleep49-comments.json +and /tmp/sleep49-review-threads.json preserve the observations. The PR body +records the passing remote gates and keeps the local embedded-database failure +visible. Slice 50 and all other remaining original obligations are incomplete. + + +Slice 50 extraction checkpoint: reassessment and its pure policy now have native +runtime owners, with the original reconciler delegating to them. All 94 status +corpus cases pass before and after extraction; all 187 governed-response cases +and server types pass afterward. No polling duty has been removed. The audit +records the critical transaction boundary: scheduled guards belong inside the +status committer, which publishes after its own transaction. Producer coverage, +worker selection, lifecycle/cost proof and a new PR remain outstanding. + +Slice 50 scheduled-commit checkpoint: nine competing source/lease/deadline/effect/ +ownership cases reproduce unsafe scheduled reassessment before the new guard. +The final 103-case status corpus and server types pass. The audit preserves two +initial invalid fixtures and the before-fix constraint rejection. The guard is +opt-in at the existing committer transaction; no outer transaction or early +publication is added. Worker discovery, producer/lifecycle/cost evidence and +periodic removal remain outstanding. No new PR or merge is claimed; all original +remaining duties and older stack gates remain open. + +Slice 50 scoped-worker checkpoint: discovery/check/register now use the shared +scheduler, with distinct dormant and retryable outcomes and exact saved deadlines. +A real retirement missed wake and a near-term deadline delay were reproduced and +fixed. Final local verification passes 116 status/worker cases plus 187 governed +response consumer cases; server types pass. Driver SQL capture proves quiet after +settlement and during drain for the tested path. Initial history cost measurements +and their synthetic-fixture limits are recorded in the owner audit. The worker is +not registered in startup and periodic reassessment remains. Producer coverage, +concurrent authority audit, selected-current/backlog cost and full acceptance are +still pending; no slice 50 PR or merge, and the full original scope stays open. + +Slice 50 producer checkpoint: six real work-product/attachment transaction misses +and three task mutation misses were reproduced and repaired. The real promotion +HTTP route also wakes reassessment. All 13 owner lifecycle tests, 79 affected +consumer cases across their documented runs, and server types pass. Initial +embedded-PostgreSQL failures and the external-database test repairs are preserved +in the owner audit. Approval/interaction/run evidence and workspace producers, +concurrent authority, backlog cost, startup wiring and periodic removal remain. +Master remains 1fbbef45a8; older stack gates and the full original scope stay open. + +Slice 50 authority checkpoint: two real commit-boundary regressions reproduced +stale completion after the source run failed or another run took task execution. +The scheduled retired-review path now rechecks both facts under existing locks. +All 115 status-corpus cases pass. A retained-history startup assertion failed in +the combined run; after waiting for bounded startup batches while enforcing the +saved retry deadline, all 28 lifecycle cases and server types pass. The owner +audit preserves failures and handoff/contract producer findings. Remaining +business-fact races, producer coverage, cost, wiring, periodic removal and PR +acceptance are still open. Nothing merged; the full stack goal is incomplete. + +Slice 50 contract checkpoint: a newer committed completion contract reproduced +stale completion; a real uncommitted writer reproduced blocking on the task lock. +The contract owner now exposes a nonblocking current-revision guard, acquired +before status row locks and held through commit. All 145 corpus/lifecycle tests +and server types pass. Failure logs and the lock-order rationale are recorded in +the owner audit. Workspace/governance races, producer coverage, cost, startup +integration, periodic removal, remote acceptance and the rest of the full stack +remain open. Nothing merged. + +Slice 50 workspace-barrier checkpoint: three stale-completion regressions now +pass after checking and share-locking the current barrier inside status commit. +The check uses existing row/FK serialization and NOWAIT contention handling. +All 148 status/lifecycle cases and server types pass. The owner audit preserves +before-fix failures and distinguishes barrier receipts from physical workspace +ownership, which still needs audit. Governance/dependencies, remaining producers, +cost, wiring, periodic removal and PR acceptance remain open; nothing merged. + +Slice 50 physical-owner checkpoint: four unsafe-completion regressions and a +subsequent release missed wake were reproduced and repaired. Scheduled retired- +review reassessment respects the existing physical lock and durable owner receipt; +normal/verified owner release wakes it through committed intent. All 217 status, +lifecycle and workspace-owner consumer cases and server types pass. Existing +orphan/stop-proof safeguards remain verified. Governance/dependencies, remaining +producers, cost, integration and remote gates remain open, along with all other +original duties and earlier stack findings. Nothing merged. + +Slice 50 governance checkpoint: seven stale-completion races reproduced changed +execution policy, approvals, interactions and question history. The governance +owner now supplies nonblocking source locks; status commit verifies the gate +used by the selected decision. All 159 status/lifecycle cases pass, the valid +question-history fixture passes on recheck, and server types pass. Dependency and +remaining authority/producer audits, task-history/query costs, wiring, periodic +removal and remote gates remain outstanding. The full stack remains incomplete; +nothing merged. + + +Slice 50 dependency checkpoint: seven stale-completion races are reproduced and +fixed through the existing dependency-readiness owner. Scheduled status commit +locks/rechecks prerequisite and workspace-finalization sources with nonblocking +contention handling; ordinary readers retain their behavior. All 181 distinct +status/lifecycle/dependency consumer cases and server types pass. The audit keeps +the initial unordered assertion and misconfigured embedded-test invocation +failures and their repairs. Remaining authority/producer coverage, cost, startup +integration, periodic removal, remote gates, earlier stack findings and the full +original scope remain open. Nothing merged. + + +Slice 50 evidence checkpoint: seven races reproduced completion using revoked, +removed or concurrently edited evidence. The assessment owner now reclassifies +and locks referenced evidence inside scheduled status commit; changed facts and +contention defer through existing retry handling. All 177 status/lifecycle/ +classifier consumer cases and server types pass. The audit preserves before-fix +failures and the remaining identity/producer and cost limits. Startup wiring, +periodic removal, remote gates, older stack findings and all remaining recurring +work are still incomplete. Nothing merged. + + +Slice 50 OAuth producer checkpoint: authorization-card creation/reopening now +signals both native review workers from the same short transaction as its card +and state link. Three missing-settlement-signal regressions were reproduced; +all 472 tool-access and 30 worker lifecycle tests plus server typecheck pass. +The audit retains before/after evidence and the remaining handoff/authority +coverage limits. Master is freshly verified at 1fbbef45a8. Discovery/cost proof, +startup wiring, periodic removal, remote acceptance, earlier stack findings and +the full original recurring-work scope remain incomplete. Nothing merged. + + +Slice 50 scale checkpoint: 10,000 separate selected completed task/run histories +produce zero company candidates (~35 ms); 10,000 actual candidates reconstruct +in 101 bounded queries (~1.56 seconds/2.05 MB), with verified drain/abort behavior. +A retired-review task with large review/comment/approval/dependency/workspace +history settles in ~126 ms and disappears from discovery. The audit preserves +two invalid benchmark-fixture attempts and the corrected fresh-task evidence. +A permanent 103-item backlog test proves real settlement and SQL quiet; all 31 +lifecycle cases and server types pass. Remaining producer/authority audit, +startup integration, periodic removal, remote gates and the full original stack +scope remain open. Nothing merged. + + +Slice 50 startup checkpoint: enabled maintenance now registers the reassessment +worker and periodic recovery skips its classification/decision path. The separate +historical supersession guard is preserved before workspace recovery. Two missing +integration regressions fail before the change; all 246 corpus/startup/lifecycle +cases, 32 existing finalization/cleanup consumer cases, and server types pass after. +The owner audit records the intentional separation of reassessment from independent +workspace settlement and the retained failure evidence. Remaining producer/authority +audit, full acceptance, publication, remote gates and all other original duties +remain incomplete. Nothing merged; the goal remains active. + + +Slice 50 is draft [#17709](https://github.com/paperclipai/paperclip/pull/17709), +based on #17702. Initial head 91a6c14170 passed all 52 CI checks; Greptile 4/5 +found direct claim-release writers missing reassessment notifications. Six real +regressions reproduced the gap. Task-scoped signals now cover their existing +transactions; the repaired selection passes 978 cases and server types, with +15 embedded-DB runtime cases excluded for the diagnosed host semaphore exhaustion. +The owner audit preserves the wrong initial ordinary-release completion assertion +and its authority-preserving correction. Fresh-head gates remain pending; nothing +merged. Earlier E2E failures on #17490/#17572 and voice findings on #17576/#17580 +were revalidated and remain unresolved. All original remaining duties stay in scope. diff --git a/doc/plans/2026-10-26-native-status-reassessment-sleep-audit.md b/doc/plans/2026-10-26-native-status-reassessment-sleep-audit.md new file mode 100644 index 00000000000..17525bba1e0 --- /dev/null +++ b/doc/plans/2026-10-26-native-status-reassessment-sleep-audit.md @@ -0,0 +1,1084 @@ +# Native status reassessment sleep audit + +Parent: governed-review response recovery, draft #17702 at `be6ca6151b`. +This audit starts the next reason in the recurring-server migration stack. +Parent #17702 has passing checks and Greptile 5/5 at that head; it remains unmerged. + +Current summary: [worker contract](../idle-native-status-reassessment.md). +Entries below preserve chronological checkpoints, including earlier incomplete +states and failures; the latest checkpoints supersede their progress status. + +## Obligation and ownership + +After a native result is assessed, changed durable facts can require a new status +assessment and decision. The current native-finalization reconciler discovers +that change by revisiting committed runs, including its rotating settled-history +page. Response publication and pending-effect delivery have separate owners in +the preceding slices. Workspace/result settlement and retained session cleanup +remain separate duties and must not disappear with this migration. + +The existing reassessment block has three activation conditions: + +- A work product was created or updated after the selected assessment. The + classifier must actually establish newly satisfied criteria and verification + before this condition changes task disposition. +- The authoritative task status/version differs from the previous assessment, + after accounting for the selected decision's projected version and preserve + semantics. A superseding decision can make the old run ineligible. +- The selected automatic review has been retired. Reassessment then checks the + latest contract, current task/run, pending child completion, dependencies, + governance and any later workspace-finalization barrier. + +These are triggers for the same append-only status reassessment operation. +Keep classification and status arbitration with native runtime. Keep task locks, +version comparison, current authority and effect creation in the status-decision +committer. Do not put evidence parsing or policy branches in startup or the shared +scheduler. Do not use the combined reconciler as a worker callback: that would +also run unrelated response, workspace and retention work. + +## Existing facts and guards + +The coordinator binds run, task, selected result, assessment and decision. The +assessment retains prior task status/version/decision and its input digest. The +result and contract are selected by matching company/task/run identities. The +committer appends the new decision and guards concurrent task changes. A +NativeStatusRaceError currently defers to later reconciliation; the new worker +must retain or receive the corresponding wake instead of silently losing it. + +The outer candidate scan respects next_attempt_at and live finalization leases. +The extracted owner must recheck these under its transaction/claim boundary; +a stale scheduler entry cannot bypass a newly acquired lease or delayed retry. +The worker must distinguish dormant changed evidence that is still insufficient +from retryable I/O or contention. Existing durable rows remain truth; no new queue +table is justified by this timer migration. + +## Producer audit to complete + +Work-product writers found in current source: + +- work-products.ts: create, update, primary replacement, batch import and removal. + Current notifications cover pull-request retention/approval work only. +- issues.ts: attachment work-product insertion. +- routes/issues.ts: trusted-source promotion changes an existing product and + inserts its promoted product inside an owned transaction. +- native-runtime/chat-attachment-reuse.ts and native-runner-file-handoff.ts: + direct metadata/title updates on existing products. + +The classifier reads more than work products: case handoffs, accepted/rejected +control-plane run events, linked approvals, resolved interactions and attachments. +Determine which mutations activate existing reassessment behavior and preserve +that contract. Do not assume that work-product notifications cover all evidence +or introduce inferred completion merely because a source row changed. + +Also audit task status/version producers; decision/effect settlement; automatic +review retirement; latest-contract changes; workspace barrier settlement; child +completion; dependency/blocker changes; agent/task authority and process claims. +Use the existing mutation/transaction owners. Reuse task/run-scoped notification +boundaries where their semantics match. Broad configuration changes may justify +broader scopes; ordinary writes must not reload every company run. + +## Acceptance before periodic removal + +- Extract and exercise the actual reassessment owner without invoking other + native recovery duties. Preserve fresh finalization and explicit recovery. +- Reproduce each missing producer wake before wiring it. Cover alternate writers, + nested outer commit/rollback and relevant uncertainty settlement. +- Verify superseding decisions, changed versions, current contract, ownership, + governance, pending children and workspace barriers with real DB state. +- Prove no duplicate decisions/effects under competing recovery and producer + changes during a sweep. A rejected or no-longer-current assessment cannot + authorize completion. +- Prove no recurring SQL for configured-idle and retained settled history, + including changed but insufficient evidence. Verify genuine deadlines, + failures, startup/resume, drain and shutdown. +- Measure selected-history and outstanding-backlog discovery before wiring it. + Bound individual queries and support interruption; disclose total reconstruction + cost and retain exact scope on ordinary changes. +- Register through existing maintenance lifecycle, disable only this periodic + duty, run relevant consumers and required checks, then obtain latest-head CI + and review acceptance on its own stacked PR. + +This is an audit, not an implemented migration. No polling entrance is removed +by this document. No Cloud alarm coverage or host sleep eligibility changes. + + +## Owner extraction checkpoint + +The status-authority corpus passes all 94 cases before extraction on a fresh +PostgreSQL database. Reassessment now lives in native-status-reassessment.ts; +its pure arbitration policy lives in native-reconciliation-status.ts. Existing +imports through the reconciler remain compatible. The combined reconciler +delegates to the extracted owner and retains all periodic behavior for now. +It avoids the legacy review-response lookup when that repair is disabled. + +The extracted owner distinguishes unchanged, superseded, deferred and committed +outcomes. Operational errors still propagate; the existing status-race behavior +remains deferred. All 94 status-authority cases and 187 governed-response cases +pass after extraction, as does server typecheck. Evidence: /tmp/sleep50-baseline.log, +/tmp/sleep50-extract-corpus.log, /tmp/sleep50-extract-responses.log and +/tmp/sleep50-extract-types.log. Each corpus run uses a fresh database because its +company prefixes persist beyond the suite. + +The scheduler entry point is not ready yet. The status committer locks the +coordinator before the task and publishes activity after its own transaction. +Wrapping it in a new outer transaction would publish before that outer commit; +do not do that. Add any scheduled source/lease/deadline guard at the committer's +existing transaction boundary, and exercise competing source/claim changes there. +The new worker must reload source facts and preserve deferred races/retries; it +must not treat an extracted snapshot-based helper as fresh scheduling authority. +Producer notifications, discovery cost, lifecycle tests and periodic removal +remain outstanding. No slice 50 PR has been published. + +## Scheduled commit guard + +Scheduled reassessment now opts into a source guard inside the existing status +committer transaction. After coordinator/task locking it compares the selected +phase/result/assessment/decision, respects a future retry and live or unknown +lease, locks and checks native run ownership, and defers while the old decision +has pending or failed effects. Explicit callers retain their existing behavior. +The lock order remains coordinator, task, then run. Publications still occur only +after the committer's transaction commits. + +The first seven-case reproduction had five behavioral failures and two invalid +fixtures: clearing the assessment/result without clearing dependent references +violated schema constraints before reaching the committer. Corrected fixtures +plus pending-effect and ownership races failed all nine before the guard: eight +incorrect commits and one attempted commit rejected by the result constraint. +After the guard, all 103 status-authority corpus cases and server typecheck pass. +Every deferred case preserves the competing state and successfully commits once +that state is released. Logs: /tmp/sleep50-guard-before.log, +/tmp/sleep50-guard-before2.log, /tmp/sleep50-guard-after.log, +/tmp/sleep50-guard-types.log; statuses are saved beside the test logs. + +This is not worker acceptance. Candidate selection, dormant-versus-retry outcome +handling, producer coverage, lifecycle/cost tests and periodic removal remain. +No slice 50 PR has been published and no additional polling has been removed. + +## Scoped worker and first producer + +A local, not-yet-registered worker now discovers the three existing activation +conditions with company/task/run/agent scopes and 100-row keyset pages. Candidate +SQL excludes superseded decisions, held native ownership, unknown finalization +leases and old pending effects. It is discovery, not mutation authority: check +reloads the source and the committer still applies the locked guard. Future retry +and live lease deadlines retain their maximum. A commit race requests a retry; +unchanged evidence or a dormant review prerequisite does not keep recurring work +scheduled. The shared scoped-deadline helper owns coalescing, fairness, admission +and retry bookkeeping. No new queue table is added. + +Nine initial discovery/owner tests pass for retired reviews, changed evidence, +authoritative task changes, future retries/leases, insufficient evidence, +ownership exclusions and pre-aborted discovery. A tenth regression exposed a +newly saved five-second deadline being pushed out to generic 30-second backoff. +The worker now returns that saved future deadline exactly; backoff applies only +when the source is still immediately eligible after a commit race. The complete +113-case status corpus and server types pass. Evidence: +/tmp/sleep50-worker-first.log, /tmp/sleep50-deadline-before.log, +/tmp/sleep50-worker-after.log and /tmp/sleep50-worker-after-types.log. + +An isolated lifecycle test reproduced a real missed wake: after the worker went +quiet, automatic review retirement left the task in review. Retirement now +signals the exact task/run inside its existing transaction, before changing the +card. The common native-finalization signal also includes reassessment, retaining +its supplied scope. All three lifecycle tests pass: real retirement after idle, +first retirement during drain followed by resume, and a saved future retry across +shutdown/restart without early dispatch. Driver-level SQL capture verifies no SQL +across two former polling minutes after completion or while drain is paused. +The existing governed-response suite passes all 187 cases after the shared +notification change. Logs: /tmp/sleep50-lifecycle-before.log, +/tmp/sleep50-lifecycle-after.log, /tmp/sleep50-lifecycle-types.log and +/tmp/sleep50-worker-responses.log. + +Initial read-only query measurement reused slice 49's 10,000 cloned history rows. +Those copies share a task and their decisions are superseded for reassessment; +this is not proof of 10,000 independently selected current assessments or an +outstanding reassessment backlog. Startup returned 62 candidates from other +retained history in one query (305 ms cold wall time; warm EXPLAIN execution about +47 ms). Company/task scopes returned none in 34/20 ms, exact run in 2.6 ms. +Plans and logs: /tmp/sleep50-query-{startup,company,issue,run}.plan and +/tmp/sleep50-query-plan.log. Selected-current history, backlog cost and cancellation +between pages still need measurement before enabling the worker. + +Remaining work is substantial: work-product and alternate writers, task changes, +classifier inputs while a retired review is dormant, current-contract/workspace/ +governance and ownership transitions, further transaction/lifecycle cases, +startup registration and periodic removal. Existing retirement prerequisite reads +precede the committer; audit their serialization with producer owners before +claiming concurrent authority coverage. In particular, completion-contract +creation uses its own advisory lock rather than the status committer's issue +lock. This checkpoint is not a claim that every business-fact race is covered. +The existing periodic reassessment path remains active and slice 50 has no PR. + +Final current-source local rerun passes 116 tests across the status corpus and +isolated worker lifecycle files (/tmp/sleep50-final-local.log). Together with the +187 governed-response cases this is 303 distinct passing cases, not whole-workspace +or remote PR acceptance. The last server typecheck also passes. + +## Work-product and task producer checkpoint + +Six real writer regressions failed before notifications: artifact update, create, +primary replacement, import, attachment creation, and nested transaction commit. +Their existing write transactions now signal task-scoped reassessment before the +mutation. Import coalesces its scopes per task. Updates signal both old and new +scope if their supported patch moves a product. Removal also signals its old task. +Attachment creation signals before asset/attachment/product writes, including +attachment evidence without a registered run. Native file handoff and attachment +reuse call that service inside their tool-authority transaction; their subsequent +same-transaction title/metadata edits share its outer settlement. Those fields +are not classifier evidence, so no second recovery wrapper was added there. +The low-trust promotion route signals inside its own atomic source/product write. +A real HTTP promotion now also proves completion through the registered worker. + +Three further regressions failed for task status changes, assignment changes, +and reasserting blocked state. The task mutation owner now signals relevant +status/version, execution, assignment and blocker writes. The regression checks +that the new assessment reflects the task's actual version and does not overwrite +the board's status. The nested work-product case verifies no decision before the +outer commit, no decision after rollback, then successful reassessment after a +later commit. All paths return to driver-observed SQL quiet. + +Final owner lifecycle tests pass all 13 cases, including the real promotion route, +and server types pass. Evidence: /tmp/sleep50-products-before.log (six failures), +/tmp/sleep50-products-after.log (nine passes), /tmp/sleep50-tasks-before.log +(three failures), /tmp/sleep50-tasks-after.log (12 passes), and +/tmp/sleep50-products-final.log (13 passes) plus its final typecheck log. + +Affected consumers pass 79 cases across five files: work products, attachment +routes, low-trust HTTP routes, attachment reuse, and runner task attachments. +The first combined run passed 53 but failed two suites during embedded PostgreSQL +bootstrap, leaving 26 cases unrun. Those suites now accept validated dedicated +external test database URLs and explicitly close their clients. Their 26 cases +then pass on separate PostgreSQL databases. Low-trust route tests use their own +dedicated database because teardown deletes shared tables. Preserve the initial +failure in /tmp/sleep50-consumers.log and the successful rerun in +/tmp/sleep50-attachment-consumers-after.log. This is 79 passing consumer cases +across the two runs, not a green initial run or a whole-workspace result. + +Master was refreshed and remains 1fbbef45a8. The stack has not been reconciled +onto it yet. The worker remains unregistered and periodic reassessment remains. +Remaining producer coverage includes approval/interaction and run-event evidence, +case handoffs, native ownership and run-status transitions, completion-contract +changes and workspace-finalization barriers. Workspace recovery writes a separate +workspace_finalize success receipt; snapshot-only rows are not that barrier. +Continue the concurrent authority audit, independent-current-history/backlog cost, +lifecycle/failure evidence, startup/periodic wiring and full PR acceptance. + + +## Approval evidence producer checkpoint + +Approval references are classified from the current approval status and its +company/task link. A product can have changed after the selected assessment +while that approval is still pending or unlinked. The worker correctly goes +quiet with insufficient evidence, but must inspect that same candidate when +an approval is resolved or linked later. This does not add a new reassessment +condition: the existing changed-product predicate must still be satisfied. + +The existing `notifyApprovalWork` boundary now sends the same task scopes to +native reassessment as to governed review responses, before the approval/link +write. It already resolves linked task IDs for status changes and accepts +explicit task scopes for link/unlink and bulk link. No new transaction wrapper, +company scan, table, or provider policy was introduced. Unlinked approval creation +has no task to reassess; the subsequent link owns activation. The production +writer audit also found budget approval resolution and gateway formal-approval +linking use this boundary with the appropriate scope. Company deletion removes +the records rather than creating a surviving reassessment obligation. + +Four real PostgreSQL regressions failed before the notification: approval +resolution, single link, bulk link, and an approval inside a caller-owned outer +transaction. With the notification they pass, including no premature decision +before outer commit, no decision after rollback, and driver-observed SQL quiet +before activation and after settlement. The full lifecycle file now passes 17 +cases. Existing approval service and route-idempotency consumers pass another +19 cases (36 total in the combined run). Server typecheck passes. + +Evidence: `/tmp/sleep50-approval-before.log` and `.status` (four failures), +`/tmp/sleep50-approval-after.log` and `.status` (36 passes), and +`/tmp/sleep50-approval-types.log` and `.status` (pass). The before and after runs +used separate freshly migrated PostgreSQL test databases. This is local producer +acceptance, not full slice or remote PR acceptance. + +Interaction evidence remains next: confirmation/question paths signal through +chat milestones, while other interaction outcomes signal governed review response +directly. Their common activation boundary must cover reassessment without teaching +the scheduler interaction policy or coupling it to a review-specific worker. +Several interaction mutations also lack either signal and require fact-level +audit before the old polling path can be removed. Run evidence, ownership, +completion contracts, workspace barriers, concurrent authority, discovery cost, +and startup registration remain outstanding as listed above. + + +## Interaction evidence producer checkpoint + +The interaction classifier reads status and result outcome, not just terminal +status. Real response regressions reproduced missed wakes for confirmation, +checkbox confirmation, question answers, completed item verdicts, and partial +item verdicts. The first checkbox fixture used `items`/`checkedItemIds` rather +than the real `options`/`selectedOptionIds` contract; after correcting it, that +case also reproduced the missed wake before production changes. + +`notifyNativeReviewState` is a small fan-out boundary for the two consumers of +changed review facts: governed response recovery and status reassessment. It +preserves their existing task/run/company scopes and delegates all discovery, +classification and authority to the owning workers. Approval, interaction, +chat-milestone, satisfied-connection and OAuth completion/rejection producers +reuse that boundary in their existing transactions. It does not add database +queries or a transaction wrapper at these call sites; the existing shared intent +registration still owns its normal transaction-tracking cost. + +Confirmation/question paths already use the milestone boundary; other interaction +outcomes use the new shared review-state boundary directly. Connection-intent +creation/supersession signals before insert in its existing transaction. Generic +non-question/non-confirmation creation signals reassessment before insert. +Partial item verdicts need reassessment even though governed response recovery +still waits for completion: their persisted `outcome: resolved` is already +accepted by the evidence classifier. This migration preserves that existing +classification policy rather than silently changing it while removing polling. + +Direct-writer audit: secret proposals, tool-action review, gateway outcome +projection, conversation promotion, and obsolete-policy review retirement already +signal milestones. Automatic completion retirement and native status commits +already signal reassessment/finalization. Confirmation comment enrichment shares +its outer transaction with ordinary service resolution. Ownership-change connection +expiry is called only through issueService's owned transaction. OAuth callback +acceptance/rejection and satisfied-connection retirement now use the shared helper. +Payload-only connection-phase updates, consumed-action stamps, retry annotations +on already accepted confirmations, and dismissal attention fields do not change +the classifier's status/outcome facts. Native response materialization uses the +existing finalization boundary. Authorization-start's reopening/creation path +and native superseded-review caller ownership still need final audit; do not infer +complete producer coverage from this checkpoint. + +All 22 reassessment lifecycle cases pass, with driver-observed SQL quiet before +and after real response activation. Server typecheck passes. Existing interaction +service, tool-access/OAuth, and satisfied-connection suites pass 625 cases across +three separate freshly migrated PostgreSQL databases. Evidence: +`/tmp/sleep50-interaction-before.log` (four missed-wake failures and one invalid +fixture), `/tmp/sleep50-interaction-checkbox-before.log` (corrected missed-wake +failure), `/tmp/sleep50-interaction-after.log` and `.status` (22 passes), +`/tmp/sleep50-interaction-types.log` and `.status` (pass), and +`/tmp/sleep50-interaction-consumers.log` and `.status` (625 passes). + +The milestone fan-out also reaches existing run-status producers, but that is not +proof that all native ownership/run-event writers are covered. Workspace barrier +producers, remaining run/contract/case-handoff facts, concurrent authority guards, +current-history/backlog query cost, startup registration and remote PR acceptance +remain incomplete. Periodic reassessment is deliberately still enabled. + + +## Workspace finalization barrier producer checkpoint + +The retired-review path defers when the latest `workspace_finalize` record has +not succeeded. It deliberately leaves no recovery timer for an in-progress or +failed barrier, so the actual producer must reactivate it when that fact changes. +Two PostgreSQL regressions reproduced missing reactivation through the normal +workspace-operation recorder: failure followed by a later successful export, +and a successful write inside a caller-owned transaction. + +The recorder now routes its initial state insert, successful settlement and failed +settlement through one local state-write boundary. Only `workspace_finalize` +operations with a run binding register reassessment intent; other phases retain +their existing writes. The added transaction contains only the state write and +notification. Log I/O, the provider callback, and progress/lease updates remain +outside it. Existing caller-owned transactions retain outer commit semantics. +The success test keeps the real callback suspended while the worker goes SQL +quiet, then completes it and observes task completion. The transaction test +proves no decision before outer commit, no decision after rollback, and successful +activation after a later commit. + +The separate retained-workspace recovery success receipt now signals in its +existing transaction before the receipt insert and generation update. A test +runs real retained recovery with temporary files and verifies that the delivered +notification's reader observes a committed `workspace_finalize: succeeded` row. +The historical unsafe-export path already had the common finalization signal; +it now registers that intent before its barrier insert rather than afterward. +Snapshot and copy-only records use other phases and do not satisfy this barrier. +Runtime-control orphan reconciliation operates on action-stamped controls; normal +native workspace finalization uses its own ownership metadata instead. + +All 24 worker lifecycle cases pass, and server typecheck passes. The workspace +operation reconciliation, retained workspace recovery and native status arbiter +corpus suites pass 170 cases across three fresh PostgreSQL databases. The two +workspace suites now accept dedicated, validated external test database URLs; +restart tests reuse that URL, and clients close before fixture teardown. This +uses the known-working local PostgreSQL installation because embedded bootstrap +failed in earlier acceptance attempts; no product database is used. + +Evidence: `/tmp/sleep50-workspace-before.log` and `.status` (two failures), +`/tmp/sleep50-workspace-after.log` and `.status` (24 passes), +`/tmp/sleep50-workspace-consumers.log` and `.status` (170 passes), and +`/tmp/sleep50-workspace-final-types.log` and `.status` (pass). All launched +commands at this checkpoint reached terminal status. These are local checks, +not remote PR acceptance or evidence of host suspension. + +Next: finish run-event/case-handoff, completion-contract, ownership and remaining +interaction producer audit; then verify commit-time business authority, discovery +cost/backlog behavior, lifecycle wiring and remote PR gates. Current scheduled +commit guards fence coordinator/run ownership and source selection, but the +retired-review business-fact checks still run before the commit transaction. +The old periodic reassessment remains enabled until this full slice is ready. + + +## Authoritative run-event producer checkpoint + +Event references may become accepted/rejected only when a persisted PRP event +has `sourceKind: control_plane` and the corresponding evidence verdict. The +classifier now exposes that existing pure verdict interpretation to the append +boundary, including its existing whitespace normalization. The append boundary +registers a run-scoped reassessment signal before the event write for native +runs. It does not look up a task context or scan evidence consumers to append +an event. Non-verdict events retain the ordinary atomic single-statement path; +runner-authored claims do not activate reassessment. + +The writer audit found three insertion paths. The native coordinator store accepts +only `sourceKind: runner` and rejects control-plane events, so its append path +cannot produce an authoritative verdict and needs no added signal. Scheduled +retry cancellation inserts ordinary lifecycle diagnostics without PRP evidence. +The shared append path owns the supported control-plane verdict writes. Event +rows are otherwise immutable in production; agent/company deletion owns removal +of their run history rather than producing new evidence for surviving runs. + +Four real-append regressions failed before the fix: ordinary append, native-source +append, outer transaction commit, and authoritative evidence after ordinary/runner +logs. The repaired tests observe task completion, outer rollback/commit fencing, +no reassessment signal for ordinary or runner-authored logs, replay idempotency, +and driver-observed SQL quiet. All 28 lifecycle cases and six existing atomic +append cases pass. The atomic tests still prove the one-statement ordinary-log +path and preserve binding/concurrency behavior. + +The initial combined run failed three sequencing cases during embedded PostgreSQL +bootstrap, before assertions. They now accept a validated dedicated external test +database URL, clean only their own company fixture between cases, and close both +clients. All three then pass, including concurrent retry receipt deduplication, +concurrent event allocation, and conflicting replay without cursor drift. Final +server typecheck passes. This is 37 distinct passing cases across the runs, not +a successful initial combined run. + +Evidence: `/tmp/sleep50-events-before.log` and `.status` (four failures), +`/tmp/sleep50-events-after.log` and `.status` (34 passes, three bootstrap failures), +`/tmp/sleep50-events-sequence.log` and `.status` (three passes), and +`/tmp/sleep50-events-final-types.log` and `.status` (pass). All test handles at +this checkpoint are terminal. Case-handoff creation/cancellation, remaining +contract/ownership/authority producers and the full acceptance tasks listed above +remain outstanding; the old periodic reassessment path is still enabled. + +## Scheduled retired-review authority checkpoint + +Two commit-boundary regressions reproduce unsafe completion: after reassessment +reads a succeeded run, a competing writer can change its outcome to failed or +bind the task to a different run without changing the task's status version. +Both previously committed a `done` decision. The scheduled source contract now +identifies retired automatic-review reassessment, and the existing status +committer rechecks run success and task execution ownership under its existing +coordinator/task/run locks. It adds no outer transaction, lock order, provider +call, or database query. Explicit legacy reconciliation remains unchanged. + +The two regressions assert that the competing state survives, no new decision +commits, and reassessment succeeds once the original authority is restored. +All 115 status-corpus cases pass. The combined corpus/lifecycle run initially +finished with 142 passes and one failed lifecycle assertion: retained test +history needs multiple startup batches, so the first registered wake can be the +next batch rather than the target's future retry. The retry test now waits for +startup batches to settle while asserting that the target stays in review, +then verifies the exact saved deadline before and after restart. All 28 lifecycle +cases pass against that same retained-history database; final server types pass. +This is 143 distinct passing cases across runs, not a green initial combined run. + +Evidence: `/tmp/sleep50-authority-before.log` and `.status` (two failures), +`/tmp/sleep50-authority-after.log` and `.status` (115 corpus passes and 27/28 +lifecycle passes), `/tmp/sleep50-authority-lifecycle.log` and `.status` (28 passes), +and `/tmp/sleep50-authority-final-types.log` and `.status` (pass). All launched +handles reached terminal status. No remote acceptance or host sleep is claimed. + +Producer audit notes for the next checkpoint: + +- Handoff evidence is accepted for every persisted non-cancelled state. Delivery, + completion, leases and retries do not change that classification. Agent-created + handoffs require the source run to be running and own the task; their generated + UUID is returned after creation. Board requests can use the current execution + binding. Do not add wake traffic to every handoff lease renewal. +- Task mutation cancellations are in the issue transaction, which also signals + reassessment. Direct handoff cancellation, delivery cancellation, and run-control + cancellation still require their own transaction/activation audit; no complete + handoff producer-coverage claim is made here. +- Superseded external-review withdrawal is called by the status committer inside + its decision transaction, which already signals native finalization and chat + milestones. OAuth authorization-start separately reopens/creates a pending + confirmation; its interaction writes remain to be audited. +- The evidence classifier checks target-agent existence, not invokable status. + Do not add status-change fanout based only on the name of that check. +- Completion-contract creation is serialized with the per-company/task advisory + lock in `ensureNativeCompletionContract`. Checking the latest revision before + the commit transaction is still insufficient. Audit its callers' lock order + before extending the scheduled commit guard. + +Latest-contract, workspace-barrier, governance, dependency and other business-fact +races remain open, along with remaining producers, query cost/backlog proof, +startup wiring, periodic removal, publication and PR gates. The full original +stack scope, older unresolved gates and master reconciliation remain outstanding. + +## Completion-contract serialization checkpoint + +The completion-contract writer is `ensureNativeCompletionContract`. Production +callers are native run preparation and heartbeat runtime selection; both select +or ensure the contract before their later run-state mutation. The writer owns a +per-company/task advisory transaction lock, and revisions are append-only. The +scheduled retired-review commit now calls the owner's +`lockCurrentNativeCompletionContract` before taking coordinator/task/run row +locks. The helper tries that same advisory lock without waiting and verifies the +expected contract is still the latest revision. A busy writer or replaced +contract returns the existing reassessment race outcome. The lock remains held +through the status transaction. Explicit legacy reconciliation is unchanged. + +This order matters: inserting a contract acquires a foreign-key lock on its task. +Trying to lock the task before recognizing an in-flight contract writer can wait +behind that writer. The before-fix tests reproduced a stale `done` decision after +a newer contract committed, and a 30-second timeout while a real contract writer +held an uncommitted revision. The concurrency fixture now also applies a two-second +statement timeout to the status transaction so a future blocking regression can +unwind and release its competing writer rather than waiting for suite timeout. +The writer uses the real service inside a caller-owned transaction; it is rolled +back only after reassessment has returned its expected retry outcome. + +The updated tests prove no status decision commits against either race, a newer +committed contract leaves the old result dormant on subsequent inspection, a +rolled-back revision leaves no row, and restoring the original authoritative +contract allows completion. The contract key is factored privately in its owning +module so the writer and guard cannot diverge. No scheduler-specific SQL lock key +or provider policy is introduced into shared scheduling. + +All 117 status-corpus and 28 worker lifecycle cases pass in one combined run +(145 total), including the retained-history startup deadline regression. Server +typecheck passes. Evidence: `/tmp/sleep50-contract-before.log` and `.status` +(stale-completion failure and writer timeout), `/tmp/sleep50-contract-after.log` +and `.status` (145 passes), `/tmp/sleep50-contract-types.log` and `.status` +(pass). All commands have terminal status. + +Workspace-barrier and governance/dependency races, remaining producer coverage, +query-cost/backlog proof, startup registration, periodic removal and full PR +acceptance remain outstanding. The existing native workspace finalization owner +already has a separate physical/durable ownership protocol; audit and reuse that +boundary before adding any barrier fence. No slice 50 PR, merge, deployed sleep +or completion of the overall stack is claimed by this checkpoint. + +## Workspace-barrier commit checkpoint + +Three regressions reproduced stale `done` decisions after the selected workspace +barrier changed to failed, after a newer running finalization operation committed, +and while another transaction held an uncommitted failure update to the selected +barrier. The scheduled retired-review guard now reads the latest finalization +barrier inside the existing status transaction and takes `FOR SHARE NOWAIT` on +that row. Non-success or lock contention yields the existing retryable race +outcome. A subsequent settled failed/running barrier is dormant until a producer +changes it. Missing-barrier semantics remain unchanged from the legacy path. + +The transaction already holds `FOR UPDATE` on the task and native run. New +`workspace_operations` rows reference the run through its foreign key, so inserts +cannot commit past those locks; the latest-row share lock additionally fences +status updates/deletion on an existing barrier. NOWAIT avoids waiting behind a +writer that could need a row already held by the status transaction. The ordinary +workspace recorder and recovery receipt producers remain unchanged by this fix; +no advisory namespace, provider call, or recurring probe was added. + +All 120 status-corpus and 28 lifecycle tests pass together (148 total), with server +types passing. The race tests preserve the competing state, assert no new status +decision, verify dormancy for a persisted failed/running barrier, and verify +completion after the original successful barrier is restored. The concurrent +writer rolls back only after recovery returns; its two-second statement timeout +bounds a future blocking regression. Existing lifecycle cases still exercise the +real recorder's failed/successful receipts and outer commit/rollback behavior. +Evidence: `/tmp/sleep50-barrier-before.log` and `.status` (three failures), +`/tmp/sleep50-barrier-after.log` and `.status` (148 passes), and +`/tmp/sleep50-barrier-types.log` and `.status` (pass). All handles are terminal. + +Physical workspace ownership is distinct from this receipt check. The existing +native owner uses an advisory lock plus a durable run-profile token, removes only +its own token after the action joins, and retains the token if cleanup cannot be +confirmed. Its admission/release interaction with reassessment still needs audit; +this checkpoint does not assert that a succeeded receipt alone proves physical +ownership released. Governance/dependency facts, remaining producers, discovery +cost, startup/periodic migration and remote acceptance remain open, as does the +full original stack scope. + +## Physical workspace ownership checkpoint + +Four regressions reproduced completion while physical workspace authority was +still outstanding: a live owner at startup, an unverified persisted owner at +startup, a real owner acquired after reassessment's read, and an owner receipt +written after that read. Scheduled retired-review reassessment now treats a +persisted owner as dormant. At commit it tries the existing physical-finalization +advisory lock before acquiring row locks, then rechecks owner presence on the +locked run. This closes the lock-before-receipt acquisition window as well as +retained ownership after a disconnected lock connection. The lock/presence +helpers remain with native workspace ownership/recovery; no provider policy is +added to scheduling. Explicit legacy reassessment remains unchanged. + +With only those guards installed, the real-owner lifecycle test reproduced a +second gap: the task stayed in review after the owner released. Normal release +now registers reassessment intent in a short transaction before removing only +its own token. Physical work, assertion of ownership and joining the copyback +callback remain outside that transaction. Failed/uncertain cleanup retains the +existing durable owner and joined-token behavior. Verified operator recovery +registers the same run/task signal in its existing transaction before removing +the verified owner. Its physical-lock check now uses the same owner's helper. +The source audit found these two removal paths; reconstruction preserves the +owner field rather than clearing it. + +The real-owner lifecycle test proves task preservation and driver-observed worker +SQL quiet while ownership remains, reactivation after release, and quiet after +completion. A separate unverified-owner case proves no completion or recurring +worker query despite a successful workspace receipt. Commit-race tests preserve +competing ownership and complete only after it is released/restored. The existing +verified-recovery test additionally reads the run from the reassessment callback +and sees the committed released profile, with the coordinator unchanged. + +All 122 status-corpus, 30 reassessment lifecycle and 65 workspace finalization +consumer cases pass together (217 total), and server typecheck passes. Existing +consumers cover live ownership, disconnected lock sessions, retained joined-owner +receipts, orphan copyback children, verified stop/replay, and current-task fencing. +The workspace suite now accepts a validated dedicated external test database URL, +uses it for restart/competing clients, and closes its primary client on teardown. +It ran in its own fresh database: its connection-termination tests must not share +a database with another suite. No production database or provider was used. + +Evidence: `/tmp/sleep50-physical-before.log` and `.status` (four failures), +`/tmp/sleep50-physical-guard.log` and `.status` (release missed wake after guard), +`/tmp/sleep50-physical-after.log` and `.status` (217 passes), and +`/tmp/sleep50-physical-types.log` and `.status` (pass). All handles are terminal. +Governance/dependency races, remaining producer audits, discovery cost/backlog, +startup registration/periodic removal and remote acceptance remain open. No PR +or merge is claimed for slice 50, and the full original stack remains incomplete. + +## Governance commit checkpoint + +Seven regressions reproduced stale completion after a pending execution-policy +hold, a newly linked pending approval, a reopened approval, a new pending +interaction, and uncommitted changes to an approval, interaction or question's +human-direction history. Reassessment's selected retired-review source now carries +the governance gate used by its policy decision. The committer rechecks that gate +under its existing task/run locks and rejects a changed kind/identity. + +`lockNativeGovernanceSources` belongs to the governance service. It share-locks +all existing interactions on the task, linked approval rows and approval-link +rows, and comments when the task has ordinary questions. It deliberately includes +resolved rows, since reopening them can introduce a new gate. Question history +depends on human comments, including deletion/trust fields, so locking only the +question row would be insufficient. Every added source lock uses NOWAIT to avoid +waiting behind a writer that may need the already-held task/run locks. New linked +rows are fenced by their task foreign keys. A pending issue-owned execution stage +is already protected by the task lock and takes priority without locking the +lower-priority source rows. Existing governance policy and ordinary callers of +`pendingNativeGovernance` are unchanged. + +The tests assert that no decision commits against changed or uncommitted facts, +the task remains in review, and completion succeeds after the gate is removed or +the competing writer rolls back. The concurrent writers have a two-second +statement bound on the recovery transaction so a future blocking regression can +unwind. All 129 status-corpus and 30 lifecycle tests pass together (159 total). +The question-history fixture was subsequently strengthened to use a valid question +payload; that case passes again and final server typecheck passes. + +Evidence: `/tmp/sleep50-governance-before.log` and `.status` (seven failures), +`/tmp/sleep50-governance-after.log` and `.status` (159 passes), +`/tmp/sleep50-governance-question.log` and `.status` (one focused pass), and +`/tmp/sleep50-governance-final-types.log` and `.status` (pass). All handles are +terminal. These guards run only for actual scheduled retired-review reassessment, +not during idle discovery. Their task-history lock/query cost must be included in +the remaining cost audit; a passing functional test does not establish that cost. + +Dependency/readiness and remaining selected-authority/evidence races, remaining +producers (including handoffs and OAuth reopening), discovery/backlog costs, +startup wiring, periodic removal and complete PR gates remain outstanding. No +slice 50 PR or merge is claimed; the original full stack scope remains open. + + +## Dependency readiness commit guard (2026-10-26) + +Seven real database races reproduced unsafe completion: a new blocker relation, +a reopened blocker, an uncommitted blocker status writer, a newly recorded +unattributed workspace operation, a changed finalize receipt, an uncommitted +receipt update, and an uncommitted unattributed operation insert. Before the +repair all seven committed a status decision despite stale dependency readiness. +Evidence: /tmp/sleep50-dependencies-before.log and .status (exit 1). + +Dependency semantics remain in issueService.getDependencyReadiness. Its opt-in +lockSources mode requires an existing transaction holding the dependent task +FOR UPDATE. It locks relation/blocker rows FOR UPDATE NOWAIT, then relevant +execution workspaces FOR UPDATE NOWAIT and their existing operation rows FOR +SHARE NOWAIT. The workspace lock fences new operations through the existing FK, +including operations without an issue attribution. Ordinary readiness consumers +retain their previous query and locking behavior. The attributed/unattributed +operation selection and later successful finalization rules are unchanged. + +The scheduled retired-review commit compares the readiness boolean used by the +policy with this locked read; contention or a changed predicate defers through +the existing NativeStatusRaceError retry. It does not create a queue/table or +introduce an independent timer. Locks only cover the current task's prerequisites +and relevant workspaces. History-dependent locking cost still needs measurement +as part of the outstanding discovery/commit cost audit. + +Initial combined verification passed 135 corpus cases (including all seven new +races), but an older effect-state assertion compared unordered SQL results and +failed when physical row order changed. Both reads now order by effect ordinal. +The lifecycle suite also failed before setup because the invocation used an +incorrect external DB variable and fell back to broken embedded PostgreSQL. +These failures remain in /tmp/sleep50-dependencies-after.log/.status; neither was +counted as passing. The dependency consumer selection passed 15 cases, and the +initial typecheck passed. The consumer suite now supports the standard validated +dedicated external test database contract and closes its pool. + +Final verification (all commands terminal): +- /tmp/sleep50-dependencies-final.log/.status: 136 status corpus plus 30 worker + lifecycle cases, all 166 passing. Fresh corpus DB and retained lifecycle history. +- /tmp/sleep50-dependencies-consumers-final.log/.status: all 15 existing dependency + readiness/blocker consumer cases passing; other suites deliberately not selected. +- /tmp/sleep50-dependencies-final-types.log/.status: server typecheck passing. + +The seven regressions also restore or roll back the competing state and prove +that reassessment can subsequently complete. Remaining authority/evidence races, +producer coverage (including handoffs and OAuth reopening), task-history/backlog +cost, startup wiring, periodic removal and remote PR acceptance remain open. +No new PR or merge, and no whole-instance sleep claim. + + +## Evidence freshness at status commit (2026-10-26) + +All seven before-fix regressions committed stale completion when supporting +work-product review, approval, interaction, attachment, event, or case-handoff +state changed after classification and before commit. The work-product case also +holds a real uncommitted rejection until recovery returns, then rolls it back. +/tmp/sleep50-evidence-before.log/.status records seven failed assertions (exit 1), +not fixture/bootstrap failures. + +The assessment owner now exposes lockCurrentNativeWorkAssessmentEvidence. It +loads the exact persisted reassessment with its bound contract/result, then calls +the existing classifier with opt-in source locking. Classification rules remain +in evidence-classifier.ts, including reference parsing, accepted/rejected policy, +question/approval outcomes and attention-target existence. The scheduled committer +only compares the resulting normalized assessment and handles contention through +its existing race/retry outcome. Both retired-review and other scheduled +reassessment paths use this guard. Ordinary classification retains its original +unlocked behavior. No extra transaction encloses the committer/publication path. + +The classifier acquires SHARE NOWAIT on the referenced evidence rows and existing +attention target agents. The already-held task/run locks fence new task/run-bound +records through their existing FKs; the reference cache still deduplicates repeat +criterion/verification references. The assessment/contract/result records are +also share-locked. This protects mutable acceptance state and evidence deletion, +rather than assuming all durable evidence is immutable. Missing arbitrary agent +IDs have no row to lock; supported agent-creation identity/producer behavior and +other authority fields remain part of the outstanding producer audit. This is +not a blanket proof for independent SQL writers. + +The added read cost is one exact assessment/contract/result query plus the normal +classifier's distinct-reference and attention-target queries. It runs only for +an actual scheduled decision commit, not on idle history. Quantitative history, +backlog and full transaction cost acceptance remains outstanding; no performance +claim is inferred from the narrow regression runtime. + +After repair all 143 status-corpus, 30 worker-lifecycle and four classifier +consumer tests pass (177 total), as does server typecheck. The seven regressions +also restore/roll back evidence and prove subsequent completion. Evidence: +/tmp/sleep50-evidence-after.log/.status and /tmp/sleep50-evidence-types.log/.status, +both exit 0; all launched commands are terminal. Ordinary classifier consumer +cases still run without a locking API in their fake query builder. + +Remaining: producer/authority coverage, query/history/backlog cost, startup +integration, removal of this periodic duty, PR publication and remote acceptance. +The full recurring-work stack and its older unresolved gates remain open. + + +## OAuth authorization-start producer checkpoint (2026-10-26) + +The direct OAuth authorization-start owner creates a confirmation or reopens its +previous accepted/rejected card by idempotency key. That bypassed the interaction +service's native-review notifications. Three real tool-access service regressions +(creation, reopening, outer rollback) reproduced absent settled notifications: +/tmp/sleep50-oauth-before.log/.status, three failed assertions, exit 1. + +The existing card lookup/mutation and OAuth-state interaction link now share a +short database transaction. It awaits notifyNativeReviewState before writing, +with company/task scope. The shared hook wakes both governed review response and +status reassessment; consumers still select only actionable durable state. +Provider discovery, client setup and authorization remain outside the new short +transaction. Browser/user flows that do not create an agent authorization card +keep their existing path. Caller-owned transactions retain their outer boundary. +No provider-specific logic moved into recovery or scheduler code. + +All 472 tool-access consumer cases pass, including the new real nested-transaction +tests. They assert the inside/outside card states, no pre-commit notification, +one settlement notification per consumer, and rollback retaining the prior card +and no OAuth state. Rollback settlement releases the intent for reconciliation; +it is not represented as a committed card. All 30 reassessment lifecycle cases +pass against retained history, and server typecheck passes. Evidence (all terminal): +/tmp/sleep50-oauth-after.log/.status, /tmp/sleep50-oauth-lifecycle.log/.status and +/tmp/sleep50-oauth-types.log/.status, all exit 0; 502 distinct tests in this checkpoint. + +A fresh origin/master fetch still resolves to +1fbbef45a8d399771add7a27a04b12f3a20e264c. No source changed during acceptance. + +Additional handoff audit evidence: direct operator cancellation flows through +cancelPendingCaseHandoffsForIssue; run-control/task cancellation uses +heartbeat/case-handoff-lifecycle; delivery can cancel a fenced claim in its own +settlement transaction. The classifier accepts all persisted non-cancelled +handoff states. Those delivery/lease transitions are not independently positive +evidence triggers, so do not fan out every lease renewal. Creation assigns its +UUID inside the owning transaction. The previous checkpoint now fences stale +handoff evidence at commit. Complete activation/quiet-state proof for the +remaining handoff/authority paths is still outstanding; no blanket producer- +coverage claim is made from this search. + +Remaining acceptance continues to include producer/authority audit, discovery +and commit costs, backlog and lifecycle coverage, startup registration, periodic +removal, PR publication and exact-head remote gates. The full original stack +scope and older unresolved gates remain active. + + +## Selected-history, backlog and commit cost checkpoint (2026-10-26) + +The earlier same-task/superseded-decision benchmark did not establish discovery +cost for selected current history. A dedicated PostgreSQL 17 database now holds +10,000 separate task/run/contract/result/assessment/decision/product bindings. +Every task initially selects its own completed decision; accepted products predate +the assessment. This is distinct from 10,000 obsolete decisions on one task. +Fixture: /tmp/sleep50-scale-fixture.sql and .log/.status; database +paperclip_sleep50_scale_20261026_test (snapshot of the completed corpus database, +plus a separate synthetic company). All fixture inserts and measurements are +local disposable test data, never deployed/customer data. + +At source 51f305f1d1 with the new backlog regression pending: +- Startup discovery: 98.7 ms, three queries, 246 candidates from the original + corpus fixtures. None of the 10,000 completed synthetic tasks was a candidate. +- Synthetic company: zero candidates, one query, 34.8 ms (warm EXPLAIN execution + 33.5 ms); exact task/run scopes: one query each, about 1.8/2.1 ms. +- Changing all synthetic tasks' authoritative status creates 10,000 actual + reassessment candidates. Reconstruction: 101 keyset queries, 1.56 seconds, + approximately 2.05 MB serialized entries. This is total startup/backlog cost, + not a constant-space claim; the coordinator retains outstanding entries. +- Drain after two pages stops at two queries; an already-aborted request makes + zero queries. Interruption is between pages, not cancellation of an active SQL. +- First/last backlog entries both commit real reassessments, disappear from + scoped discovery, and do not replay: 37/36 SQL statements, about 32/15 ms. + +Evidence: /tmp/sleep50-scale-measure.mts, .log/.status (exit 0), and +/tmp/sleep50-scale-{startup,company,issue,run}.plan. No new index was justified by +this selected-history measurement. It is local PostgreSQL evidence, not a remote +network latency or fleet savings claim. + +A second fixture exercises retired-review reassessment with 10,000 resolved +interactions, 10,000 comments, 10,000 linked approved approvals, 1,000 completed +prerequisites sharing a workspace, and 10,000 successful workspace operations. +One historical question forces the comment source-lock path too. Scoped discovery +is one query/about 5.9 ms; complete guarded reassessment uses 83 SQL statements +(including nine NOWAIT queries), takes about 126 ms, completes the task, and then +returns zero candidates. Source locks are scoped to this task and its referenced +prerequisites/workspaces; work grows with that history. The measurement does not +assert that a remote database round trip has the same cost. + +Two failed benchmark attempts are retained. The first manually assigned version +2 while a status-update trigger advanced the actual version to 4, so it measured +the stale-status path and failed the intended completion assertion. A repair that +rewound that already-processed fixture then reused its existing append-only +assessment and hit status_decisions_company_assessment_uq. Neither failure was +waived or changed in production code. A fresh untouched task with the actual +trigger-generated version exercised the intended retired-review path. Evidence: +/tmp/sleep50-task-history-measure.log/.status and +/tmp/sleep50-task-history-final.log/.status (failures), fixture/repair logs under +/tmp/sleep50-task-history-*, and /tmp/sleep50-task-history-fresh.log/.status (exit 0). + +The permanent lifecycle suite now seeds 103 actual retired-review obligations, +verifies full discovery, one-page stop on admission loss, zero-query abort, +coordinator dispatch across batches, exactly one additional decision per task, +no rediscovery, and SQL quiet after settlement. All 31 lifecycle cases pass against +retained history, and server typecheck passes: +/tmp/sleep50-scale-lifecycle.log/.status and /tmp/sleep50-scale-types.log/.status. +All handles are terminal. Startup registration/periodic removal, remaining +producer/authority audit and remote acceptance still remain; the full stack +objective and its earlier unresolved gates are not complete. + + +## Startup integration and periodic ownership, 2026-10-26 + +The worker now registers through heartbeat under enabled maintenance, with the +same warm-standby, idle-drain and stopped gates as its sibling owners. The shared +coordinator owns shutdown and restart reconstruction. The periodic orphan reaper +passes `skipNativeStatusReassessment`; explicit/manual and one-time startup +recovery retain the existing reassessment entry point. + +The old reassessment call also supplied a superseded-decision early return before +workspace recovery. Removing the whole call without preserving that guard would +allow obsolete history into unrelated recovery. The reassessment owner now shares +that pure authority predicate and exposes a narrow guard for the periodic path. +It makes no queries for the usual matching-decision case, and reads only selected +decision/assessment authority when IDs differ; it does not classify evidence or +create assessments. Independent workspace settlement and committed result replay +remain owned by finalization recovery. Reassessment's dormant outcome no longer +implicitly suppresses those separate duties in the periodic path; their own +ownership and physical-resource guards remain applicable. + +Before the wiring change, the new regressions failed because periodic recovery +completed the task itself and startup never registered the worker. Historical +supersession was verified with the new option both enabled and disabled. Evidence: +`/tmp/sleep50-startup-before.log/.status` (two failures, four passing selected cases). +After the repair, all 246 corpus/startup/lifecycle cases pass and server typecheck +passes: `/tmp/sleep50-startup-after.log/.status` and +`/tmp/sleep50-startup-types.log/.status`. The existing native finalization and +retained cleanup consumer suite passes all 32 cases in a dedicated database: +`/tmp/sleep50-startup-consumers.log/.status`. All command handles are terminal. + +This integrates only reassessment ownership. The aggregate orphan/finalization +sweep still runs for the other listed duties, so this is not whole-process SQL +quiet or host suspension. Remaining producer/authority audit, final acceptance, +publication and remote PR gates remain open, along with the older stack findings +and every remaining recurring-server obligation. No merge occurred. + + +## Producer closeout and PR validation checkpoint + +The final source audit distinguishes reactivation from negative evidence. Handoff +creation uses a fresh identity in its transaction; delivery/renewal/completion +preserve accepted evidence. Cancellation can only revoke it and is locked by the +scheduled evidence guard. Existing task cancellation/assignment writers signal +through the task boundary. No permanent lease-renewal wake was introduced. +Target-agent existence affects attention validation, not criterion/verification +acceptance, so lifecycle status changes do not activate the three reassessment +predicates. Arbitrary missing identities have no row lock; the worker contract +states this limit rather than claiming a general identity-insertion fence. +Child completion remains a fresh-turn obligation enforced at status commit. +Latest-contract creation can invalidate an old result, not make it eligible again. +The only dormant retired-review prerequisites that can be restored in normal +operation are current run/task authority and workspace settlement/ownership; +their real transaction producers and lifecycle tests are recorded above. + +At 0337779e6d, workspace typecheck and build pass, in addition to the 278 selected +integration/consumer tests. Evidence: /tmp/sleep50-workspace-types.log/.status and +/tmp/sleep50-build.log/.status. All handles are terminal. Fresh origin/master is +still 1fbbef45a8d399771add7a27a04b12f3a20e264c; the PR base remains slice 49. +No duplicate reassessment PR was found; related automatic-review migrations are +#17679 and #17680. Full-stack master reconciliation remains separate unfinished +work; the PR is not represented as a direct-master diff. + +A direct disposable embedded initdb probe explains earlier full-suite failures: +FATAL could not create semaphores, No space left on device (semget), before tests. +/tmp/sleep50-pg-init-probe.log/.status preserves the failure. Read-only IPC/process +inspection found a candidate orphan group, but automatic approval review rejected +its removal because it changes shared OS resources and PID checks can race. +Explicit approval for those exact IDs is pending; no cleanup was executed. This +blocks local embedded-database acceptance, not dedicated PostgreSQL validation or +remote PR validation. The full goal is still incomplete and no merge is authorized. + + +## Review repair: direct claim release + +Draft #17709 at 91a6c14170 passed all 52 CI checks but Greptile scored 4/5 and +identified a missing adminForceRelease wake (discussion 4376190896). This invalidates +the preceding producer-closeout claim for direct task-claim writers. The ordinary +task update method did not cover their separate transactions. + +Six real regressions fail before repair: operator force-release, ordinary release, +terminal execution cleanup, terminal checkout cleanup, the queue's actual release +adapter, and outer force-release commit after rollback. Each starts with a retired +review dormant behind a different execution owner. The missing notification leaves +it untouched after that owner is released. /tmp/sleep50-release-before.log/.status +retains all six failures. + +The release owners now await task-scoped reassessment intent before their existing +writes. The audit also covers queued-comment release, watchdog settlement, legacy +and native recovery, safe replacement, tree restoration and stale-lock cleanup. +Existing run-dispatch and queue branches that already notify task-wide milestones +retain that common hook. Case-handoff release already delegates to issueService +update. Rare member-archive bulk reassignment signals its company. No database +transaction wrappers, recurring retries or provider logic are added. + +The first repaired run passed 36/37 cases. Ordinary release deliberately clears +the assignee, which advances task authority; the worker correctly appended a +preserve decision instead of completing the task. The test now requires that +preserved status, cleared assignee and exactly one additional decision. The initial +incorrect completion expectation remains recorded in /tmp/sleep50-release-after.log. +All 37 lifecycle cases now pass, including outer rollback/commit and SQL quiet. + +Affected consumer suites pass: queue 25, queued comments 5, execution control 5, +safe replacement 59, access 18; native executor/restart, legacy recovery and tree +unit selection 829 pass with 15 known embedded-DB cases excluded. Thus 978 distinct +selected cases pass after repair. This is not full local-suite acceptance. Server +typecheck passes. Evidence: /tmp/sleep50-release-final.log/.status, +/tmp/sleep50-release-{queue,queued_comment,control,safe,access,runtime,types}.log +and .status. All launched command handles are terminal. Fresh-head remote CI and +review remain required. The full stack and its earlier unresolved gates remain open. + + +## Current master and landed parent qualification (2026-10-29) + +Parent #17702 merged from exact reviewed head +`902817c46675d71fb17324683f39b9a39d6135e0` as +`79ce86630a79dad65e3b753c13e3d49e22bbca2f` with all 57 checks clean and +Greptile 5/5. No Ponytail review was present, which the authorized merge policy +allows. The normal parent merge retains 12661 parent/unrelated tree entries and +35 owned entries exactly. Twenty-one overlapping paths retain Git's automatic +three-way result. Five original conflicts are reviewed explicitly: scoped wake +fanout, conversation/session source locks, the extracted reassessment block, +and the unsafe-workspace omission-barrier notification. The complete held-child +tree is unchanged when reconciling the parent's actual squash ancestry. + +The current parent's required quality-review policy also moves with the +reassessment owner. A required review is a business gate, not an operational +failure. Two real regressions exercise this on retired-review reassessment and +on committed work-product approval. Each optional control passes before repair; +each required case instead retains retries, reports repeated +`required_quality_review_incomplete` exceptions and reaches the unchanged +15-second limit. Porting the retired-review policy fixes the first path. +Reading the current task's quality policy for all completion-producing +reassessments fixes the evidence path too. Both required cases now record the +intended in-progress decision and become SQL quiet; neither bypasses the review. +The same assertions and time limits are retained. + +Two more real SQL regressions cover dormant reassessment after three coarse +company-only milestone/finalization hints. A controlled negative baseline removes +only the ordinary issue/run guards; it reproduces nonzero response-history +reconstruction. Restoring the guards byte for byte passes the same cases with +zero reconstruction queries. Routine wake paths now remain scoped across all +three workers; the explicit company authority boundary still wakes genuine global +changes. Actual coalesced input and concurrent Photon recovery exercise all three +queues. Ten active Photon assertions produce zero hints/SELECTs per worker; +three concurrent attention assertions produce one wake/SELECT, then return quiet. +Paperclip callbacks and PostgreSQL are real; external Photon transport is mocked. + +Broad source `9fddb3f0b1157df32b64fb1f22e9d6e9624470e8` passes **2731 +distinct cases across 28 complete files**, with no skipped cases: 2458 across +27 producer/owner/authority/workspace/quality files in 394.43 seconds, then all +273 governed-response cases in 53.74 seconds. Nineteen dedicated PostgreSQL 17 +fixtures migrate through 0413. Eighteen embedded-only fixture invocations use +the existing startRecoveryPostgres external-host helper, preserving a fresh +owned database per call and normal cleanup. All 22 temporary import adaptations +restore exactly and all 28 captured source hashes match. This qualifies external +PostgreSQL behavior, not embedded bootstrap on this semaphore-exhausted host. + +The later evidence-trigger fix changes only the reassessment owner and adds its +two regressions. Final runtime source +`5ea3b0fd016bf7e63ba82a7044825a46a0cd2082` passes **1211 distinct cases in +five complete affected files** in 108.23 seconds: 43 worker cases, 145 status +corpus cases, 32 finalization-recovery cases, 968 native-session cases and 23 +quality-review cases. Four dedicated fresh 0413 databases plus the quality +suite's own fresh helper fixture use PostgreSQL 17. Its one temporary import +restores exactly; all five hashes match. The other 27 broad-run test files are +unchanged. These results keep their separate source scopes; overlapping cases +are not cumulatively counted or described as a fresh complete 2731-case rerun. + +Database build, production server types/build, module boundaries and full-SHA +migration order pass. Fourteen added comments in six files were removed with +identical nontrivia token streams and no directives removed. The final added- +comment check is empty. The database tree is identical to the landed parent: +no migration, snapshot generation or queue table is introduced here. + +Earlier failures remain part of the record, including the two quality retry +loops, the controlled scope negatives, and inherited historical host/fixture +limitations. The local evidence-trigger runner reused an earlier composition-tree +metadata field; its addendum identifies the precise 9fddb production baseline +and before/after owner hashes without altering outcomes or test hashes. The first +actual-squash proof attempt lacked the newly created Git object; fetching the +object and rerunning establishes exact parent/tree identity before mutation. +No application failure was classified as infrastructure from a successful retry. + +This is targeted local acceptance, not a full repository-suite or deployed-sleep +claim. Fresh exact-head CI, Greptile 5/5, and resolution of actionable reviews +remain required before merging this PR. Other workspace/session/cleanup duties +and Cloud sleep eligibility remain separate. diff --git a/server/src/__tests__/heartbeat-run-event-sequencing.test.ts b/server/src/__tests__/heartbeat-run-event-sequencing.test.ts index d8e7bd0b9cb..98a002f8557 100644 --- a/server/src/__tests__/heartbeat-run-event-sequencing.test.ts +++ b/server/src/__tests__/heartbeat-run-event-sequencing.test.ts @@ -1,3 +1,4 @@ +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; import { describe, expect, it } from "vitest"; import { eq, sql } from "drizzle-orm"; import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; @@ -15,6 +16,15 @@ import { } from "../services/heartbeat-run-events.js"; import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_EVENT_SEQUENCING_TEST_DATABASE_URL"); +async function clearExternalFixture(db: ReturnType, companyId: string) { + if (!externalUrl) return; + await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.companyId, companyId)); + await db.delete(heartbeatRuns).where(eq(heartbeatRuns.companyId, companyId)); + await db.delete(agents).where(eq(agents.companyId, companyId)); + await db.delete(companies).where(eq(companies.id, companyId)); +} + describe("P6-11..13 / P6-17 canonical event allocator", () => { it("keeps native cursor responses bounded with long histories and a late gap fill", async () => { const temporary = await startEmbeddedPostgresTestDatabase("paperclip-native-cursor-budget-"); @@ -73,9 +83,9 @@ describe("P6-11..13 / P6-17 canonical event allocator", () => { }, 60_000); it.each([false, true])("deduplicates concurrent retry exhaustion (historical receipt: %s)", async (historical) => { - const temporary = await startEmbeddedPostgresTestDatabase("paperclip-exhaustion-events-"); - const db = createDb(temporary.connectionString); - const otherDb = createDb(temporary.connectionString); + const temporary = externalUrl ? undefined : await startEmbeddedPostgresTestDatabase("paperclip-exhaustion-events-"); + const db = createDb(externalUrl ?? temporary!.connectionString); + const otherDb = createDb(externalUrl ?? temporary!.connectionString); const companyId = "20000000-0000-4000-8000-000000000001"; const agentId = "20000000-0000-4000-8000-000000000002"; const runId = "20000000-0000-4000-8000-000000000003"; @@ -129,13 +139,16 @@ describe("P6-11..13 / P6-17 canonical event allocator", () => { ...event, companyId: "20000000-0000-4000-8000-000000000099", retryExhaustion: exhaustion, })).rejects.toThrow("heartbeat_run_event_binding_mismatch"); } finally { - await temporary.cleanup(); + await clearExternalFixture(db, companyId); + await otherDb.$client.end({ timeout: 1 }); + await db.$client.end({ timeout: 1 }); + await temporary?.cleanup(); } }, 60_000); it("serializes concurrent writers and rejects conflicting replay without cursor drift", async () => { - const temporary = await startEmbeddedPostgresTestDatabase("paperclip-native-events-"); - const db = createDb(temporary.connectionString); + const temporary = externalUrl ? undefined : await startEmbeddedPostgresTestDatabase("paperclip-native-events-"); + const db = createDb(externalUrl ?? temporary!.connectionString); const companyId = "20000000-0000-4000-8000-000000000001"; const agentId = "20000000-0000-4000-8000-000000000002"; const runId = "20000000-0000-4000-8000-000000000003"; @@ -219,7 +232,9 @@ describe("P6-11..13 / P6-17 canonical event allocator", () => { })).rejects.toBeInstanceOf(HeartbeatRunEventConflictError); expect(await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, runId))).toHaveLength(48); } finally { - await temporary.cleanup(); + await clearExternalFixture(db, companyId); + await db.$client.end({ timeout: 1 }); + await temporary?.cleanup(); } }, 60_000); diff --git a/server/src/__tests__/issues-service.test.ts b/server/src/__tests__/issues-service.test.ts index 737039c5463..bbf94717ff3 100644 --- a/server/src/__tests__/issues-service.test.ts +++ b/server/src/__tests__/issues-service.test.ts @@ -57,6 +57,8 @@ import { } from "../services/execution-workspace-policy.ts"; import { buildAgentMentionHref, buildProjectMentionHref, MAX_ISSUE_REQUEST_DEPTH, type IssueWorkMode } from "@paperclipai/shared"; +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; + const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; @@ -4111,14 +4113,15 @@ describeEmbeddedPostgres("issueService.create workspace inheritance", () => { }); }); -describeEmbeddedPostgres("issueService blockers and dependency wake readiness", () => { +const dependencyTestDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_ISSUE_DEPENDENCIES_TEST_DATABASE_URL"); +(dependencyTestDatabaseUrl ? describe : describeEmbeddedPostgres)("issueService blockers and dependency wake readiness", () => { let db!: ReturnType; let svc!: ReturnType; let tempDb: Awaited> | null = null; beforeAll(async () => { - tempDb = await startEmbeddedPostgresTestDatabase("paperclip-issues-blockers-"); - db = createDb(tempDb.connectionString); + if (!dependencyTestDatabaseUrl) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-issues-blockers-"); + db = createDb(dependencyTestDatabaseUrl ?? tempDb!.connectionString); svc = issueService(db); await ensureIssueRelationsTable(db); }, 20_000); @@ -4139,6 +4142,7 @@ describeEmbeddedPostgres("issueService blockers and dependency wake readiness", }); afterAll(async () => { + await db?.$client.end({ timeout: 5 }); await tempDb?.cleanup(); }); diff --git a/server/src/__tests__/low-trust-red-team-routes.test.ts b/server/src/__tests__/low-trust-red-team-routes.test.ts index 68ef4c60ce3..598cdfd81cf 100644 --- a/server/src/__tests__/low-trust-red-team-routes.test.ts +++ b/server/src/__tests__/low-trust-red-team-routes.test.ts @@ -1,3 +1,4 @@ +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; import { randomUUID } from "node:crypto"; import { createServer } from "node:http"; import express from "express"; @@ -55,12 +56,13 @@ import { issueService } from "../services/issues.js"; import { heartbeatService } from "../services/heartbeat.js"; import { LOW_TRUST_QUARANTINED_BODY } from "../services/source-trust.js"; +const externalTestDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_LOW_TRUST_TEST_DATABASE_URL"); const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); -const describeEmbeddedPostgres = embeddedPostgresSupport.supported +const describeEmbeddedPostgres = externalTestDatabaseUrl || embeddedPostgresSupport.supported ? describe : describe.skip; -if (!embeddedPostgresSupport.supported) { +if (!externalTestDatabaseUrl && !embeddedPostgresSupport.supported) { console.warn( `Skipping embedded Postgres low-trust route tests on this host: ${ embeddedPostgresSupport.reason ?? "unsupported environment" @@ -885,10 +887,10 @@ describeEmbeddedPostgres( } beforeAll(async () => { - tempDb = await startEmbeddedPostgresTestDatabase( + if (!externalTestDatabaseUrl) tempDb = await startEmbeddedPostgresTestDatabase( "paperclip-low-trust-red-team-routes-", ); - db = createDb(tempDb.connectionString); + db = createDb(externalTestDatabaseUrl ?? tempDb!.connectionString); }, 20_000); afterEach(async () => { @@ -927,6 +929,7 @@ describeEmbeddedPostgres( }); afterAll(async () => { + await db?.$client.end({ timeout: 1 }); await tempDb?.cleanup(); }); diff --git a/server/src/__tests__/native-status-arbiter-corpus.test.ts b/server/src/__tests__/native-status-arbiter-corpus.test.ts index cf330f4105a..6a0d69db056 100644 --- a/server/src/__tests__/native-status-arbiter-corpus.test.ts +++ b/server/src/__tests__/native-status-arbiter-corpus.test.ts @@ -1,3 +1,8 @@ +import { pendingNativeGovernance } from "../services/native-runtime/native-governance.js"; +import { withNativeWorkspaceFinalizationOwnership } from "../services/native-runtime/native-workspace-finalization-ownership.js"; +import { ensureNativeCompletionContract } from "../services/native-runtime/completion-contracts.js"; +import { nativeStatusReassessmentWork } from "../services/native-runtime/native-status-reassessment-work.js"; +import { reassessNativeStatusFromSource } from "../services/native-runtime/native-status-reassessment.js"; import { NativeRunCoordinatorStore } from "../services/native-runtime/native-run-coordinator-store.js"; import { registerNativePendingEffectWork } from "../services/native-runtime/native-pending-effect-work.js"; import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; @@ -9,12 +14,22 @@ import { randomUUID } from "node:crypto"; import { readFileSync } from "node:fs"; import { fileURLToPath } from "node:url"; import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; -import { and, eq } from "drizzle-orm"; +import { and, eq, sql } from "drizzle-orm"; import { agents, + assets, + caseHandoffs, + cases, + heartbeatRunEvents, + issueAttachments, + approvals, + issueApprovals, + issueComments, agentWakeupRequests, companies, completionContracts, + executionWorkspaces, + issueRelations, createDb, heartbeatRuns, issueThreadInteractions, @@ -2451,6 +2466,514 @@ describe("P6-31 Section 18.13 executable status-authority corpus", () => { return seeded; } + it.each(["future", "claim", "unknown claim", "assessment", "result", "phase", "decision", "effects", "ownership", "run outcome", "task execution", "contract revision", "contract writer", "workspace status", "workspace operation", "workspace writer", "workspace owner", "workspace owner receipt"] as const)( + "guards scheduled reassessment against a changed %s at the commit boundary", async changed => { + const seeded = await seedAutomaticReview(); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + const [source] = await db.select({ + runId: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + issueId: issues.id, issueStatus: issues.status, issueStatusVersion: issues.statusVersion, + issueDecisionId: issues.lastStatusDecisionId, coordinatorPhase: nativeRunFinalizations.phase, + assessmentId: nativeRunFinalizations.assessmentId, decisionId: nativeRunFinalizations.decisionId, + contextSnapshot: heartbeatRuns.contextSnapshot, + }).from(nativeRunFinalizations).innerJoin(heartbeatRuns, eq(heartbeatRuns.id, nativeRunFinalizations.runId)) + .innerJoin(issues, eq(issues.id, nativeRunFinalizations.issueId)) + .where(eq(nativeRunFinalizations.runId, seeded.runId)); + const selected = { ...source, assessmentId: source.assessmentId! }; + const future = new Date(Date.now() + 60_000); + const change = changed === "future" ? { nextAttemptAt: future } + : changed === "claim" ? { leaseOwner: "other-finalizer", leaseExpiresAt: future } + : changed === "unknown claim" ? { leaseOwner: "unknown-finalizer", leaseExpiresAt: null } + : changed === "assessment" ? { assessmentId: null, decisionId: null } + : changed === "result" ? { resultId: null, assessmentId: null, decisionId: null } + : changed === "phase" ? { phase: "terminal_failure" } + : changed === "decision" ? { decisionId: null } : null; + const [originalRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.runId)); + const [originalIssue] = await db.select().from(issues).where(eq(issues.id, seeded.issueId)); + const [originalBarrier] = await db.select().from(workspaceOperations).where(and( + eq(workspaceOperations.heartbeatRunId, seeded.runId), eq(workspaceOperations.phase, "workspace_finalize"))); + const newBarrierId = randomUUID(); + const replacementRunId = randomUUID(); + if (changed === "task execution") await db.insert(heartbeatRuns).values({ + id: replacementRunId, companyId, agentId, status: "running", runtimeMode: "native", + nativeIssueId: seeded.issueId, contextSnapshot: { issueId: seeded.issueId }, + }); + const originalEffects = await db.select().from(statusDecisionEffects).where(eq(statusDecisionEffects.decisionId, selected.decisionId!)).orderBy(statusDecisionEffects.ordinal); + expect(originalEffects.length).toBeGreaterThan(0); + const transaction = db.transaction.bind(db); + let newContractId: string | undefined; + let releaseWriter: (() => void) | undefined; + let writer: Promise | undefined; + const writeContract = async (owner: typeof db) => { + const saved = await ensureNativeCompletionContract({ db: owner, companyId, + issue: { id: seeded.issueId, title: "New direction", description: "Verify the revised objective" }, actorId: agentId }); + newContractId = saved.row.id; + }; + const intercept = vi.spyOn(db, "transaction").mockImplementationOnce((async (callback, options) => { + if (change) await db.update(nativeRunFinalizations).set(change).where(eq(nativeRunFinalizations.runId, seeded.runId)); + if (changed === "effects") await db.update(statusDecisionEffects).set({ deliveryState: "pending" }) + .where(eq(statusDecisionEffects.decisionId, selected.decisionId!)); + if (changed === "ownership") await db.update(heartbeatRuns).set({ status: "running", + runtimeMode: "native", errorCode: "native_execution_ownership_unverified", nativePhase: "terminal_failure" }) + .where(eq(heartbeatRuns.id, seeded.runId)); + if (changed === "run outcome") await db.update(heartbeatRuns).set({ status: "failed" }) + .where(eq(heartbeatRuns.id, seeded.runId)); + if (changed === "task execution") await db.update(issues).set({ executionRunId: replacementRunId }) + .where(eq(issues.id, seeded.issueId)); + if (changed === "workspace status") await db.update(workspaceOperations).set({ status: "failed" }) + .where(eq(workspaceOperations.id, originalBarrier.id)); + if (changed === "workspace operation") await db.insert(workspaceOperations).values({ + id: newBarrierId, companyId, heartbeatRunId: seeded.runId, issueId: seeded.issueId, + phase: "workspace_finalize", status: "running", cwd: process.cwd(), + createdAt: new Date(Math.max(Date.now(), originalBarrier.createdAt.getTime() + 1)), + }); + if (changed === "workspace owner receipt") await db.update(heartbeatRuns).set({ + runnerProfileJson: { ...originalRun.runnerProfileJson, nativeWorkspaceFinalizationOwner: { + token: randomUUID(), hostname: "unverified-controller", pid: 12345, processStartedAt: new Date().toISOString(), + } }, + }).where(eq(heartbeatRuns.id, seeded.runId)); + if (changed === "workspace owner") { + let entered!: () => void; + const ready = new Promise(resolve => { entered = resolve; }); + const released = new Promise(resolve => { releaseWriter = resolve; }); + writer = withNativeWorkspaceFinalizationOwnership({ db, companyId, runId: seeded.runId }, async () => { + entered(); await released; + }).then(() => undefined); + await Promise.race([ready, writer]); + } + if (changed === "contract revision") await writeContract(db); + if (changed === "contract writer" || changed === "workspace writer") { + let writerReady!: () => void; + const ready = new Promise(resolve => { writerReady = resolve; }); + const released = new Promise(resolve => { releaseWriter = resolve; }); + const rollback = new Error("rollback competing writer"); + writer = transaction(async tx => { + if (changed === "contract writer") await writeContract(tx as unknown as typeof db); + else await tx.update(workspaceOperations).set({ status: "failed" }) + .where(eq(workspaceOperations.id, originalBarrier.id)); + writerReady(); + await released; + throw rollback; + }).catch(error => { if (error !== rollback) throw error; }); + await Promise.race([ready, writer]); + } + return transaction(async tx => { + if (changed === "contract writer" || changed === "workspace writer") await tx.execute(sql`set local statement_timeout = '2s'`); + return callback(tx); + }, options); + }) as typeof db.transaction); + try { + expect(await reassessNativeStatusFromSource(db, selected, { onlyWhenDue: true })).toEqual({ kind: "deferred", retry: true }); + } finally { intercept.mockRestore(); releaseWriter?.(); await writer; } + expect((await issueService(db).getById(seeded.issueId))!.status).toBe("in_review"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(1); + if (change) expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seeded.runId)))[0]).toMatchObject(change); + if (changed === "effects") { + expect(await db.select().from(statusDecisionEffects).where(eq(statusDecisionEffects.decisionId, selected.decisionId!)).orderBy(statusDecisionEffects.ordinal)) + .toEqual(originalEffects.map(effect => ({ ...effect, deliveryState: "pending" }))); + for (const effect of originalEffects) await db.update(statusDecisionEffects).set({ deliveryState: effect.deliveryState }) + .where(eq(statusDecisionEffects.id, effect.id)); + } + if (changed === "ownership") { + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.runId)))[0]) + .toMatchObject({ status: "running", errorCode: "native_execution_ownership_unverified" }); + await db.update(heartbeatRuns).set({ status: originalRun.status, runtimeMode: originalRun.runtimeMode, + errorCode: originalRun.errorCode, nativePhase: originalRun.nativePhase }).where(eq(heartbeatRuns.id, seeded.runId)); + } + if (changed === "run outcome") { + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.runId)))[0].status).toBe("failed"); + await db.update(heartbeatRuns).set({ status: originalRun.status }).where(eq(heartbeatRuns.id, seeded.runId)); + } + if (changed === "task execution") { + expect((await issueService(db).getById(seeded.issueId))!.executionRunId).toBe(replacementRunId); + await db.update(issues).set({ executionRunId: originalIssue.executionRunId }).where(eq(issues.id, seeded.issueId)); + await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, replacementRunId)); + } + if (changed === "contract revision") { + expect(await reassessNativeStatusFromSource(db, selected, { onlyWhenDue: true })).toEqual({ kind: "deferred" }); + await db.delete(completionContracts).where(eq(completionContracts.id, newContractId!)); + } + if (changed === "contract writer") expect(await db.select().from(completionContracts) + .where(eq(completionContracts.id, newContractId!))).toHaveLength(0); + if (changed === "workspace status" || changed === "workspace operation") { + expect(await reassessNativeStatusFromSource(db, selected, { onlyWhenDue: true })).toEqual({ kind: "deferred" }); + if (changed === "workspace status") await db.update(workspaceOperations).set({ status: originalBarrier.status }) + .where(eq(workspaceOperations.id, originalBarrier.id)); + else await db.delete(workspaceOperations).where(eq(workspaceOperations.id, newBarrierId)); + } + if (changed === "workspace writer") expect((await db.select().from(workspaceOperations) + .where(eq(workspaceOperations.id, originalBarrier.id)))[0].status).toBe(originalBarrier.status); + if (changed === "workspace owner receipt") { + expect(await reassessNativeStatusFromSource(db, selected, { onlyWhenDue: true })).toEqual({ kind: "deferred" }); + await db.update(heartbeatRuns).set({ runnerProfileJson: originalRun.runnerProfileJson }) + .where(eq(heartbeatRuns.id, seeded.runId)); + } + await db.update(nativeRunFinalizations).set({ phase: selected.coordinatorPhase, + resultId: seeded.resultId, assessmentId: selected.assessmentId, decisionId: selected.decisionId, + nextAttemptAt: null, leaseOwner: null, leaseExpiresAt: null }).where(eq(nativeRunFinalizations.runId, seeded.runId)); + expect(await reassessNativeStatusFromSource(db, selected, { onlyWhenDue: true })).toMatchObject({ kind: "committed" }); + expect((await issueService(db).getById(seeded.issueId))!.status).toBe("done"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2); + }, 30_000, + ); + + it.each(["execution policy", "approval link", "approval reopening", "interaction", "approval writer", "interaction writer", "question history writer"] as const)( + "fences reassessment against changed governance: %s", async changed => { + const seeded = await seedAutomaticReview(); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + const [source] = await db.select({ + runId: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + issueId: issues.id, issueStatus: issues.status, issueStatusVersion: issues.statusVersion, + issueDecisionId: issues.lastStatusDecisionId, coordinatorPhase: nativeRunFinalizations.phase, + assessmentId: nativeRunFinalizations.assessmentId, decisionId: nativeRunFinalizations.decisionId, + contextSnapshot: heartbeatRuns.contextSnapshot, + }).from(nativeRunFinalizations).innerJoin(heartbeatRuns, eq(heartbeatRuns.id, nativeRunFinalizations.runId)) + .innerJoin(issues, eq(issues.id, nativeRunFinalizations.issueId)) + .where(eq(nativeRunFinalizations.runId, seeded.runId)); + const selected = { ...source, assessmentId: source.assessmentId! }; + const approvalId = randomUUID(), interactionId = randomUUID(), commentId = randomUUID(); + const approvalCase = changed.startsWith("approval"); + if (approvalCase) { + await db.insert(approvals).values({ id: approvalId, companyId, type: "approve_ceo_strategy", payload: {}, + status: changed === "approval link" ? "pending" : "approved" }); + if (changed !== "approval link") await db.insert(issueApprovals).values({ companyId, issueId: seeded.issueId, approvalId }); + } + if (changed === "interaction writer" || changed === "question history writer") { + await db.insert(issueThreadInteractions).values({ id: interactionId, companyId, issueId: seeded.issueId, + sourceRunId: seeded.runId, kind: changed === "question history writer" ? "ask_user_questions" : "request_confirmation", + status: changed === "question history writer" ? "pending" : "accepted", + payload: changed === "question history writer" + ? { version: 1, questions: [{ id: "review", prompt: "Continue?", selectionMode: "single", + options: [{ id: "yes", label: "Yes" }] }] } + : { version: 1, prompt: "Review" }, + createdAt: new Date(Date.now() - 2000) }); + if (changed === "question history writer") await db.insert(issueComments).values({ id: commentId, + companyId, issueId: seeded.issueId, authorUserId: "operator", authorType: "user", body: "Continue with the new direction" }); + } + const scope = { companyId, issueId: seeded.issueId, runId: seeded.runId, executionState: null }; + expect(await pendingNativeGovernance({ db, ...scope })).toBeNull(); + const transaction = db.transaction.bind(db); + let release!: () => void; + let writer: Promise | undefined; + const intercept = vi.spyOn(db, "transaction").mockImplementationOnce((async (callback, options) => { + if (changed === "execution policy") await db.update(issues).set({ executionState: { status: "pending" } }) + .where(eq(issues.id, seeded.issueId)); + if (changed === "approval link") await db.insert(issueApprovals).values({ companyId, issueId: seeded.issueId, approvalId }); + if (changed === "approval reopening") await db.update(approvals).set({ status: "revision_requested" }).where(eq(approvals.id, approvalId)); + if (changed === "interaction") await db.insert(issueThreadInteractions).values({ id: interactionId, + companyId, issueId: seeded.issueId, sourceRunId: seeded.runId, kind: "request_confirmation", + status: "pending", payload: { version: 1, prompt: "Review" } }); + if (changed.endsWith("writer")) { + let entered!: () => void; + const ready = new Promise(resolve => { entered = resolve; }); + const released = new Promise(resolve => { release = resolve; }); + const rollback = new Error("rollback governance writer"); + writer = transaction(async tx => { + if (changed === "approval writer") await tx.update(approvals).set({ status: "revision_requested" }).where(eq(approvals.id, approvalId)); + if (changed === "interaction writer") await tx.update(issueThreadInteractions).set({ status: "pending" }).where(eq(issueThreadInteractions.id, interactionId)); + if (changed === "question history writer") await tx.update(issueComments).set({ deletedAt: new Date() }).where(eq(issueComments.id, commentId)); + entered(); await released; throw rollback; + }).catch(error => { if (error !== rollback) throw error; }); + await Promise.race([ready, writer]); + } + return transaction(async tx => { + await tx.execute(sql`set local statement_timeout = '2s'`); + return callback(tx); + }, options); + }) as typeof db.transaction); + try { + expect(await reassessNativeStatusFromSource(db, selected, { onlyWhenDue: true })).toEqual({ kind: "deferred", retry: true }); + } finally { intercept.mockRestore(); release?.(); await writer; } + expect((await issueService(db).getById(seeded.issueId))!.status).toBe("in_review"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(1); + if (changed === "execution policy") await db.update(issues).set({ executionState: null }).where(eq(issues.id, seeded.issueId)); + if (changed === "approval link") await db.delete(issueApprovals).where(eq(issueApprovals.approvalId, approvalId)); + if (changed === "approval reopening") await db.update(approvals).set({ status: "approved" }).where(eq(approvals.id, approvalId)); + if (changed === "interaction") await db.delete(issueThreadInteractions).where(eq(issueThreadInteractions.id, interactionId)); + expect(await reassessNativeStatusFromSource(db, selected, { onlyWhenDue: true })).toMatchObject({ kind: "committed" }); + expect((await issueService(db).getById(seeded.issueId))!.status).toBe("done"); + }, 30_000, + ); + + it.each(["new blocker", "reopened blocker", "blocker writer", "workspace operation", "workspace status", "workspace writer", "workspace insert writer"] as const)( + "fences reassessment against changed dependency readiness: %s", async changed => { + const seeded = await seedAutomaticReview(); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + const [source] = await db.select({ + runId: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + issueId: issues.id, issueStatus: issues.status, issueStatusVersion: issues.statusVersion, + issueDecisionId: issues.lastStatusDecisionId, coordinatorPhase: nativeRunFinalizations.phase, + assessmentId: nativeRunFinalizations.assessmentId, decisionId: nativeRunFinalizations.decisionId, + contextSnapshot: heartbeatRuns.contextSnapshot, + }).from(nativeRunFinalizations).innerJoin(heartbeatRuns, eq(heartbeatRuns.id, nativeRunFinalizations.runId)) + .innerJoin(issues, eq(issues.id, nativeRunFinalizations.issueId)) + .where(eq(nativeRunFinalizations.runId, seeded.runId)); + const selected = { ...source, assessmentId: source.assessmentId! }; + const blockerId = randomUUID(), workspaceId = randomUUID(), operationId = randomUUID(); + const workspaceCase = changed.startsWith("workspace"); + if (workspaceCase) await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, + mode: "isolated", strategyType: "git_worktree", name: "Dependency workspace" }); + await db.insert(issues).values({ id: blockerId, companyId, title: "Prerequisite", + status: changed === "new blocker" ? "todo" : "done", executionWorkspaceId: workspaceCase ? workspaceId : null }); + if (changed !== "new blocker") await db.insert(issueRelations).values({ companyId, issueId: blockerId, relatedIssueId: seeded.issueId, type: "blocks" }); + if (changed === "workspace status" || changed === "workspace writer") await db.insert(workspaceOperations).values({ + id: operationId, companyId, issueId: blockerId, executionWorkspaceId: workspaceId, phase: "workspace_finalize", status: "succeeded" }); + expect((await issueService(db).getDependencyReadiness(seeded.issueId)).isDependencyReady).toBe(true); + const transaction = db.transaction.bind(db); + let release!: () => void; + let writer: Promise | undefined; + const pendingOperation = { id: operationId, companyId, executionWorkspaceId: workspaceId, + phase: "workspace_finalize", status: "running" }; + const intercept = vi.spyOn(db, "transaction").mockImplementationOnce((async (callback, options) => { + if (changed === "new blocker") await db.insert(issueRelations).values({ companyId, issueId: blockerId, relatedIssueId: seeded.issueId, type: "blocks" }); + if (changed === "reopened blocker") await db.update(issues).set({ status: "todo" }).where(eq(issues.id, blockerId)); + if (changed === "workspace operation") await db.insert(workspaceOperations).values(pendingOperation); + if (changed === "workspace status") await db.update(workspaceOperations).set({ status: "failed" }).where(eq(workspaceOperations.id, operationId)); + if (changed.endsWith("writer")) { + let entered!: () => void; + const ready = new Promise(resolve => { entered = resolve; }); + const released = new Promise(resolve => { release = resolve; }); + const rollback = new Error("rollback dependency writer"); + writer = transaction(async tx => { + if (changed === "blocker writer") await tx.update(issues).set({ status: "todo" }).where(eq(issues.id, blockerId)); + if (changed === "workspace writer") await tx.update(workspaceOperations).set({ status: "failed" }).where(eq(workspaceOperations.id, operationId)); + if (changed === "workspace insert writer") await tx.insert(workspaceOperations).values(pendingOperation); + entered(); await released; throw rollback; + }).catch(error => { if (error !== rollback) throw error; }); + await Promise.race([ready, writer]); + } + return transaction(async tx => { + await tx.execute(sql`set local statement_timeout = '2s'`); + return callback(tx); + }, options); + }) as typeof db.transaction); + try { + expect(await reassessNativeStatusFromSource(db, selected, { onlyWhenDue: true })).toEqual({ kind: "deferred", retry: true }); + } finally { intercept.mockRestore(); release?.(); await writer; } + expect((await issueService(db).getById(seeded.issueId))!.status).toBe("in_review"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(1); + if (changed === "new blocker" || changed === "reopened blocker") await db.update(issues).set({ status: "done" }).where(eq(issues.id, blockerId)); + if (changed === "workspace operation" || changed === "workspace status") await db.update(workspaceOperations).set({ status: "succeeded" }).where(eq(workspaceOperations.id, operationId)); + expect(await reassessNativeStatusFromSource(db, selected, { onlyWhenDue: true })).toMatchObject({ kind: "committed" }); + expect((await issueService(db).getById(seeded.issueId))!.status).toBe("done"); + }, 30_000, + ); + + it.each(["work product", "work product writer", "approval", "interaction", "attachment", "event", "handoff"] as const)( + "fences reassessment against changed evidence: %s", async kind => { + const seeded = await seedAutomaticReview(); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + const evidenceId = randomUUID(), assetId = randomUUID(), caseId = randomUUID(); + let ref = `work_product:${seeded.workProductId}`; + const attachment = { id: evidenceId, companyId, issueId: seeded.issueId, assetId }; + const event = { companyId, agentId, runId: seeded.runId, seq: 90001, eventType: "evidence.recorded", + payload: { prpEvent: { sourceKind: "control_plane", payload: { evidenceOutcome: "accepted" } } } }; + if (kind === "approval") { + ref = `approval:${evidenceId}`; + await db.insert(approvals).values({ id: evidenceId, companyId, type: "hire_agent", status: "approved", payload: {} }); + await db.insert(issueApprovals).values({ companyId, issueId: seeded.issueId, approvalId: evidenceId }); + } + if (kind === "interaction") { + ref = `interaction:${evidenceId}`; + await db.insert(issueThreadInteractions).values({ id: evidenceId, companyId, issueId: seeded.issueId, + kind: "request_confirmation", status: "accepted", payload: { version: 1, prompt: "Evidence accepted" } }); + } + if (kind === "attachment") { + ref = `attachment:${evidenceId}`; + await db.insert(assets).values({ id: assetId, companyId, provider: "local_disk", objectKey: assetId, + contentType: "text/plain", byteSize: 1, sha256: assetId }); + await db.insert(issueAttachments).values(attachment); + } + if (kind === "event") { ref = "event:90001"; await db.insert(heartbeatRunEvents).values(event); } + if (kind === "handoff") { + ref = `case-handoff:${evidenceId}`; + const existing = await db.select({ id: cases.id }).from(cases).where(eq(cases.companyId, companyId)); + await db.insert(cases).values({ id: caseId, companyId, caseNumber: existing.length + 1, + identifier: caseId, caseType: "evidence", title: "Handoff evidence" }); + await db.insert(caseHandoffs).values({ id: evidenceId, companyId, caseId, issueId: seeded.issueId, + version: 1, destinationKind: "done", phase: "complete", instructions: "Complete", actorType: "system", + actorId: "test", sourceRunId: seeded.runId, wakeupRequestId: randomUUID(), idempotencyKey: evidenceId, + inputFingerprint: evidenceId, state: "completed" }); + } + const [storedResult] = await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, seeded.resultId!)); + const result = storedResult.resultJson.result as Record; + await db.update(nativeRunResults).set({ resultJson: { ...storedResult.resultJson, result: { ...result, + completionClaim: { ...(result.completionClaim as object), criteria: [{ criterionId: "objective", status: "satisfied", evidenceRefs: [ref] }] }, + verification: [{ commandOrCheck: "evidence", status: "passed", artifactRef: ref }], + } } }).where(eq(nativeRunResults.id, seeded.resultId!)); + const [source] = await db.select({ + runId: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + issueId: issues.id, issueStatus: issues.status, issueStatusVersion: issues.statusVersion, + issueDecisionId: issues.lastStatusDecisionId, coordinatorPhase: nativeRunFinalizations.phase, + assessmentId: nativeRunFinalizations.assessmentId, decisionId: nativeRunFinalizations.decisionId, + contextSnapshot: heartbeatRuns.contextSnapshot, + }).from(nativeRunFinalizations).innerJoin(heartbeatRuns, eq(heartbeatRuns.id, nativeRunFinalizations.runId)) + .innerJoin(issues, eq(issues.id, nativeRunFinalizations.issueId)).where(eq(nativeRunFinalizations.runId, seeded.runId)); + const selected = { ...source, assessmentId: source.assessmentId! }; + async function change(database: typeof db, revoke: boolean) { + if (kind.startsWith("work product")) await database.update(issueWorkProducts) + .set({ reviewState: revoke ? "rejected" : "approved" }).where(eq(issueWorkProducts.id, seeded.workProductId)); + if (kind === "approval") await database.update(approvals).set({ status: revoke ? "rejected" : "approved" }).where(eq(approvals.id, evidenceId)); + if (kind === "interaction") await database.update(issueThreadInteractions).set({ status: revoke ? "rejected" : "accepted" }).where(eq(issueThreadInteractions.id, evidenceId)); + if (kind === "handoff") await database.update(caseHandoffs).set({ state: revoke ? "cancelled" : "completed" }).where(eq(caseHandoffs.id, evidenceId)); + if (kind === "attachment") { + if (revoke) await database.delete(issueAttachments).where(eq(issueAttachments.id, evidenceId)); + else await database.insert(issueAttachments).values(attachment); + } + if (kind === "event") { + if (revoke) await database.delete(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.runId, seeded.runId), eq(heartbeatRunEvents.seq, event.seq))); + else await database.insert(heartbeatRunEvents).values(event); + } + } + const transaction = db.transaction.bind(db); + let release!: () => void; + let writer: Promise | undefined; + const intercept = vi.spyOn(db, "transaction").mockImplementationOnce((async (callback, options) => { + if (kind === "work product writer") { + let entered!: () => void; + const ready = new Promise(resolve => { entered = resolve; }); + const released = new Promise(resolve => { release = resolve; }); + const rollback = new Error("rollback evidence writer"); + writer = transaction(async tx => { await change(tx as unknown as typeof db, true); entered(); await released; throw rollback; }) + .catch(error => { if (error !== rollback) throw error; }); + await Promise.race([ready, writer]); + } else await change(db, true); + return transaction(async tx => { + await tx.execute(sql`set local statement_timeout = '2s'`); + return callback(tx); + }, options); + }) as typeof db.transaction); + try { + expect(await reassessNativeStatusFromSource(db, selected, { onlyWhenDue: true })).toEqual({ kind: "deferred", retry: true }); + } finally { intercept.mockRestore(); release?.(); await writer; } + expect((await issueService(db).getById(seeded.issueId))!.status).toBe("in_review"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(1); + if (kind !== "work product writer") await change(db, false); + expect(await reassessNativeStatusFromSource(db, selected, { onlyWhenDue: true })).toMatchObject({ kind: "committed" }); + expect((await issueService(db).getById(seeded.issueId))!.status).toBe("done"); + }, 30_000, + ); + + it("leaves reassessment to its scheduled owner while periodic finalization continues", async () => { + const seeded = await seedAutomaticReview(); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + const options = { skipNativeStatusReassessment: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true }; + const result = await reconcileNativeFinalizations(db, [seeded.runId], options); + expect(result).toEqual([expect.objectContaining({ runId: seeded.runId, phase: "committed" })]); + expect((await issueService(db).getById(seeded.issueId))!.status).toBe("in_review"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(1); + const work = nativeStatusReassessmentWork(db); + const [pending] = await work.load({ companyId, issueId: seeded.issueId, runId: seeded.runId }); + expect(pending).toBeDefined(); + expect(await work.check(pending)).toBeNull(); + expect((await issueService(db).getById(seeded.issueId))!.status).toBe("done"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2); + }); + + describe("native status reassessment work", () => { + const selectedScope = (seeded: { issueId: string; runId: string }) => ({ companyId, issueId: seeded.issueId, runId: seeded.runId }); + + it.each(["retired review", "new evidence", "task status"] as const)("discovers and settles %s without revisiting settled history", async trigger => { + const seeded = await seedAutomaticReview(); + const work = nativeStatusReassessmentWork(db), scope = selectedScope(seeded); + expect(await work.load(scope)).toEqual([]); + if (trigger === "retired review") await dismissAutomaticCompletionReviews(db, seeded.issueId); + if (trigger === "new evidence") { + const [assessment] = await db.select().from(workAssessments).where(eq(workAssessments.id, seeded.assessmentId)); + await db.update(workAssessments).set({ createdAt: new Date(Date.now() - 2000), + assessmentJson: { ...assessment.assessmentJson, allCriteriaSatisfied: false } }).where(eq(workAssessments.id, seeded.assessmentId)); + await db.update(issueWorkProducts).set({ updatedAt: new Date() }).where(eq(issueWorkProducts.id, seeded.workProductId)); + } + if (trigger === "task status") await db.update(issues).set({ status: "blocked", statusVersion: 2 }).where(eq(issues.id, seeded.issueId)); + const pending = await work.load(scope); + expect(pending).toMatchObject([{ id: seeded.runId, dueAt: 0 }]); + expect(await work.check(pending[0])).toBeNull(); + expect((await issueService(db).getById(seeded.issueId))!.status).toBe(trigger === "task status" ? "blocked" : "done"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2); + expect(await work.load(scope)).toEqual([]); + expect(await work.check(pending[0])).toBeNull(); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2); + }); + + it("respects a saved retry and live lease, then reloads the released owner", async () => { + const seeded = await seedAutomaticReview(); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + const work = nativeStatusReassessmentWork(db), scope = selectedScope(seeded); + const future = new Date(Date.now() + 60_000), lease = new Date(Date.now() + 120_000); + await db.update(nativeRunFinalizations).set({ nextAttemptAt: future, leaseOwner: "finalizer", leaseExpiresAt: lease }) + .where(eq(nativeRunFinalizations.runId, seeded.runId)); + const [pending] = await work.load(scope); + expect(pending.dueAt).toBe(lease.getTime()); + expect(await work.check(pending)).toEqual(pending); + expect((await issueService(db).getById(seeded.issueId))!.status).toBe("in_review"); + await db.update(nativeRunFinalizations).set({ nextAttemptAt: null, leaseOwner: null, leaseExpiresAt: null }) + .where(eq(nativeRunFinalizations.runId, seeded.runId)); + expect(await work.check(pending)).toBeNull(); + expect((await issueService(db).getById(seeded.issueId))!.status).toBe("done"); + }); + + it("leaves insufficient evidence dormant until a later change satisfies the contract", async () => { + const seeded = await seedAutomaticReview(); + const work = nativeStatusReassessmentWork(db), scope = selectedScope(seeded); + const [assessment] = await db.select().from(workAssessments).where(eq(workAssessments.id, seeded.assessmentId)); + await db.update(workAssessments).set({ createdAt: new Date(Date.now() - 2000), + assessmentJson: { ...assessment.assessmentJson, allCriteriaSatisfied: false } }).where(eq(workAssessments.id, seeded.assessmentId)); + await db.update(issueWorkProducts).set({ reviewState: "changes_requested", updatedAt: new Date() }) + .where(eq(issueWorkProducts.id, seeded.workProductId)); + const [pending] = await work.load(scope); + expect(pending).toBeDefined(); + expect(await work.check(pending)).toBeNull(); + expect((await issueService(db).getById(seeded.issueId))!.status).toBe("in_review"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(1); + await db.update(issueWorkProducts).set({ reviewState: "approved", updatedAt: new Date() }) + .where(eq(issueWorkProducts.id, seeded.workProductId)); + const [changed] = await work.load(scope); + expect(await work.check(changed)).toBeNull(); + expect((await issueService(db).getById(seeded.issueId))!.status).toBe("done"); + }); + + it.each(["unknown lease", "pending effect", "ownership"] as const)("waits for the %s owner to release its obligation", async held => { + const seeded = await seedAutomaticReview(); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + const work = nativeStatusReassessmentWork(db), scope = selectedScope(seeded); + const [pending] = await work.load(scope); + if (held === "unknown lease") await db.update(nativeRunFinalizations).set({ leaseOwner: "unknown", leaseExpiresAt: null }) + .where(eq(nativeRunFinalizations.runId, seeded.runId)); + if (held === "pending effect") await db.update(statusDecisionEffects).set({ deliveryState: "pending" }) + .where(eq(statusDecisionEffects.decisionId, seeded.decision.id)); + if (held === "ownership") await db.update(heartbeatRuns).set({ status: "running", errorCode: "native_execution_ownership_unverified", nativePhase: "terminal_failure" }) + .where(eq(heartbeatRuns.id, seeded.runId)); + expect(await work.load(scope)).toEqual([]); + expect(await work.check(pending)).toBeNull(); + expect((await issueService(db).getById(seeded.issueId))!.status).toBe("in_review"); + }); + + it("keeps a newly saved near-term deadline after a competing commit", async () => { + const seeded = await seedAutomaticReview(); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + const work = nativeStatusReassessmentWork(db), scope = selectedScope(seeded); + const [pending] = await work.load(scope); + const future = new Date(Date.now() + 5000); + const transaction = db.transaction.bind(db); + const intercept = vi.spyOn(db, "transaction").mockImplementationOnce((async (callback, options) => { + await db.update(nativeRunFinalizations).set({ nextAttemptAt: future }).where(eq(nativeRunFinalizations.runId, seeded.runId)); + return transaction(callback, options); + }) as typeof db.transaction); + try { expect(await work.check(pending)).toEqual({ ...pending, dueAt: future.getTime() }); } + finally { intercept.mockRestore(); } + expect((await issueService(db).getById(seeded.issueId))!.status).toBe("in_review"); + }); + + it("does not query after discovery was cancelled", async () => { + const controller = new AbortController(); controller.abort(); + const select = vi.spyOn(db, "select"); + try { + await expect(nativeStatusReassessmentWork(db).load(undefined, controller.signal)).rejects.toThrow("discovery stopped"); + expect(select).not.toHaveBeenCalled(); + } finally { select.mockRestore(); } + }); + }); + it("retires a proven automatic review and applies the current successful completion exactly once", async () => { const seeded = await seedAutomaticReview(); // The persisted result is done, while its old assessment/decision required a review. @@ -3126,7 +3649,7 @@ describe("P6-31 Section 18.13 executable status-authority corpus", () => { ]); }); - it("ignores historical committed finalizations superseded by a newer authoritative decision", async () => { + it.each([false, true])("ignores superseded historical finalizations with scheduled reassessment %s", async scheduled => { const fixture = corpus.fixtures.find((candidate) => candidate.mode === "native"); if (!fixture) throw new Error("native corpus fixture missing"); const seeded = await seedFixture(fixture); @@ -3188,7 +3711,7 @@ describe("P6-31 Section 18.13 executable status-authority corpus", () => { decisionId: historicalDecision!.id, }).where(eq(nativeRunFinalizations.runId, seeded.runId)); - await expect(reconcileNativeFinalizations(db, [seeded.runId])).resolves.toEqual([]); + await expect(reconcileNativeFinalizations(db, [seeded.runId], { skipNativeStatusReassessment: scheduled })).resolves.toEqual([]); await expect(db.select({ decisionId: nativeRunFinalizations.decisionId, }).from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seeded.runId))).resolves.toEqual([ diff --git a/server/src/__tests__/native-status-reassessment-work.test.ts b/server/src/__tests__/native-status-reassessment-work.test.ts new file mode 100644 index 00000000000..667f9c2bfe9 --- /dev/null +++ b/server/src/__tests__/native-status-reassessment-work.test.ts @@ -0,0 +1,640 @@ +import { notifyChatRunMilestones } from "../services/chat-run-milestone-notifications.js"; +import { notifyNativeFinalizationWork } from "../services/native-finalization-notifications.js"; +import { createPostgresWakeQueueAdapter } from "../modules/wake-queue/adapters/postgres.js"; +import { withNativeWorkspaceFinalizationOwnership } from "../services/native-runtime/native-workspace-finalization-ownership.js"; +import express from "express"; +import { appendHeartbeatRunEvent } from "../services/heartbeat-run-events.js"; +import { DELIVERY_QUEUES, subscribeDeliveryWork } from "../services/delivery-work-notifications.js"; +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { workspaceOperationService } from "../services/workspace-operations.js"; +import request from "supertest"; +import { LOW_TRUST_REVIEW_PRESET } from "@paperclipai/shared"; +import { issueRoutes } from "../routes/issues.js"; +import { errorHandler } from "../middleware/index.js"; +import { workProductService } from "../services/work-products.js"; +import { issueService } from "../services/issues.js"; +import { approvalService } from "../services/approvals.js"; +import { issueThreadInteractionService } from "../services/issue-thread-interactions.js"; +import { issueApprovalService } from "../services/issue-approvals.js"; +import { randomUUID } from "node:crypto"; +import { setTimeout as delay } from "node:timers/promises"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { eq } from "drizzle-orm"; +import { agents, companies, completionContracts, createDb, heartbeatRuns, issues, issueWorkProducts, issueThreadInteractions, + nativeRunFinalizations, nativeRunResults, statusDecisions, workAssessments, workspaceOperations } from "@paperclipai/db"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; +import { idleWorkSnapshot } from "../services/task-admission.js"; +import { nativeStatusReassessmentWork } from "../services/native-runtime/native-status-reassessment-work.js"; +import { dismissAutomaticCompletionReviews } from "../services/native-runtime/automatic-completion-reviews.js"; +import { commitNativeStatusDecision } from "../services/native-runtime/status-decision-committer.js"; +import { NATIVE_STATUS_ARBITER_POLICY_VERSION } from "../services/native-runtime/status-arbiter.js"; +import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; + +describe("native reassessment worker lifecycle", () => { + let temporary: Awaited> | undefined; + let db: ReturnType; + let coordinator: ReturnType | undefined; + const companyId = randomUUID(), agentId = randomUUID(); + let logRoot: string; + const priorLogRoot = process.env.WORKSPACE_OPERATION_LOG_BASE_PATH; + let admitted = true; + const errors: unknown[] = []; + beforeAll(async () => { + logRoot = await mkdtemp(join(tmpdir(), "reassessment-workspace-logs-")); + process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = logRoot; + const external = resolveExternalTestDatabaseUrl("PAPERCLIP_STATUS_REASSESSMENT_TEST_DATABASE_URL"); + if (!external) temporary = await startEmbeddedPostgresTestDatabase("paperclip-status-reassessment-"); + db = createDb(external ?? temporary!.connectionString); + await db.insert(companies).values({ id: companyId, name: "Reassessment lifecycle", issuePrefix: `RS${randomUUID().slice(0, 8)}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Reassessment agent", adapterType: "codex_local", status: "idle" }); + }, 30_000); + afterAll(async () => { + await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); + await rm(logRoot, { recursive: true, force: true }); + if (priorLogRoot === undefined) delete process.env.WORKSPACE_OPERATION_LOG_BASE_PATH; + else process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = priorLogRoot; + }); + beforeEach(() => { + admitted = true; errors.length = 0; + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true }); + }); + afterEach(async () => { + await coordinator?.stop(); coordinator = undefined; + vi.useRealTimers(); vi.restoreAllMocks(); + expect(errors).toEqual([]); + }); + async function start() { + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => admitted, canReconcile: () => admitted, + onError: error => errors.push(error) }); + await nativeStatusReassessmentWork(db).register(coordinator, () => admitted).ready; + } + async function flush() { + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(1000); + expect(idleWorkSnapshot().active).toBe(0); + expect(coordinator!.nextWakeAt()).toBeNull(); + }, { timeout: 15_000 }); + } + async function quiet() { + const original = db.$client.options.debug; + const queries: string[] = []; + db.$client.options.debug = (_id, query) => { queries.push(query); }; + try { + await vi.advanceTimersByTimeAsync(120_000); + await delay(10); + expect(queries).toEqual([]); + } finally { db.$client.options.debug = original; } + } + async function status(issueId: string) { + return (await db.select({ status: issues.status }).from(issues).where(eq(issues.id, issueId)))[0].status; + } + async function seedReview(evidenceRef?: string) { + const issueId = randomUUID(), runId = randomUUID(), resultId = randomUUID(), contractId = randomUUID(); + const assessmentId = randomUUID(), workProductId = randomUUID(); + await db.insert(issues).values({ id: issueId, companyId, title: "Persisted native result", status: "in_progress", assigneeAgentId: agentId }); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "succeeded", runtimeMode: "native", + nativeIssueId: issueId, completionContractId: contractId, contextSnapshot: { issueId }, finishedAt: new Date() }); + await db.insert(completionContracts).values({ id: contractId, companyId, issueId, revision: 1, + schemaVersion: "paperclip.completion-contract.v1", policyVersion: "phase6-v1", risk: "low", + completionAuthority: "agent_claim_policy", incompleteCriteriaPolicy: "preserve_non_terminal", + contractJson: { revision: "test-v1", criteria: [{ id: "objective", requirement: "Verified artifact" }] }, + canonicalSha256: contractId, createdByActorType: "system", createdByActorId: "reassessment-test" }); + await db.insert(nativeRunResults).values({ id: resultId, companyId, issueId, runId, completionContractId: contractId, + serverFingerprint: resultId, schemaStatus: "accepted", canonicalSha256: resultId, + resultJson: { result: { reportedWorkDisposition: "done", summary: "Completed", attentionRequests: [], + completionClaim: { contractRevision: "test-v1", objectiveSatisfied: true, remainingWork: [], + criteria: [{ criterionId: "objective", status: "satisfied", evidenceRefs: [evidenceRef ?? `work_product:${workProductId}`] }] }, + verification: [{ commandOrCheck: "artifact", status: "passed", artifactRef: evidenceRef ?? `work_product:${workProductId}` }], + }, terminal: { runTerminalState: "succeeded" } } }); + await db.insert(issueWorkProducts).values({ id: workProductId, companyId, issueId, type: "artifact", provider: "paperclip", + title: "Verified artifact", status: "ready_for_review", reviewState: "approved", + createdAt: new Date(Date.now() - 1000), updatedAt: new Date(Date.now() - 1000) }); + await db.insert(workAssessments).values({ id: assessmentId, companyId, issueId, runId, contractId, resultId, + triggerKind: "native_result", triggerActorCompanyId: companyId, priorIssueStatus: "in_progress", priorStatusVersion: 0, + policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, inputDigest: assessmentId, + assessmentJson: { reportedDisposition: "needs_review", attentionRequests: [], allCriteriaSatisfied: false } }); + await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "assessing", resultId, assessmentId }); + const committed = await commitNativeStatusDecision({ db, companyId, issueId, runId, assessmentId, + priorStatus: "in_progress", priorStatusVersion: 0, priorDecisionId: null, + decision: { policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, statusAction: "in_review", toStatus: "in_review", + reasonCode: "completion_claim_incomplete", unblockDescriptor: null, + effects: [{ kind: "bind_reviewer", ownerUserId: null, + prompt: "Review the persisted native-run evidence and confirm whether this issue may be completed." }] } }); + await db.insert(workspaceOperations).values({ companyId, heartbeatRunId: runId, issueId, phase: "workspace_finalize", + status: "succeeded", exitCode: 0, cwd: process.cwd(), finishedAt: new Date() }); + return { issueId, runId, workProductId, decisionId: committed.decision.id }; + } + + it.each(["milestone", "finalization"] as const)("ignores repeated unscoped routine %s hints without reconstructing dormant reassessment history", async source => { + const seeded = await seedReview(); + const [other] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "failed", + runtimeMode: "legacy", contextSnapshot: { issueId: seeded.issueId }, finishedAt: new Date() }).returning(); + await db.update(issues).set({ executionRunId: other.id }).where(eq(issues.id, seeded.issueId)); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + await start(); + await flush(); + await quiet(); + expect(await nativeStatusReassessmentWork(db).load({ companyId, runId: seeded.runId })).toHaveLength(1); + const debug = db.$client.options.debug; + const queries: string[] = []; + db.$client.options.debug = (_connection, query) => { + if (query.startsWith("select") && query.includes("native_run_finalizations")) queries.push(query); + }; + try { + for (let i = 0; i < 3; i++) { + await db.transaction(tx => source === "milestone" ? notifyChatRunMilestones(tx, { companyId }) + : notifyNativeFinalizationWork(tx, { companyId })); + await flush(); + } + expect(queries).toEqual([]); + expect(await status(seeded.issueId)).toBe("in_review"); + } finally { db.$client.options.debug = debug; } + await quiet(); + }); + + it.each([false, true])("preserves the current required quality-review gate during retired-review reassessment (required=%s)", async required => { + const seeded = await seedReview(); + if (required) await db.update(issues).set({ executionPolicy: { requiredQualityReview: { + version: 1, deliverables: [{ key: "output", label: "Verified output" }], + } } }).where(eq(issues.id, seeded.issueId)); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + await start(); + await flush(); + expect(await status(seeded.issueId)).toBe(required ? "in_progress" : "done"); + const decisions = await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId)); + if (required) expect(decisions.find(row => row.id !== seeded.decisionId)?.reasonCode).toBe("required_quality_review_incomplete"); + await quiet(); + }); + + it.each(["force release", "release", "terminal execution", "terminal checkout", "queue release"] as const)( + "wakes dormant reassessment from real %s", async action => { + const seeded = await seedReview(); + const [other] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "failed", + runtimeMode: "legacy", contextSnapshot: { issueId: seeded.issueId }, finishedAt: new Date() }).returning(); + await db.update(issues).set({ executionRunId: other.id, checkoutRunId: other.id }).where(eq(issues.id, seeded.issueId)); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + await start(); await flush(); await quiet(); + expect(await status(seeded.issueId)).toBe("in_review"); + const service = issueService(db); + if (action === "force release") await service.adminForceRelease(seeded.issueId); + if (action === "release") await service.release(seeded.issueId); + if (action === "terminal execution") await service.clearExecutionRunIfTerminal(seeded.issueId); + if (action === "terminal checkout") await service.clearCheckoutRunIfTerminal(seeded.issueId); + if (action === "queue release") { + const adapter = createPostgresWakeQueueAdapter(db, { + resolveResponsibleUserId: async () => "reassessment-user", + getRoutineEnv: async () => ({ routineId: null, env: null, responsibleUserId: null }), + resolveSessionBeforeForWakeup: async () => null, + }); + await adapter.withIssueExecutionLock({ companyId, runId: other.id, now: new Date() }, + async () => ({ outcome: { kind: "released" }, postCommitEffects: [] })); + } + await flush(); + expect(await status(seeded.issueId)).toBe(action === "release" ? "in_review" : "done"); + const decisions = await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId)); + expect(decisions).toHaveLength(2); + if (action === "release") { + expect((await service.getById(seeded.issueId))!.assigneeAgentId).toBeNull(); + expect(decisions.find(decision => decision.id !== seeded.decisionId)?.reasonCode).toBe("prior_status_terminal_preserved"); + } + await quiet(); + }, + ); + + it("retains dormant reassessment through outer force-release rollback and commit", async () => { + const seeded = await seedReview(); + const [other] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "failed", + runtimeMode: "legacy", contextSnapshot: { issueId: seeded.issueId }, finishedAt: new Date() }).returning(); + await db.update(issues).set({ executionRunId: other.id }).where(eq(issues.id, seeded.issueId)); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + await start(); await flush(); await quiet(); + const rollback = new Error("rollback release"); + await expect(db.transaction(async tx => { + await issueService(tx as unknown as typeof db).adminForceRelease(seeded.issueId); + await vi.advanceTimersByTimeAsync(2000); + expect(await status(seeded.issueId)).toBe("in_review"); + throw rollback; + })).rejects.toBe(rollback); + await flush(); await quiet(); + expect((await issueService(db).getById(seeded.issueId))!.executionRunId).toBe(other.id); + await db.transaction(async tx => { + await issueService(tx as unknown as typeof db).adminForceRelease(seeded.issueId); + await vi.advanceTimersByTimeAsync(2000); + expect(await status(seeded.issueId)).toBe("in_review"); + }); + await flush(); + expect(await status(seeded.issueId)).toBe("done"); + await quiet(); + }); + + it("bounds discovery by pages and drains a real reassessment backlog to SQL quiet", async () => { + const backlog = []; + for (let index = 0; index < 103; index++) { + const seeded = await seedReview(); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + backlog.push(seeded); + } + const work = nativeStatusReassessmentWork(db), scope = { companyId }; + const expected = new Set(backlog.map(row => row.runId)); + const discovered = await work.load(scope); + expect(discovered.filter(row => expected.has(row.id))).toHaveLength(103); + const debug = db.$client.options.debug; + let queries = 0, admissionChecks = 0; + db.$client.options.debug = (_id, query) => { + if (query.startsWith("select") && query.includes("native_run_finalizations")) queries++; + }; + try { + await expect(work.load(scope, undefined, () => ++admissionChecks <= 1)) + .rejects.toThrow("Native status reassessment discovery stopped"); + expect(queries).toBe(1); + queries = 0; + await expect(work.load(scope, AbortSignal.abort())).rejects.toThrow("Native status reassessment discovery stopped"); + expect(queries).toBe(0); + } finally { db.$client.options.debug = debug; } + await start(); await flush(); + for (const seeded of backlog) { + expect(await status(seeded.issueId)).toBe("done"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2); + } + expect((await work.load(scope)).filter(row => expected.has(row.id))).toEqual([]); + await quiet(); + }, 60_000); + + it("waits quietly for physical workspace ownership to release before reassessment", async () => { + const seeded = await seedReview(); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + let enter!: () => void, release!: () => void; + const entered = new Promise(resolve => { enter = resolve; }); + const released = new Promise(resolve => { release = resolve; }); + const owner = withNativeWorkspaceFinalizationOwnership({ db, companyId, runId: seeded.runId }, async () => { + enter(); await released; + }); + try { + await Promise.race([entered, owner]); + await start(); await flush(); + expect(await status(seeded.issueId)).toBe("in_review"); + await quiet(); + } finally { release(); await owner; } + await flush(); + expect(await status(seeded.issueId)).toBe("done"); + await quiet(); + }); + + it("keeps an unverified workspace owner dormant despite a successful receipt", async () => { + const seeded = await seedReview(); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + const owner = { token: randomUUID(), hostname: "unverified-controller", pid: 12345, + processStartedAt: new Date().toISOString() }; + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: owner } }) + .where(eq(heartbeatRuns.id, seeded.runId)); + await start(); await flush(); + expect(await status(seeded.issueId)).toBe("in_review"); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.runId)))[0] + .runnerProfileJson?.nativeWorkspaceFinalizationOwner).toEqual(owner); + await quiet(); + }); + + it.each(["ordinary", "native source"] as const)("wakes from a committed control-plane evidence event through the %s append path", async path => { + const seeded = await seedReview("event:1"); + await vi.advanceTimersByTimeAsync(1000); + await db.update(issueWorkProducts).set({ updatedAt: new Date() }).where(eq(issueWorkProducts.id, seeded.workProductId)); + await start(); await flush(); await quiet(); + const prpEvent = { sourceKind: "control_plane", payload: { evidenceOutcome: " accepted " } }; + const input = { companyId, agentId, runId: seeded.runId, eventType: "evidence.recorded", payload: { prpEvent }, + ...(path === "native source" ? { nativeSource: { sourceInstanceId: randomUUID(), sourceEventId: randomUUID(), + sourceSeq: 1, protocolSchemaVersion: 1, canonicalPayload: prpEvent } } : {}) }; + expect((await appendHeartbeatRunEvent(db, input)).disposition).toBe("committed"); + await flush(); + expect(await status(seeded.issueId)).toBe("done"); + if (path === "native source") expect((await appendHeartbeatRunEvent(db, input)).disposition).toBe("duplicate"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2); + await quiet(); + }); + + it("preserves event evidence until outer commit and ignores rollback", async () => { + const seeded = await seedReview("event:1"); + await vi.advanceTimersByTimeAsync(1000); + await db.update(issueWorkProducts).set({ updatedAt: new Date() }).where(eq(issueWorkProducts.id, seeded.workProductId)); + await start(); await flush(); await quiet(); + const write = async (tx: typeof db) => { + await appendHeartbeatRunEvent(tx, { companyId, agentId, runId: seeded.runId, eventType: "evidence.recorded", + payload: { prpEvent: { sourceKind: "control_plane", payload: { evidenceOutcome: "accepted" } } } }); + await vi.advanceTimersByTimeAsync(1000); + expect(await status(seeded.issueId)).toBe("in_review"); + }; + await expect(db.transaction(async tx => { + await write(tx as unknown as typeof db); throw new Error("rollback event evidence"); + })).rejects.toThrow("rollback event evidence"); + await flush(); await quiet(); + expect(await status(seeded.issueId)).toBe("in_review"); + await db.transaction(tx => write(tx as unknown as typeof db)); + await flush(); + expect(await status(seeded.issueId)).toBe("done"); + await quiet(); + }); + + it("keeps ordinary and runner-authored logs from waking reassessment", async () => { + const seeded = await seedReview("event:3"); + await vi.advanceTimersByTimeAsync(1000); + await db.update(issueWorkProducts).set({ updatedAt: new Date() }).where(eq(issueWorkProducts.id, seeded.workProductId)); + await start(); await flush(); await quiet(); + const wakes = vi.fn(); + const stop = subscribeDeliveryWork(db, DELIVERY_QUEUES.nativeStatusReassessment, wakes); + try { + await appendHeartbeatRunEvent(db, { companyId, agentId, runId: seeded.runId, eventType: "log", message: "Still working" }); + await appendHeartbeatRunEvent(db, { companyId, agentId, runId: seeded.runId, eventType: "evidence.recorded", + payload: { prpEvent: { sourceKind: "runner", payload: { evidenceOutcome: "accepted" } } } }); + expect(wakes).not.toHaveBeenCalled(); + await quiet(); + expect(await status(seeded.issueId)).toBe("in_review"); + await appendHeartbeatRunEvent(db, { companyId, agentId, runId: seeded.runId, eventType: "evidence.recorded", + payload: { prpEvent: { sourceKind: "control_plane", payload: { evidenceOutcome: "accepted" } } } }); + await flush(); + expect(await status(seeded.issueId)).toBe("done"); + await quiet(); + } finally { stop(); } + }); + + it("wakes when a real workspace finalization settles after a quiet failure", async () => { + const seeded = await seedReview(); + await db.update(workspaceOperations).set({ status: "running" }).where(eq(workspaceOperations.heartbeatRunId, seeded.runId)); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + await start(); await flush(); await quiet(); + const recorder = workspaceOperationService(db).createRecorder({ companyId, issueId: seeded.issueId, heartbeatRunId: seeded.runId }); + await expect(recorder.recordOperation({ phase: "workspace_finalize", run: async () => { throw new Error("export interrupted"); } })) + .rejects.toThrow("export interrupted"); + await flush(); await quiet(); + expect(await status(seeded.issueId)).toBe("in_review"); + let release!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + let started!: () => void; + const running = new Promise(resolve => { started = resolve; }); + const operation = recorder.recordOperation({ phase: "workspace_finalize", run: async () => { + started(); await gate; return { status: "succeeded" }; + } }); + try { + await running; await flush(); await quiet(); + expect(await status(seeded.issueId)).toBe("in_review"); + } finally { release(); await operation; } + await flush(); + expect(await status(seeded.issueId)).toBe("done"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2); + await quiet(); + }); + + it("defers workspace barrier writes until outer commit and ignores rollback", async () => { + const seeded = await seedReview(); + await db.update(workspaceOperations).set({ status: "running" }).where(eq(workspaceOperations.heartbeatRunId, seeded.runId)); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + await start(); await flush(); await quiet(); + const write = async (tx: typeof db) => { + await workspaceOperationService(tx).createRecorder({ companyId, issueId: seeded.issueId, heartbeatRunId: seeded.runId }) + .recordOperation({ phase: "workspace_finalize", run: async () => ({ status: "succeeded" }) }); + await vi.advanceTimersByTimeAsync(1000); + expect(await status(seeded.issueId)).toBe("in_review"); + }; + await expect(db.transaction(async tx => { + await write(tx as unknown as typeof db); throw new Error("rollback workspace barrier"); + })).rejects.toThrow("rollback workspace barrier"); + await flush(); await quiet(); + expect(await status(seeded.issueId)).toBe("in_review"); + await db.transaction(tx => write(tx as unknown as typeof db)); + await flush(); + expect(await status(seeded.issueId)).toBe("done"); + await quiet(); + }); + + it.each(["confirmation", "checkbox", "question", "verdict", "partial verdict"] as const)("wakes dormant evidence from real interaction %s", async mutation => { + const interactionId = randomUUID(); + const seeded = await seedReview(`interaction:${interactionId}`); + const kind = mutation === "confirmation" ? "request_confirmation" : mutation === "checkbox" ? "request_checkbox_confirmation" + : mutation === "question" ? "ask_user_questions" : "request_item_verdicts"; + const payload = kind === "ask_user_questions" + ? { version: 1 as const, questions: [{ id: "verified", prompt: "Verified?", selectionMode: "single" as const, + options: [{ id: "yes", label: "Yes" }] }] } + : kind === "request_item_verdicts" + ? { version: 1 as const, prompt: "Review evidence", items: [{ id: "artifact", label: "Artifact" }, + ...(mutation === "partial verdict" ? [{ id: "other", label: "Other" }] : [])] } + : kind === "request_checkbox_confirmation" + ? { version: 1 as const, prompt: "Accept evidence", options: [{ id: "verified", label: "Verified" }] } + : { version: 1 as const, prompt: "Accept evidence" }; + await db.insert(issueThreadInteractions).values({ id: interactionId, companyId, issueId: seeded.issueId, kind, + sourceRunId: seeded.runId, continuationPolicy: "none", createdByUserId: "board-user", payload }); + await db.update(issues).set({ createdByUserId: "board-user" }).where(eq(issues.id, seeded.issueId)); + await vi.advanceTimersByTimeAsync(1000); + await db.update(issueWorkProducts).set({ updatedAt: new Date() }).where(eq(issueWorkProducts.id, seeded.workProductId)); + await start(); await flush(); await quiet(); + expect(await status(seeded.issueId)).toBe("in_review"); + const scope = { id: seeded.issueId, companyId, projectId: null, goalId: null }; + const interactions = issueThreadInteractionService(db); + if (mutation === "confirmation" || mutation === "checkbox") { + await interactions.acceptInteraction(scope, interactionId, mutation === "checkbox" ? { selectedOptionIds: ["verified"] } : {}, { userId: "board-user" }); + } else if (mutation === "question") { + await interactions.answerQuestions(scope, interactionId, { answers: [{ questionId: "verified", optionIds: ["yes"] }] }, { userId: "board-user" }); + } else { + await interactions.submitItemVerdicts(scope, interactionId, { verdicts: [{ id: "artifact", verdict: "approve" }] }, { userId: "board-user" }); + } + await flush(); + expect(await status(seeded.issueId)).toBe("done"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2); + await quiet(); + }); + + it.each(["resolution", "link", "bulk link"] as const)("wakes dormant evidence from real approval %s", async mutation => { + const approvalId = randomUUID(); + const seeded = await seedReview(`approval:${approvalId}`); + await approvalService(db).create(companyId, { id: approvalId, type: "approve_ceo_strategy", payload: {}, + status: mutation === "resolution" ? "pending" : "approved" }); + if (mutation === "resolution") await issueApprovalService(db).link(seeded.issueId, approvalId); + await vi.advanceTimersByTimeAsync(1000); + await db.update(issueWorkProducts).set({ updatedAt: new Date() }).where(eq(issueWorkProducts.id, seeded.workProductId)); + await start(); await flush(); await quiet(); + expect(await status(seeded.issueId)).toBe("in_review"); + if (mutation === "resolution") await approvalService(db).approve(approvalId, "board-user"); + if (mutation === "link") await issueApprovalService(db).link(seeded.issueId, approvalId); + if (mutation === "bulk link") await issueApprovalService(db).linkManyForApproval(approvalId, [seeded.issueId]); + await flush(); + expect(await status(seeded.issueId)).toBe("done"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2); + await quiet(); + }); + + it("defers approval evidence until outer commit and ignores rolled-back approval", async () => { + const approvalId = randomUUID(); + const seeded = await seedReview(`approval:${approvalId}`); + await approvalService(db).create(companyId, { id: approvalId, type: "approve_ceo_strategy", payload: {} }); + await issueApprovalService(db).link(seeded.issueId, approvalId); + await vi.advanceTimersByTimeAsync(1000); + await db.update(issueWorkProducts).set({ updatedAt: new Date() }).where(eq(issueWorkProducts.id, seeded.workProductId)); + await start(); await flush(); await quiet(); + await expect(db.transaction(async tx => { + await approvalService(tx as unknown as typeof db).approve(approvalId, "board-user"); + await vi.advanceTimersByTimeAsync(1000); + expect(await status(seeded.issueId)).toBe("in_review"); + throw new Error("rollback approval evidence"); + })).rejects.toThrow("rollback approval evidence"); + await flush(); await quiet(); + expect(await status(seeded.issueId)).toBe("in_review"); + await db.transaction(async tx => { + await approvalService(tx as unknown as typeof db).approve(approvalId, "board-user"); + await vi.advanceTimersByTimeAsync(1000); + expect(await status(seeded.issueId)).toBe("in_review"); + }); + await flush(); + expect(await status(seeded.issueId)).toBe("done"); + await quiet(); + }); + + it.each([false, true])("preserves the required quality-review gate after real evidence approval (required=%s)", async required => { + const seeded = await seedReview(); + await db.update(issueWorkProducts).set({ reviewState: "none" }).where(eq(issueWorkProducts.id, seeded.workProductId)); + if (required) await db.update(issues).set({ executionPolicy: { requiredQualityReview: { + version: 1, deliverables: [{ key: "output", label: "Verified output" }], + } } }).where(eq(issues.id, seeded.issueId)); + await start(); + await flush(); + await quiet(); + await workProductService(db).update(seeded.workProductId, { reviewState: "approved" }); + await flush(); + expect(await status(seeded.issueId)).toBe(required ? "in_progress" : "done"); + const decisions = await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId)); + if (required) expect(decisions.find(row => row.id !== seeded.decisionId)?.reasonCode).toBe("required_quality_review_incomplete"); + await quiet(); + }); + + it.each(["update", "create", "primary replacement", "import", "attachment", "source promotion"] as const)("wakes from the real work-product %s writer", async writer => { + const seeded = await seedReview(); + if (writer === "update") await db.update(issueWorkProducts).set({ reviewState: "none" }).where(eq(issueWorkProducts.id, seeded.workProductId)); + if (writer === "create") await db.delete(issueWorkProducts).where(eq(issueWorkProducts.id, seeded.workProductId)); + if (writer === "source promotion") await db.update(issueWorkProducts).set({ sourceTrust: { + preset: LOW_TRUST_REVIEW_PRESET, disposition: "quarantined", sourceIssueId: seeded.issueId, + sourceRunId: seeded.runId, sourceAgentId: agentId, + } }).where(eq(issueWorkProducts.id, seeded.workProductId)); + await start(); await flush(); await quiet(); + expect(await status(seeded.issueId)).toBe("in_review"); + const products = workProductService(db); + if (writer === "update") await products.update(seeded.workProductId, { reviewState: "approved" }); + if (writer === "create" || writer === "primary replacement") await products.createForIssue(seeded.issueId, companyId, { + ...(writer === "create" ? { id: seeded.workProductId } : {}), + type: "artifact", provider: "paperclip", title: "Verified artifact", status: "approved", reviewState: "approved", + isPrimary: writer === "primary replacement", + }); + if (writer === "import") await products.createManyForImport([{ companyId, issueId: seeded.issueId, projectId: null, + type: "artifact", provider: "paperclip", externalId: null, title: "Imported evidence", url: null, status: "approved", + reviewState: "approved", isPrimary: false, healthStatus: "unknown", summary: null, metadata: null, + executionWorkspaceId: null, runtimeServiceId: null, createdByRunId: null, sourceTrust: null }]); + if (writer === "attachment") await issueService(db).createAttachment({ issueId: seeded.issueId, provider: "local_disk", + objectKey: randomUUID(), contentType: "text/plain", byteSize: 4, sha256: "a".repeat(64), originalFilename: "result.txt", + createdByAgentId: agentId, createdByRunId: seeded.runId }); + if (writer === "source promotion") { + const app = express(); app.use(express.json()); + app.use((req, _res, next) => { + req.actor = { type: "board", userId: "board-user", companyIds: [companyId], + memberships: [{ companyId, membershipRole: "operator", status: "active" }], + isInstanceAdmin: true, source: "local_implicit" }; next(); + }); + app.use("/api", issueRoutes(db, {} as any)); app.use(errorHandler); + const promoted = await request(app).post(`/api/issues/${seeded.issueId}/low-trust/promotions`).send({ + sourceArtifactKind: "work_product", sourceArtifactId: seeded.workProductId, + title: "Reviewed artifact", summary: "Reviewed completion evidence", + }); + expect(promoted.status, JSON.stringify(promoted.body)).toBe(201); + } + await flush(); + expect(await status(seeded.issueId)).toBe("done"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2); + await quiet(); + }); + + it("waits for an outer work-product transaction and ignores rollback", async () => { + const seeded = await seedReview(); + await db.update(issueWorkProducts).set({ reviewState: "none" }).where(eq(issueWorkProducts.id, seeded.workProductId)); + await start(); await flush(); await quiet(); + await expect(db.transaction(async tx => { + await workProductService(tx as unknown as typeof db).update(seeded.workProductId, { reviewState: "approved" }); + await vi.advanceTimersByTimeAsync(1000); + expect(await status(seeded.issueId)).toBe("in_review"); + throw new Error("rollback evidence approval"); + })).rejects.toThrow("rollback evidence approval"); + await flush(); await quiet(); + expect(await status(seeded.issueId)).toBe("in_review"); + await db.transaction(async tx => { + await workProductService(tx as unknown as typeof db).update(seeded.workProductId, { reviewState: "approved" }); + await vi.advanceTimersByTimeAsync(1000); + expect(await status(seeded.issueId)).toBe("in_review"); + }); + await flush(); + expect(await status(seeded.issueId)).toBe("done"); + await quiet(); + }); + + it.each(["status", "blocked reassertion", "assignment"] as const)("reassesses a real task %s mutation after quiet", async mutation => { + const seeded = await seedReview(); + if (mutation === "blocked reassertion") await issueService(db).update(seeded.issueId, { status: "blocked" }); + await start(); await flush(); await quiet(); + const before = await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId)); + await issueService(db).update(seeded.issueId, mutation === "assignment" ? { assigneeAgentId: null } : { status: "blocked" }); + await flush(); + const after = await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId)); + expect(after).toHaveLength(before.length + 1); + expect(await status(seeded.issueId)).toBe(mutation === "assignment" ? "in_review" : "blocked"); + const [current] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seeded.runId)); + const [assessment] = await db.select().from(workAssessments).where(eq(workAssessments.id, current.assessmentId!)); + const [issue] = await db.select().from(issues).where(eq(issues.id, seeded.issueId)); + expect(assessment.priorStatusVersion).toBe(issue.statusVersion); + await quiet(); + }); + + it("retains the first review-retirement wake through idle drain", async () => { + const seeded = await seedReview(); + await start(); await flush(); await quiet(); + admitted = false; + await dismissAutomaticCompletionReviews(db, seeded.issueId); + await quiet(); + expect(await status(seeded.issueId)).toBe("in_review"); + admitted = true; + await vi.advanceTimersByTimeAsync(30_000); await flush(); + expect(await status(seeded.issueId)).toBe("done"); + await quiet(); + }); + + it("reconstructs the saved retry after shutdown and never dispatches it early", async () => { + const seeded = await seedReview(); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + const future = new Date(Date.now() + 60_000); + await db.update(nativeRunFinalizations).set({ nextAttemptAt: future }).where(eq(nativeRunFinalizations.runId, seeded.runId)); + const waitForSavedDeadline = async () => { + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(1000); + expect(idleWorkSnapshot().active).toBe(0); + expect(coordinator!.nextWakeAt()).toBe(future.getTime()); + expect(await status(seeded.issueId)).toBe("in_review"); + }, { timeout: 15_000 }); + }; + await start(); + await waitForSavedDeadline(); + await vi.advanceTimersByTimeAsync(20_000); + expect(await status(seeded.issueId)).toBe("in_review"); + await coordinator!.stop(); coordinator = undefined; + await start(); + await waitForSavedDeadline(); + await vi.advanceTimersByTimeAsync(Math.max(0, future.getTime() - Date.now())); await flush(); + expect(await status(seeded.issueId)).toBe("done"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2); + await quiet(); + }); + + it("wakes from real review retirement after idle and becomes SQL quiet with completed history", async () => { + const seeded = await seedReview(); + await start(); await flush(); await quiet(); + expect(await status(seeded.issueId)).toBe("in_review"); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + await flush(); + expect(await status(seeded.issueId)).toBe("done"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2); + await quiet(); + }); +}); diff --git a/server/src/__tests__/photon/channel.integration.test.ts b/server/src/__tests__/photon/channel.integration.test.ts index 84985e0face..47e93a2d90a 100644 --- a/server/src/__tests__/photon/channel.integration.test.ts +++ b/server/src/__tests__/photon/channel.integration.test.ts @@ -1,3 +1,4 @@ +import { nativeStatusReassessmentWork } from "../../services/native-runtime/native-status-reassessment-work.js"; import { nativeReviewResponseWork } from "../../services/native-runtime/native-review-response-work.js"; import { nativeResponsePresentationWork } from "../../services/native-runtime/native-response-presentation-work.js"; import { DELIVERY_QUEUES } from "../../services/delivery-work-notifications.js"; @@ -377,7 +378,7 @@ describe("iMessage Photon channel control plane", () => { }, }; } - it.each([DELIVERY_QUEUES.nativeResponsePresentation, DELIVERY_QUEUES.nativeReviewResponse])( + it.each([DELIVERY_QUEUES.nativeResponsePresentation, DELIVERY_QUEUES.nativeReviewResponse, DELIVERY_QUEUES.nativeStatusReassessment])( "keeps repeated active Photon ownership assertions out of %s reconstruction and wakes once on recovery", async queue => { const t = await setup(); await t.start(); @@ -385,7 +386,8 @@ describe("iMessage Photon channel control plane", () => { const errors: unknown[] = []; const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) }); coordinators.push(coordinator); - const work = queue === DELIVERY_QUEUES.nativeResponsePresentation ? nativeResponsePresentationWork(db) : nativeReviewResponseWork(db); + const work = queue === DELIVERY_QUEUES.nativeResponsePresentation ? nativeResponsePresentationWork(db) + : queue === DELIVERY_QUEUES.nativeReviewResponse ? nativeReviewResponseWork(db) : nativeStatusReassessmentWork(db); const worker = work.register(coordinator, () => true); const debug = db.$client.options.debug; const hints: unknown[] = []; diff --git a/server/src/__tests__/server-startup-feedback-export.test.ts b/server/src/__tests__/server-startup-feedback-export.test.ts index feb240892e4..1198f45960e 100644 --- a/server/src/__tests__/server-startup-feedback-export.test.ts +++ b/server/src/__tests__/server-startup-feedback-export.test.ts @@ -116,6 +116,7 @@ const { reapOrphanedRuns: vi.fn(async () => ({ reaped: 0, runIds: [] })), registerScheduledRetryCancellationWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), registerStoppedInstructionCollectionWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), + registerNativeStatusReassessmentWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), registerNativeReviewResponseWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), registerNativeResponsePresentationWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve(), stop: vi.fn() })), registerNativePendingEffectWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), @@ -571,6 +572,13 @@ describe("startServer feedback export wiring", () => { } }); + it.each([false, true])("registers native reassessment only when maintenance is enabled (%s)", async enabled => { + loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled })); + await startServer(); await settleStartupHousekeeping(); + expect(heartbeatServiceMock.registerNativeStatusReassessmentWork).toHaveBeenCalledTimes(enabled ? 1 : 0); + if (enabled) expect(heartbeatServiceMock.registerNativeStatusReassessmentWork.mock.calls[0][1]()).toBe(true); + }); + it.each([false, true])("registers governed-response recovery only when maintenance is enabled (%s)", async enabled => { loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled })); await startServer(); await settleStartupHousekeeping(); @@ -1025,7 +1033,7 @@ describe("startServer feedback export wiring", () => { intervalCallback?.(); await new Promise((resolve) => setImmediate(resolve)); expect(heartbeatServiceMock.sweepStaleIssueLocks).toHaveBeenCalledTimes(2); - expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true }); + expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true, skipNativeStatusReassessment: true }); expect(heartbeatServiceMock.reconcileCostAccounting).toHaveBeenCalledTimes(1); expect(retiredDetector).not.toHaveBeenCalled(); } finally { diff --git a/server/src/__tests__/tool-access-service.test.ts b/server/src/__tests__/tool-access-service.test.ts index 9aaf534dc1a..37a00be75e6 100644 --- a/server/src/__tests__/tool-access-service.test.ts +++ b/server/src/__tests__/tool-access-service.test.ts @@ -1,3 +1,4 @@ +import { DELIVERY_QUEUES, subscribeDeliveryWork } from "../services/delivery-work-notifications.js"; import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; import * as fs from "node:fs/promises"; import { createHash, generateKeyPairSync, randomUUID } from "node:crypto"; @@ -10088,6 +10089,45 @@ describeEmbeddedPostgres("tool access service", () => { }); + it.each(["creation", "reopening", "rollback"] as const)("signals native review recovery after OAuth card %s settles", async mode => { + vi.stubEnv("PAPERCLIP_TOOL_OAUTH_SLACK_CLIENT_ID", "slack-client-id"); + vi.stubEnv("PAPERCLIP_TOOL_OAUTH_SLACK_CLIENT_SECRET", "slack-client-secret"); + const company = await createCompany(db); + const agent = await createAgent(db, company.id); + const { issue, run } = await createIssueAndRun(db, company.id, agent.id); + const connected = await createTestToolAccessService(db).connectGalleryApp(company.id, { galleryKey: "slack", name: "Slack user auth" }); + if (mode !== "creation") await db.insert(issueThreadInteractions).values({ companyId: company.id, + issueId: issue.id, kind: "request_confirmation", status: "accepted", payload: { version: 1, prompt: "Previous authorization" }, + idempotencyKey: `connection-authorization:${connected.connectionId}:user-for-run`, sourceRunId: run.id }); + const reassessment = vi.fn(), reviewResponse = vi.fn(); + const unsubscribe = [subscribeDeliveryWork(db, DELIVERY_QUEUES.nativeStatusReassessment, reassessment), + subscribeDeliveryWork(db, DELIVERY_QUEUES.nativeReviewResponse, reviewResponse)]; + const rollback = new Error("rollback OAuth reopening"); + try { + const operation = db.transaction(async tx => { + const started = await createTestToolAccessService(tx as unknown as typeof db).startOAuth(company.id, connected.connectionId, { + redirectUri: "https://paperclip.example/api/tools/oauth/callback", actor: { actorType: "agent", actorId: agent.id, sessionId: run.id }, + issueId: issue.id, subjectUserId: "user-for-run", scopes: ["channels:read"], + }); + const [inside] = await tx.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.issueId, issue.id)); + expect(inside.status).toBe("pending"); + expect(inside.payload).toMatchObject({ target: { href: started.authorizationUrl } }); + expect(reassessment).not.toHaveBeenCalled(); expect(reviewResponse).not.toHaveBeenCalled(); + const [outside] = await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.issueId, issue.id)); + expect(outside?.status).toBe(mode === "creation" ? undefined : "accepted"); + if (mode === "rollback") throw rollback; + }); + if (mode === "rollback") await expect(operation).rejects.toBe(rollback); else await operation; + + + expect(reassessment).toHaveBeenCalledTimes(1); expect(reviewResponse).toHaveBeenCalledTimes(1); + const [persisted] = await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.issueId, issue.id)); + expect(persisted.status).toBe(mode === "rollback" ? "accepted" : "pending"); + expect(await db.select().from(toolOauthStates).where(eq(toolOauthStates.issueId, issue.id))) + .toHaveLength(mode === "rollback" ? 0 : 1); + } finally { unsubscribe.forEach(stop => stop()); } + }); + it("creates and resolves an agent-initiated user authorization grant card", async () => { vi.stubEnv("PAPERCLIP_TOOL_OAUTH_SLACK_CLIENT_ID", "slack-client-id"); vi.stubEnv( diff --git a/server/src/__tests__/workspace-operations-reconciliation.test.ts b/server/src/__tests__/workspace-operations-reconciliation.test.ts index 03d1d80bbeb..4414d511432 100644 --- a/server/src/__tests__/workspace-operations-reconciliation.test.ts +++ b/server/src/__tests__/workspace-operations-reconciliation.test.ts @@ -1,3 +1,4 @@ +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; import { randomUUID } from "node:crypto"; import fs from "node:fs/promises"; import os from "node:os"; @@ -65,7 +66,8 @@ describe("workspace runtime control serialization", () => { }); }); -const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); +const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_WORKSPACE_OPERATIONS_TEST_DATABASE_URL"); +const embeddedPostgresSupport = externalUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; if (!embeddedPostgresSupport.supported) { @@ -81,8 +83,8 @@ describeEmbeddedPostgres("workspace operation reconciliation", () => { let previousLogRoot: string | undefined; beforeAll(async () => { - tempDb = await startEmbeddedPostgresTestDatabase("paperclip-workspace-operation-reconcile-"); - db = createDb(tempDb.connectionString); + if (!externalUrl) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-workspace-operation-reconcile-"); + db = createDb(externalUrl ?? tempDb!.connectionString); logRoot = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-workspace-operation-logs-")); previousLogRoot = process.env.WORKSPACE_OPERATION_LOG_BASE_PATH; process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = logRoot; @@ -99,6 +101,7 @@ describeEmbeddedPostgres("workspace operation reconciliation", () => { if (previousLogRoot === undefined) delete process.env.WORKSPACE_OPERATION_LOG_BASE_PATH; else process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = previousLogRoot; await fs.rm(logRoot, { recursive: true, force: true }); + await db?.$client.end({ timeout: 1 }); await tempDb?.cleanup(); }); diff --git a/server/src/index.ts b/server/src/index.ts index 9e25bd0a284..7202f5f7361 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1825,6 +1825,8 @@ async function startServerWithDatabaseTeardown( () => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready; if (config.heartbeatSchedulerEnabled) await heartbeat.registerInstructionDirectoryCleanupWork(deliveryWork, () => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready; + if (config.heartbeatSchedulerEnabled) await heartbeat.registerNativeStatusReassessmentWork(deliveryWork, + () => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready; if (config.heartbeatSchedulerEnabled) await heartbeat.registerNativeReviewResponseWork(deliveryWork, () => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready; if (config.heartbeatSchedulerEnabled) { @@ -1974,7 +1976,7 @@ async function startServerWithDatabaseTeardown( // Periodically reap orphaned runs (5-min staleness threshold) and make sure // persisted queued work is still being driven forward. runPeriodicRecovery(() => heartbeat - .reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true }) + .reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true, skipNativeStatusReassessment: true }) .then(() => heartbeat.promoteDueScheduledRetries()) .then(async (promotion) => { await heartbeat.resumeQueuedRuns(); diff --git a/server/src/modules/active-run-watchdog/adapters/postgres.ts b/server/src/modules/active-run-watchdog/adapters/postgres.ts index 42267449092..94eb3b154bc 100644 --- a/server/src/modules/active-run-watchdog/adapters/postgres.ts +++ b/server/src/modules/active-run-watchdog/adapters/postgres.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "../../../services/native-status-reassessment-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "../../../services/unsafe-workspace-export-notifications.js"; import { notifyCancelledWorkspacePreservation } from "../../../services/cancelled-workspace-preservation-notifications.js"; import { notifyInstructionCollection } from "../../../services/instruction-collection-notifications.js"; @@ -343,6 +344,7 @@ export function createPostgresWatchdogAdapter(db: Db): WatchdogRunReader & Watch await notifyInstructionCollection(tx, { companyId, runId: input.run.id }); await notifyCancelledWorkspacePreservation(tx, { companyId, runId: input.run.id }); await notifyUnsafeWorkspaceExportRecovery(tx, { companyId, issueId: input.sourceIssue.id }); + await notifyNativeStatusReassessment(tx, { companyId, issueId: input.sourceIssue.id }); const [updatedRun] = await tx .update(heartbeatRuns) .set({ diff --git a/server/src/modules/wake-queue/adapters/postgres.test.ts b/server/src/modules/wake-queue/adapters/postgres.test.ts index 79f5ea78968..068a21f2420 100644 --- a/server/src/modules/wake-queue/adapters/postgres.test.ts +++ b/server/src/modules/wake-queue/adapters/postgres.test.ts @@ -1224,7 +1224,7 @@ describeEmbeddedPostgres("wake-queue postgres adapter", () => { expect(wakeRow?.payload).not.toHaveProperty("foo"); }); - it.each([DELIVERY_QUEUES.nativeResponsePresentation, DELIVERY_QUEUES.nativeReviewResponse])( + it.each([DELIVERY_QUEUES.nativeResponsePresentation, DELIVERY_QUEUES.nativeReviewResponse, DELIVERY_QUEUES.nativeStatusReassessment])( "scopes a real coalesced-input %s response wake to its active run after commit", async queue => { const companyId = await seedCompany(); const agentId = await seedAgent({ companyId }); diff --git a/server/src/modules/wake-queue/adapters/postgres.ts b/server/src/modules/wake-queue/adapters/postgres.ts index 0c80fd6f5d9..bc2c06f4cea 100644 --- a/server/src/modules/wake-queue/adapters/postgres.ts +++ b/server/src/modules/wake-queue/adapters/postgres.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "../../../services/native-status-reassessment-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "../../../services/unsafe-workspace-export-notifications.js"; import { notifyChatInboundWakeNotices } from "../../../services/chat-inbound-wake-notice-notifications.js"; import { notifyChatRunMilestones } from "../../../services/chat-run-milestone-notifications.js"; @@ -1125,6 +1126,7 @@ export function createPostgresWakeQueueAdapter(db: Db, deps: WakeQueuePostgresAd for (const issue of candidateIssues) { if (issue.executionRunId === run.id || issue.checkoutRunId === run.id) { await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: input.companyId, issueId: issue.id }); + await notifyNativeStatusReassessment(tx, { companyId: input.companyId, issueId: issue.id }); } } // Two separate updates: a retry can move `executionRunId` to a new run diff --git a/server/src/modules/wake-queue/adapters/queued-comment-postgres.ts b/server/src/modules/wake-queue/adapters/queued-comment-postgres.ts index 455c2109050..02b02622968 100644 --- a/server/src/modules/wake-queue/adapters/queued-comment-postgres.ts +++ b/server/src/modules/wake-queue/adapters/queued-comment-postgres.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "../../../services/native-status-reassessment-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "../../../services/unsafe-workspace-export-notifications.js"; import { notifyCancelledWorkspacePreservation } from "../../../services/cancelled-workspace-preservation-notifications.js"; import { notifyInstructionCollection } from "../../../services/instruction-collection-notifications.js"; @@ -120,6 +121,7 @@ function buildTransaction(tx: Db, companyId: string, issueId: string, deps: Queu }, async clearExecutionLockAndTouchIssue({ issueId, executionRunId, updatedAt }) { + await notifyNativeStatusReassessment(tx, { companyId, issueId }); await notifyUnsafeWorkspaceExportRecovery(tx, { companyId, issueId }); await tx .update(issues) diff --git a/server/src/routes/issues.ts b/server/src/routes/issues.ts index 74241e50965..e191452d97a 100644 --- a/server/src/routes/issues.ts +++ b/server/src/routes/issues.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "../services/native-status-reassessment-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "../services/unsafe-workspace-export-notifications.js"; import { notifyChatInboundWakeNotices } from "../services/chat-inbound-wake-notice-notifications.js"; import { completeIssueCommentReceipt } from "../services/issue-comment-receipt.js"; @@ -11816,6 +11817,7 @@ export function issueRoutes( promotedAt, }); const product = await db.transaction(async (tx) => { + await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId: issue.id }); const markPromoted = { sourceTrust: promotionTrust, updatedAt: promotedAt, diff --git a/server/src/services/access.ts b/server/src/services/access.ts index 64f57cb1f1a..81a0c7222c7 100644 --- a/server/src/services/access.ts +++ b/server/src/services/access.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js"; import { notifyChatCompanyAuthorityChange } from "./chat-run-milestone-notifications.js"; import { notifyVoiceChange } from "./voice/voice-recovery-notifications.js"; @@ -740,6 +741,7 @@ export function accessService(db: Db) { eq(issues.assigneeUserId, existing.principalId), sql`${issues.status} not in ('done', 'cancelled')`, ); + await notifyNativeStatusReassessment(tx, { companyId }); const resetInProgress = await tx .update(issues) .set({ diff --git a/server/src/services/approval-work-notifications.ts b/server/src/services/approval-work-notifications.ts index 6b74cb6c665..a4308d32522 100644 --- a/server/src/services/approval-work-notifications.ts +++ b/server/src/services/approval-work-notifications.ts @@ -1,6 +1,6 @@ import { and, eq } from "drizzle-orm"; import { approvals, issueApprovals, type Db } from "@paperclipai/db"; -import { notifyNativeReviewResponse } from "./native-review-response-notifications.js"; +import { notifyNativeReviewState } from "./native-review-state-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; export const notifyApprovalQueueWork = (transaction: object, companyId: string) => notifyDeliveryWork(transaction, DELIVERY_QUEUES.approvalQueueSeeding, companyId); @@ -21,7 +21,7 @@ export async function notifyApprovalWork(transaction: object, companyId: string, .where(and(eq(issueApprovals.companyId, companyId), eq(issueApprovals.approvalId, scope.approvalId))); for (const link of links) issueIds.add(link.issueId); } - for (const issueId of issueIds) await notifyNativeReviewResponse(transaction, { companyId, issueId }); + for (const issueId of issueIds) await notifyNativeReviewState(transaction, { companyId, issueId }); } export function writeApprovalChange(db: Db, companyId: string, write: (transaction: Db) => Promise, scope?: ApprovalChangeScope): Promise { diff --git a/server/src/services/chat-run-milestone-notifications.ts b/server/src/services/chat-run-milestone-notifications.ts index 6736b0e6ffc..62e5cb32ac4 100644 --- a/server/src/services/chat-run-milestone-notifications.ts +++ b/server/src/services/chat-run-milestone-notifications.ts @@ -1,4 +1,4 @@ -import { notifyNativeReviewResponse } from "./native-review-response-notifications.js"; +import { notifyNativeReviewState } from "./native-review-state-notifications.js"; import { notifyNativeResponsePresentation } from "./native-response-presentation-notifications.js"; import { notifyChatResponseAvailability } from "./chat-response-availability.js"; import { notifyChatConfirmationWork } from "./chat-confirmation-notifications.js"; @@ -12,7 +12,7 @@ export type ChatRunMilestoneScope = { companyId: string; issueId?: string; runId export async function notifyChatRunMilestones(tx: object, scope: ChatRunMilestoneScope) { if (scope.issueId || scope.runId) { - await notifyNativeReviewResponse(tx, { companyId: scope.companyId, issueId: scope.issueId, + await notifyNativeReviewState(tx, { companyId: scope.companyId, issueId: scope.issueId, runId: scope.issueId ? undefined : scope.runId }); await notifyNativeResponsePresentation(tx, { companyId: scope.companyId, issueId: scope.issueId, runId: scope.issueId ? undefined : scope.runId }); @@ -33,5 +33,5 @@ export async function notifyChatRunMilestones(tx: object, scope: ChatRunMileston export async function notifyChatCompanyAuthorityChange(tx: object, scope: { companyId: string }) { await notifyChatRunMilestones(tx, scope); await notifyNativeResponsePresentation(tx, scope); - await notifyNativeReviewResponse(tx, scope); + await notifyNativeReviewState(tx, scope); } diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index b1b72704e53..057f9a6c242 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -2,6 +2,7 @@ import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db"; /** Topics for existing durable queues; register intent before their writes. */ export const DELIVERY_QUEUES = { + nativeStatusReassessment: "native-status-reassessment", nativeReviewResponse: "native-review-response", nativeResponsePresentation: "native-response-presentation", nativePendingEffects: "native-pending-effects", diff --git a/server/src/services/execution-control-reconciliation.ts b/server/src/services/execution-control-reconciliation.ts index 3bbe612e612..7f57bcbb8e1 100644 --- a/server/src/services/execution-control-reconciliation.ts +++ b/server/src/services/execution-control-reconciliation.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js"; import { notifyNativeFinalizationWork } from "./native-finalization-notifications.js"; import { notifyUnsafeWorkspaceExportRunChange } from "./unsafe-workspace-export-notifications.js"; import { notifyCancelledWorkspacePreservation } from "./cancelled-workspace-preservation-notifications.js"; @@ -246,6 +247,7 @@ export async function reconcileAbandonedExecutionControl( if (updatedRun) await preserveNativeFailedWorkspace(tx as unknown as Db, updatedRun); return true; } + await notifyNativeStatusReassessment(tx, { companyId: task.companyId, issueId: task.id }); await tx .update(issues) .set({ executionRunId: null, checkoutRunId: null, updatedAt: now }) diff --git a/server/src/services/execution-recovery-resolution.ts b/server/src/services/execution-recovery-resolution.ts index 2d190156d6c..725fba0eb55 100644 --- a/server/src/services/execution-recovery-resolution.ts +++ b/server/src/services/execution-recovery-resolution.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js"; import { notifyChatResponseAvailability } from "./chat-response-availability.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; @@ -591,6 +592,7 @@ export async function settleUnrecoverableExecutions( let nativeFailureBlock = action.evidence.nativeFailureBlock; let nativeBootstrapFailureBlock = action.evidence.nativeBootstrapFailureBlock; if (current) { + await notifyNativeStatusReassessment(tx, { companyId: task.companyId, issueId: task.id }); if (task.status !== "blocked") await notifyChatResponseAvailability(tx, { companyId: task.companyId, issueId: task.id }); const [projected] = await tx .update(issues) diff --git a/server/src/services/heartbeat-run-events.ts b/server/src/services/heartbeat-run-events.ts index b6b1b694a49..ebce78a8266 100644 --- a/server/src/services/heartbeat-run-events.ts +++ b/server/src/services/heartbeat-run-events.ts @@ -1,3 +1,5 @@ +import { readNativeEventEvidenceOutcome } from "./native-runtime/evidence-classifier.js"; +import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js"; import { isNativeResponseEvidence, notifyNativeResponsePresentation } from "./native-response-presentation-notifications.js"; import { isSafeNativeChatProgressEventType } from "./safe-native-chat-progress.js"; import { notifyChatNativeProgress } from "./chat-native-progress-notifications.js"; @@ -72,9 +74,10 @@ export async function appendHeartbeatRunEvent( db: Db, input: AppendHeartbeatRunEventInput, ): Promise { + const projectsReassessment = readNativeEventEvidenceOutcome(input.payload?.prpEvent) !== null; const projectsProgress = isSafeNativeChatProgressEventType(input.eventType); const projectsResponse = isNativeResponseEvidence(input.eventType, input.payload?.prpEvent); - if (!input.nativeSource && !input.retryExhaustion && !projectsProgress && !projectsResponse) { + if (!input.nativeSource && !input.retryExhaustion && !projectsProgress && !projectsResponse && !projectsReassessment) { const allocated = db.$with("allocated_run_event", {}).as(sql` update ${heartbeatRuns} set next_event_seq = ${heartbeatRuns.nextEventSeq} + 1, @@ -103,7 +106,7 @@ export async function appendHeartbeatRunEvent( .select({ companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, // Only progress needs the task binding. Other native/recovery events // must not read a potentially large context snapshot to append a log. - ...(projectsProgress || projectsResponse ? { runtimeMode: heartbeatRuns.runtimeMode } : {}), + ...(projectsProgress || projectsResponse || projectsReassessment ? { runtimeMode: heartbeatRuns.runtimeMode } : {}), ...(projectsProgress ? { issueId: boundRunIssueIdSql() } : {}) }) .from(heartbeatRuns) .where(eq(heartbeatRuns.id, input.runId)) @@ -175,6 +178,9 @@ export async function appendHeartbeatRunEvent( } } + if (projectsReassessment && run.runtimeMode === "native") { + await notifyNativeStatusReassessment(tx, { companyId: run.companyId, runId: input.runId }); + } if (projectsProgress && run.runtimeMode === "native" && run.issueId) { await notifyChatNativeProgress(tx, { companyId: run.companyId, issueId: run.issueId }); } diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 2aab7492427..17574152073 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -1,3 +1,4 @@ +import { nativeStatusReassessmentWork } from "./native-runtime/native-status-reassessment-work.js"; import { registerNativeReviewResponseWork } from "./native-runtime/native-review-response-work.js"; import { registerNativeResponsePresentationWork } from "./native-runtime/native-response-presentation-work.js"; import { registerNativePendingEffectWork } from "./native-runtime/native-pending-effect-work.js"; @@ -4091,6 +4092,8 @@ export function heartbeatService( triggerIssueMonitor, reportRunActivity: clearDetachedRunWarning, + registerNativeStatusReassessmentWork: (coordinator: Parameters["register"]>[0], canRun: () => boolean) => + nativeStatusReassessmentWork(db).register(coordinator, () => canRun() && readTaskDrain(new Date()) === null), registerNativeReviewResponseWork: (coordinator: Parameters[1], canRun: () => boolean) => registerNativeReviewResponseWork(db, coordinator, () => canRun() && readTaskDrain(new Date()) === null), registerNativeResponsePresentationWork: (coordinator: Parameters[1], canRun: () => boolean) => diff --git a/server/src/services/heartbeat/queue.ts b/server/src/services/heartbeat/queue.ts index 2be0fd457f7..4de5b9fb07d 100644 --- a/server/src/services/heartbeat/queue.ts +++ b/server/src/services/heartbeat/queue.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery, notifyUnsafeWorkspaceExportRunChange } from "../unsafe-workspace-export-notifications.js"; import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js"; import { notifyInstructionCollection } from "../instruction-collection-notifications.js"; @@ -1803,6 +1804,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende await db.transaction(async tx => { await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: run.companyId, issueId }); + await notifyNativeStatusReassessment(tx, { companyId: run.companyId, issueId }); await tx .update(issues) .set({ @@ -3694,6 +3696,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende } if (!activeExecutionRun && issue.executionRunId) { + await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId: issue.id }); await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: issue.companyId, issueId: issue.id }); await tx .update(issues) @@ -3893,6 +3896,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende `- Next action: ${WORKSPACE_WORKTREE_REQUIRES_PROJECT_REMEDIATION}`, ].join("\n"); await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: issue.companyId, issueId: issue.id }); + await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId: issue.id }); if (issue.status !== "blocked") await notifyChatResponseAvailability(tx, { companyId: issue.companyId, issueId: issue.id }); await tx .update(issues) diff --git a/server/src/services/heartbeat/recovery.ts b/server/src/services/heartbeat/recovery.ts index 7225dba15b5..e6c3e03c084 100644 --- a/server/src/services/heartbeat/recovery.ts +++ b/server/src/services/heartbeat/recovery.ts @@ -1947,7 +1947,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD let settledProcessesDiscovered = false; - async function reapOrphanedRuns(opts?: { staleThresholdMs?: number; skipScheduledRetryCancellation?: boolean; skipCapturedInstructionCommit?: boolean; skipInstructionDirectoryCleanup?: boolean; skipStoppedInstructionCollection?: boolean; skipCancelledWorkspacePreservation?: boolean; skipObsoletePolicyReviews?: boolean; skipAutomaticCompletionReviews?: boolean; skipUnsafeWorkspaceExports?: boolean; skipNativePendingEffects?: boolean; skipNativeResponsePresentation?: boolean; skipNativeReviewResponse?: boolean }) { + async function reapOrphanedRuns(opts?: { staleThresholdMs?: number; skipScheduledRetryCancellation?: boolean; skipCapturedInstructionCommit?: boolean; skipInstructionDirectoryCleanup?: boolean; skipStoppedInstructionCollection?: boolean; skipCancelledWorkspacePreservation?: boolean; skipObsoletePolicyReviews?: boolean; skipAutomaticCompletionReviews?: boolean; skipUnsafeWorkspaceExports?: boolean; skipNativePendingEffects?: boolean; skipNativeResponsePresentation?: boolean; skipNativeReviewResponse?: boolean; skipNativeStatusReassessment?: boolean }) { if (!opts?.skipCancelledWorkspacePreservation) await preserveCancelledNativeWorkspaces(db); const staleThresholdMs = opts?.staleThresholdMs ?? 0; const now = new Date(); @@ -1971,6 +1971,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD skipNativePendingEffects: opts?.skipNativePendingEffects, skipNativeResponsePresentation: opts?.skipNativeResponsePresentation, skipNativeReviewResponse: opts?.skipNativeReviewResponse, + skipNativeStatusReassessment: opts?.skipNativeStatusReassessment, environmentRuntime, pluginWorkerManager: dependencies.pluginWorkerManager, onWorkspaceSettled: settleRecoveredNativeWorkspace, diff --git a/server/src/services/issue-thread-interactions.ts b/server/src/services/issue-thread-interactions.ts index 295cf894a36..38bbdb657c8 100644 --- a/server/src/services/issue-thread-interactions.ts +++ b/server/src/services/issue-thread-interactions.ts @@ -1,4 +1,5 @@ -import { notifyNativeReviewResponse } from "./native-review-response-notifications.js"; +import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js"; +import { notifyNativeReviewState } from "./native-review-state-notifications.js"; import { notifyChatConfirmationResolution } from "./chat-confirmation-notifications.js"; import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js"; import { notifyToolReviewChange } from "./tool-review-work-notifications.js"; @@ -1945,7 +1946,7 @@ async function expireStaleRequestConfirmationTarget( const expireInTransaction = async (tx: Db | any) => { await notifyChatConfirmationResolution(tx, args.row); if (args.row.kind === "request_confirmation") await notifyChatRunMilestones(tx, args.row); - else await notifyNativeReviewResponse(tx, args.row); + else await notifyNativeReviewState(tx, args.row); const [updated] = await tx .update(issueThreadInteractions) .set({ @@ -2225,7 +2226,7 @@ export function issueThreadInteractionService( await assertInteractionRunWriteAllowed(tx as unknown as Db, args.issue, args.actor); await notifyChatConfirmationResolution(tx, args.current); if (args.current.kind === "request_confirmation") await notifyChatRunMilestones(tx, args.current); - else await notifyNativeReviewResponse(tx, args.current); + else await notifyNativeReviewState(tx, args.current); await args.mutationOptions?.beforeResolveInTransaction?.(tx); // Policy mutations and review transitions use the same issue-row lock, // so the authoritative review policy and requester are stable through @@ -2516,7 +2517,7 @@ export function issueThreadInteractionService( await assertInteractionRunWriteAllowed(tx as unknown as Db, args.issue, args.actor); await notifyChatConfirmationResolution(tx, args.current); if (args.current.kind === "request_confirmation") await notifyChatRunMilestones(tx, args.current); - else await notifyNativeReviewResponse(tx, args.current); + else await notifyNativeReviewState(tx, args.current); await args.mutationOptions?.beforeResolveInTransaction?.(tx); const issueContext = await tx .select({ @@ -2753,6 +2754,7 @@ export function issueThreadInteractionService( .where(and(eq(heartbeatRuns.id, input.sourceRunId), eq(heartbeatRuns.companyId, issue.companyId))); if (!sourceRun) throw unprocessable("Interaction source run is unavailable"); const originCommentIds = await currentContinuationOrigins(tx as unknown as Db, issue.companyId, issue.id, sourceRun.context); + await notifyNativeReviewState(tx, { companyId: issue.companyId, issueId: issue.id }); const [row] = await tx .insert(issueThreadInteractions) .values({ @@ -2909,7 +2911,7 @@ export function issueThreadInteractionService( const updated = await db.transaction(async (tx) => { await notifyChatConfirmationResolution(tx, current); await mutationOptions.beforeResolveInTransaction?.(tx); - await notifyNativeReviewResponse(tx, current); + await notifyNativeReviewState(tx, current); const [row] = await tx .update(issueThreadInteractions) .set({ @@ -3345,7 +3347,7 @@ export function issueThreadInteractionService( const updated = await db.transaction(async (tx) => { await notifyChatConfirmationResolution(tx, row); if (["ask_user_questions", "request_confirmation"].includes(row.kind)) await notifyChatRunMilestones(tx, row); - else await notifyNativeReviewResponse(tx, row); + else await notifyNativeReviewState(tx, row); if (row.kind === "request_confirmation") await notifyToolProjectionForInteractions(tx, row.companyId, [row.id]); const [resolved] = await tx .update(issueThreadInteractions) @@ -3433,7 +3435,7 @@ export function issueThreadInteractionService( const updated = await db.transaction(async (tx) => { await notifyChatConfirmationResolution(tx, row); if (["ask_user_questions", "request_confirmation"].includes(row.kind)) await notifyChatRunMilestones(tx, row); - else await notifyNativeReviewResponse(tx, row); + else await notifyNativeReviewState(tx, row); if (row.kind === "request_confirmation") await notifyToolProjectionForInteractions(tx, row.companyId, [row.id]); const [updatedRow] = await tx .update(issueThreadInteractions) @@ -3863,6 +3865,8 @@ export function issueThreadInteractionService( } if (data.kind === "ask_user_questions" || data.kind === "request_confirmation") { await notifyChatRunMilestones(tx, { companyId: issue.companyId, issueId: issue.id }); + } else { + await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId: issue.id }); } const [row] = await tx .insert(issueThreadInteractions) @@ -4215,7 +4219,7 @@ export function issueThreadInteractionService( } } } - await notifyNativeReviewResponse(tx, { companyId: issue.companyId, issueId: issue.id }); + await notifyNativeReviewState(tx, { companyId: issue.companyId, issueId: issue.id }); const resolvedAt = new Date(); const [claimed] = await tx .update(issueThreadInteractions) @@ -4479,7 +4483,8 @@ export function issueThreadInteractionService( complete, items, } satisfies RequestItemVerdictsResult; - if (complete) await notifyNativeReviewResponse(tx, { companyId: issue.companyId, issueId: issue.id }); + if (complete) await notifyNativeReviewState(tx, { companyId: issue.companyId, issueId: issue.id }); + else await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId: issue.id }); const [updated] = await tx .update(issueThreadInteractions) .set({ @@ -4546,7 +4551,7 @@ export function issueThreadInteractionService( await notifyChatConfirmationResolution(tx, current); await mutationOptions.beforeResolveInTransaction?.(tx); await assertInteractionRunWriteAllowed(tx as unknown as Db, issue, actor); - await notifyNativeReviewResponse(tx, { companyId: issue.companyId, issueId: issue.id }); + await notifyNativeReviewState(tx, { companyId: issue.companyId, issueId: issue.id }); const rows = await tx.update(issueThreadInteractions) .set({ status: "rejected", @@ -4638,7 +4643,7 @@ export function issueThreadInteractionService( const updated = await db.transaction(async (tx) => { await notifyChatConfirmationResolution(tx, row); if (["ask_user_questions", "request_confirmation"].includes(row.kind)) await notifyChatRunMilestones(tx, row); - else await notifyNativeReviewResponse(tx, row); + else await notifyNativeReviewState(tx, row); if (row.kind === "connection_intent") { await tx .delete(toolOauthStates) @@ -4816,7 +4821,7 @@ export function issueThreadInteractionService( const updatedInteractions = await db.transaction(async (tx) => { await notifyChatConfirmationResolution(tx, sampleConfirmationRow); if (commentRows.some(row => row.kind === "request_confirmation")) await notifyChatRunMilestones(tx, sampleConfirmationRow); - else await notifyNativeReviewResponse(tx, sampleConfirmationRow); + else await notifyNativeReviewState(tx, sampleConfirmationRow); await notifyToolProjectionForInteractions(tx, issue.companyId, confirmationRowIds); const updatedRows = await tx .update(issueThreadInteractions) @@ -4852,7 +4857,7 @@ export function issueThreadInteractionService( for (const row of itemVerdictRows) { const updated = await db.transaction(async tx => { - await notifyNativeReviewResponse(tx, row); + await notifyNativeReviewState(tx, row); const [retired] = await tx .update(issueThreadInteractions) .set({ @@ -4878,7 +4883,7 @@ export function issueThreadInteractionService( for (const row of connectionIntentRows) { const [updated] = await db.transaction(async (tx) => { - await notifyNativeReviewResponse(tx, row); + await notifyNativeReviewState(tx, row); await notifyChatConfirmationResolution(tx, row); await tx .delete(toolOauthStates) @@ -4967,7 +4972,7 @@ export function issueThreadInteractionService( const updated = await db.transaction(async (tx) => { await notifyChatConfirmationResolution(tx, row); if (row.kind === "request_confirmation") await notifyChatRunMilestones(tx, row); - else await notifyNativeReviewResponse(tx, row); + else await notifyNativeReviewState(tx, row); const [resolved] = await tx .update(issueThreadInteractions) .set({ @@ -5031,7 +5036,7 @@ export function issueThreadInteractionService( ? "the AI account" : `the ${payload?.serviceName?.trim() || "requested"} connection`; const updated = await db.transaction(async tx => { - await notifyNativeReviewResponse(tx, { companyId: issue.companyId, issueId: issue.id }); + await notifyNativeReviewState(tx, { companyId: issue.companyId, issueId: issue.id }); await notifyChatConfirmationResolution(tx, { companyId: issue.companyId, issueId: issue.id, kind: "connection_intent" }); const [resolved] = await tx .update(issueThreadInteractions) @@ -5093,7 +5098,7 @@ export function issueThreadInteractionService( .transaction(async (tx) => { await notifyChatConfirmationResolution(tx, row); if (["ask_user_questions", "request_confirmation"].includes(row.kind)) await notifyChatRunMilestones(tx, row); - else await notifyNativeReviewResponse(tx, row); + else await notifyNativeReviewState(tx, row); if (row.kind === "connection_intent") { await tx .delete(toolOauthStates) @@ -5186,7 +5191,7 @@ export function issueThreadInteractionService( await assertInteractionRunWriteAllowed(tx as unknown as Db, issue, actor); await notifyChatConfirmationResolution(tx, current); if (["ask_user_questions", "request_confirmation"].includes(current.kind)) await notifyChatRunMilestones(tx, current); - else await notifyNativeReviewResponse(tx, current); + else await notifyNativeReviewState(tx, current); await resolveLinkedToolActionRequests(tx, current, { status: "cancelled", fromStatuses: ["pending", "approved"], @@ -5449,7 +5454,7 @@ export function issueThreadInteractionService( await assertInteractionRunWriteAllowed(tx as unknown as Db, issue, actor); await notifyChatConfirmationResolution(tx, current); if (["ask_user_questions", "request_confirmation"].includes(current.kind)) await notifyChatRunMilestones(tx, current); - else await notifyNativeReviewResponse(tx, current); + else await notifyNativeReviewState(tx, current); await resolveLinkedToolActionRequests(tx, current, { status: "cancelled", fromStatuses: ["pending", "approved"], diff --git a/server/src/services/issue-tree-control.ts b/server/src/services/issue-tree-control.ts index 5697e19449c..d66d3f0732a 100644 --- a/server/src/services/issue-tree-control.ts +++ b/server/src/services/issue-tree-control.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js"; import { notifyChatResponseAvailability } from "./chat-response-availability.js"; import { notifyChatInboundWakeNotices } from "./chat-inbound-wake-notice-notifications.js"; @@ -1002,6 +1003,7 @@ export function issueTreeControlService(db: Db) { for (const [status, issueIdsForStatus] of issueIdsByStatus) { if (issueIdsForStatus.length === 0) continue; for (const issueId of issueIdsForStatus) { + await notifyNativeStatusReassessment(tx, { companyId, issueId }); await notifyTerminalWorkspaceWork(tx, `issue:${issueId}`); await notifyChatResponseAvailability(tx, { companyId, issueId }); if (status === "blocked" || status === "in_review") await notifyUnsafeWorkspaceExportRecovery(tx, { companyId, issueId }); diff --git a/server/src/services/issues.ts b/server/src/services/issues.ts index 23a0f9c1c99..8deb995df11 100644 --- a/server/src/services/issues.ts +++ b/server/src/services/issues.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js"; import { notifyChatResponseAvailability } from "./chat-response-availability.js"; import { notifyChatInboundWakeNotices } from "./chat-inbound-wake-notice-notifications.js"; @@ -2513,6 +2514,7 @@ async function listPendingFinalizeBlockerIssueIds( blockerIssueId: string; executionWorkspaceId: string; }>, + lockSources = false, ): Promise> { const pending = new Set(); const blockerIssueIds = [ @@ -2529,7 +2531,14 @@ async function listPendingFinalizeBlockerIssueIds( ), ); - const rows = await dbOrTx + if (lockSources) { + + + await dbOrTx.select({ id: executionWorkspaces.id }).from(executionWorkspaces) + .where(and(eq(executionWorkspaces.companyId, companyId), inArray(executionWorkspaces.id, executionWorkspaceIds))) + .for("update", { noWait: true }); + } + const query = dbOrTx .select({ issueId: workspaceOperations.issueId, executionWorkspaceId: workspaceOperations.executionWorkspaceId, @@ -2548,6 +2557,8 @@ async function listPendingFinalizeBlockerIssueIds( ), ); + const rows = await (lockSources ? query.for("share", { noWait: true }) : query); + const latestAttributedByBlockerWorkspace = new Map< string, { phase: string; status: string; startedAt: Date } @@ -2697,6 +2708,7 @@ async function listIssueDependencyReadinessMap( dbOrTx: Pick, companyId: string, issueIds: string[], + lockSources = false, ) { const uniqueIssueIds = [...new Set(issueIds.filter(Boolean))]; const readinessMap = new Map(); @@ -2705,7 +2717,7 @@ async function listIssueDependencyReadinessMap( } if (uniqueIssueIds.length === 0) return readinessMap; - const blockerRows = await dbOrTx + const blockerQuery = dbOrTx .select({ issueId: issueRelations.relatedIssueId, blockerIssueId: issueRelations.issueId, @@ -2722,6 +2734,10 @@ async function listIssueDependencyReadinessMap( ), ); + + + const blockerRows = await (lockSources ? blockerQuery.for("update", { noWait: true }) : blockerQuery); + // Collect issue/workspace pairs of "done" blockers — these are the only ones // subject to the workspace-finalize barrier. Blockers that aren't done already // mark the dependent as not-ready and don't need a finalize check. @@ -2742,6 +2758,7 @@ async function listIssueDependencyReadinessMap( dbOrTx, companyId, doneBlockerWorkspacePairs, + lockSources, ); for (const row of blockerRows) { @@ -7823,6 +7840,7 @@ export function issueService(db: Db) { if (isExplicitContinuationRetryClaim(issue, run)) return false; await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: issue.companyId, issueId }); + await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId }); const updated = await tx .update(issues) .set({ @@ -7897,6 +7915,7 @@ export function issueService(db: Db) { } await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: issue.companyId, issueId }); + await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId }); const updated = await tx .update(issues) .set({ @@ -9311,7 +9330,13 @@ export function issueService(db: Db) { }; }, - getDependencyReadiness: async (issueId: string, dbOrTx: any = db) => { + + + getDependencyReadiness: async ( + issueId: string, + dbOrTx: any = db, + options: { lockSources?: boolean } = {}, + ) => { const issue = await dbOrTx .select({ id: issues.id, companyId: issues.companyId }) .from(issues) @@ -9324,6 +9349,7 @@ export function issueService(db: Db) { dbOrTx, issue.companyId, [issueId], + options.lockSources, ); return readiness.get(issueId) ?? createIssueDependencyReadiness(issueId); }, @@ -11473,6 +11499,11 @@ export function issueService(db: Db) { || (patch.checkoutRunId !== undefined && patch.checkoutRunId !== receiptExisting.checkoutRunId)) { await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: receiptExisting.companyId, issueId: id }); } + if (patch.status !== undefined || patch.statusVersion !== undefined || patch.executionState !== undefined + || patch.executionRunId !== undefined || patch.checkoutRunId !== undefined + || patch.assigneeAgentId !== undefined || patch.assigneeUserId !== undefined || blockedByIssueIds !== undefined) { + await notifyNativeStatusReassessment(tx, { companyId: receiptExisting.companyId, issueId: id }); + } const updated = await tx .update(issues) .set(patch) @@ -12383,6 +12414,7 @@ export function issueService(db: Db) { const releaseStatus = existing.status === "in_progress" ? "todo" : existing.status; await notifyTerminalWorkspaceWork(tx, `issue:${id}`); + await notifyNativeStatusReassessment(tx, { companyId: existing.companyId, issueId: id }); const updated = await tx .update(issues) .set({ @@ -12434,6 +12466,9 @@ export function issueService(db: Db) { } await notifyTerminalWorkspaceWork(tx, `issue:${id}`); + if (existing.checkoutRunId || existing.executionRunId || options.clearAssignee) { + await notifyNativeStatusReassessment(tx, { companyId: existing.companyId, issueId: id }); + } if (existing.checkoutRunId || existing.executionRunId) { await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: existing.companyId, issueId: id }); } @@ -13497,6 +13532,7 @@ export function issueService(db: Db) { } } } + await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId: issue.id }); const [asset] = await tx .insert(assets) .values({ diff --git a/server/src/services/legacy-execution-recovery.ts b/server/src/services/legacy-execution-recovery.ts index 22461cf38b3..427704de63f 100644 --- a/server/src/services/legacy-execution-recovery.ts +++ b/server/src/services/legacy-execution-recovery.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js"; import { notifyInstructionCollection } from "./instruction-collection-notifications.js"; import { notifyExecutionRunWork } from "./execution-control-notifications.js"; @@ -172,6 +173,9 @@ export async function terminalizeLegacyExecution(input: { ? retainedLeaseIds.length > 0 || hasRequiredWorkspaceRecovery(updated.resultJson) : !input.reconcileIfNeeded || await legacyExecutionNeedsReconciliationWithEvidence(tx as unknown as Db, updated); if (!needsRecovery) return updated; + if (!input.recordRestoreFailureOnly && task && (task.executionRunId === run.id || task.checkoutRunId === run.id)) { + await notifyNativeStatusReassessment(tx, { companyId: task.companyId, issueId: task.id }); + } if (!input.recordRestoreFailureOnly && task?.executionRunId === run.id) await tx .update(issues) diff --git a/server/src/services/native-finalization-notifications.ts b/server/src/services/native-finalization-notifications.ts index 91ccfc2023d..3f95a4a67a4 100644 --- a/server/src/services/native-finalization-notifications.ts +++ b/server/src/services/native-finalization-notifications.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js"; import { notifyNativeReviewResponse } from "./native-review-response-notifications.js"; import { notifyNativeResponsePresentation } from "./native-response-presentation-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; @@ -6,6 +7,7 @@ export type NativeFinalizationWorkScope = { companyId: string; runId?: string; i export async function notifyNativeFinalizationWork(transaction: object, scope: NativeFinalizationWorkScope) { if (scope.issueId || scope.runId) { + await notifyNativeStatusReassessment(transaction, scope); await notifyNativeResponsePresentation(transaction, scope); await notifyNativeReviewResponse(transaction, scope); } diff --git a/server/src/services/native-review-state-notifications.ts b/server/src/services/native-review-state-notifications.ts new file mode 100644 index 00000000000..43e61a0fdba --- /dev/null +++ b/server/src/services/native-review-state-notifications.ts @@ -0,0 +1,7 @@ +import { notifyNativeReviewResponse } from "./native-review-response-notifications.js"; +import { notifyNativeStatusReassessment, type NativeStatusReassessmentScope } from "./native-status-reassessment-notifications.js"; + +export async function notifyNativeReviewState(tx: object, scope: NativeStatusReassessmentScope) { + await notifyNativeReviewResponse(tx, scope); + await notifyNativeStatusReassessment(tx, scope); +} diff --git a/server/src/services/native-runtime/automatic-completion-reviews.ts b/server/src/services/native-runtime/automatic-completion-reviews.ts index 9f7eea8c326..42be62845cc 100644 --- a/server/src/services/native-runtime/automatic-completion-reviews.ts +++ b/server/src/services/native-runtime/automatic-completion-reviews.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js"; import type { AutomaticCompletionReviewScope } from "../automatic-completion-review-notifications.js"; import { DELIVERY_QUEUES } from "../delivery-work-notifications.js"; import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js"; @@ -138,6 +139,7 @@ async function dismissCandidates(db: Db, candidates: Awaited { ); beforeAll(async () => { - temporary = await startEmbeddedPostgresTestDatabase("native-chat-reuse-"); - db = createDb(temporary.connectionString); + const external = resolveExternalTestDatabaseUrl("PAPERCLIP_CHAT_REUSE_TEST_DATABASE_URL"); + if (!external) temporary = await startEmbeddedPostgresTestDatabase("native-chat-reuse-"); + db = createDb(external ?? temporary!.connectionString); const storageRoot = await mkdtemp( path.join(tmpdir(), "paperclip-chat-reuse-"), ); @@ -295,6 +297,7 @@ describe("native same-conversation chat attachment reuse", () => { }); afterAll(async () => { + await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); }); diff --git a/server/src/services/native-runtime/completion-contracts.ts b/server/src/services/native-runtime/completion-contracts.ts index f8a642769db..7a4fc728bfd 100644 --- a/server/src/services/native-runtime/completion-contracts.ts +++ b/server/src/services/native-runtime/completion-contracts.ts @@ -133,6 +133,22 @@ export function buildNativeCompletionContract( }; } +function nativeCompletionContractLockKey(companyId: string, issueId: string): string { + return `paperclip:native-completion-contract:${companyId}:${issueId}`; +} + +export async function lockCurrentNativeCompletionContract(tx: Db, input: { + companyId: string; issueId: string; contractId: string; +}): Promise { + const [lock] = await tx.execute(sql`select pg_try_advisory_xact_lock(hashtextextended( + ${nativeCompletionContractLockKey(input.companyId, input.issueId)}, 0)) as acquired`) as unknown as Array<{ acquired: boolean }>; + if (!lock.acquired) return false; + const [latest] = await tx.select({ id: completionContracts.id }).from(completionContracts) + .where(and(eq(completionContracts.companyId, input.companyId), eq(completionContracts.issueId, input.issueId))) + .orderBy(desc(completionContracts.revision)).limit(1); + return latest?.id === input.contractId; +} + export async function ensureNativeCompletionContract(input: { db: Db; companyId: string; @@ -151,11 +167,8 @@ export async function ensureNativeCompletionContract(input: { humanResponseId?: string | null; }) { return input.db.transaction(async (tx) => { - await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${[ - "paperclip:native-completion-contract", - input.companyId, - input.issue.id, - ].join(":")}, 0))`); + await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended( + ${nativeCompletionContractLockKey(input.companyId, input.issue.id)}, 0))`); const policy = resolveNativeCompletionPolicy(input.issue); const latest = await tx .select() diff --git a/server/src/services/native-runtime/evidence-classifier.ts b/server/src/services/native-runtime/evidence-classifier.ts index 835dacf57dd..66182c1c9f8 100644 --- a/server/src/services/native-runtime/evidence-classifier.ts +++ b/server/src/services/native-runtime/evidence-classifier.ts @@ -102,26 +102,35 @@ function uuidRef(ref: string, prefixes: string[]): string | null { return null; } +export function readNativeEventEvidenceOutcome(value: unknown): "accepted" | "rejected" | null { + const event = record(value); + if (event.sourceKind !== "control_plane") return null; + const outcome = text(record(event.payload).evidenceOutcome); + return outcome === "accepted" || outcome === "rejected" ? outcome : null; +} + async function classifyEvidenceRef(input: { db: Db; companyId: string; issueId: string; runId: string; ref: string; + lockSources?: boolean; }): Promise { const handoffId = uuidRef(input.ref, ["case-handoff"]); if (handoffId) { - const [row] = await input.db.select({ id: caseHandoffs.id, state: caseHandoffs.state }).from(caseHandoffs).where(and( + const query = input.db.select({ id: caseHandoffs.id, state: caseHandoffs.state }).from(caseHandoffs).where(and( eq(caseHandoffs.id, handoffId), eq(caseHandoffs.companyId, input.companyId), eq(caseHandoffs.issueId, input.issueId), eq(caseHandoffs.sourceRunId, input.runId), )).limit(1); + const [row] = await (input.lockSources ? query.for("share", { noWait: true }) : query); return { ref: input.ref, kind: "case_handoff", outcome: !row ? "missing" : row.state === "cancelled" ? "rejected" : "accepted", reasonCode: !row ? "case_handoff_missing" : row.state === "cancelled" ? "case_handoff_cancelled" : "case_handoff_persisted", durableRecordId: row?.id ?? null }; } const eventMatch = /^event:(\d+)$/.exec(input.ref); if (eventMatch) { const cursor = Number(eventMatch[1]); - const event = await input.db.select({ + const query = input.db.select({ id: heartbeatRunEvents.id, eventType: heartbeatRunEvents.eventType, payload: heartbeatRunEvents.payload, @@ -131,16 +140,16 @@ async function classifyEvidenceRef(input: { eq(heartbeatRunEvents.companyId, input.companyId), eq(heartbeatRunEvents.runId, input.runId), or(eq(heartbeatRunEvents.seq, cursor), eq(heartbeatRunEvents.sourceSeq, cursor)), - )).limit(1).then((rows) => rows[0] ?? null); + )).limit(1); + const [event] = await (input.lockSources ? query.for("share", { noWait: true }) : query); if (!event) { return { ref: input.ref, kind: "event", outcome: "missing", reasonCode: "durable_run_event_missing", durableRecordId: null }; } - const prpEvent = record(record(event.payload).prpEvent); - const evidenceOutcome = text(record(prpEvent.payload).evidenceOutcome); - if (prpEvent.sourceKind === "control_plane" && evidenceOutcome === "accepted") { + const evidenceOutcome = readNativeEventEvidenceOutcome(record(event.payload).prpEvent); + if (evidenceOutcome === "accepted") { return { ref: input.ref, kind: "event", outcome: "accepted", reasonCode: "control_plane_evidence_accepted", durableRecordId: String(event.id) }; } - if (prpEvent.sourceKind === "control_plane" && evidenceOutcome === "rejected") { + if (evidenceOutcome === "rejected") { return { ref: input.ref, kind: "event", outcome: "rejected", reasonCode: "control_plane_evidence_rejected", durableRecordId: String(event.id) }; } return { @@ -156,7 +165,7 @@ async function classifyEvidenceRef(input: { const workProductId = uuidRef(input.ref, ["work_product", "work-product", "artifact"]); if (workProductId) { - const row = await input.db.select({ + const query = input.db.select({ id: issueWorkProducts.id, status: issueWorkProducts.status, reviewState: issueWorkProducts.reviewState, @@ -164,7 +173,8 @@ async function classifyEvidenceRef(input: { eq(issueWorkProducts.id, workProductId), eq(issueWorkProducts.companyId, input.companyId), eq(issueWorkProducts.issueId, input.issueId), - )).limit(1).then((rows) => rows[0] ?? null); + )).limit(1); + const [row] = await (input.lockSources ? query.for("share", { noWait: true }) : query); if (!row) return { ref: input.ref, kind: "work_product", outcome: "missing", reasonCode: "work_product_missing", durableRecordId: null }; if ( ["failed", "error", "cancelled", "rejected", "changes_requested"].includes(row.status) @@ -183,7 +193,7 @@ async function classifyEvidenceRef(input: { const approvalId = uuidRef(input.ref, ["approval"]); if (approvalId) { - const row = await input.db.select({ id: approvals.id, status: approvals.status }) + const query = input.db.select({ id: approvals.id, status: approvals.status }) .from(issueApprovals) .innerJoin(approvals, eq(issueApprovals.approvalId, approvals.id)) .where(and( @@ -191,7 +201,8 @@ async function classifyEvidenceRef(input: { eq(issueApprovals.issueId, input.issueId), eq(approvals.companyId, input.companyId), eq(approvals.id, approvalId), - )).limit(1).then((rows) => rows[0] ?? null); + )).limit(1); + const [row] = await (input.lockSources ? query.for("share", { noWait: true }) : query); if (!row) return { ref: input.ref, kind: "approval", outcome: "missing", reasonCode: "approval_missing", durableRecordId: null }; if (row.status === "approved") return { ref: input.ref, kind: "approval", outcome: "accepted", reasonCode: "approval_approved", durableRecordId: row.id }; if (["rejected", "cancelled"].includes(row.status)) return { ref: input.ref, kind: "approval", outcome: "rejected", reasonCode: "approval_rejected", durableRecordId: row.id }; @@ -200,7 +211,7 @@ async function classifyEvidenceRef(input: { const interactionId = uuidRef(input.ref, ["interaction"]); if (interactionId) { - const row = await input.db.select({ + const query = input.db.select({ id: issueThreadInteractions.id, status: issueThreadInteractions.status, result: issueThreadInteractions.result, @@ -208,7 +219,8 @@ async function classifyEvidenceRef(input: { eq(issueThreadInteractions.id, interactionId), eq(issueThreadInteractions.companyId, input.companyId), eq(issueThreadInteractions.issueId, input.issueId), - )).limit(1).then((rows) => rows[0] ?? null); + )).limit(1); + const [row] = await (input.lockSources ? query.for("share", { noWait: true }) : query); if (!row) return { ref: input.ref, kind: "interaction", outcome: "missing", reasonCode: "interaction_missing", durableRecordId: null }; const outcome = text(record(row.result).outcome); if (["accepted", "answered", "resolved"].includes(outcome ?? "") || ["accepted", "answered"].includes(row.status)) { @@ -222,11 +234,12 @@ async function classifyEvidenceRef(input: { const attachmentId = uuidRef(input.ref, ["attachment"]); if (attachmentId) { - const row = await input.db.select({ id: issueAttachments.id }).from(issueAttachments).where(and( + const query = input.db.select({ id: issueAttachments.id }).from(issueAttachments).where(and( eq(issueAttachments.id, attachmentId), eq(issueAttachments.companyId, input.companyId), eq(issueAttachments.issueId, input.issueId), - )).limit(1).then((rows) => rows[0] ?? null); + )).limit(1); + const [row] = await (input.lockSources ? query.for("share", { noWait: true }) : query); return row ? { ref: input.ref, kind: "attachment", outcome: "accepted", reasonCode: "attachment_persisted", durableRecordId: row.id } : { ref: input.ref, kind: "attachment", outcome: "missing", reasonCode: "attachment_missing", durableRecordId: null }; @@ -241,10 +254,10 @@ async function classifyEvidenceRef(input: { }; } -/** - * Classifies model claims against immutable server-owned records. A model's - * `satisfied` or `passed` value is retained as a claim but never becomes proof. - */ + + + + export async function classifyNativeEvidence(input: { db: Db; companyId: string; @@ -252,6 +265,8 @@ export async function classifyNativeEvidence(input: { runId: string; contract: Record; result: Record; + + lockSources?: boolean; }): Promise { const claim = record(input.result.completionClaim); const claimedCriteria = Array.isArray(claim.criteria) ? claim.criteria.map(record) : []; @@ -355,11 +370,12 @@ export async function classifyNativeEvidence(input: { const ignoredAttentionRequests = [...normalizedSignals.ignoredAttentionRequests]; for (const request of normalizedSignals.actionableAttentionRequests) { if (request.ownerClass === "agent" && request.targetAgentId) { - const targetExists = await input.db.select({ id: agents.id }).from(agents).where(and( + const targetQuery = input.db.select({ id: agents.id }).from(agents).where(and( eq(agents.id, request.targetAgentId), eq(agents.companyId, input.companyId), - )).limit(1).then((rows) => rows.length > 0); - if (!targetExists) { + )).limit(1); + const targets = await (input.lockSources ? targetQuery.for("share", { noWait: true }) : targetQuery); + if (targets.length === 0) { ignoredAttentionRequests.push({ sourceIndex: request.sourceIndex, sourceKind: request.sourceKind, diff --git a/server/src/services/native-runtime/native-finalization-reconciler.ts b/server/src/services/native-runtime/native-finalization-reconciler.ts index cc97199948b..48a31e52cc0 100644 --- a/server/src/services/native-runtime/native-finalization-reconciler.ts +++ b/server/src/services/native-runtime/native-finalization-reconciler.ts @@ -1,43 +1,37 @@ +import { nativeStatusReassessmentSourceIsSuperseded, reassessNativeStatusFromSource } from "./native-status-reassessment.js"; +import { resolveNativeReconciliationStatus } from "./native-reconciliation-status.js"; +export { resolveNativeReconciliationStatus, type NativeReconciliationFacts } from "./native-reconciliation-status.js"; import { pendingNativeStatusEffectsCondition } from "./native-pending-effect-work.js"; import { notifyUnsafeWorkspaceExportRunChange } from "../unsafe-workspace-export-notifications.js"; import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js"; import { notifyInstructionCollection } from "../instruction-collection-notifications.js"; -import { readRequiredQualityReview } from "../required-quality-review.js"; import type { WorkspaceWorkers } from "../plugin-workspaces.js"; import { recoverLegacyUnsafeWorkspaceExports } from "./native-workspace-export-recovery.js"; -import { hasPendingNativeChildCompletion } from "./native-child-completion-delivery.js"; -import { dismissAutomaticCompletionReviews, decisionHasRetiredAutomaticReview } from "./automatic-completion-reviews.js"; +import { dismissAutomaticCompletionReviews } from "./automatic-completion-reviews.js"; import { logger } from "../../middleware/logger.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js"; import { createHash, randomUUID } from "node:crypto"; import { and, asc, desc, eq, gt, inArray, isNotNull, isNull, lt, lte, ne, not, notInArray, or, sql, type SQL } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { - completionContracts, agents, heartbeatRunEvents, heartbeatRuns, - issueWorkProducts, issues, nativeRunFinalizations, nativeRunResults, statusDecisionEffects, statusDecisions, - workAssessments, workspaceOperations, } from "@paperclipai/db"; import { finalizeNativeRun, - pendingNativeGovernance, - resolveNativeFinalizerStatus, recordNativeFinalizationFailure, repairCommittedNativeReviewResponse, repairCommittedNativeChatResponse, } from "./native-run-finalizer.js"; import { - commitNativeStatusDecision, dispatchPendingNativeStatusEffects, - NativeStatusRaceError, } from "./status-decision-committer.js"; import { issueRecoveryActionService } from "../issue-recovery-actions.js"; import { issueService } from "../issues.js"; @@ -50,17 +44,13 @@ import { readNativeWorkspaceSyncReference, } from "./native-workspace-sync.js"; import type { EnvironmentRuntimeService } from "../environment-runtime.js"; -import { classifyNativeEvidence } from "./evidence-classifier.js"; import { createNativeReconciliationPassTiming } from "./native-reconciliation-timing.js"; -import { recordNativeWorkAssessment } from "./work-assessments.js"; import { isNativeRunnerOwnershipHeld, nativeRunnerOwnershipNotHeldCondition, } from "./native-runner-ownership.js"; import { - NATIVE_STATUS_ARBITER_POLICY_VERSION, type NativeAuthoritativeIssueStatus, - type NativeStatusDecision, } from "./status-arbiter.js"; import { getNativeReviewAssignment } from "./native-review-participant.js"; @@ -70,125 +60,12 @@ import { readRunCancellation } from "../run-cancellation.js"; import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js"; import { retireSupersededNativeRetry } from "./native-superseded-retry.js"; -export type NativeReconciliationFacts = { - equivalentAttentionFamily?: boolean; - canonicalReplay?: boolean; - callerMaterialConflict?: boolean; - workspaceOperationPending?: boolean; - undeliveredEffectCount?: number; - authoritativeStatusChanged?: boolean; - newEvidenceSatisfiesContract?: boolean; - hasPendingChildCompletion?: boolean; - dependencyResolved?: boolean; - authorizedResume?: boolean; - statusVersionAdvanced?: boolean; -}; - function record(value: unknown): Record { return value && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; } -/** Append-only recovery/reconciliation re-arbitration from durable facts. */ -export function resolveNativeReconciliationStatus(input: { - facts: NativeReconciliationFacts; - priorIssueStatus: NativeAuthoritativeIssueStatus; - agentId: string; -}): NativeStatusDecision { - const preserve = (reasonCode: string, effects: NativeStatusDecision["effects"]): NativeStatusDecision => ({ - policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, - statusAction: "preserve", - toStatus: input.priorIssueStatus, - reasonCode, - unblockDescriptor: null, - effects, - }); - const continuation = (reasonCode: string, effects: NativeStatusDecision["effects"] = []): NativeStatusDecision => ({ - policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, - statusAction: "in_progress", - toStatus: "in_progress", - reasonCode, - unblockDescriptor: null, - effects: [{ - kind: "enqueue_continuation", - continuationKind: "same_agent", - summary: `Continue native reconciliation after ${reasonCode}.`, - idempotencyKey: `native-reconciliation:${reasonCode}`, - agentId: input.agentId, - }, ...effects], - }); - - if (input.facts.equivalentAttentionFamily) { - return preserve("attention_duplicate_suppressed", [{ kind: "link_canonical_request" }]); - } - - if (input.facts.callerMaterialConflict) { - return preserve("structured_result_replay_conflict", [{ - kind: "record_finalization_error", - cause: "structured_result_replay_conflict", - nextAction: "Use a fresh canonical result identity.", - agentId: input.agentId, - }]); - } - if (input.facts.authoritativeStatusChanged) { - return preserve("prior_status_terminal_preserved", [{ kind: "append_superseding_assessment" }]); - } - if (input.facts.statusVersionAdvanced) { - return preserve("arbitration_conflict_reloaded", [ - { kind: "increment_status_version" }, - { kind: "schedule_reconciliation" }, - ]); - } - if (input.facts.newEvidenceSatisfiesContract) { - if (input.facts.hasPendingChildCompletion) { - return { - policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, - statusAction: "in_progress", - toStatus: "in_progress", - reasonCode: "native_child_completion_pending", - unblockDescriptor: null, - // The existing child-result wake owns the continuation. - effects: [{ kind: "release_checkout" }], - }; - } - return { - policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, - statusAction: "done", - toStatus: "done", - reasonCode: "decision_superseded_by_new_evidence", - unblockDescriptor: null, - effects: [{ kind: "release_checkout" }], - }; - } - if (input.facts.dependencyResolved) return continuation("dependency_resolved"); - if (input.facts.authorizedResume) return continuation("authorized_resume"); - if ((input.facts.undeliveredEffectCount ?? 0) > 0) { - return continuation("live_continuation_registered", [{ kind: "dispatch_pending_effect" }]); - } - if (input.facts.workspaceOperationPending) { - return { - policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, - statusAction: "done", - toStatus: "done", - reasonCode: "completion_contract_satisfied", - unblockDescriptor: null, - effects: [{ kind: "resume_workspace_operation" }, { kind: "release_checkout" }], - }; - } - if (input.facts.canonicalReplay) { - return { - policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, - statusAction: "done", - toStatus: "done", - reasonCode: "completion_contract_satisfied", - unblockDescriptor: null, - effects: [{ kind: "release_checkout" }], - }; - } - throw new Error("native_reconciliation_facts_invalid"); -} - export type NativeSessionResumeClaim = { runId: string; leaseOwner: string }; type NativeCleanupOutcome = { @@ -646,6 +523,7 @@ type NativeReconciliationOptions = { skipNativePendingEffects?: boolean; skipNativeResponsePresentation?: boolean; skipNativeReviewResponse?: boolean; + skipNativeStatusReassessment?: boolean; pluginWorkerManager?: WorkspaceWorkers; environmentRuntime?: EnvironmentRuntimeService; onWorkspaceSettled?: (input: { @@ -830,39 +708,12 @@ async function reconcileNativeFinalizationsPass( continue; } if (row.assessmentId) { - const assessment = await db.select({ - id: workAssessments.id, - contractId: workAssessments.contractId, - resultId: workAssessments.resultId, - policyVersion: workAssessments.policyVersion, - priorIssueStatus: workAssessments.priorIssueStatus, - priorStatusVersion: workAssessments.priorStatusVersion, - priorDecisionId: workAssessments.priorDecisionId, - assessmentJson: workAssessments.assessmentJson, - createdAt: workAssessments.createdAt, - }).from(workAssessments).where(and( - eq(workAssessments.id, row.assessmentId), - eq(workAssessments.companyId, row.companyId), - eq(workAssessments.issueId, row.issueId), - )).limit(1).then((entries) => entries[0] ?? null); - const changedEvidence = assessment - ? await db.select({ id: issueWorkProducts.id }).from(issueWorkProducts).where(and( - eq(issueWorkProducts.companyId, row.companyId), - eq(issueWorkProducts.issueId, row.issueId), - or( - gt(issueWorkProducts.createdAt, assessment.createdAt), - gt(issueWorkProducts.updatedAt, assessment.createdAt), - ), - )).limit(1).then((entries) => entries[0] ?? null) - : null; - const currentDecision = row.decisionId + const currentDecision = !options.skipNativeReviewResponse && row.coordinatorPhase === "committed" && row.decisionId ? await db.select().from(statusDecisions).where(and( - eq(statusDecisions.id, row.decisionId), - eq(statusDecisions.companyId, row.companyId), + eq(statusDecisions.id, row.decisionId), eq(statusDecisions.companyId, row.companyId), eq(statusDecisions.issueId, row.issueId), - )).limit(1).then((entries) => entries[0] ?? null) + )).limit(1).then(entries => entries[0] ?? null) : null; - const currentDecisionJson = record(currentDecision?.decisionJson); if (row.coordinatorPhase === "committed" && !options.skipNativeResponsePresentation) { // A later decision can supersede this run's task status, but cannot // erase its accepted, still-authorized response. Repair presentation @@ -872,12 +723,12 @@ async function reconcileNativeFinalizationsPass( }); } if ( + !options.skipNativeReviewResponse && row.coordinatorPhase === "committed" && row.decisionId && row.resultId && row.assessmentId && - currentDecisionJson.externalChatReviewPresentation && - !options.skipNativeReviewResponse + record(currentDecision?.decisionJson).externalChatReviewPresentation ) { // A later chat turn may retain the same pending review. Recover the // earlier response independently before status reconciliation skips @@ -891,168 +742,15 @@ async function reconcileNativeFinalizationsPass( assessmentId: row.assessmentId, }); } - if (row.coordinatorPhase === "committed" && row.decisionId && !currentDecision) { - throw new Error("native_committed_decision_missing"); - } - const supersededCommittedDecision = row.coordinatorPhase === "committed" - && row.decisionId !== null - && row.issueDecisionId !== null - && row.issueDecisionId !== row.decisionId - && !( - currentDecisionJson.statusAction === "preserve" - && assessment?.priorDecisionId === row.issueDecisionId - ); - if (supersededCommittedDecision) continue; - const persistedProjectedVersion = Number(currentDecisionJson.projectedStatusVersion); - const expectedProjectedVersion = currentDecision - ? Number.isFinite(persistedProjectedVersion) - ? persistedProjectedVersion - : Number(currentDecision.decisionVersion) - (currentDecisionJson.statusAction === "preserve" ? 1 : 0) - : null; - const issueMatchesCurrentDecision = !!assessment - && !!currentDecision - && row.issueDecisionId === row.decisionId - && currentDecision.assessmentId === assessment.id - && currentDecision.toStatus === row.issueStatus - && expectedProjectedVersion === Number(row.issueStatusVersion); - const authoritativeStatusChanged = !!assessment && !issueMatchesCurrentDecision && ( - assessment.priorIssueStatus !== row.issueStatus - || Number(assessment.priorStatusVersion) !== Number(row.issueStatusVersion) - ); - const retiredAutomaticReview = issueMatchesCurrentDecision && currentDecision - ? await decisionHasRetiredAutomaticReview(db, currentDecision) : false; - let reassessment = null; - let resultRow = null; - let contractRow = null; - if (assessment && (authoritativeStatusChanged || changedEvidence || retiredAutomaticReview)) { - [resultRow, contractRow] = await Promise.all([ - db.select().from(nativeRunResults).where(and( - eq(nativeRunResults.id, assessment.resultId), - eq(nativeRunResults.companyId, row.companyId), - eq(nativeRunResults.issueId, row.issueId), - eq(nativeRunResults.runId, row.runId), - )).limit(1).then((entries) => entries[0] ?? null), - db.select().from(completionContracts).where(and( - eq(completionContracts.id, assessment.contractId), - eq(completionContracts.companyId, row.companyId), - eq(completionContracts.issueId, row.issueId), - )).limit(1).then((entries) => entries[0] ?? null), - ]); - if (!resultRow || !contractRow) throw new Error("native_reconciliation_evidence_binding_missing"); - reassessment = await classifyNativeEvidence({ - db, - companyId: row.companyId, - issueId: row.issueId, - runId: row.runId, - contract: record(contractRow.contractJson), - result: record(record(resultRow.resultJson).result), - }); - } - const previousAssessment = record(assessment?.assessmentJson); - const newEvidenceSatisfiesContract = !!changedEvidence - && previousAssessment.allCriteriaSatisfied !== true - && reassessment?.allCriteriaSatisfied === true - && reassessment.verificationPassed === true; - const facts: NativeReconciliationFacts = authoritativeStatusChanged - ? { authoritativeStatusChanged: true } - : newEvidenceSatisfiesContract - ? { newEvidenceSatisfiesContract: true } - : {}; - if (Object.keys(facts).length > 0 || retiredAutomaticReview) { - if (!assessment || !reassessment || !resultRow || !contractRow) { - throw new Error("native_reconciliation_reassessment_missing"); + const source = { ...row, assessmentId: row.assessmentId }; + if (options.skipNativeStatusReassessment) { + if (await nativeStatusReassessmentSourceIsSuperseded(db, source)) continue; + } else { + const reassessed = await reassessNativeStatusFromSource(db, source); + if (reassessed.kind !== "unchanged") { + if (reassessed.kind === "committed") results.push(reassessed.result); + continue; } - const currentIssue = retiredAutomaticReview - ? await db.select().from(issues).where(and(eq(issues.id, row.issueId), eq(issues.companyId, row.companyId))).then((entries) => entries[0]) - : null; - const readiness = retiredAutomaticReview ? await issueService(db).getDependencyReadiness(row.issueId, db) : null; - const currentRun = retiredAutomaticReview ? await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, row.runId)).then((entries) => entries[0]) : null; - // Never replay an old result over a new run or a later task contract. - const latestContract = retiredAutomaticReview ? await db.select({ id: completionContracts.id }).from(completionContracts) - .where(and(eq(completionContracts.companyId, row.companyId), eq(completionContracts.issueId, row.issueId))) - .orderBy(desc(completionContracts.revision)).limit(1).then((entries) => entries[0]) : null; - if (retiredAutomaticReview && (!currentIssue || currentRun?.status !== "succeeded" - || latestContract?.id !== contractRow.id - || (currentIssue.executionRunId && currentIssue.executionRunId !== row.runId))) continue; - // A committed decision proves the original barrier passed. If a later - // workspace operation exists, do not ignore a pending or failed retry. - const reviewBarrier = retiredAutomaticReview ? await db.select({ status: workspaceOperations.status }) - .from(workspaceOperations).where(and(eq(workspaceOperations.companyId, row.companyId), - eq(workspaceOperations.heartbeatRunId, row.runId), eq(workspaceOperations.phase, "workspace_finalize"))) - .orderBy(desc(workspaceOperations.createdAt)).limit(1).then((entries) => entries[0]) : null; - if (reviewBarrier && reviewBarrier.status !== "succeeded") continue; - const childCompletionRecipient = { - companyId: row.companyId, issueId: row.issueId, agentId: row.agentId, runId: row.runId, - sourceIntentId: typeof row.nativeStatusWakeIntentId === "string" ? row.nativeStatusWakeIntentId : null, - }; - const hasPendingChildCompletion = await hasPendingNativeChildCompletion(db, childCompletionRecipient); - const reassessmentRow = await recordNativeWorkAssessment({ - db, - companyId: row.companyId, - issueId: row.issueId, - runId: row.runId, - turnId: resultRow.turnId, - contractId: contractRow.id, - contractCanonicalSha256: contractRow.canonicalSha256, - resultId: resultRow.id, - resultCanonicalSha256: resultRow.canonicalSha256, - priorIssueStatus: row.issueStatus, - priorStatusVersion: Number(row.issueStatusVersion), - priorDecisionId: row.issueDecisionId, - policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, - assessment: reassessment, - supersedesAssessmentId: assessment.id, - }); - const qualityReview = currentIssue ? await readRequiredQualityReview(db, currentIssue) : null; - let decision = retiredAutomaticReview && currentIssue - ? resolveNativeFinalizerStatus({ - hasPendingChildCompletion, - requiredQualityReviewSatisfied: qualityReview?.satisfied, - assessment: reassessment, terminalState: "succeeded", workspaceFinalizeStatus: "succeeded", - governanceGate: await pendingNativeGovernance({ db, companyId: row.companyId, issueId: row.issueId, - runId: row.runId, executionState: record(currentIssue.executionState) }), - completionClaimPolicyAccepted: contractRow.risk === "low" && contractRow.completionAuthority === "agent_claim_policy", - hasUnresolvedIssueBlockers: (readiness?.unresolvedBlockerCount ?? 0) > 0, - reviewOwnerUserId: currentIssue.responsibleUserId ?? currentIssue.createdByUserId, - priorIssueStatus: row.issueStatus as NativeAuthoritativeIssueStatus, agentId: row.agentId, - }) - : resolveNativeReconciliationStatus({ - facts: { ...facts, hasPendingChildCompletion }, - priorIssueStatus: row.issueStatus as NativeAuthoritativeIssueStatus, agentId: row.agentId, - }); - if (decision.statusAction === "done" && qualityReview?.satisfied === false) { - decision = resolveNativeFinalizerStatus({ assessment: reassessment, terminalState: "succeeded", workspaceFinalizeStatus: "succeeded", - requiredQualityReviewSatisfied: false, priorIssueStatus: row.issueStatus as NativeAuthoritativeIssueStatus, agentId: row.agentId }); - } - let committed: Awaited>; - try { - committed = await commitNativeStatusDecision({ - db, - companyId: row.companyId, - issueId: row.issueId, - runId: row.runId, - assessmentId: reassessmentRow.id, - priorStatus: row.issueStatus, - priorStatusVersion: Number(row.issueStatusVersion), - priorDecisionId: row.issueDecisionId, - decision, - requireNoPendingChildCompletion: decision.statusAction === "done" ? childCompletionRecipient : undefined, - supersedesCommittedDecisionId: row.coordinatorPhase === "committed" - ? row.decisionId ?? undefined - : undefined, - }); - } catch (error) { - if (error instanceof NativeStatusRaceError) continue; - throw error; - } - results.push({ - runId: row.runId, - phase: "committed", - reconciliationAction: decision.reasonCode, - decisionId: committed.decision.id, - reconciliationDecision: decision, - }); - continue; } } const barrier = await db.select({ status: workspaceOperations.status }) diff --git a/server/src/services/native-runtime/native-governance.ts b/server/src/services/native-runtime/native-governance.ts index 2b71ffcf24f..dd8f2a27835 100644 --- a/server/src/services/native-runtime/native-governance.ts +++ b/server/src/services/native-runtime/native-governance.ts @@ -1,5 +1,5 @@ import { and, eq, inArray, notInArray } from "drizzle-orm"; -import { approvals, issueApprovals, issueThreadInteractions, issues, type Db } from "@paperclipai/db"; +import { approvals, issueApprovals, issueComments, issueThreadInteractions, issues, type Db } from "@paperclipai/db"; import { activeIssueInteractionCondition } from "../issue-question-context.js"; import { isConversation } from "../agent-conversations.js"; import type { NativeGovernanceGate } from "./status-arbiter.js"; @@ -8,6 +8,22 @@ function record(value: unknown): Record { return typeof value === "object" && value !== null ? value as Record : {}; } + +export async function lockNativeGovernanceSources(input: { db: Db; companyId: string; issueId: string }): Promise { + const interactions = await input.db.select({ kind: issueThreadInteractions.kind }).from(issueThreadInteractions) + .where(and(eq(issueThreadInteractions.companyId, input.companyId), eq(issueThreadInteractions.issueId, input.issueId))) + .for("share", { noWait: true }); + if (interactions.some(interaction => interaction.kind === "ask_user_questions")) { + await input.db.select({ id: issueComments.id }).from(issueComments) + .where(and(eq(issueComments.companyId, input.companyId), eq(issueComments.issueId, input.issueId))) + .for("share", { noWait: true }); + } + await input.db.select({ id: approvals.id }).from(issueApprovals) + .innerJoin(approvals, eq(issueApprovals.approvalId, approvals.id)) + .where(and(eq(issueApprovals.companyId, input.companyId), eq(issueApprovals.issueId, input.issueId), + eq(approvals.companyId, input.companyId))).for("share", { noWait: true }); +} + export async function pendingNativeGovernance(input: { db: Db; companyId: string; diff --git a/server/src/services/native-runtime/native-reconciliation-status.ts b/server/src/services/native-runtime/native-reconciliation-status.ts new file mode 100644 index 00000000000..dc9edc794d8 --- /dev/null +++ b/server/src/services/native-runtime/native-reconciliation-status.ts @@ -0,0 +1,113 @@ +import { NATIVE_STATUS_ARBITER_POLICY_VERSION, type NativeAuthoritativeIssueStatus, type NativeStatusDecision } from "./status-arbiter.js"; + +export type NativeReconciliationFacts = { + equivalentAttentionFamily?: boolean; + canonicalReplay?: boolean; + callerMaterialConflict?: boolean; + workspaceOperationPending?: boolean; + undeliveredEffectCount?: number; + authoritativeStatusChanged?: boolean; + newEvidenceSatisfiesContract?: boolean; + hasPendingChildCompletion?: boolean; + dependencyResolved?: boolean; + authorizedResume?: boolean; + statusVersionAdvanced?: boolean; +}; + + +export function resolveNativeReconciliationStatus(input: { + facts: NativeReconciliationFacts; + priorIssueStatus: NativeAuthoritativeIssueStatus; + agentId: string; +}): NativeStatusDecision { + const preserve = (reasonCode: string, effects: NativeStatusDecision["effects"]): NativeStatusDecision => ({ + policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, + statusAction: "preserve", + toStatus: input.priorIssueStatus, + reasonCode, + unblockDescriptor: null, + effects, + }); + const continuation = (reasonCode: string, effects: NativeStatusDecision["effects"] = []): NativeStatusDecision => ({ + policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, + statusAction: "in_progress", + toStatus: "in_progress", + reasonCode, + unblockDescriptor: null, + effects: [{ + kind: "enqueue_continuation", + continuationKind: "same_agent", + summary: `Continue native reconciliation after ${reasonCode}.`, + idempotencyKey: `native-reconciliation:${reasonCode}`, + agentId: input.agentId, + }, ...effects], + }); + + if (input.facts.equivalentAttentionFamily) { + return preserve("attention_duplicate_suppressed", [{ kind: "link_canonical_request" }]); + } + + if (input.facts.callerMaterialConflict) { + return preserve("structured_result_replay_conflict", [{ + kind: "record_finalization_error", + cause: "structured_result_replay_conflict", + nextAction: "Use a fresh canonical result identity.", + agentId: input.agentId, + }]); + } + if (input.facts.authoritativeStatusChanged) { + return preserve("prior_status_terminal_preserved", [{ kind: "append_superseding_assessment" }]); + } + if (input.facts.statusVersionAdvanced) { + return preserve("arbitration_conflict_reloaded", [ + { kind: "increment_status_version" }, + { kind: "schedule_reconciliation" }, + ]); + } + if (input.facts.newEvidenceSatisfiesContract) { + if (input.facts.hasPendingChildCompletion) { + return { + policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, + statusAction: "in_progress", + toStatus: "in_progress", + reasonCode: "native_child_completion_pending", + unblockDescriptor: null, + effects: [{ kind: "release_checkout" }], + }; + } + return { + policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, + statusAction: "done", + toStatus: "done", + reasonCode: "decision_superseded_by_new_evidence", + unblockDescriptor: null, + effects: [{ kind: "release_checkout" }], + }; + } + if (input.facts.dependencyResolved) return continuation("dependency_resolved"); + if (input.facts.authorizedResume) return continuation("authorized_resume"); + if ((input.facts.undeliveredEffectCount ?? 0) > 0) { + return continuation("live_continuation_registered", [{ kind: "dispatch_pending_effect" }]); + } + if (input.facts.workspaceOperationPending) { + return { + policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, + statusAction: "done", + toStatus: "done", + reasonCode: "completion_contract_satisfied", + unblockDescriptor: null, + effects: [{ kind: "resume_workspace_operation" }, { kind: "release_checkout" }], + }; + } + if (input.facts.canonicalReplay) { + return { + policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, + statusAction: "done", + toStatus: "done", + reasonCode: "completion_contract_satisfied", + unblockDescriptor: null, + effects: [{ kind: "release_checkout" }], + }; + } + throw new Error("native_reconciliation_facts_invalid"); +} diff --git a/server/src/services/native-runtime/native-restart-recovery.ts b/server/src/services/native-runtime/native-restart-recovery.ts index 0b7c8bf41ed..82a202c5ff7 100644 --- a/server/src/services/native-runtime/native-restart-recovery.ts +++ b/server/src/services/native-runtime/native-restart-recovery.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js"; import { notifyUnsafeWorkspaceExportRunChange } from "../unsafe-workspace-export-notifications.js"; import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js"; import { notifyInstructionCollection } from "../instruction-collection-notifications.js"; @@ -942,6 +943,7 @@ export async function claimNativeRestartRecoveries(input: { if (updatedRun && isNewTerminalRunFailure(row.run, updatedRun)) { terminalRunToReport = updatedRun; } + await notifyNativeStatusReassessment(tx, { companyId: row.run.companyId, issueId: row.coordinator.issueId }); await tx .update(issues) .set({ executionRunId: null, updatedAt: now }) diff --git a/server/src/services/native-runtime/native-safe-replacement.ts b/server/src/services/native-runtime/native-safe-replacement.ts index 78b889b169a..7fb789f3304 100644 --- a/server/src/services/native-runtime/native-safe-replacement.ts +++ b/server/src/services/native-runtime/native-safe-replacement.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js"; import { notifyExecutionRunWork } from "../execution-control-notifications.js"; import { logger } from "../../middleware/logger.js"; @@ -487,6 +488,7 @@ export async function reconcileSafeNativeReplacements( .update(heartbeatRuns) .set({ executionStatusDeliveryId: randomUUID() }) .where(eq(heartbeatRuns.id, run.id)); + await notifyNativeStatusReassessment(tx, { companyId: run.companyId, issueId: task.id }); // The successor gets task authority only through the existing admission gate. await tx .update(issues) diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index 4e73bfcfa45..fe662bd7e09 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js"; import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js"; import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js"; @@ -9882,6 +9883,7 @@ async function executePaperclipNativeSessionWithinScope( stillOwnsTask && phase === "terminal_failure" ) { + await notifyNativeStatusReassessment(tx, { companyId: input.execution.binding.companyId, issueId: input.execution.binding.issueId }); await tx .update(issues) .set({ executionRunId: null, checkoutRunId: null, updatedAt: now }) diff --git a/server/src/services/native-runtime/native-status-reassessment-work.ts b/server/src/services/native-runtime/native-status-reassessment-work.ts new file mode 100644 index 00000000000..13611a91de4 --- /dev/null +++ b/server/src/services/native-runtime/native-status-reassessment-work.ts @@ -0,0 +1,117 @@ +import { and, asc, eq, exists, gt, inArray, isNotNull, isNull, ne, not, notExists, or, sql } from "drizzle-orm"; +import { heartbeatRuns, issues, issueThreadInteractions, issueWorkProducts, nativeRunFinalizations, + statusDecisionEffects, statusDecisions, workAssessments, type Db } from "@paperclipai/db"; +import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js"; +import { DELIVERY_QUEUES } from "../delivery-work-notifications.js"; +import type { NativeStatusReassessmentScope } from "../native-status-reassessment-notifications.js"; +import { registerScopedDeadlineWork } from "../scoped-deadline-work.js"; +import { nativeRunnerOwnershipNotHeldCondition } from "./native-runner-ownership.js"; +import { reassessNativeStatusFromSource } from "./native-status-reassessment.js"; + +type Entry = { id: string; companyId: string; issueId: string; agentId: string; dueAt: number }; + +export function nativeStatusReassessmentWork(db: Db) { + function candidates(scope?: NativeStatusReassessmentScope, after?: string, runId?: string) { + const projectedVersion = sql`case + when jsonb_typeof(${statusDecisions.decisionJson}->'projectedStatusVersion') = 'number' + then (${statusDecisions.decisionJson}->>'projectedStatusVersion')::numeric + else ${statusDecisions.decisionVersion} - case when ${statusDecisions.decisionJson}->>'statusAction' = 'preserve' then 1 else 0 end + end`; + const matchesDecision = sql`coalesce(${issues.lastStatusDecisionId} = ${nativeRunFinalizations.decisionId} + and ${statusDecisions.assessmentId} = ${workAssessments.id} and ${statusDecisions.toStatus} = ${issues.status} + and ${projectedVersion} = ${issues.statusVersion}, false)`; + const statusChanged = and(not(matchesDecision), or(ne(workAssessments.priorIssueStatus, issues.status), + ne(workAssessments.priorStatusVersion, issues.statusVersion))); + const evidenceChanged = exists(db.select({ id: issueWorkProducts.id }).from(issueWorkProducts).where(and( + eq(issueWorkProducts.companyId, nativeRunFinalizations.companyId), eq(issueWorkProducts.issueId, nativeRunFinalizations.issueId), + or(gt(issueWorkProducts.createdAt, workAssessments.createdAt), gt(issueWorkProducts.updatedAt, workAssessments.createdAt)), + ))); + const retiredReview = exists(db.select({ id: issueThreadInteractions.id }).from(issueThreadInteractions).where(and( + eq(issueThreadInteractions.companyId, nativeRunFinalizations.companyId), eq(issueThreadInteractions.issueId, nativeRunFinalizations.issueId), + eq(issueThreadInteractions.status, "cancelled"), sql`${issueThreadInteractions.result}->>'reason' = 'automatic_completion_review_removed'`, + sql`(${issueThreadInteractions.payload}->'target'->>'revisionId' = ${statusDecisions.id}::text + or ${statusDecisions.decisionJson}->'effects' @> jsonb_build_array(jsonb_build_object('gate', + jsonb_build_object('kind', 'interaction', 'id', ${issueThreadInteractions.id}::text))))`, + ))); + return db.select({ runId: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + issueId: issues.id, issueStatus: issues.status, issueStatusVersion: issues.statusVersion, + issueDecisionId: issues.lastStatusDecisionId, coordinatorPhase: nativeRunFinalizations.phase, + assessmentId: workAssessments.id, decisionId: nativeRunFinalizations.decisionId, nativeStatusWakeIntentId: sql`${heartbeatRuns.contextSnapshot}->'nativeStatusWakeIntentId'`, + nextAttemptAt: nativeRunFinalizations.nextAttemptAt, leaseOwner: nativeRunFinalizations.leaseOwner, + leaseExpiresAt: nativeRunFinalizations.leaseExpiresAt, + }).from(nativeRunFinalizations) + .innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, nativeRunFinalizations.runId), + eq(heartbeatRuns.companyId, nativeRunFinalizations.companyId), eq(heartbeatRuns.nativeIssueId, nativeRunFinalizations.issueId))) + .innerJoin(issues, and(eq(issues.id, nativeRunFinalizations.issueId), eq(issues.companyId, nativeRunFinalizations.companyId))) + .innerJoin(workAssessments, and(eq(workAssessments.id, nativeRunFinalizations.assessmentId), + eq(workAssessments.companyId, nativeRunFinalizations.companyId), eq(workAssessments.issueId, nativeRunFinalizations.issueId), + eq(workAssessments.runId, nativeRunFinalizations.runId))) + .leftJoin(statusDecisions, and(eq(statusDecisions.id, nativeRunFinalizations.decisionId), + eq(statusDecisions.companyId, nativeRunFinalizations.companyId), eq(statusDecisions.issueId, nativeRunFinalizations.issueId))) + .where(and(eq(heartbeatRuns.runtimeMode, "native"), nativeRunnerOwnershipNotHeldCondition(), + ne(nativeRunFinalizations.phase, "terminal_failure"), isNotNull(nativeRunFinalizations.resultId), + or(isNull(nativeRunFinalizations.leaseOwner), isNotNull(nativeRunFinalizations.leaseExpiresAt)), + sql`not coalesce(${nativeRunFinalizations.phase} = 'committed' + and ${nativeRunFinalizations.decisionId} is not null and ${issues.lastStatusDecisionId} is not null + and ${issues.lastStatusDecisionId} <> ${nativeRunFinalizations.decisionId} + and not coalesce(${statusDecisions.decisionJson}->>'statusAction' = 'preserve' + and ${workAssessments.priorDecisionId} = ${issues.lastStatusDecisionId}, false), false)`, + or(statusChanged, evidenceChanged, and(matchesDecision, retiredReview)), + notExists(db.select({ id: statusDecisionEffects.id }).from(statusDecisionEffects).where(and( + eq(statusDecisionEffects.companyId, nativeRunFinalizations.companyId), eq(statusDecisionEffects.issueId, nativeRunFinalizations.issueId), + eq(statusDecisionEffects.decisionId, nativeRunFinalizations.decisionId), inArray(statusDecisionEffects.deliveryState, ["pending", "failed"]), + ))), + scope ? eq(nativeRunFinalizations.companyId, scope.companyId) : undefined, + scope?.issueId ? eq(nativeRunFinalizations.issueId, scope.issueId) : undefined, + scope?.runId ? eq(nativeRunFinalizations.runId, scope.runId) : undefined, + scope?.agentId ? eq(heartbeatRuns.agentId, scope.agentId) : undefined, + after ? gt(nativeRunFinalizations.runId, after) : undefined, + runId ? eq(nativeRunFinalizations.runId, runId) : undefined, + )).orderBy(asc(nativeRunFinalizations.runId)).limit(100); + } + function entry(row: Awaited>[number]): Entry { + return { id: row.runId, companyId: row.companyId, issueId: row.issueId, agentId: row.agentId, + dueAt: Math.max(row.nextAttemptAt?.getTime() ?? 0, row.leaseOwner !== null ? row.leaseExpiresAt!.getTime() : 0) }; + } + async function load(scope?: NativeStatusReassessmentScope, signal?: AbortSignal, canRun = () => true): Promise { + const entries: Entry[] = []; + let after: string | undefined; + while (canRun() && !signal?.aborted) { + const rows = await candidates(scope, after); + entries.push(...rows.map(entry)); + if (rows.length < 100) return entries; + after = rows[rows.length - 1].runId; + } + throw new Error("Native status reassessment discovery stopped"); + } + async function check(current: Entry, signal?: AbortSignal): Promise { + if (signal?.aborted) throw new Error("Native status reassessment stopped"); + const scope = { companyId: current.companyId, issueId: current.issueId, runId: current.id }; + const [source] = await candidates(scope); + if (!source) return null; + const due = entry(source); + if (due.dueAt > Date.now()) return due; + if (signal?.aborted) throw new Error("Native status reassessment stopped"); + const outcome = await reassessNativeStatusFromSource(db, source, { onlyWhenDue: true }); + if (outcome.kind !== "deferred" || !outcome.retry) return null; + const [remaining] = await candidates(scope); + if (!remaining) return null; + const retry = entry(remaining); + return retry.dueAt > Date.now() ? retry : { ...retry, dueAt: Date.now() + 30_000 }; + } + function register(coordinator: ReturnType, canRun: () => boolean) { + return registerScopedDeadlineWork(coordinator, { + queue: DELIVERY_QUEUES.nativeStatusReassessment, + decodeScope: (hint: string): NativeStatusReassessmentScope | undefined => hint === "*" ? undefined : JSON.parse(hint), + matches: (item: Entry, scope?: NativeStatusReassessmentScope) => !scope || item.companyId === scope.companyId + && (!scope.issueId || item.issueId === scope.issueId) && (!scope.runId || item.id === scope.runId) + && (!scope.agentId || item.agentId === scope.agentId), + load: (scope, signal) => load(scope, signal, canRun), + check: (item, signal) => { + if (!canRun()) throw new Error("Native status reassessment paused"); + return check(item, signal); + }, + }, canRun); + } + return { load, check, register }; +} diff --git a/server/src/services/native-runtime/native-status-reassessment.ts b/server/src/services/native-runtime/native-status-reassessment.ts new file mode 100644 index 00000000000..41bc23f45a7 --- /dev/null +++ b/server/src/services/native-runtime/native-status-reassessment.ts @@ -0,0 +1,266 @@ +import { readRequiredQualityReview } from "../required-quality-review.js"; +import { hasNativeWorkspaceFinalizationOwner } from "./native-workspace-finalization-recovery-state.js"; +import { and, desc, eq, gt, or } from "drizzle-orm"; +import { completionContracts, heartbeatRuns, issueWorkProducts, issues, nativeRunResults, + statusDecisions, workAssessments, workspaceOperations, type Db } from "@paperclipai/db"; +import { issueService } from "../issues.js"; +import { decisionHasRetiredAutomaticReview } from "./automatic-completion-reviews.js"; +import { classifyNativeEvidence } from "./evidence-classifier.js"; +import { hasPendingNativeChildCompletion } from "./native-child-completion-delivery.js"; +import { resolveNativeReconciliationStatus, type NativeReconciliationFacts } from "./native-reconciliation-status.js"; +import { pendingNativeGovernance, resolveNativeFinalizerStatus } from "./native-run-finalizer.js"; +import { NATIVE_STATUS_ARBITER_POLICY_VERSION, type NativeAuthoritativeIssueStatus, type NativeStatusDecision } from "./status-arbiter.js"; +import { commitNativeStatusDecision, NativeStatusRaceError } from "./status-decision-committer.js"; +import { recordNativeWorkAssessment } from "./work-assessments.js"; + +export type NativeStatusReassessmentSource = { + runId: string; companyId: string; agentId: string; issueId: string; + issueStatus: string; issueStatusVersion: number; issueDecisionId: string | null; + coordinatorPhase: string; assessmentId: string; decisionId: string | null; + nativeStatusWakeIntentId: unknown; +}; + +type Outcome = { kind: "unchanged" | "superseded" } | { kind: "deferred"; retry?: boolean } | { + kind: "committed"; + result: { runId: string; phase: "committed"; reconciliationAction: string | null; + decisionId: string; reconciliationDecision: NativeStatusDecision }; +}; + +function record(value: unknown): Record { + return value && typeof value === "object" && !Array.isArray(value) + ? value as Record + : {}; +} + +function isSupersededSource( + row: NativeStatusReassessmentSource, + decision: Record, + priorDecisionId: string | null | undefined, +): boolean { + return row.coordinatorPhase === "committed" + && row.decisionId !== null + && row.issueDecisionId !== null + && row.issueDecisionId !== row.decisionId + && !( + decision.statusAction === "preserve" + && priorDecisionId === row.issueDecisionId + ); +} + + + +export async function nativeStatusReassessmentSourceIsSuperseded( + db: Db, + row: NativeStatusReassessmentSource, +): Promise { + if (row.coordinatorPhase !== "committed" || row.decisionId === null + || row.issueDecisionId === null || row.issueDecisionId === row.decisionId) return false; + const [decision] = await db.select({ decisionJson: statusDecisions.decisionJson }) + .from(statusDecisions).where(and(eq(statusDecisions.id, row.decisionId), + eq(statusDecisions.companyId, row.companyId), eq(statusDecisions.issueId, row.issueId))).limit(1); + if (!decision) throw new Error("native_committed_decision_missing"); + const [assessment] = await db.select({ priorDecisionId: workAssessments.priorDecisionId }) + .from(workAssessments).where(and(eq(workAssessments.id, row.assessmentId), + eq(workAssessments.companyId, row.companyId), eq(workAssessments.issueId, row.issueId))).limit(1); + return isSupersededSource(row, record(decision.decisionJson), assessment?.priorDecisionId); +} + +export async function reassessNativeStatusFromSource( + db: Db, + row: NativeStatusReassessmentSource, + options: { onlyWhenDue?: boolean } = {}, +): Promise { + const assessment = await db.select({ + id: workAssessments.id, + contractId: workAssessments.contractId, + resultId: workAssessments.resultId, + policyVersion: workAssessments.policyVersion, + priorIssueStatus: workAssessments.priorIssueStatus, + priorStatusVersion: workAssessments.priorStatusVersion, + priorDecisionId: workAssessments.priorDecisionId, + assessmentJson: workAssessments.assessmentJson, + createdAt: workAssessments.createdAt, + }).from(workAssessments).where(and( + eq(workAssessments.id, row.assessmentId), + eq(workAssessments.companyId, row.companyId), + eq(workAssessments.issueId, row.issueId), + )).limit(1).then((entries) => entries[0] ?? null); + const changedEvidence = assessment + ? await db.select({ id: issueWorkProducts.id }).from(issueWorkProducts).where(and( + eq(issueWorkProducts.companyId, row.companyId), + eq(issueWorkProducts.issueId, row.issueId), + or( + gt(issueWorkProducts.createdAt, assessment.createdAt), + gt(issueWorkProducts.updatedAt, assessment.createdAt), + ), + )).limit(1).then((entries) => entries[0] ?? null) + : null; + const currentDecision = row.decisionId + ? await db.select().from(statusDecisions).where(and( + eq(statusDecisions.id, row.decisionId), + eq(statusDecisions.companyId, row.companyId), + eq(statusDecisions.issueId, row.issueId), + )).limit(1).then((entries) => entries[0] ?? null) + : null; + const currentDecisionJson = record(currentDecision?.decisionJson); + if (row.coordinatorPhase === "committed" && row.decisionId && !currentDecision) { + throw new Error("native_committed_decision_missing"); + } + if (isSupersededSource(row, currentDecisionJson, assessment?.priorDecisionId)) return { kind: "superseded" }; + const persistedProjectedVersion = Number(currentDecisionJson.projectedStatusVersion); + const expectedProjectedVersion = currentDecision + ? Number.isFinite(persistedProjectedVersion) + ? persistedProjectedVersion + : Number(currentDecision.decisionVersion) - (currentDecisionJson.statusAction === "preserve" ? 1 : 0) + : null; + const issueMatchesCurrentDecision = !!assessment + && !!currentDecision + && row.issueDecisionId === row.decisionId + && currentDecision.assessmentId === assessment.id + && currentDecision.toStatus === row.issueStatus + && expectedProjectedVersion === Number(row.issueStatusVersion); + const authoritativeStatusChanged = !!assessment && !issueMatchesCurrentDecision && ( + assessment.priorIssueStatus !== row.issueStatus + || Number(assessment.priorStatusVersion) !== Number(row.issueStatusVersion) + ); + const retiredAutomaticReview = issueMatchesCurrentDecision && currentDecision + ? await decisionHasRetiredAutomaticReview(db, currentDecision) : false; + let reassessment = null; + let resultRow = null; + let contractRow = null; + if (assessment && (authoritativeStatusChanged || changedEvidence || retiredAutomaticReview)) { + [resultRow, contractRow] = await Promise.all([ + db.select().from(nativeRunResults).where(and( + eq(nativeRunResults.id, assessment.resultId), + eq(nativeRunResults.companyId, row.companyId), + eq(nativeRunResults.issueId, row.issueId), + eq(nativeRunResults.runId, row.runId), + )).limit(1).then((entries) => entries[0] ?? null), + db.select().from(completionContracts).where(and( + eq(completionContracts.id, assessment.contractId), + eq(completionContracts.companyId, row.companyId), + eq(completionContracts.issueId, row.issueId), + )).limit(1).then((entries) => entries[0] ?? null), + ]); + if (!resultRow || !contractRow) throw new Error("native_reconciliation_evidence_binding_missing"); + reassessment = await classifyNativeEvidence({ + db, + companyId: row.companyId, + issueId: row.issueId, + runId: row.runId, + contract: record(contractRow.contractJson), + result: record(record(resultRow.resultJson).result), + }); + } + const previousAssessment = record(assessment?.assessmentJson); + const newEvidenceSatisfiesContract = !!changedEvidence + && previousAssessment.allCriteriaSatisfied !== true + && reassessment?.allCriteriaSatisfied === true + && reassessment.verificationPassed === true; + const facts: NativeReconciliationFacts = authoritativeStatusChanged + ? { authoritativeStatusChanged: true } + : newEvidenceSatisfiesContract + ? { newEvidenceSatisfiesContract: true } + : {}; + if (Object.keys(facts).length > 0 || retiredAutomaticReview) { + if (!assessment || !reassessment || !resultRow || !contractRow) { + throw new Error("native_reconciliation_reassessment_missing"); + } + const currentIssue = await db.select().from(issues) + .where(and(eq(issues.id, row.issueId), eq(issues.companyId, row.companyId))).then((entries) => entries[0]); + const readiness = retiredAutomaticReview ? await issueService(db).getDependencyReadiness(row.issueId, db) : null; + const currentRun = retiredAutomaticReview ? await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, row.runId)).then((entries) => entries[0]) : null; + const latestContract = retiredAutomaticReview ? await db.select({ id: completionContracts.id }).from(completionContracts) + .where(and(eq(completionContracts.companyId, row.companyId), eq(completionContracts.issueId, row.issueId))) + .orderBy(desc(completionContracts.revision)).limit(1).then((entries) => entries[0]) : null; + if (retiredAutomaticReview && (!currentIssue || currentRun?.status !== "succeeded" + || (options.onlyWhenDue && hasNativeWorkspaceFinalizationOwner(currentRun.runnerProfileJson)) + || latestContract?.id !== contractRow.id + || (currentIssue.executionRunId && currentIssue.executionRunId !== row.runId))) return { kind: "deferred" }; + const reviewBarrier = retiredAutomaticReview ? await db.select({ status: workspaceOperations.status }) + .from(workspaceOperations).where(and(eq(workspaceOperations.companyId, row.companyId), + eq(workspaceOperations.heartbeatRunId, row.runId), eq(workspaceOperations.phase, "workspace_finalize"))) + .orderBy(desc(workspaceOperations.createdAt)).limit(1).then((entries) => entries[0]) : null; + if (reviewBarrier && reviewBarrier.status !== "succeeded") return { kind: "deferred" }; + const childCompletionRecipient = { + companyId: row.companyId, issueId: row.issueId, agentId: row.agentId, runId: row.runId, + sourceIntentId: typeof row.nativeStatusWakeIntentId === "string" ? row.nativeStatusWakeIntentId : null, + }; + const hasPendingChildCompletion = await hasPendingNativeChildCompletion(db, childCompletionRecipient); + const reassessmentRow = await recordNativeWorkAssessment({ + db, + companyId: row.companyId, + issueId: row.issueId, + runId: row.runId, + turnId: resultRow.turnId, + contractId: contractRow.id, + contractCanonicalSha256: contractRow.canonicalSha256, + resultId: resultRow.id, + resultCanonicalSha256: resultRow.canonicalSha256, + priorIssueStatus: row.issueStatus, + priorStatusVersion: Number(row.issueStatusVersion), + priorDecisionId: row.issueDecisionId, + policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, + assessment: reassessment, + supersedesAssessmentId: assessment.id, + }); + const hasUnresolvedIssueBlockers = (readiness?.unresolvedBlockerCount ?? 0) > 0; + const governanceGate = retiredAutomaticReview && currentIssue + ? await pendingNativeGovernance({ db, companyId: row.companyId, issueId: row.issueId, + runId: row.runId, executionState: record(currentIssue.executionState) }) : null; + const qualityReview = currentIssue ? await readRequiredQualityReview(db, currentIssue) : null; + let decision = retiredAutomaticReview && currentIssue + ? resolveNativeFinalizerStatus({ + hasPendingChildCompletion, + requiredQualityReviewSatisfied: qualityReview?.satisfied, + assessment: reassessment, terminalState: "succeeded", workspaceFinalizeStatus: "succeeded", + governanceGate, + completionClaimPolicyAccepted: contractRow.risk === "low" && contractRow.completionAuthority === "agent_claim_policy", + hasUnresolvedIssueBlockers, + reviewOwnerUserId: currentIssue.responsibleUserId ?? currentIssue.createdByUserId, + priorIssueStatus: row.issueStatus as NativeAuthoritativeIssueStatus, agentId: row.agentId, + }) + : resolveNativeReconciliationStatus({ + facts: { ...facts, hasPendingChildCompletion }, + priorIssueStatus: row.issueStatus as NativeAuthoritativeIssueStatus, agentId: row.agentId, + }); + if (decision.statusAction === "done" && qualityReview?.satisfied === false) { + decision = resolveNativeFinalizerStatus({ assessment: reassessment, terminalState: "succeeded", workspaceFinalizeStatus: "succeeded", + requiredQualityReviewSatisfied: false, priorIssueStatus: row.issueStatus as NativeAuthoritativeIssueStatus, agentId: row.agentId }); + } + let committed: Awaited>; + try { + committed = await commitNativeStatusDecision({ + db, + companyId: row.companyId, + issueId: row.issueId, + runId: row.runId, + assessmentId: reassessmentRow.id, + priorStatus: row.issueStatus, + priorStatusVersion: Number(row.issueStatusVersion), + priorDecisionId: row.issueDecisionId, + decision, + requireReassessmentSource: options.onlyWhenDue ? { + phase: row.coordinatorPhase, resultId: resultRow.id, assessmentId: assessment.id, + decisionId: row.decisionId, agentId: row.agentId, retiredAutomaticReview: retiredAutomaticReview ? { contractId: contractRow.id, governanceGate, hasUnresolvedIssueBlockers } : null, + } : undefined, + requireNoPendingChildCompletion: decision.statusAction === "done" ? childCompletionRecipient : undefined, + supersedesCommittedDecisionId: row.coordinatorPhase === "committed" + ? row.decisionId ?? undefined + : undefined, + }); + } catch (error) { + if (error instanceof NativeStatusRaceError) return options.onlyWhenDue + ? { kind: "deferred", retry: true } : { kind: "deferred" }; + throw error; + } + return { kind: "committed", result: { + runId: row.runId, + phase: "committed", + reconciliationAction: decision.reasonCode, + decisionId: committed.decision.id, + reconciliationDecision: decision, + } }; + } + return { kind: "unchanged" }; +} diff --git a/server/src/services/native-runtime/native-workspace-export-recovery.ts b/server/src/services/native-runtime/native-workspace-export-recovery.ts index c638f3b3e94..41e51f2ddc7 100644 --- a/server/src/services/native-runtime/native-workspace-export-recovery.ts +++ b/server/src/services/native-runtime/native-workspace-export-recovery.ts @@ -95,13 +95,11 @@ async function recoverCandidate(db: Db, candidate: { runId: string; companyId: s let updatedRun: typeof heartbeatRuns.$inferSelect | undefined; if (eligible) { await ownership.assertHeld(); - - + await notifyNativeFinalizationWork(tx, { companyId: run.companyId, runId: run.id }); await tx.insert(workspaceOperations).values({ companyId: run.companyId, heartbeatRunId: run.id, issueId: issue.id, phase: "workspace_finalize", status: "succeeded", exitCode: 0, finishedAt: now, metadata: { owningService: "native_workspace_finalizer", workspaceSync: { omitted: true, reason: "restore_unsafe_archive", legacy: true } }, }); - await notifyNativeFinalizationWork(tx, { companyId: run.companyId, runId: run.id }); await tx.update(nativeRunFinalizations).set({ phase: "result_accepted", failureCode: null, nextAttemptAt: null, failureDetail: { workspaceFinalizeAttempt: 0, legacyUnsafeExportOmitted: { resultId: result.id, recoveredAt: now.toISOString() } }, updatedAt: now, }).where(eq(nativeRunFinalizations.runId, run.id)); diff --git a/server/src/services/native-runtime/native-workspace-finalization-ownership.ts b/server/src/services/native-runtime/native-workspace-finalization-ownership.ts index 7ab2cbf1edb..60db370812e 100644 --- a/server/src/services/native-runtime/native-workspace-finalization-ownership.ts +++ b/server/src/services/native-runtime/native-workspace-finalization-ownership.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js"; import { beginRunPhase } from "../execution-preparation-timing.js"; import { logActivity } from "../activity-log.js"; import { issueRecoveryActionService } from "../issue-recovery-actions.js"; @@ -21,6 +22,12 @@ const OWNER_KEY = "nativeWorkspaceFinalizationOwner"; export function nativeWorkspaceFinalizationLockKey(companyId: string, runId: string): string { return `native-workspace-finalization:${companyId}:${runId}`; } +export async function tryLockNativeWorkspaceFinalization(tx: Db, input: { companyId: string; runId: string }): Promise { + const [lock] = await tx.execute(sql`select pg_try_advisory_xact_lock(hashtextextended( + ${nativeWorkspaceFinalizationLockKey(input.companyId, input.runId)}, 0)) as acquired`) as unknown as Array<{ acquired: boolean }>; + return lock.acquired; +} + const activeOwnerTokens = new Set(); // Positive same-boot evidence, retained only when durable cleanup could not be // confirmed. Absence from the active set alone never proves another token joined. @@ -258,9 +265,12 @@ export async function withNativeWorkspaceFinalizationOwnership( activeOwnerTokens.delete(owner.token); joinedOwnerTokens.add(owner.token); try { - await input.db.update(heartbeatRuns).set({ - runnerProfileJson: sql`coalesce(${heartbeatRuns.runnerProfileJson}, '{}'::jsonb) - ${OWNER_KEY}`, - }).where(owned); + await input.db.transaction(async tx => { + await notifyNativeStatusReassessment(tx, { companyId: input.companyId, runId: input.runId }); + await tx.update(heartbeatRuns).set({ + runnerProfileJson: sql`coalesce(${heartbeatRuns.runnerProfileJson}, '{}'::jsonb) - ${OWNER_KEY}`, + }).where(owned); + }); joinedOwnerTokens.delete(owner.token); } catch { // A reconnect in this exact controller boot can use the positive join diff --git a/server/src/services/native-runtime/native-workspace-finalization-recovery-state.ts b/server/src/services/native-runtime/native-workspace-finalization-recovery-state.ts index 71bc4f426c5..a4798b23e55 100644 --- a/server/src/services/native-runtime/native-workspace-finalization-recovery-state.ts +++ b/server/src/services/native-runtime/native-workspace-finalization-recovery-state.ts @@ -5,6 +5,10 @@ import { nativeSha256 } from "./canonical.js"; export const NATIVE_WORKSPACE_OWNER_UNVERIFIED = "native_workspace_finalization_owner_unverified"; +export function hasNativeWorkspaceFinalizationOwner(profile: Record | null | undefined): boolean { + return profile?.nativeWorkspaceFinalizationOwner != null; +} + /** A terminal task does not prove that an older controller's copyback stopped. */ export async function hasUnverifiedNativeWorkspaceFinalizationOwner( db: Db, @@ -20,5 +24,5 @@ export async function hasUnverifiedNativeWorkspaceFinalizationOwner( eq(heartbeatRuns.runtimeMode, "native"), )).limit(1); const owner = run?.profile?.nativeWorkspaceFinalizationOwner; - return owner != null && action.fingerprint === nativeSha256({ runId, owner }); + return hasNativeWorkspaceFinalizationOwner(run?.profile) && action.fingerprint === nativeSha256({ runId, owner }); } diff --git a/server/src/services/native-runtime/native-workspace-finalizer.test.ts b/server/src/services/native-runtime/native-workspace-finalizer.test.ts index f79698d5b9a..327515304d7 100644 --- a/server/src/services/native-runtime/native-workspace-finalizer.test.ts +++ b/server/src/services/native-runtime/native-workspace-finalizer.test.ts @@ -1,3 +1,5 @@ +import { DELIVERY_QUEUES, subscribeDeliveryWork } from "../delivery-work-notifications.js"; +import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js"; import * as pluginWorkspaces from "../plugin-workspaces.js"; import * as platformProof from "./native-workspace-platform-proof.js"; import { currentNativeControllerIdentity } from "./native-restart-recovery.js"; @@ -38,7 +40,8 @@ import { resumeNativeWorkspaceFinalization } from "./native-workspace-finalizer. import { reconcileNativeFinalizations } from "./native-finalization-reconciler.js"; describe("native workspace finalization recovery", () => { - let temporary: Awaited>; + let temporary: Awaited> | undefined; + let connectionString: string; let db: ReturnType; let workspaceRoot: string; let priorLogRoot: string | undefined; @@ -125,10 +128,10 @@ describe("native workspace finalization recovery", () => { workspaceRoot, "operation-logs", ); - temporary = await startEmbeddedPostgresTestDatabase( - "paperclip-native-workspace-finalizer-", - ); - db = createDb(temporary.connectionString); + const external = resolveExternalTestDatabaseUrl("PAPERCLIP_NATIVE_WORKSPACE_FINALIZER_TEST_DATABASE_URL"); + if (!external) temporary = await startEmbeddedPostgresTestDatabase("paperclip-native-workspace-finalizer-"); + connectionString = external ?? temporary!.connectionString; + db = createDb(connectionString); await db.insert(companies).values([ { id: companyId, name: "Native workspace owner", issuePrefix: "NWO" }, @@ -162,7 +165,8 @@ describe("native workspace finalization recovery", () => { }, 30_000); afterAll(async () => { - await temporary.cleanup(); + await db?.$client.end({ timeout: 1 }); + await temporary?.cleanup(); await fs.rm(workspaceRoot, { recursive: true, force: true }); if (priorLogRoot === undefined) { delete process.env.WORKSPACE_OPERATION_LOG_BASE_PATH; @@ -210,7 +214,17 @@ describe("native workspace finalization recovery", () => { const seed = await seedAbandonedCopyback(); const [before] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId)); expect(await withNativeWorkspaceFinalizationOwnership(seed, async () => "unexpected")).toEqual({ acquired: false }); - const receipt = await resumeNativeWorkspaceAfterOwnerStop(seed); + const observed: Array> = []; + const stop = subscribeDeliveryWork(db, DELIVERY_QUEUES.nativeStatusReassessment, () => { + observed.push(db.select({ profile: heartbeatRuns.runnerProfileJson }).from(heartbeatRuns) + .where(eq(heartbeatRuns.id, seed.runId)).then(rows => rows[0].profile)); + }); + let receipt: Awaited>; + try { + receipt = await resumeNativeWorkspaceAfterOwnerStop(seed); + await vi.waitFor(() => expect(observed).toHaveLength(1)); + expect(await Promise.all(observed)).toEqual([{ preserved: "workspace descriptor" }]); + } finally { stop(); } expect(receipt).toEqual({ runId: seed.runId, resultId: before.resultId, status: "queued" }); const [after] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId)); expect(after).toEqual(before); @@ -425,7 +439,7 @@ describe("native workspace finalization recovery", () => { it("keeps ordinary progress writable even when the application pool has one connection", async () => { const seeded = await seedRun({ executionWorkspaceId: randomUUID(), title: "Small application pool" }); - const smallPool = createDb(temporary.connectionString, { maxConnections: 1 }); + const smallPool = createDb(connectionString, { maxConnections: 1 }); const result = await withNativeWorkspaceFinalizationOwnership({ db: smallPool, companyId, runId: seeded.runId }, async (ownership) => { await ownership.assertHeld(); await smallPool.update(heartbeatRuns).set({ updatedAt: new Date() }).where(eq(heartbeatRuns.id, seeded.runId)); @@ -438,7 +452,7 @@ describe("native workspace finalization recovery", () => { const seeded = await seedRun({ executionWorkspaceId: randomUUID(), title: "Live copyback" }); const live = await withNativeWorkspaceFinalizationOwnership({ db, companyId, runId: seeded.runId }, async (ownership) => { await ownership.assertHeld(); - expect(await resumeNativeWorkspaceFinalization({ db: createDb(temporary.connectionString), runId: seeded.runId })).toBeNull(); + expect(await resumeNativeWorkspaceFinalization({ db: createDb(connectionString), runId: seeded.runId })).toBeNull(); expect(await db.select().from(workspaceOperations).where(eq(workspaceOperations.heartbeatRunId, seeded.runId))).toEqual([]); return "copied"; }); @@ -462,7 +476,7 @@ describe("native workspace finalization recovery", () => { try { await entered; await db.execute(sql`select pg_terminate_backend(pid) from pg_locks where locktype = 'advisory' and database = (select oid from pg_database where datname = current_database()) and pid <> pg_backend_pid()`); - const second = await withNativeWorkspaceFinalizationOwnership({ db: createDb(temporary.connectionString), companyId, runId: seeded.runId }, async () => "must not run"); + const second = await withNativeWorkspaceFinalizationOwnership({ db: createDb(connectionString), companyId, runId: seeded.runId }, async () => "must not run"); expect(second).toEqual({ acquired: false }); const [held] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.runId)); expect(held.runnerProfileJson?.nativeWorkspaceFinalizationOwner).toBeTruthy(); @@ -475,7 +489,7 @@ describe("native workspace finalization recovery", () => { it("recovers its own joined copyback receipt after the application pool disconnects during cleanup", async () => { const seeded = await seedRun({ executionWorkspaceId: randomUUID(), title: "Reconnect joined copyback" }); - const interruptedDb = createDb(temporary.connectionString); + const interruptedDb = createDb(connectionString); let joined = false; await expect(withNativeWorkspaceFinalizationOwnership({ db: interruptedDb, companyId, runId: seeded.runId }, async () => { await fs.writeFile(path.join(workspaceRoot, "joined-copyback.txt"), "durable work"); @@ -493,7 +507,7 @@ describe("native workspace finalization recovery", () => { .toEqual({ acquired: false }); } await db.update(heartbeatRuns).set({ runnerProfileJson: beforeReconnect.runnerProfileJson }).where(eq(heartbeatRuns.id, seeded.runId)); - expect(await withNativeWorkspaceFinalizationOwnership({ db: createDb(temporary.connectionString), companyId, runId: seeded.runId }, async () => "resumed")) + expect(await withNativeWorkspaceFinalizationOwnership({ db: createDb(connectionString), companyId, runId: seeded.runId }, async () => "resumed")) .toEqual({ acquired: true, value: "resumed" }); }); diff --git a/server/src/services/native-runtime/native-workspace-owner-recovery.ts b/server/src/services/native-runtime/native-workspace-owner-recovery.ts index 4b3e19e3311..ae122b5edf7 100644 --- a/server/src/services/native-runtime/native-workspace-owner-recovery.ts +++ b/server/src/services/native-runtime/native-workspace-owner-recovery.ts @@ -1,3 +1,5 @@ +import { tryLockNativeWorkspaceFinalization } from "./native-workspace-finalization-ownership.js"; +import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js"; import os from "node:os"; import { and, eq, sql } from "drizzle-orm"; import { heartbeatRuns, issues, issueRecoveryActions, nativeRunFinalizations, type Db } from "@paperclipai/db"; @@ -24,8 +26,7 @@ export async function resumeNativeWorkspaceAfterOwnerStop(input: { const admitted = await input.db.transaction(async tx => { // Share the physical copyback lock. Do not wait behind an active exporter // and then apply a confirmation that described an earlier owner. - const locks = await tx.execute(sql`select pg_try_advisory_xact_lock(hashtextextended(${`native-workspace-finalization:${input.companyId}:${input.runId}`}, 0)) as acquired`); - if (locks[0]?.acquired !== true) throw conflict("Workspace copyback is still active. Wait for it to stop before confirming recovery."); + if (!await tryLockNativeWorkspaceFinalization(tx as unknown as Db, input)) throw conflict("Workspace copyback is still active. Wait for it to stop before confirming recovery."); // Match the status arbiter's coordinator -> issue -> run lock order. const [coordinator] = await tx.select().from(nativeRunFinalizations).where(and(eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.issueId), eq(nativeRunFinalizations.runId, input.runId))).for("update"); const [issue] = await tx.select().from(issues).where(and(eq(issues.companyId, input.companyId), eq(issues.id, input.issueId))).for("update"); @@ -65,6 +66,7 @@ export async function resumeNativeWorkspaceAfterOwnerStop(input: { const now = new Date(); const proof = { owner: prior, resultId: result.id, actorId: input.actorId, stopEvidence: confirmation.stopEvidence, confirmedAt: now.toISOString() }; + await notifyNativeStatusReassessment(tx, { companyId: input.companyId, issueId: input.issueId, runId: input.runId }); await tx.update(heartbeatRuns).set({ runnerProfileJson: sql`${heartbeatRuns.runnerProfileJson} - ${OWNER_KEY}`, updatedAt: now, }).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, run.id), diff --git a/server/src/services/native-runtime/runner-task-attachments.test.ts b/server/src/services/native-runtime/runner-task-attachments.test.ts index 88bea30420e..d122e301eb9 100644 --- a/server/src/services/native-runtime/runner-task-attachments.test.ts +++ b/server/src/services/native-runtime/runner-task-attachments.test.ts @@ -1,3 +1,4 @@ +import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js"; import { createHash, randomUUID } from "node:crypto"; import { Readable } from "node:stream"; import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; @@ -11,10 +12,14 @@ import { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority. const sha = (body: Buffer) => createHash("sha256").update(body).digest("hex"); describe("Dot assigned-task attachment reading", () => { - let temporary: Awaited>; + let temporary: Awaited> | undefined; let db: ReturnType; - beforeAll(async () => { temporary = await startEmbeddedPostgresTestDatabase("dot-task-files-"); db = createDb(temporary.connectionString); }); - afterAll(async () => { await temporary?.cleanup(); }); + beforeAll(async () => { + const external = resolveExternalTestDatabaseUrl("PAPERCLIP_RUNNER_ATTACHMENTS_TEST_DATABASE_URL"); + if (!external) temporary = await startEmbeddedPostgresTestDatabase("dot-task-files-"); + db = createDb(external ?? temporary!.connectionString); + }); + afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); }); afterEach(() => { vi.unstubAllEnvs(); vi.unstubAllGlobals(); }); async function fixture(body = Buffer.from("only in the file 🌍"), enabled = true) { const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(); diff --git a/server/src/services/native-runtime/status-decision-committer.ts b/server/src/services/native-runtime/status-decision-committer.ts index 04696bc1c85..cedd7130e8c 100644 --- a/server/src/services/native-runtime/status-decision-committer.ts +++ b/server/src/services/native-runtime/status-decision-committer.ts @@ -1,3 +1,8 @@ +import { lockCurrentNativeWorkAssessmentEvidence } from "./work-assessments.js"; +import { hasNativeWorkspaceFinalizationOwner } from "./native-workspace-finalization-recovery-state.js"; +import { tryLockNativeWorkspaceFinalization } from "./native-workspace-finalization-ownership.js"; +import { lockCurrentNativeCompletionContract } from "./completion-contracts.js"; +import { isNativeRunnerOwnershipHeld } from "./native-runner-ownership.js"; import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js"; import { notifyAutomaticCompletionReviews } from "../automatic-completion-review-notifications.js"; @@ -8,7 +13,7 @@ import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js" import { scheduledIssueMonitorWait } from "../issue-monitors.js"; import { onboardingFirstTaskCompletionAllowed } from "../onboarding-first-task-admission.js"; import { hasPublishedGitHubReview, readInheritedGitHubReviewWait } from "../chat-channels/github/review-wait.js"; -import { pendingNativeGovernance } from "./native-governance.js"; +import { lockNativeGovernanceSources, pendingNativeGovernance } from "./native-governance.js"; import { findSupersededExternalReviewRequests, supersedeExternalReviewRequestsInTransaction } from "./superseded-external-review-requests.js"; import { normalizePrpResultSignals, type PrpStructuredRunResult } from "../../vendor/paperclip-runner/index.js"; import { readNativeMonitorDelivery, readNativePreservedMonitorWait, type NativeMonitorDelivery, type NativePreservedMonitorWait } from "./native-monitor-delivery.js"; @@ -38,6 +43,7 @@ import { } from "@paperclipai/db"; import { NATIVE_STATUS_ARBITER_POLICY_VERSION, + type NativeGovernanceGate, type NativeStatusDecision, type NativeStatusEffect, } from "./status-arbiter.js"; @@ -1603,6 +1609,10 @@ export async function commitNativeStatusDecision(input: { requireMonitorDelivery?: NativeMonitorDelivery; requirePreservedMonitorWait?: NativePreservedMonitorWait; requireNoPendingChildCompletion?: NativeChildCompletionRecipient; + requireReassessmentSource?: { + phase: string; resultId: string; assessmentId: string; decisionId: string | null; agentId: string; + retiredAutomaticReview: { contractId: string; governanceGate: NativeGovernanceGate | null; hasUnresolvedIssueBlockers: boolean } | null; + }; requireModelRejectionOwner?: { agentId: string; reviewContext: NativeReviewAssignmentContext | null }; requireProviderFailureOwner?: { agentId: string; reviewContext: NativeReviewAssignmentContext | null }; requireBoardResponseWaitSource?: NativeBoardResponseWaitSource; @@ -1626,6 +1636,11 @@ export async function commitNativeStatusDecision(input: { const publications: ActivityPublication[] = []; const terminalRunsToEmit: (typeof heartbeatRuns.$inferSelect)[] = []; const committed = await input.db.transaction(async (tx) => { + const retiredReview = input.requireReassessmentSource?.retiredAutomaticReview; + if (retiredReview && !await lockCurrentNativeCompletionContract(tx as unknown as Db, { + companyId: input.companyId, issueId: input.issueId, contractId: retiredReview.contractId, + })) throw new NativeStatusRaceError(); + if (retiredReview && !await tryLockNativeWorkspaceFinalization(tx as unknown as Db, input)) throw new NativeStatusRaceError(); const coordinator = await tx .select() .from(nativeRunFinalizations) @@ -1654,6 +1669,67 @@ export async function commitNativeStatusDecision(input: { .limit(1) .then((rows) => rows[0] ?? null); if (!issue) throw new NativeStatusRaceError(); + if (input.requireReassessmentSource) { + const expected = input.requireReassessmentSource; + const now = Date.now(); + if (coordinator.phase === "terminal_failure" || coordinator.phase !== expected.phase || + coordinator.resultId !== expected.resultId || coordinator.assessmentId !== expected.assessmentId || + coordinator.decisionId !== expected.decisionId || + (coordinator.nextAttemptAt !== null && coordinator.nextAttemptAt.getTime() > now) || + (coordinator.leaseOwner !== null && + (coordinator.leaseExpiresAt === null || coordinator.leaseExpiresAt.getTime() > now))) { + throw new NativeStatusRaceError(); + } + const run = await tx.select().from(heartbeatRuns).where(and( + eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId), + eq(heartbeatRuns.nativeIssueId, input.issueId), eq(heartbeatRuns.agentId, expected.agentId), + )).for("update").limit(1).then(rows => rows[0]); + if (!run || run.runtimeMode !== "native" || isNativeRunnerOwnershipHeld(run)) throw new NativeStatusRaceError(); + if (expected.retiredAutomaticReview && (run.status !== "succeeded" || + hasNativeWorkspaceFinalizationOwner(run.runnerProfileJson) || + (issue.executionRunId !== null && issue.executionRunId !== input.runId))) { + throw new NativeStatusRaceError(); + } + if (expected.retiredAutomaticReview) { + try { + const [barrier] = await tx.select({ status: workspaceOperations.status }).from(workspaceOperations) + .where(and(eq(workspaceOperations.companyId, input.companyId), + eq(workspaceOperations.heartbeatRunId, input.runId), eq(workspaceOperations.phase, "workspace_finalize"))) + .orderBy(desc(workspaceOperations.createdAt)).limit(1).for("share", { noWait: true }); + if (barrier && barrier.status !== "succeeded") throw new NativeStatusRaceError(); + const executionState = record(issue.executionState); + if (executionState.status !== "pending") await lockNativeGovernanceSources({ db: tx as unknown as Db, + companyId: input.companyId, issueId: input.issueId }); + const gate = await pendingNativeGovernance({ db: tx as unknown as Db, companyId: input.companyId, + issueId: input.issueId, runId: input.runId, executionState }); + const expectedGate = expected.retiredAutomaticReview.governanceGate; + if (gate?.kind !== expectedGate?.kind || gate?.id !== expectedGate?.id) throw new NativeStatusRaceError(); + const readiness = await issueService(tx as unknown as Db).getDependencyReadiness(input.issueId, tx, { lockSources: true }); + if ((readiness.unresolvedBlockerCount > 0) !== expected.retiredAutomaticReview.hasUnresolvedIssueBlockers) { + throw new NativeStatusRaceError(); + } + } catch (error) { + if (record(error).code === "55P03" || record(record(error).cause).code === "55P03") throw new NativeStatusRaceError(); + throw error; + } + } + const pendingEffect = expected.decisionId ? await tx.select({ id: statusDecisionEffects.id }) + .from(statusDecisionEffects).where(and( + eq(statusDecisionEffects.companyId, input.companyId), eq(statusDecisionEffects.issueId, input.issueId), + eq(statusDecisionEffects.decisionId, expected.decisionId), + inArray(statusDecisionEffects.deliveryState, ["pending", "failed"]), + )).limit(1).then(rows => rows[0]) : null; + if (pendingEffect) throw new NativeStatusRaceError(); + try { + if (!await lockCurrentNativeWorkAssessmentEvidence(tx as unknown as Db, { + companyId: input.companyId, issueId: input.issueId, runId: input.runId, + resultId: expected.resultId, assessmentId: input.assessmentId, + })) throw new NativeStatusRaceError(); + } catch (error) { + if (record(error).code === "55P03" || record(record(error).cause).code === "55P03") throw new NativeStatusRaceError(); + throw error; + } + } // A completed model turn cannot close a persistent conversation. Preserve // the task here; the response finalizer records its durable waiting state. if (issue.conversationAgentId && input.decision.statusAction === "done") { diff --git a/server/src/services/native-runtime/work-assessments.ts b/server/src/services/native-runtime/work-assessments.ts index 58975d83b6d..fe554d33edd 100644 --- a/server/src/services/native-runtime/work-assessments.ts +++ b/server/src/services/native-runtime/work-assessments.ts @@ -1,7 +1,7 @@ import { and, eq } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; -import { workAssessments } from "@paperclipai/db"; -import type { NativeEvidenceAssessment } from "./evidence-classifier.js"; +import { completionContracts, nativeRunResults, workAssessments } from "@paperclipai/db"; +import { classifyNativeEvidence, type NativeEvidenceAssessment } from "./evidence-classifier.js"; import { nativeSha256 } from "./canonical.js"; import type { NativeMonitorDelivery, NativePreservedMonitorWait } from "./native-monitor-delivery.js"; @@ -144,3 +144,33 @@ export async function recordNativeAttentionAssessment(input: { if (!row) throw new Error("native_attention_assessment_not_persisted"); return row; } + + + +export async function lockCurrentNativeWorkAssessmentEvidence(db: Db, input: { + companyId: string; + issueId: string; + runId: string; + resultId: string; + assessmentId: string; +}): Promise { + const [row] = await db.select({ + assessment: workAssessments.assessmentJson, + contract: completionContracts.contractJson, + envelope: nativeRunResults.resultJson, + }).from(workAssessments) + .innerJoin(completionContracts, and(eq(completionContracts.id, workAssessments.contractId), + eq(completionContracts.companyId, input.companyId), eq(completionContracts.issueId, input.issueId))) + .innerJoin(nativeRunResults, and(eq(nativeRunResults.id, workAssessments.resultId), + eq(nativeRunResults.companyId, input.companyId), eq(nativeRunResults.issueId, input.issueId), + eq(nativeRunResults.runId, input.runId))) + .where(and(eq(workAssessments.id, input.assessmentId), eq(workAssessments.companyId, input.companyId), + eq(workAssessments.issueId, input.issueId), eq(workAssessments.runId, input.runId), + eq(workAssessments.resultId, input.resultId))) + .limit(1).for("share", { noWait: true }); + if (!row) return false; + const current = await classifyNativeEvidence({ db, companyId: input.companyId, + issueId: input.issueId, runId: input.runId, contract: row.contract, + result: row.envelope.result as Record, lockSources: true }); + return nativeSha256(current) === nativeSha256(row.assessment); +} diff --git a/server/src/services/native-status-reassessment-notifications.ts b/server/src/services/native-status-reassessment-notifications.ts new file mode 100644 index 00000000000..fb845523bb4 --- /dev/null +++ b/server/src/services/native-status-reassessment-notifications.ts @@ -0,0 +1,8 @@ +import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; + +export type NativeStatusReassessmentScope = { companyId: string; issueId?: string; runId?: string; agentId?: string }; + +export function notifyNativeStatusReassessment(tx: object, scope: NativeStatusReassessmentScope) { + return notifyDeliveryWork(tx, DELIVERY_QUEUES.nativeStatusReassessment, + JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId, runId: scope.runId, agentId: scope.agentId })); +} diff --git a/server/src/services/recovery/service.ts b/server/src/services/recovery/service.ts index 3870dbb66c0..460c02f4462 100644 --- a/server/src/services/recovery/service.ts +++ b/server/src/services/recovery/service.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js"; import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js"; import { notifyInstructionCollection } from "../instruction-collection-notifications.js"; @@ -6453,6 +6454,7 @@ export function recoveryService( const updated = await db.transaction(async tx => { await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: issue.companyId, issueId: issue.id }); + await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId: issue.id }); return tx .update(issues) .set({ diff --git a/server/src/services/satisfied-connection-intents.ts b/server/src/services/satisfied-connection-intents.ts index ba8c74205d4..e99b3289395 100644 --- a/server/src/services/satisfied-connection-intents.ts +++ b/server/src/services/satisfied-connection-intents.ts @@ -1,4 +1,4 @@ -import { notifyNativeReviewResponse } from "./native-review-response-notifications.js"; +import { notifyNativeReviewState } from "./native-review-state-notifications.js"; import { notifyChatConfirmationResolution } from "./chat-confirmation-notifications.js"; import { enqueueTerminalIssueInteractionChatPublications } from "./chat-interaction-publications.js"; import { connectionManagementService } from "./connection-management.js"; @@ -156,7 +156,7 @@ export function satisfiedConnectionIntentService(db: Db) { } const connection = await findSatisfiedToolConnection(txDb, task, intent); if (!connection) return false; - await notifyNativeReviewResponse(tx, candidate); + await notifyNativeReviewState(tx, candidate); const now = new Date(); // Retire the obsolete request without inventing a human approval. await notifyChatConfirmationResolution(tx, intent); diff --git a/server/src/services/tool-access/oauth/authorization-start.ts b/server/src/services/tool-access/oauth/authorization-start.ts index 0972611c16b..6af84ff5838 100644 --- a/server/src/services/tool-access/oauth/authorization-start.ts +++ b/server/src/services/tool-access/oauth/authorization-start.ts @@ -1,3 +1,4 @@ +import { notifyNativeReviewState } from "../../native-review-state-notifications.js"; import { vercelConnectHttpError, vercelCredentialFor, vercelConnectResourcesFor } from "../vercel-credentials.js"; import { credentialScope, managedConnectorProfile } from "./client-settings.js"; import { createOAuthClientRegistration } from "./client-registration.js"; @@ -563,65 +564,69 @@ export function createOAuthAuthorizationStart(db: Db, dependencies: Dependencies href: authorizationUrl.toString(), }, }; - const [existingInteraction] = await db - .select() - .from(issueThreadInteractions) - .where( - and( - eq(issueThreadInteractions.companyId, companyId), - eq(issueThreadInteractions.issueId, input.issueId), - eq(issueThreadInteractions.idempotencyKey, idempotencyKey), - ), - ) - .limit(1); - const [interaction] = existingInteraction - ? await db - .update(issueThreadInteractions) - .set({ - status: "pending", - requestedResolverPolicy: "human_only", - effectiveResolverPolicy: "human_only", - resolverPolicyProvenance: "explicit", - effectiveResolverPolicySource: "requested", - addresseeUserId: authorizationSubjectUserId, - payload, - result: null, - resolvedAt: null, - updatedAt: new Date(), - }) - .where(eq(issueThreadInteractions.id, existingInteraction.id)) - .returning() - : await db - .insert(issueThreadInteractions) - .values({ - companyId, - issueId: input.issueId, - kind: "request_confirmation", - status: "pending", - continuationPolicy: "none", - requestedResolverPolicy: "human_only", - effectiveResolverPolicy: "human_only", - resolverPolicyProvenance: "explicit", - effectiveResolverPolicySource: "requested", - addresseeUserId: authorizationSubjectUserId, - idempotencyKey, - sourceRunId: - binding.actorType === "agent" - ? (input.actor.sessionId ?? null) - : null, - title: `Connect your ${providerName} to continue`, - summary: `${requestingAgent?.name ?? "An agent"} needs your ${providerName} identity for work running as you.`, - createdByAgentId: - binding.actorType === "agent" ? binding.actorId : null, - payload, - }) - .returning(); - if (interaction) { - await db - .update(toolOauthStates) - .set({ interactionId: interaction.id }) - .where(eq(toolOauthStates.state, state)); - } + const issueId = input.issueId; + await db.transaction(async tx => { + await notifyNativeReviewState(tx, { companyId, issueId }); + const [existingInteraction] = await tx + .select() + .from(issueThreadInteractions) + .where( + and( + eq(issueThreadInteractions.companyId, companyId), + eq(issueThreadInteractions.issueId, issueId), + eq(issueThreadInteractions.idempotencyKey, idempotencyKey), + ), + ) + .limit(1); + const [interaction] = existingInteraction + ? await tx + .update(issueThreadInteractions) + .set({ + status: "pending", + requestedResolverPolicy: "human_only", + effectiveResolverPolicy: "human_only", + resolverPolicyProvenance: "explicit", + effectiveResolverPolicySource: "requested", + addresseeUserId: authorizationSubjectUserId, + payload, + result: null, + resolvedAt: null, + updatedAt: new Date(), + }) + .where(eq(issueThreadInteractions.id, existingInteraction.id)) + .returning() + : await tx + .insert(issueThreadInteractions) + .values({ + companyId, + issueId: issueId, + kind: "request_confirmation", + status: "pending", + continuationPolicy: "none", + requestedResolverPolicy: "human_only", + effectiveResolverPolicy: "human_only", + resolverPolicyProvenance: "explicit", + effectiveResolverPolicySource: "requested", + addresseeUserId: authorizationSubjectUserId, + idempotencyKey, + sourceRunId: + binding.actorType === "agent" + ? (input.actor.sessionId ?? null) + : null, + title: `Connect your ${providerName} to continue`, + summary: `${requestingAgent?.name ?? "An agent"} needs your ${providerName} identity for work running as you.`, + createdByAgentId: + binding.actorType === "agent" ? binding.actorId : null, + payload, + }) + .returning(); + if (interaction) { + await tx + .update(toolOauthStates) + .set({ interactionId: interaction.id }) + .where(eq(toolOauthStates.state, state)); + } + }); } const nextConfig = { diff --git a/server/src/services/tool-access/oauth/cloud-callback.ts b/server/src/services/tool-access/oauth/cloud-callback.ts index 3c777fc7969..41ce659847d 100644 --- a/server/src/services/tool-access/oauth/cloud-callback.ts +++ b/server/src/services/tool-access/oauth/cloud-callback.ts @@ -1,4 +1,4 @@ -import { notifyNativeReviewResponse } from "../../native-review-response-notifications.js"; +import { notifyNativeReviewState } from "../../native-review-state-notifications.js"; import { notifyToolConnectionHealth } from "../../tool-connection-health-notifications.js"; import { managedConnectorProfile } from "./client-settings.js"; import type { ActorInfo } from "../types.js"; @@ -421,7 +421,7 @@ export function createCloudOAuthCallback(db: Db, dependencies: Dependencies) { stateRow.interactionId && linkedInteraction?.kind === "request_confirmation" ) { - await notifyNativeReviewResponse(tx, { companyId: connection.companyId, issueId: linkedInteraction.issueId }); + await notifyNativeReviewState(tx, { companyId: connection.companyId, issueId: linkedInteraction.issueId }); await tx .update(issueThreadInteractions) .set({ diff --git a/server/src/services/tool-access/oauth/direct-callback.ts b/server/src/services/tool-access/oauth/direct-callback.ts index e3f68f3ba40..5649cd36d64 100644 --- a/server/src/services/tool-access/oauth/direct-callback.ts +++ b/server/src/services/tool-access/oauth/direct-callback.ts @@ -1,4 +1,4 @@ -import { notifyNativeReviewResponse } from "../../native-review-response-notifications.js"; +import { notifyNativeReviewState } from "../../native-review-state-notifications.js"; import { notifyToolConnectionHealth } from "../../tool-connection-health-notifications.js"; import { credentialScope } from "./client-settings.js"; import { withoutOAuthRefreshLease } from "./refresh-leases.js"; @@ -387,7 +387,7 @@ export function createDirectOAuthCallback(db: Db, dependencies: Dependencies) { stateRow.interactionId && linkedInteraction?.kind === "request_confirmation" ) { - await notifyNativeReviewResponse(tx, { companyId: connection.companyId, issueId: linkedInteraction.issueId }); + await notifyNativeReviewState(tx, { companyId: connection.companyId, issueId: linkedInteraction.issueId }); await tx .update(issueThreadInteractions) .set({ diff --git a/server/src/services/tool-access/oauth/state.ts b/server/src/services/tool-access/oauth/state.ts index efa92bc139f..b9e0ccdfcb0 100644 --- a/server/src/services/tool-access/oauth/state.ts +++ b/server/src/services/tool-access/oauth/state.ts @@ -1,4 +1,4 @@ -import { notifyNativeReviewResponse } from "../../native-review-response-notifications.js"; +import { notifyNativeReviewState } from "../../native-review-state-notifications.js"; import type { ActorInfo } from "../types.js"; import { and, eq } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; @@ -164,7 +164,7 @@ async function rejectPendingOAuthInteraction( } if (!linked) return; await db.transaction(async tx => { - await notifyNativeReviewResponse(tx, { companyId: stateRow.companyId, issueId: linked.issueId }); + await notifyNativeReviewState(tx, { companyId: stateRow.companyId, issueId: linked.issueId }); await tx .update(issueThreadInteractions) .set({ diff --git a/server/src/services/work-products.ts b/server/src/services/work-products.ts index 4aa44ef7f49..b993c9d656a 100644 --- a/server/src/services/work-products.ts +++ b/server/src/services/work-products.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js"; import { notifyApprovalQueueWork } from "./approval-work-notifications.js"; import { and, desc, eq, inArray } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; @@ -274,6 +275,7 @@ export function workProductService( createForIssue: async (issueId: string, companyId: string, data: Omit) => { const row = await db.transaction(async (tx) => { + await notifyNativeStatusReassessment(tx, { companyId, issueId }); if (data.type === "pull_request") { await notifyTerminalWorkspaceWork(tx, `issue:${issueId}`); await notifyApprovalQueueWork(tx, companyId); @@ -312,6 +314,10 @@ export function workProductService( .where(eq(issueWorkProducts.id, id)) .then((rows) => rows[0] ?? null); if (!existing) return null; + await notifyNativeStatusReassessment(tx, { companyId: existing.companyId, issueId: existing.issueId }); + if ((patch.companyId && patch.companyId !== existing.companyId) || (patch.issueId && patch.issueId !== existing.issueId)) { + await notifyNativeStatusReassessment(tx, { companyId: patch.companyId ?? existing.companyId, issueId: patch.issueId ?? existing.issueId }); + } if (existing.type === "pull_request" || patch.type === "pull_request") { await notifyTerminalWorkspaceWork(tx, `issue:${existing.issueId}`); @@ -385,6 +391,9 @@ export function workProductService( // prefix behind (which a retry would then duplicate). Mirrors the // per-writer transaction the batched issue/document writers use. await db.transaction(async (tx) => { + for (const scope of new Map(values.map(value => [value.issueId, { companyId: value.companyId, issueId: value.issueId }])).values()) { + await notifyNativeStatusReassessment(tx, scope); + } for (const companyId of new Set(values.filter(value => value.type === "pull_request").map(value => value.companyId))) { await notifyApprovalQueueWork(tx, companyId); } @@ -399,6 +408,7 @@ export function workProductService( const row = await db.transaction(async tx => { const [existing] = await tx.select({ type: issueWorkProducts.type, issueId: issueWorkProducts.issueId, companyId: issueWorkProducts.companyId }) .from(issueWorkProducts).where(eq(issueWorkProducts.id, id)); + if (existing) await notifyNativeStatusReassessment(tx, { companyId: existing.companyId, issueId: existing.issueId }); if (existing?.type === "pull_request") { await notifyTerminalWorkspaceWork(tx, `issue:${existing.issueId}`); await notifyApprovalQueueWork(tx, existing.companyId); diff --git a/server/src/services/workspace-operations.ts b/server/src/services/workspace-operations.ts index 53fcbf5292c..60103c15fbd 100644 --- a/server/src/services/workspace-operations.ts +++ b/server/src/services/workspace-operations.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js"; import { randomUUID } from "node:crypto"; import type { Db } from "@paperclipai/db"; import { workspaceOperations } from "@paperclipai/db"; @@ -476,6 +477,14 @@ export function workspaceOperationService(db: Db) { }, async recordOperation(recordInput) { + async function writeState(write: (database: Db) => Promise): Promise { + if (recordInput.phase !== "workspace_finalize" || !input.heartbeatRunId) return write(db); + return db.transaction(async tx => { + await notifyNativeStatusReassessment(tx, { companyId: input.companyId, + runId: input.heartbeatRunId!, issueId: input.issueId ?? undefined }); + return write(tx as unknown as Db); + }); + } const currentUserRedactionOptions = { enabled: (await instanceSettings.getGeneral()).censorUsernameInLogs, }; @@ -521,7 +530,7 @@ export function workspaceOperationService(db: Db) { if (runtimeControlAction) liveRuntimeControlOperationIds.add(id); try { - await db.insert(workspaceOperations).values({ + await writeState(database => database.insert(workspaceOperations).values({ id, companyId: input.companyId, executionWorkspaceId, @@ -539,7 +548,7 @@ export function workspaceOperationService(db: Db) { ) as Record | null, startedAt, updatedAt: startedAt, - }); + })); } catch (insertError) { liveRuntimeControlOperationIds.delete(id); throw insertError; @@ -630,7 +639,7 @@ export function workspaceOperationService(db: Db) { await append("stderr", result.stderr ?? null); const finalized = await logStore.finalize(handle); const finishedAt = new Date(); - const row = await db + const row = await writeState(database => database .update(workspaceOperations) .set({ executionWorkspaceId, @@ -650,7 +659,7 @@ export function workspaceOperationService(db: Db) { }) .where(eq(workspaceOperations.id, id)) .returning() - .then((rows) => rows[0] ?? null); + .then((rows) => rows[0] ?? null)); if (!row) throw notFound("Workspace operation not found"); return toWorkspaceOperation(row); } catch (error) { @@ -660,7 +669,7 @@ export function workspaceOperationService(db: Db) { await append("stderr", error instanceof Error ? error.message : String(error)).catch(() => undefined); const finalized = await logStore.finalize(handle).catch(() => null); const finishedAt = new Date(); - await db + await writeState(database => database .update(workspaceOperations) .set({ executionWorkspaceId, @@ -687,7 +696,7 @@ export function workspaceOperationService(db: Db) { finishedAt, updatedAt: finishedAt, }) - .where(eq(workspaceOperations.id, id)); + .where(eq(workspaceOperations.id, id))); throw error; } finally { // Releasing the in-process claim before the request unwinds is what lets the very diff --git a/server/src/services/workspace-recovery.test.ts b/server/src/services/workspace-recovery.test.ts index 3cb3089b196..1c3e976b091 100644 --- a/server/src/services/workspace-recovery.test.ts +++ b/server/src/services/workspace-recovery.test.ts @@ -1,3 +1,5 @@ +import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js"; +import { DELIVERY_QUEUES, subscribeDeliveryWork } from "./delivery-work-notifications.js"; import { prepareHeartbeatWorkspace } from "./heartbeat/workspace-preparation.js"; import { createHeartbeatWorkspaceResolver, resolveExecutionWorkspaceReuseRequestForIssue } from "./heartbeat/workspaces.js"; import { environmentService } from "./environments.js"; @@ -29,7 +31,6 @@ import { resumePreservedResultFinalization } from "./preserved-result-finalizati import type { AdapterSandboxExecutionTarget } from "@paperclipai/adapter-utils/execution-target"; import type { EnvironmentLease } from "@paperclipai/shared"; import { startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js"; -import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js"; import { prepareNativeWorkspaceSync } from "./native-runtime/native-workspace-sync.js"; import { nativeWorkspaceFinalizationLockKey } from "./native-runtime/native-workspace-finalization-ownership.js"; import { remoteTerminationReceipt } from "./remote-execution-termination.js"; @@ -54,7 +55,8 @@ vi.mock("./environment-execution-target.js", async importOriginal => { }); describe("automatic retained workspace recovery", () => { - let temporary: Awaited>; + let temporary: Awaited> | undefined; + let connectionString: string; let db: ReturnType; let root: string; const priorHome = process.env.PAPERCLIP_HOME, priorInstance = process.env.PAPERCLIP_INSTANCE_ID; @@ -66,7 +68,8 @@ describe("automatic retained workspace recovery", () => { await applyPendingMigrations(external); temporary = { connectionString: external, cleanup: async () => {} }; } else temporary = await startEmbeddedPostgresTestDatabase("workspace-recovery-"); - db = createDb(temporary.connectionString); + connectionString = temporary.connectionString; + db = createDb(connectionString); if (external) await db.execute(sql`TRUNCATE TABLE companies CASCADE`); }, 30_000); afterAll(async () => { @@ -478,6 +481,21 @@ describe("automatic retained workspace recovery", () => { systemRecovery: { ...state, nextAttemptAt: 0 } } }).where(eq(issueRecoveryActions.id, actionId)); } + it("publishes the committed finalization barrier after real retained workspace recovery", async () => { + const f = await seed(); + const observed: Array>> = []; + const stop = subscribeDeliveryWork(db, DELIVERY_QUEUES.nativeStatusReassessment, () => { + observed.push(db.select({ status: workspaceOperations.status }).from(workspaceOperations).where(and( + eq(workspaceOperations.companyId, f.companyId), eq(workspaceOperations.heartbeatRunId, f.runId), + eq(workspaceOperations.phase, "workspace_finalize"), + )).then(rows => rows)); + }); + try { await recoverRetainedWorkspace(f.input); } finally { stop(); } + expect(readSystemRecovery((await action(f.actionId)).evidence)?.state).toBe("complete"); + expect(observed.length).toBeGreaterThan(0); + expect((await Promise.all(observed)).some(rows => rows.some(row => row.status === "succeeded"))).toBe(true); + }); + it("re-enters only the existing finalizer for an accepted result, once per intent, preserving failure history", async () => { const f = await seed(); const contractId = randomUUID(), resultId = randomUUID(); @@ -582,7 +600,7 @@ describe("automatic retained workspace recovery", () => { expect(pending.status).toBe("running"); expect(uploads).toBe(2); await makeRetryDue(f.actionId); - const restarted = createDb(temporary.connectionString); + const restarted = createDb(connectionString); try { await recoverRetainedWorkspace({ ...f.input, db: restarted }); } finally { await restarted.$client.end(); } const operations = await db.select().from(workspaceOperations).where(and( @@ -791,7 +809,7 @@ describe("automatic retained workspace recovery", () => { const pending = await action(f.actionId); await db.update(issueRecoveryActions).set({ evidence: { ...pending.evidence, systemRecovery: { ...readSystemRecovery(pending.evidence), nextAttemptAt: 0 } } }).where(eq(issueRecoveryActions.id, f.actionId)); - const restarted = createDb(temporary.connectionString); + const restarted = createDb(connectionString); try { await recoverRetainedWorkspace({ ...f.input, db: restarted }); expect(await hasVerifiedWorkspaceRepair(restarted, await run(f.runId))).toBe(true); @@ -1143,7 +1161,7 @@ describe("automatic retained workspace recovery", () => { } expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toHaveLength(0); expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)))[0].status).toBe("released"); - const restarted = createDb(temporary.connectionString), onRecovered = vi.fn(async (_source: { id: string }) => {}); + const restarted = createDb(connectionString), onRecovered = vi.fn(async (_source: { id: string }) => {}); try { await expect.poll(async () => { await reconcileRetainedWorkspaces(restarted, f.input.environmentRuntime, onRecovered); diff --git a/server/src/services/workspace-recovery.ts b/server/src/services/workspace-recovery.ts index 0cf699ab239..20ca973c20b 100644 --- a/server/src/services/workspace-recovery.ts +++ b/server/src/services/workspace-recovery.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; import { beginIdleTrackedWork } from "./task-admission.js"; import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js"; @@ -246,6 +247,7 @@ export async function recoverRetainedWorkspace(input: Source & { db: Db; environ const next = { ...current!, revision: current!.revision + 1, phase: "reconcile_effects" as const, state: "complete" as const, nextAttemptAt: null }; const activity = await db.transaction(async tx => { + await notifyNativeStatusReassessment(tx, input); await tx.insert(workspaceOperations).values({ companyId: input.companyId, issueId: input.issueId, heartbeatRunId: input.runId, phase: "workspace_finalize", status: "succeeded", finishedAt: new Date(), metadata: { owningService: "workspace_recovery", preservationOperationIds: next.preservationOperationIds } });