diff --git a/doc/idle-native-status-reassessment.md b/doc/idle-native-status-reassessment.md new file mode 100644 index 00000000000..b9875615653 --- /dev/null +++ b/doc/idle-native-status-reassessment.md @@ -0,0 +1,100 @@ +# Native status reassessment without idle polling + +Native status reassessment uses committed changes and the shared delivery +scheduler. It no longer runs on every periodic orphan-recovery pass. + +## What creates work + +Existing native result, assessment, decision, task and review rows remain the +source of truth. No new queue table or database migration is required. + +| Change | Existing producer boundary | +| --- | --- | +| Work-product creation, update, import, removal, attachment or trusted promotion | Work-product/task services and the promotion transaction | +| Task status/version, assignment, execution state or blocker configuration | Task mutation transaction | +| Direct claim release, terminal/stale lock cleanup, queue release, recovery or tree restoration | Existing task-scoped claim mutation transaction | +| Automatic review retirement | Automatic completion review owner | +| Approval/link or interaction outcome, including OAuth card creation/reopening | Approval/review-state notification boundaries | +| Accepted/rejected control-plane run evidence | Run-event append transaction | +| Workspace barrier or physical ownership settlement | Workspace operation, finalizer and verified recovery owners | +| Coordinator claim/retry, effect settlement or run projection | Native finalization and run-milestone boundaries | + +These signals request inspection. They do not grant permission to complete a +task. Discovery retains the three existing conditions: changed work-product +history, changed authoritative task status/version, or a retired selected +completion review. The owner still classifies the selected result and appends +assessments and decisions through the existing status committer. + +A settled worker makes no recurring queries. Insufficient evidence is dormant. +Real contention or operational failure keeps a retry. A saved retry or lease +expiry keeps its actual future deadline. Startup reconstructs candidates in +100-row pages; the coordinator processes bounded batches and retains changes +through drain/resume and concurrent dispatch. Shutdown uses its existing owner. + +## Authority and separate responsibilities + +Scheduled commit checks the selected source, current task/run, retry and lease, +pending effects and native process ownership inside the status transaction. +Retired-review completion also checks the latest contract, workspace barrier, +physical workspace ownership, governance and dependencies. Evidence is +reclassified under source locks. Contention defers instead of blocking an idle +worker indefinitely or committing stale completion. + +The legacy reconciler retains explicit recovery and a one-time startup pass. +Its periodic call skips reassessment but preserves the superseded historical +source guard before independent workspace recovery. Workspace/result settlement, +retained session cleanup, native wake delivery and generic orphan recovery are +separate duties; their recurring scans have not been removed by this slice. + +Case-handoff delivery and lease changes do not improve evidence: every persisted +non-cancelled state already counts as accepted. Cancellation is negative evidence +and is fenced by the source lock at scheduled commit. Creation generates the +handoff identity inside its transaction before returning it. No reassessment wake +is added to every lease renewal. Target-agent status is also not a classification +trigger: attention validation checks existence, not whether an agent is invokable. +Creating an agent does not satisfy result criteria or verification. Missing +arbitrary target identities have no row to lock; this change does not provide a +general predicate-lock guarantee for independent identity insertion. + +Pending child completion is a guard and a fresh parent-turn obligation. It is not +silently consumed by completing the old result. New contracts supersede the old +result's eligibility; creating one does not revive that old result. Direct SQL +writers and independent replicas do not receive process-local notifications. +The supported producer model remains the active server and integrated transaction +owners; this change does not add a distributed notification transport. + +## Evidence and limits + +Current qualification is recorded in the [audit](plans/2026-10-26-native-status-reassessment-sleep-audit.md). +The broad 9fddb source passes 2731 cases in 28 complete files. The final 5ea3 +runtime source, after the evidence-trigger quality fix, passes 1211 cases in five +complete affected files. Their overlapping counts are not added together. Real +producer/transaction coverage includes first writes, rollback/outer commit, +authority races, saved deadlines, drain/resume, claim release and retained history. +The final worker suite includes all 43 cases. + +Required quality review remains mandatory on both retired-review and evidence +reassessment. An unmet review produces its existing nonterminal decision and +review recovery action, rather than repeatedly attempting completion as an +operational retry. Real required/optional regressions and controlled coarse-hint +negatives fail before repair and pass afterward without changing time limits. + +Historical discovery measurements in the audit retain their original revision: +10,000 selected-history tasks return no company candidates in about 35 ms; +10,000 real candidates require 101 queries and about 1.56 seconds to discover. +This is discovery cost, not complete backlog execution. The current worker still +retains candidates in memory proportional to backlog size. Its permanent +103-item lifecycle case verifies settlement across batches and SQL quiet. +Cancellation is checked between pages, not during an already-running SQL call. + +Database build, server typecheck/build, module boundaries and migration-order +checks pass. The host's embedded PostgreSQL bootstrap remains limited by +exhausted System V semaphores; acceptance uses fresh owned PostgreSQL 17 databases +and preserves embedded-only fresh-per-call isolation through the existing recovery +helper. Adapted test imports restore exactly; assertions are unchanged. No full +local repository-suite or live external-provider qualification is claimed. + +Remote exact-head CI and review still gate merging. The +[stack ledger](plans/2026-10-24-recurring-server-sleep-stack.md) tracks other duties. +Worker quiet does not establish database suspension, Cloud alarm coverage, host +sleep or fleet cost savings. diff --git a/doc/plans/2026-10-24-recurring-server-sleep-stack.md b/doc/plans/2026-10-24-recurring-server-sleep-stack.md index 3b5b3b9f45b..ff918b88dfc 100644 --- a/doc/plans/2026-10-24-recurring-server-sleep-stack.md +++ b/doc/plans/2026-10-24-recurring-server-sleep-stack.md @@ -2876,3 +2876,179 @@ between pages on drain. Fresh migration and interrupted/concurrent replay pass. The audit preserves the initial query, migration and test-clock failures. Full local tests are running; remote acceptance, older stack findings, master reconciliation and every remaining original duty are still open. Nothing merged. + + +## Slice 49 publication; slice 50 owner audit + +Draft #17702 at `be6ca6151b` is open on #17696. Greptile reviewed that head +at 5/5 with zero unresolved threads. CI remains in progress. The full local +suite was stopped after repeated embedded PostgreSQL bootstrap failures and +many skipped DB suites. One isolated native-session-executor failure reproduces +in startEmbeddedPostgresTestDatabase before recovery behavior runs; logs are +/tmp/sleep49-full-tests.log and /tmp/sleep49-full-failure-isolated.log. The +255-case real-PostgreSQL owner/startup selection, workspace types/build and +migration replay remain passing. This does not make full local acceptance green. + +Slice 50 audits native status reassessment: changed evidence, authoritative task +state and retired automatic reviews all feed the existing append-only assessment +and decision owner. Its producer/guard inventory is in the dated reassessment +audit. No implementation or periodic removal is claimed for slice 50 yet. Earlier +stack gates, master reconciliation and the full remaining scope remain open. + + +Slice 49 remote acceptance: #17702 at +`be6ca6151bb04e25f691a830734ddb5856701d27` now has all 53 checks passing, +Greptile 5/5 at that exact head and zero unresolved review threads. It remains +draft and unmerged. /tmp/sleep49-final-gates.json, /tmp/sleep49-comments.json +and /tmp/sleep49-review-threads.json preserve the observations. The PR body +records the passing remote gates and keeps the local embedded-database failure +visible. Slice 50 and all other remaining original obligations are incomplete. + + +Slice 50 extraction checkpoint: reassessment and its pure policy now have native +runtime owners, with the original reconciler delegating to them. All 94 status +corpus cases pass before and after extraction; all 187 governed-response cases +and server types pass afterward. No polling duty has been removed. The audit +records the critical transaction boundary: scheduled guards belong inside the +status committer, which publishes after its own transaction. Producer coverage, +worker selection, lifecycle/cost proof and a new PR remain outstanding. + +Slice 50 scheduled-commit checkpoint: nine competing source/lease/deadline/effect/ +ownership cases reproduce unsafe scheduled reassessment before the new guard. +The final 103-case status corpus and server types pass. The audit preserves two +initial invalid fixtures and the before-fix constraint rejection. The guard is +opt-in at the existing committer transaction; no outer transaction or early +publication is added. Worker discovery, producer/lifecycle/cost evidence and +periodic removal remain outstanding. No new PR or merge is claimed; all original +remaining duties and older stack gates remain open. + +Slice 50 scoped-worker checkpoint: discovery/check/register now use the shared +scheduler, with distinct dormant and retryable outcomes and exact saved deadlines. +A real retirement missed wake and a near-term deadline delay were reproduced and +fixed. Final local verification passes 116 status/worker cases plus 187 governed +response consumer cases; server types pass. Driver SQL capture proves quiet after +settlement and during drain for the tested path. Initial history cost measurements +and their synthetic-fixture limits are recorded in the owner audit. The worker is +not registered in startup and periodic reassessment remains. Producer coverage, +concurrent authority audit, selected-current/backlog cost and full acceptance are +still pending; no slice 50 PR or merge, and the full original scope stays open. + +Slice 50 producer checkpoint: six real work-product/attachment transaction misses +and three task mutation misses were reproduced and repaired. The real promotion +HTTP route also wakes reassessment. All 13 owner lifecycle tests, 79 affected +consumer cases across their documented runs, and server types pass. Initial +embedded-PostgreSQL failures and the external-database test repairs are preserved +in the owner audit. Approval/interaction/run evidence and workspace producers, +concurrent authority, backlog cost, startup wiring and periodic removal remain. +Master remains 1fbbef45a8; older stack gates and the full original scope stay open. + +Slice 50 authority checkpoint: two real commit-boundary regressions reproduced +stale completion after the source run failed or another run took task execution. +The scheduled retired-review path now rechecks both facts under existing locks. +All 115 status-corpus cases pass. A retained-history startup assertion failed in +the combined run; after waiting for bounded startup batches while enforcing the +saved retry deadline, all 28 lifecycle cases and server types pass. The owner +audit preserves failures and handoff/contract producer findings. Remaining +business-fact races, producer coverage, cost, wiring, periodic removal and PR +acceptance are still open. Nothing merged; the full stack goal is incomplete. + +Slice 50 contract checkpoint: a newer committed completion contract reproduced +stale completion; a real uncommitted writer reproduced blocking on the task lock. +The contract owner now exposes a nonblocking current-revision guard, acquired +before status row locks and held through commit. All 145 corpus/lifecycle tests +and server types pass. Failure logs and the lock-order rationale are recorded in +the owner audit. Workspace/governance races, producer coverage, cost, startup +integration, periodic removal, remote acceptance and the rest of the full stack +remain open. Nothing merged. + +Slice 50 workspace-barrier checkpoint: three stale-completion regressions now +pass after checking and share-locking the current barrier inside status commit. +The check uses existing row/FK serialization and NOWAIT contention handling. +All 148 status/lifecycle cases and server types pass. The owner audit preserves +before-fix failures and distinguishes barrier receipts from physical workspace +ownership, which still needs audit. Governance/dependencies, remaining producers, +cost, wiring, periodic removal and PR acceptance remain open; nothing merged. + +Slice 50 physical-owner checkpoint: four unsafe-completion regressions and a +subsequent release missed wake were reproduced and repaired. Scheduled retired- +review reassessment respects the existing physical lock and durable owner receipt; +normal/verified owner release wakes it through committed intent. All 217 status, +lifecycle and workspace-owner consumer cases and server types pass. Existing +orphan/stop-proof safeguards remain verified. Governance/dependencies, remaining +producers, cost, integration and remote gates remain open, along with all other +original duties and earlier stack findings. Nothing merged. + +Slice 50 governance checkpoint: seven stale-completion races reproduced changed +execution policy, approvals, interactions and question history. The governance +owner now supplies nonblocking source locks; status commit verifies the gate +used by the selected decision. All 159 status/lifecycle cases pass, the valid +question-history fixture passes on recheck, and server types pass. Dependency and +remaining authority/producer audits, task-history/query costs, wiring, periodic +removal and remote gates remain outstanding. The full stack remains incomplete; +nothing merged. + + +Slice 50 dependency checkpoint: seven stale-completion races are reproduced and +fixed through the existing dependency-readiness owner. Scheduled status commit +locks/rechecks prerequisite and workspace-finalization sources with nonblocking +contention handling; ordinary readers retain their behavior. All 181 distinct +status/lifecycle/dependency consumer cases and server types pass. The audit keeps +the initial unordered assertion and misconfigured embedded-test invocation +failures and their repairs. Remaining authority/producer coverage, cost, startup +integration, periodic removal, remote gates, earlier stack findings and the full +original scope remain open. Nothing merged. + + +Slice 50 evidence checkpoint: seven races reproduced completion using revoked, +removed or concurrently edited evidence. The assessment owner now reclassifies +and locks referenced evidence inside scheduled status commit; changed facts and +contention defer through existing retry handling. All 177 status/lifecycle/ +classifier consumer cases and server types pass. The audit preserves before-fix +failures and the remaining identity/producer and cost limits. Startup wiring, +periodic removal, remote gates, older stack findings and all remaining recurring +work are still incomplete. Nothing merged. + + +Slice 50 OAuth producer checkpoint: authorization-card creation/reopening now +signals both native review workers from the same short transaction as its card +and state link. Three missing-settlement-signal regressions were reproduced; +all 472 tool-access and 30 worker lifecycle tests plus server typecheck pass. +The audit retains before/after evidence and the remaining handoff/authority +coverage limits. Master is freshly verified at 1fbbef45a8. Discovery/cost proof, +startup wiring, periodic removal, remote acceptance, earlier stack findings and +the full original recurring-work scope remain incomplete. Nothing merged. + + +Slice 50 scale checkpoint: 10,000 separate selected completed task/run histories +produce zero company candidates (~35 ms); 10,000 actual candidates reconstruct +in 101 bounded queries (~1.56 seconds/2.05 MB), with verified drain/abort behavior. +A retired-review task with large review/comment/approval/dependency/workspace +history settles in ~126 ms and disappears from discovery. The audit preserves +two invalid benchmark-fixture attempts and the corrected fresh-task evidence. +A permanent 103-item backlog test proves real settlement and SQL quiet; all 31 +lifecycle cases and server types pass. Remaining producer/authority audit, +startup integration, periodic removal, remote gates and the full original stack +scope remain open. Nothing merged. + + +Slice 50 startup checkpoint: enabled maintenance now registers the reassessment +worker and periodic recovery skips its classification/decision path. The separate +historical supersession guard is preserved before workspace recovery. Two missing +integration regressions fail before the change; all 246 corpus/startup/lifecycle +cases, 32 existing finalization/cleanup consumer cases, and server types pass after. +The owner audit records the intentional separation of reassessment from independent +workspace settlement and the retained failure evidence. Remaining producer/authority +audit, full acceptance, publication, remote gates and all other original duties +remain incomplete. Nothing merged; the goal remains active. + + +Slice 50 is draft [#17709](https://github.com/paperclipai/paperclip/pull/17709), +based on #17702. Initial head 91a6c14170 passed all 52 CI checks; Greptile 4/5 +found direct claim-release writers missing reassessment notifications. Six real +regressions reproduced the gap. Task-scoped signals now cover their existing +transactions; the repaired selection passes 978 cases and server types, with +15 embedded-DB runtime cases excluded for the diagnosed host semaphore exhaustion. +The owner audit preserves the wrong initial ordinary-release completion assertion +and its authority-preserving correction. Fresh-head gates remain pending; nothing +merged. Earlier E2E failures on #17490/#17572 and voice findings on #17576/#17580 +were revalidated and remain unresolved. All original remaining duties stay in scope. diff --git a/doc/plans/2026-10-26-native-status-reassessment-sleep-audit.md b/doc/plans/2026-10-26-native-status-reassessment-sleep-audit.md new file mode 100644 index 00000000000..17525bba1e0 --- /dev/null +++ b/doc/plans/2026-10-26-native-status-reassessment-sleep-audit.md @@ -0,0 +1,1084 @@ +# Native status reassessment sleep audit + +Parent: governed-review response recovery, draft #17702 at `be6ca6151b`. +This audit starts the next reason in the recurring-server migration stack. +Parent #17702 has passing checks and Greptile 5/5 at that head; it remains unmerged. + +Current summary: [worker contract](../idle-native-status-reassessment.md). +Entries below preserve chronological checkpoints, including earlier incomplete +states and failures; the latest checkpoints supersede their progress status. + +## Obligation and ownership + +After a native result is assessed, changed durable facts can require a new status +assessment and decision. The current native-finalization reconciler discovers +that change by revisiting committed runs, including its rotating settled-history +page. Response publication and pending-effect delivery have separate owners in +the preceding slices. Workspace/result settlement and retained session cleanup +remain separate duties and must not disappear with this migration. + +The existing reassessment block has three activation conditions: + +- A work product was created or updated after the selected assessment. The + classifier must actually establish newly satisfied criteria and verification + before this condition changes task disposition. +- The authoritative task status/version differs from the previous assessment, + after accounting for the selected decision's projected version and preserve + semantics. A superseding decision can make the old run ineligible. +- The selected automatic review has been retired. Reassessment then checks the + latest contract, current task/run, pending child completion, dependencies, + governance and any later workspace-finalization barrier. + +These are triggers for the same append-only status reassessment operation. +Keep classification and status arbitration with native runtime. Keep task locks, +version comparison, current authority and effect creation in the status-decision +committer. Do not put evidence parsing or policy branches in startup or the shared +scheduler. Do not use the combined reconciler as a worker callback: that would +also run unrelated response, workspace and retention work. + +## Existing facts and guards + +The coordinator binds run, task, selected result, assessment and decision. The +assessment retains prior task status/version/decision and its input digest. The +result and contract are selected by matching company/task/run identities. The +committer appends the new decision and guards concurrent task changes. A +NativeStatusRaceError currently defers to later reconciliation; the new worker +must retain or receive the corresponding wake instead of silently losing it. + +The outer candidate scan respects next_attempt_at and live finalization leases. +The extracted owner must recheck these under its transaction/claim boundary; +a stale scheduler entry cannot bypass a newly acquired lease or delayed retry. +The worker must distinguish dormant changed evidence that is still insufficient +from retryable I/O or contention. Existing durable rows remain truth; no new queue +table is justified by this timer migration. + +## Producer audit to complete + +Work-product writers found in current source: + +- work-products.ts: create, update, primary replacement, batch import and removal. + Current notifications cover pull-request retention/approval work only. +- issues.ts: attachment work-product insertion. +- routes/issues.ts: trusted-source promotion changes an existing product and + inserts its promoted product inside an owned transaction. +- native-runtime/chat-attachment-reuse.ts and native-runner-file-handoff.ts: + direct metadata/title updates on existing products. + +The classifier reads more than work products: case handoffs, accepted/rejected +control-plane run events, linked approvals, resolved interactions and attachments. +Determine which mutations activate existing reassessment behavior and preserve +that contract. Do not assume that work-product notifications cover all evidence +or introduce inferred completion merely because a source row changed. + +Also audit task status/version producers; decision/effect settlement; automatic +review retirement; latest-contract changes; workspace barrier settlement; child +completion; dependency/blocker changes; agent/task authority and process claims. +Use the existing mutation/transaction owners. Reuse task/run-scoped notification +boundaries where their semantics match. Broad configuration changes may justify +broader scopes; ordinary writes must not reload every company run. + +## Acceptance before periodic removal + +- Extract and exercise the actual reassessment owner without invoking other + native recovery duties. Preserve fresh finalization and explicit recovery. +- Reproduce each missing producer wake before wiring it. Cover alternate writers, + nested outer commit/rollback and relevant uncertainty settlement. +- Verify superseding decisions, changed versions, current contract, ownership, + governance, pending children and workspace barriers with real DB state. +- Prove no duplicate decisions/effects under competing recovery and producer + changes during a sweep. A rejected or no-longer-current assessment cannot + authorize completion. +- Prove no recurring SQL for configured-idle and retained settled history, + including changed but insufficient evidence. Verify genuine deadlines, + failures, startup/resume, drain and shutdown. +- Measure selected-history and outstanding-backlog discovery before wiring it. + Bound individual queries and support interruption; disclose total reconstruction + cost and retain exact scope on ordinary changes. +- Register through existing maintenance lifecycle, disable only this periodic + duty, run relevant consumers and required checks, then obtain latest-head CI + and review acceptance on its own stacked PR. + +This is an audit, not an implemented migration. No polling entrance is removed +by this document. No Cloud alarm coverage or host sleep eligibility changes. + + +## Owner extraction checkpoint + +The status-authority corpus passes all 94 cases before extraction on a fresh +PostgreSQL database. Reassessment now lives in native-status-reassessment.ts; +its pure arbitration policy lives in native-reconciliation-status.ts. Existing +imports through the reconciler remain compatible. The combined reconciler +delegates to the extracted owner and retains all periodic behavior for now. +It avoids the legacy review-response lookup when that repair is disabled. + +The extracted owner distinguishes unchanged, superseded, deferred and committed +outcomes. Operational errors still propagate; the existing status-race behavior +remains deferred. All 94 status-authority cases and 187 governed-response cases +pass after extraction, as does server typecheck. Evidence: /tmp/sleep50-baseline.log, +/tmp/sleep50-extract-corpus.log, /tmp/sleep50-extract-responses.log and +/tmp/sleep50-extract-types.log. Each corpus run uses a fresh database because its +company prefixes persist beyond the suite. + +The scheduler entry point is not ready yet. The status committer locks the +coordinator before the task and publishes activity after its own transaction. +Wrapping it in a new outer transaction would publish before that outer commit; +do not do that. Add any scheduled source/lease/deadline guard at the committer's +existing transaction boundary, and exercise competing source/claim changes there. +The new worker must reload source facts and preserve deferred races/retries; it +must not treat an extracted snapshot-based helper as fresh scheduling authority. +Producer notifications, discovery cost, lifecycle tests and periodic removal +remain outstanding. No slice 50 PR has been published. + +## Scheduled commit guard + +Scheduled reassessment now opts into a source guard inside the existing status +committer transaction. After coordinator/task locking it compares the selected +phase/result/assessment/decision, respects a future retry and live or unknown +lease, locks and checks native run ownership, and defers while the old decision +has pending or failed effects. Explicit callers retain their existing behavior. +The lock order remains coordinator, task, then run. Publications still occur only +after the committer's transaction commits. + +The first seven-case reproduction had five behavioral failures and two invalid +fixtures: clearing the assessment/result without clearing dependent references +violated schema constraints before reaching the committer. Corrected fixtures +plus pending-effect and ownership races failed all nine before the guard: eight +incorrect commits and one attempted commit rejected by the result constraint. +After the guard, all 103 status-authority corpus cases and server typecheck pass. +Every deferred case preserves the competing state and successfully commits once +that state is released. Logs: /tmp/sleep50-guard-before.log, +/tmp/sleep50-guard-before2.log, /tmp/sleep50-guard-after.log, +/tmp/sleep50-guard-types.log; statuses are saved beside the test logs. + +This is not worker acceptance. Candidate selection, dormant-versus-retry outcome +handling, producer coverage, lifecycle/cost tests and periodic removal remain. +No slice 50 PR has been published and no additional polling has been removed. + +## Scoped worker and first producer + +A local, not-yet-registered worker now discovers the three existing activation +conditions with company/task/run/agent scopes and 100-row keyset pages. Candidate +SQL excludes superseded decisions, held native ownership, unknown finalization +leases and old pending effects. It is discovery, not mutation authority: check +reloads the source and the committer still applies the locked guard. Future retry +and live lease deadlines retain their maximum. A commit race requests a retry; +unchanged evidence or a dormant review prerequisite does not keep recurring work +scheduled. The shared scoped-deadline helper owns coalescing, fairness, admission +and retry bookkeeping. No new queue table is added. + +Nine initial discovery/owner tests pass for retired reviews, changed evidence, +authoritative task changes, future retries/leases, insufficient evidence, +ownership exclusions and pre-aborted discovery. A tenth regression exposed a +newly saved five-second deadline being pushed out to generic 30-second backoff. +The worker now returns that saved future deadline exactly; backoff applies only +when the source is still immediately eligible after a commit race. The complete +113-case status corpus and server types pass. Evidence: +/tmp/sleep50-worker-first.log, /tmp/sleep50-deadline-before.log, +/tmp/sleep50-worker-after.log and /tmp/sleep50-worker-after-types.log. + +An isolated lifecycle test reproduced a real missed wake: after the worker went +quiet, automatic review retirement left the task in review. Retirement now +signals the exact task/run inside its existing transaction, before changing the +card. The common native-finalization signal also includes reassessment, retaining +its supplied scope. All three lifecycle tests pass: real retirement after idle, +first retirement during drain followed by resume, and a saved future retry across +shutdown/restart without early dispatch. Driver-level SQL capture verifies no SQL +across two former polling minutes after completion or while drain is paused. +The existing governed-response suite passes all 187 cases after the shared +notification change. Logs: /tmp/sleep50-lifecycle-before.log, +/tmp/sleep50-lifecycle-after.log, /tmp/sleep50-lifecycle-types.log and +/tmp/sleep50-worker-responses.log. + +Initial read-only query measurement reused slice 49's 10,000 cloned history rows. +Those copies share a task and their decisions are superseded for reassessment; +this is not proof of 10,000 independently selected current assessments or an +outstanding reassessment backlog. Startup returned 62 candidates from other +retained history in one query (305 ms cold wall time; warm EXPLAIN execution about +47 ms). Company/task scopes returned none in 34/20 ms, exact run in 2.6 ms. +Plans and logs: /tmp/sleep50-query-{startup,company,issue,run}.plan and +/tmp/sleep50-query-plan.log. Selected-current history, backlog cost and cancellation +between pages still need measurement before enabling the worker. + +Remaining work is substantial: work-product and alternate writers, task changes, +classifier inputs while a retired review is dormant, current-contract/workspace/ +governance and ownership transitions, further transaction/lifecycle cases, +startup registration and periodic removal. Existing retirement prerequisite reads +precede the committer; audit their serialization with producer owners before +claiming concurrent authority coverage. In particular, completion-contract +creation uses its own advisory lock rather than the status committer's issue +lock. This checkpoint is not a claim that every business-fact race is covered. +The existing periodic reassessment path remains active and slice 50 has no PR. + +Final current-source local rerun passes 116 tests across the status corpus and +isolated worker lifecycle files (/tmp/sleep50-final-local.log). Together with the +187 governed-response cases this is 303 distinct passing cases, not whole-workspace +or remote PR acceptance. The last server typecheck also passes. + +## Work-product and task producer checkpoint + +Six real writer regressions failed before notifications: artifact update, create, +primary replacement, import, attachment creation, and nested transaction commit. +Their existing write transactions now signal task-scoped reassessment before the +mutation. Import coalesces its scopes per task. Updates signal both old and new +scope if their supported patch moves a product. Removal also signals its old task. +Attachment creation signals before asset/attachment/product writes, including +attachment evidence without a registered run. Native file handoff and attachment +reuse call that service inside their tool-authority transaction; their subsequent +same-transaction title/metadata edits share its outer settlement. Those fields +are not classifier evidence, so no second recovery wrapper was added there. +The low-trust promotion route signals inside its own atomic source/product write. +A real HTTP promotion now also proves completion through the registered worker. + +Three further regressions failed for task status changes, assignment changes, +and reasserting blocked state. The task mutation owner now signals relevant +status/version, execution, assignment and blocker writes. The regression checks +that the new assessment reflects the task's actual version and does not overwrite +the board's status. The nested work-product case verifies no decision before the +outer commit, no decision after rollback, then successful reassessment after a +later commit. All paths return to driver-observed SQL quiet. + +Final owner lifecycle tests pass all 13 cases, including the real promotion route, +and server types pass. Evidence: /tmp/sleep50-products-before.log (six failures), +/tmp/sleep50-products-after.log (nine passes), /tmp/sleep50-tasks-before.log +(three failures), /tmp/sleep50-tasks-after.log (12 passes), and +/tmp/sleep50-products-final.log (13 passes) plus its final typecheck log. + +Affected consumers pass 79 cases across five files: work products, attachment +routes, low-trust HTTP routes, attachment reuse, and runner task attachments. +The first combined run passed 53 but failed two suites during embedded PostgreSQL +bootstrap, leaving 26 cases unrun. Those suites now accept validated dedicated +external test database URLs and explicitly close their clients. Their 26 cases +then pass on separate PostgreSQL databases. Low-trust route tests use their own +dedicated database because teardown deletes shared tables. Preserve the initial +failure in /tmp/sleep50-consumers.log and the successful rerun in +/tmp/sleep50-attachment-consumers-after.log. This is 79 passing consumer cases +across the two runs, not a green initial run or a whole-workspace result. + +Master was refreshed and remains 1fbbef45a8. The stack has not been reconciled +onto it yet. The worker remains unregistered and periodic reassessment remains. +Remaining producer coverage includes approval/interaction and run-event evidence, +case handoffs, native ownership and run-status transitions, completion-contract +changes and workspace-finalization barriers. Workspace recovery writes a separate +workspace_finalize success receipt; snapshot-only rows are not that barrier. +Continue the concurrent authority audit, independent-current-history/backlog cost, +lifecycle/failure evidence, startup/periodic wiring and full PR acceptance. + + +## Approval evidence producer checkpoint + +Approval references are classified from the current approval status and its +company/task link. A product can have changed after the selected assessment +while that approval is still pending or unlinked. The worker correctly goes +quiet with insufficient evidence, but must inspect that same candidate when +an approval is resolved or linked later. This does not add a new reassessment +condition: the existing changed-product predicate must still be satisfied. + +The existing `notifyApprovalWork` boundary now sends the same task scopes to +native reassessment as to governed review responses, before the approval/link +write. It already resolves linked task IDs for status changes and accepts +explicit task scopes for link/unlink and bulk link. No new transaction wrapper, +company scan, table, or provider policy was introduced. Unlinked approval creation +has no task to reassess; the subsequent link owns activation. The production +writer audit also found budget approval resolution and gateway formal-approval +linking use this boundary with the appropriate scope. Company deletion removes +the records rather than creating a surviving reassessment obligation. + +Four real PostgreSQL regressions failed before the notification: approval +resolution, single link, bulk link, and an approval inside a caller-owned outer +transaction. With the notification they pass, including no premature decision +before outer commit, no decision after rollback, and driver-observed SQL quiet +before activation and after settlement. The full lifecycle file now passes 17 +cases. Existing approval service and route-idempotency consumers pass another +19 cases (36 total in the combined run). Server typecheck passes. + +Evidence: `/tmp/sleep50-approval-before.log` and `.status` (four failures), +`/tmp/sleep50-approval-after.log` and `.status` (36 passes), and +`/tmp/sleep50-approval-types.log` and `.status` (pass). The before and after runs +used separate freshly migrated PostgreSQL test databases. This is local producer +acceptance, not full slice or remote PR acceptance. + +Interaction evidence remains next: confirmation/question paths signal through +chat milestones, while other interaction outcomes signal governed review response +directly. Their common activation boundary must cover reassessment without teaching +the scheduler interaction policy or coupling it to a review-specific worker. +Several interaction mutations also lack either signal and require fact-level +audit before the old polling path can be removed. Run evidence, ownership, +completion contracts, workspace barriers, concurrent authority, discovery cost, +and startup registration remain outstanding as listed above. + + +## Interaction evidence producer checkpoint + +The interaction classifier reads status and result outcome, not just terminal +status. Real response regressions reproduced missed wakes for confirmation, +checkbox confirmation, question answers, completed item verdicts, and partial +item verdicts. The first checkbox fixture used `items`/`checkedItemIds` rather +than the real `options`/`selectedOptionIds` contract; after correcting it, that +case also reproduced the missed wake before production changes. + +`notifyNativeReviewState` is a small fan-out boundary for the two consumers of +changed review facts: governed response recovery and status reassessment. It +preserves their existing task/run/company scopes and delegates all discovery, +classification and authority to the owning workers. Approval, interaction, +chat-milestone, satisfied-connection and OAuth completion/rejection producers +reuse that boundary in their existing transactions. It does not add database +queries or a transaction wrapper at these call sites; the existing shared intent +registration still owns its normal transaction-tracking cost. + +Confirmation/question paths already use the milestone boundary; other interaction +outcomes use the new shared review-state boundary directly. Connection-intent +creation/supersession signals before insert in its existing transaction. Generic +non-question/non-confirmation creation signals reassessment before insert. +Partial item verdicts need reassessment even though governed response recovery +still waits for completion: their persisted `outcome: resolved` is already +accepted by the evidence classifier. This migration preserves that existing +classification policy rather than silently changing it while removing polling. + +Direct-writer audit: secret proposals, tool-action review, gateway outcome +projection, conversation promotion, and obsolete-policy review retirement already +signal milestones. Automatic completion retirement and native status commits +already signal reassessment/finalization. Confirmation comment enrichment shares +its outer transaction with ordinary service resolution. Ownership-change connection +expiry is called only through issueService's owned transaction. OAuth callback +acceptance/rejection and satisfied-connection retirement now use the shared helper. +Payload-only connection-phase updates, consumed-action stamps, retry annotations +on already accepted confirmations, and dismissal attention fields do not change +the classifier's status/outcome facts. Native response materialization uses the +existing finalization boundary. Authorization-start's reopening/creation path +and native superseded-review caller ownership still need final audit; do not infer +complete producer coverage from this checkpoint. + +All 22 reassessment lifecycle cases pass, with driver-observed SQL quiet before +and after real response activation. Server typecheck passes. Existing interaction +service, tool-access/OAuth, and satisfied-connection suites pass 625 cases across +three separate freshly migrated PostgreSQL databases. Evidence: +`/tmp/sleep50-interaction-before.log` (four missed-wake failures and one invalid +fixture), `/tmp/sleep50-interaction-checkbox-before.log` (corrected missed-wake +failure), `/tmp/sleep50-interaction-after.log` and `.status` (22 passes), +`/tmp/sleep50-interaction-types.log` and `.status` (pass), and +`/tmp/sleep50-interaction-consumers.log` and `.status` (625 passes). + +The milestone fan-out also reaches existing run-status producers, but that is not +proof that all native ownership/run-event writers are covered. Workspace barrier +producers, remaining run/contract/case-handoff facts, concurrent authority guards, +current-history/backlog query cost, startup registration and remote PR acceptance +remain incomplete. Periodic reassessment is deliberately still enabled. + + +## Workspace finalization barrier producer checkpoint + +The retired-review path defers when the latest `workspace_finalize` record has +not succeeded. It deliberately leaves no recovery timer for an in-progress or +failed barrier, so the actual producer must reactivate it when that fact changes. +Two PostgreSQL regressions reproduced missing reactivation through the normal +workspace-operation recorder: failure followed by a later successful export, +and a successful write inside a caller-owned transaction. + +The recorder now routes its initial state insert, successful settlement and failed +settlement through one local state-write boundary. Only `workspace_finalize` +operations with a run binding register reassessment intent; other phases retain +their existing writes. The added transaction contains only the state write and +notification. Log I/O, the provider callback, and progress/lease updates remain +outside it. Existing caller-owned transactions retain outer commit semantics. +The success test keeps the real callback suspended while the worker goes SQL +quiet, then completes it and observes task completion. The transaction test +proves no decision before outer commit, no decision after rollback, and successful +activation after a later commit. + +The separate retained-workspace recovery success receipt now signals in its +existing transaction before the receipt insert and generation update. A test +runs real retained recovery with temporary files and verifies that the delivered +notification's reader observes a committed `workspace_finalize: succeeded` row. +The historical unsafe-export path already had the common finalization signal; +it now registers that intent before its barrier insert rather than afterward. +Snapshot and copy-only records use other phases and do not satisfy this barrier. +Runtime-control orphan reconciliation operates on action-stamped controls; normal +native workspace finalization uses its own ownership metadata instead. + +All 24 worker lifecycle cases pass, and server typecheck passes. The workspace +operation reconciliation, retained workspace recovery and native status arbiter +corpus suites pass 170 cases across three fresh PostgreSQL databases. The two +workspace suites now accept dedicated, validated external test database URLs; +restart tests reuse that URL, and clients close before fixture teardown. This +uses the known-working local PostgreSQL installation because embedded bootstrap +failed in earlier acceptance attempts; no product database is used. + +Evidence: `/tmp/sleep50-workspace-before.log` and `.status` (two failures), +`/tmp/sleep50-workspace-after.log` and `.status` (24 passes), +`/tmp/sleep50-workspace-consumers.log` and `.status` (170 passes), and +`/tmp/sleep50-workspace-final-types.log` and `.status` (pass). All launched +commands at this checkpoint reached terminal status. These are local checks, +not remote PR acceptance or evidence of host suspension. + +Next: finish run-event/case-handoff, completion-contract, ownership and remaining +interaction producer audit; then verify commit-time business authority, discovery +cost/backlog behavior, lifecycle wiring and remote PR gates. Current scheduled +commit guards fence coordinator/run ownership and source selection, but the +retired-review business-fact checks still run before the commit transaction. +The old periodic reassessment remains enabled until this full slice is ready. + + +## Authoritative run-event producer checkpoint + +Event references may become accepted/rejected only when a persisted PRP event +has `sourceKind: control_plane` and the corresponding evidence verdict. The +classifier now exposes that existing pure verdict interpretation to the append +boundary, including its existing whitespace normalization. The append boundary +registers a run-scoped reassessment signal before the event write for native +runs. It does not look up a task context or scan evidence consumers to append +an event. Non-verdict events retain the ordinary atomic single-statement path; +runner-authored claims do not activate reassessment. + +The writer audit found three insertion paths. The native coordinator store accepts +only `sourceKind: runner` and rejects control-plane events, so its append path +cannot produce an authoritative verdict and needs no added signal. Scheduled +retry cancellation inserts ordinary lifecycle diagnostics without PRP evidence. +The shared append path owns the supported control-plane verdict writes. Event +rows are otherwise immutable in production; agent/company deletion owns removal +of their run history rather than producing new evidence for surviving runs. + +Four real-append regressions failed before the fix: ordinary append, native-source +append, outer transaction commit, and authoritative evidence after ordinary/runner +logs. The repaired tests observe task completion, outer rollback/commit fencing, +no reassessment signal for ordinary or runner-authored logs, replay idempotency, +and driver-observed SQL quiet. All 28 lifecycle cases and six existing atomic +append cases pass. The atomic tests still prove the one-statement ordinary-log +path and preserve binding/concurrency behavior. + +The initial combined run failed three sequencing cases during embedded PostgreSQL +bootstrap, before assertions. They now accept a validated dedicated external test +database URL, clean only their own company fixture between cases, and close both +clients. All three then pass, including concurrent retry receipt deduplication, +concurrent event allocation, and conflicting replay without cursor drift. Final +server typecheck passes. This is 37 distinct passing cases across the runs, not +a successful initial combined run. + +Evidence: `/tmp/sleep50-events-before.log` and `.status` (four failures), +`/tmp/sleep50-events-after.log` and `.status` (34 passes, three bootstrap failures), +`/tmp/sleep50-events-sequence.log` and `.status` (three passes), and +`/tmp/sleep50-events-final-types.log` and `.status` (pass). All test handles at +this checkpoint are terminal. Case-handoff creation/cancellation, remaining +contract/ownership/authority producers and the full acceptance tasks listed above +remain outstanding; the old periodic reassessment path is still enabled. + +## Scheduled retired-review authority checkpoint + +Two commit-boundary regressions reproduce unsafe completion: after reassessment +reads a succeeded run, a competing writer can change its outcome to failed or +bind the task to a different run without changing the task's status version. +Both previously committed a `done` decision. The scheduled source contract now +identifies retired automatic-review reassessment, and the existing status +committer rechecks run success and task execution ownership under its existing +coordinator/task/run locks. It adds no outer transaction, lock order, provider +call, or database query. Explicit legacy reconciliation remains unchanged. + +The two regressions assert that the competing state survives, no new decision +commits, and reassessment succeeds once the original authority is restored. +All 115 status-corpus cases pass. The combined corpus/lifecycle run initially +finished with 142 passes and one failed lifecycle assertion: retained test +history needs multiple startup batches, so the first registered wake can be the +next batch rather than the target's future retry. The retry test now waits for +startup batches to settle while asserting that the target stays in review, +then verifies the exact saved deadline before and after restart. All 28 lifecycle +cases pass against that same retained-history database; final server types pass. +This is 143 distinct passing cases across runs, not a green initial combined run. + +Evidence: `/tmp/sleep50-authority-before.log` and `.status` (two failures), +`/tmp/sleep50-authority-after.log` and `.status` (115 corpus passes and 27/28 +lifecycle passes), `/tmp/sleep50-authority-lifecycle.log` and `.status` (28 passes), +and `/tmp/sleep50-authority-final-types.log` and `.status` (pass). All launched +handles reached terminal status. No remote acceptance or host sleep is claimed. + +Producer audit notes for the next checkpoint: + +- Handoff evidence is accepted for every persisted non-cancelled state. Delivery, + completion, leases and retries do not change that classification. Agent-created + handoffs require the source run to be running and own the task; their generated + UUID is returned after creation. Board requests can use the current execution + binding. Do not add wake traffic to every handoff lease renewal. +- Task mutation cancellations are in the issue transaction, which also signals + reassessment. Direct handoff cancellation, delivery cancellation, and run-control + cancellation still require their own transaction/activation audit; no complete + handoff producer-coverage claim is made here. +- Superseded external-review withdrawal is called by the status committer inside + its decision transaction, which already signals native finalization and chat + milestones. OAuth authorization-start separately reopens/creates a pending + confirmation; its interaction writes remain to be audited. +- The evidence classifier checks target-agent existence, not invokable status. + Do not add status-change fanout based only on the name of that check. +- Completion-contract creation is serialized with the per-company/task advisory + lock in `ensureNativeCompletionContract`. Checking the latest revision before + the commit transaction is still insufficient. Audit its callers' lock order + before extending the scheduled commit guard. + +Latest-contract, workspace-barrier, governance, dependency and other business-fact +races remain open, along with remaining producers, query cost/backlog proof, +startup wiring, periodic removal, publication and PR gates. The full original +stack scope, older unresolved gates and master reconciliation remain outstanding. + +## Completion-contract serialization checkpoint + +The completion-contract writer is `ensureNativeCompletionContract`. Production +callers are native run preparation and heartbeat runtime selection; both select +or ensure the contract before their later run-state mutation. The writer owns a +per-company/task advisory transaction lock, and revisions are append-only. The +scheduled retired-review commit now calls the owner's +`lockCurrentNativeCompletionContract` before taking coordinator/task/run row +locks. The helper tries that same advisory lock without waiting and verifies the +expected contract is still the latest revision. A busy writer or replaced +contract returns the existing reassessment race outcome. The lock remains held +through the status transaction. Explicit legacy reconciliation is unchanged. + +This order matters: inserting a contract acquires a foreign-key lock on its task. +Trying to lock the task before recognizing an in-flight contract writer can wait +behind that writer. The before-fix tests reproduced a stale `done` decision after +a newer contract committed, and a 30-second timeout while a real contract writer +held an uncommitted revision. The concurrency fixture now also applies a two-second +statement timeout to the status transaction so a future blocking regression can +unwind and release its competing writer rather than waiting for suite timeout. +The writer uses the real service inside a caller-owned transaction; it is rolled +back only after reassessment has returned its expected retry outcome. + +The updated tests prove no status decision commits against either race, a newer +committed contract leaves the old result dormant on subsequent inspection, a +rolled-back revision leaves no row, and restoring the original authoritative +contract allows completion. The contract key is factored privately in its owning +module so the writer and guard cannot diverge. No scheduler-specific SQL lock key +or provider policy is introduced into shared scheduling. + +All 117 status-corpus and 28 worker lifecycle cases pass in one combined run +(145 total), including the retained-history startup deadline regression. Server +typecheck passes. Evidence: `/tmp/sleep50-contract-before.log` and `.status` +(stale-completion failure and writer timeout), `/tmp/sleep50-contract-after.log` +and `.status` (145 passes), `/tmp/sleep50-contract-types.log` and `.status` +(pass). All commands have terminal status. + +Workspace-barrier and governance/dependency races, remaining producer coverage, +query-cost/backlog proof, startup registration, periodic removal and full PR +acceptance remain outstanding. The existing native workspace finalization owner +already has a separate physical/durable ownership protocol; audit and reuse that +boundary before adding any barrier fence. No slice 50 PR, merge, deployed sleep +or completion of the overall stack is claimed by this checkpoint. + +## Workspace-barrier commit checkpoint + +Three regressions reproduced stale `done` decisions after the selected workspace +barrier changed to failed, after a newer running finalization operation committed, +and while another transaction held an uncommitted failure update to the selected +barrier. The scheduled retired-review guard now reads the latest finalization +barrier inside the existing status transaction and takes `FOR SHARE NOWAIT` on +that row. Non-success or lock contention yields the existing retryable race +outcome. A subsequent settled failed/running barrier is dormant until a producer +changes it. Missing-barrier semantics remain unchanged from the legacy path. + +The transaction already holds `FOR UPDATE` on the task and native run. New +`workspace_operations` rows reference the run through its foreign key, so inserts +cannot commit past those locks; the latest-row share lock additionally fences +status updates/deletion on an existing barrier. NOWAIT avoids waiting behind a +writer that could need a row already held by the status transaction. The ordinary +workspace recorder and recovery receipt producers remain unchanged by this fix; +no advisory namespace, provider call, or recurring probe was added. + +All 120 status-corpus and 28 lifecycle tests pass together (148 total), with server +types passing. The race tests preserve the competing state, assert no new status +decision, verify dormancy for a persisted failed/running barrier, and verify +completion after the original successful barrier is restored. The concurrent +writer rolls back only after recovery returns; its two-second statement timeout +bounds a future blocking regression. Existing lifecycle cases still exercise the +real recorder's failed/successful receipts and outer commit/rollback behavior. +Evidence: `/tmp/sleep50-barrier-before.log` and `.status` (three failures), +`/tmp/sleep50-barrier-after.log` and `.status` (148 passes), and +`/tmp/sleep50-barrier-types.log` and `.status` (pass). All handles are terminal. + +Physical workspace ownership is distinct from this receipt check. The existing +native owner uses an advisory lock plus a durable run-profile token, removes only +its own token after the action joins, and retains the token if cleanup cannot be +confirmed. Its admission/release interaction with reassessment still needs audit; +this checkpoint does not assert that a succeeded receipt alone proves physical +ownership released. Governance/dependency facts, remaining producers, discovery +cost, startup/periodic migration and remote acceptance remain open, as does the +full original stack scope. + +## Physical workspace ownership checkpoint + +Four regressions reproduced completion while physical workspace authority was +still outstanding: a live owner at startup, an unverified persisted owner at +startup, a real owner acquired after reassessment's read, and an owner receipt +written after that read. Scheduled retired-review reassessment now treats a +persisted owner as dormant. At commit it tries the existing physical-finalization +advisory lock before acquiring row locks, then rechecks owner presence on the +locked run. This closes the lock-before-receipt acquisition window as well as +retained ownership after a disconnected lock connection. The lock/presence +helpers remain with native workspace ownership/recovery; no provider policy is +added to scheduling. Explicit legacy reassessment remains unchanged. + +With only those guards installed, the real-owner lifecycle test reproduced a +second gap: the task stayed in review after the owner released. Normal release +now registers reassessment intent in a short transaction before removing only +its own token. Physical work, assertion of ownership and joining the copyback +callback remain outside that transaction. Failed/uncertain cleanup retains the +existing durable owner and joined-token behavior. Verified operator recovery +registers the same run/task signal in its existing transaction before removing +the verified owner. Its physical-lock check now uses the same owner's helper. +The source audit found these two removal paths; reconstruction preserves the +owner field rather than clearing it. + +The real-owner lifecycle test proves task preservation and driver-observed worker +SQL quiet while ownership remains, reactivation after release, and quiet after +completion. A separate unverified-owner case proves no completion or recurring +worker query despite a successful workspace receipt. Commit-race tests preserve +competing ownership and complete only after it is released/restored. The existing +verified-recovery test additionally reads the run from the reassessment callback +and sees the committed released profile, with the coordinator unchanged. + +All 122 status-corpus, 30 reassessment lifecycle and 65 workspace finalization +consumer cases pass together (217 total), and server typecheck passes. Existing +consumers cover live ownership, disconnected lock sessions, retained joined-owner +receipts, orphan copyback children, verified stop/replay, and current-task fencing. +The workspace suite now accepts a validated dedicated external test database URL, +uses it for restart/competing clients, and closes its primary client on teardown. +It ran in its own fresh database: its connection-termination tests must not share +a database with another suite. No production database or provider was used. + +Evidence: `/tmp/sleep50-physical-before.log` and `.status` (four failures), +`/tmp/sleep50-physical-guard.log` and `.status` (release missed wake after guard), +`/tmp/sleep50-physical-after.log` and `.status` (217 passes), and +`/tmp/sleep50-physical-types.log` and `.status` (pass). All handles are terminal. +Governance/dependency races, remaining producer audits, discovery cost/backlog, +startup registration/periodic removal and remote acceptance remain open. No PR +or merge is claimed for slice 50, and the full original stack remains incomplete. + +## Governance commit checkpoint + +Seven regressions reproduced stale completion after a pending execution-policy +hold, a newly linked pending approval, a reopened approval, a new pending +interaction, and uncommitted changes to an approval, interaction or question's +human-direction history. Reassessment's selected retired-review source now carries +the governance gate used by its policy decision. The committer rechecks that gate +under its existing task/run locks and rejects a changed kind/identity. + +`lockNativeGovernanceSources` belongs to the governance service. It share-locks +all existing interactions on the task, linked approval rows and approval-link +rows, and comments when the task has ordinary questions. It deliberately includes +resolved rows, since reopening them can introduce a new gate. Question history +depends on human comments, including deletion/trust fields, so locking only the +question row would be insufficient. Every added source lock uses NOWAIT to avoid +waiting behind a writer that may need the already-held task/run locks. New linked +rows are fenced by their task foreign keys. A pending issue-owned execution stage +is already protected by the task lock and takes priority without locking the +lower-priority source rows. Existing governance policy and ordinary callers of +`pendingNativeGovernance` are unchanged. + +The tests assert that no decision commits against changed or uncommitted facts, +the task remains in review, and completion succeeds after the gate is removed or +the competing writer rolls back. The concurrent writers have a two-second +statement bound on the recovery transaction so a future blocking regression can +unwind. All 129 status-corpus and 30 lifecycle tests pass together (159 total). +The question-history fixture was subsequently strengthened to use a valid question +payload; that case passes again and final server typecheck passes. + +Evidence: `/tmp/sleep50-governance-before.log` and `.status` (seven failures), +`/tmp/sleep50-governance-after.log` and `.status` (159 passes), +`/tmp/sleep50-governance-question.log` and `.status` (one focused pass), and +`/tmp/sleep50-governance-final-types.log` and `.status` (pass). All handles are +terminal. These guards run only for actual scheduled retired-review reassessment, +not during idle discovery. Their task-history lock/query cost must be included in +the remaining cost audit; a passing functional test does not establish that cost. + +Dependency/readiness and remaining selected-authority/evidence races, remaining +producers (including handoffs and OAuth reopening), discovery/backlog costs, +startup wiring, periodic removal and complete PR gates remain outstanding. No +slice 50 PR or merge is claimed; the original full stack scope remains open. + + +## Dependency readiness commit guard (2026-10-26) + +Seven real database races reproduced unsafe completion: a new blocker relation, +a reopened blocker, an uncommitted blocker status writer, a newly recorded +unattributed workspace operation, a changed finalize receipt, an uncommitted +receipt update, and an uncommitted unattributed operation insert. Before the +repair all seven committed a status decision despite stale dependency readiness. +Evidence: /tmp/sleep50-dependencies-before.log and .status (exit 1). + +Dependency semantics remain in issueService.getDependencyReadiness. Its opt-in +lockSources mode requires an existing transaction holding the dependent task +FOR UPDATE. It locks relation/blocker rows FOR UPDATE NOWAIT, then relevant +execution workspaces FOR UPDATE NOWAIT and their existing operation rows FOR +SHARE NOWAIT. The workspace lock fences new operations through the existing FK, +including operations without an issue attribution. Ordinary readiness consumers +retain their previous query and locking behavior. The attributed/unattributed +operation selection and later successful finalization rules are unchanged. + +The scheduled retired-review commit compares the readiness boolean used by the +policy with this locked read; contention or a changed predicate defers through +the existing NativeStatusRaceError retry. It does not create a queue/table or +introduce an independent timer. Locks only cover the current task's prerequisites +and relevant workspaces. History-dependent locking cost still needs measurement +as part of the outstanding discovery/commit cost audit. + +Initial combined verification passed 135 corpus cases (including all seven new +races), but an older effect-state assertion compared unordered SQL results and +failed when physical row order changed. Both reads now order by effect ordinal. +The lifecycle suite also failed before setup because the invocation used an +incorrect external DB variable and fell back to broken embedded PostgreSQL. +These failures remain in /tmp/sleep50-dependencies-after.log/.status; neither was +counted as passing. The dependency consumer selection passed 15 cases, and the +initial typecheck passed. The consumer suite now supports the standard validated +dedicated external test database contract and closes its pool. + +Final verification (all commands terminal): +- /tmp/sleep50-dependencies-final.log/.status: 136 status corpus plus 30 worker + lifecycle cases, all 166 passing. Fresh corpus DB and retained lifecycle history. +- /tmp/sleep50-dependencies-consumers-final.log/.status: all 15 existing dependency + readiness/blocker consumer cases passing; other suites deliberately not selected. +- /tmp/sleep50-dependencies-final-types.log/.status: server typecheck passing. + +The seven regressions also restore or roll back the competing state and prove +that reassessment can subsequently complete. Remaining authority/evidence races, +producer coverage (including handoffs and OAuth reopening), task-history/backlog +cost, startup wiring, periodic removal and remote PR acceptance remain open. +No new PR or merge, and no whole-instance sleep claim. + + +## Evidence freshness at status commit (2026-10-26) + +All seven before-fix regressions committed stale completion when supporting +work-product review, approval, interaction, attachment, event, or case-handoff +state changed after classification and before commit. The work-product case also +holds a real uncommitted rejection until recovery returns, then rolls it back. +/tmp/sleep50-evidence-before.log/.status records seven failed assertions (exit 1), +not fixture/bootstrap failures. + +The assessment owner now exposes lockCurrentNativeWorkAssessmentEvidence. It +loads the exact persisted reassessment with its bound contract/result, then calls +the existing classifier with opt-in source locking. Classification rules remain +in evidence-classifier.ts, including reference parsing, accepted/rejected policy, +question/approval outcomes and attention-target existence. The scheduled committer +only compares the resulting normalized assessment and handles contention through +its existing race/retry outcome. Both retired-review and other scheduled +reassessment paths use this guard. Ordinary classification retains its original +unlocked behavior. No extra transaction encloses the committer/publication path. + +The classifier acquires SHARE NOWAIT on the referenced evidence rows and existing +attention target agents. The already-held task/run locks fence new task/run-bound +records through their existing FKs; the reference cache still deduplicates repeat +criterion/verification references. The assessment/contract/result records are +also share-locked. This protects mutable acceptance state and evidence deletion, +rather than assuming all durable evidence is immutable. Missing arbitrary agent +IDs have no row to lock; supported agent-creation identity/producer behavior and +other authority fields remain part of the outstanding producer audit. This is +not a blanket proof for independent SQL writers. + +The added read cost is one exact assessment/contract/result query plus the normal +classifier's distinct-reference and attention-target queries. It runs only for +an actual scheduled decision commit, not on idle history. Quantitative history, +backlog and full transaction cost acceptance remains outstanding; no performance +claim is inferred from the narrow regression runtime. + +After repair all 143 status-corpus, 30 worker-lifecycle and four classifier +consumer tests pass (177 total), as does server typecheck. The seven regressions +also restore/roll back evidence and prove subsequent completion. Evidence: +/tmp/sleep50-evidence-after.log/.status and /tmp/sleep50-evidence-types.log/.status, +both exit 0; all launched commands are terminal. Ordinary classifier consumer +cases still run without a locking API in their fake query builder. + +Remaining: producer/authority coverage, query/history/backlog cost, startup +integration, removal of this periodic duty, PR publication and remote acceptance. +The full recurring-work stack and its older unresolved gates remain open. + + +## OAuth authorization-start producer checkpoint (2026-10-26) + +The direct OAuth authorization-start owner creates a confirmation or reopens its +previous accepted/rejected card by idempotency key. That bypassed the interaction +service's native-review notifications. Three real tool-access service regressions +(creation, reopening, outer rollback) reproduced absent settled notifications: +/tmp/sleep50-oauth-before.log/.status, three failed assertions, exit 1. + +The existing card lookup/mutation and OAuth-state interaction link now share a +short database transaction. It awaits notifyNativeReviewState before writing, +with company/task scope. The shared hook wakes both governed review response and +status reassessment; consumers still select only actionable durable state. +Provider discovery, client setup and authorization remain outside the new short +transaction. Browser/user flows that do not create an agent authorization card +keep their existing path. Caller-owned transactions retain their outer boundary. +No provider-specific logic moved into recovery or scheduler code. + +All 472 tool-access consumer cases pass, including the new real nested-transaction +tests. They assert the inside/outside card states, no pre-commit notification, +one settlement notification per consumer, and rollback retaining the prior card +and no OAuth state. Rollback settlement releases the intent for reconciliation; +it is not represented as a committed card. All 30 reassessment lifecycle cases +pass against retained history, and server typecheck passes. Evidence (all terminal): +/tmp/sleep50-oauth-after.log/.status, /tmp/sleep50-oauth-lifecycle.log/.status and +/tmp/sleep50-oauth-types.log/.status, all exit 0; 502 distinct tests in this checkpoint. + +A fresh origin/master fetch still resolves to +1fbbef45a8d399771add7a27a04b12f3a20e264c. No source changed during acceptance. + +Additional handoff audit evidence: direct operator cancellation flows through +cancelPendingCaseHandoffsForIssue; run-control/task cancellation uses +heartbeat/case-handoff-lifecycle; delivery can cancel a fenced claim in its own +settlement transaction. The classifier accepts all persisted non-cancelled +handoff states. Those delivery/lease transitions are not independently positive +evidence triggers, so do not fan out every lease renewal. Creation assigns its +UUID inside the owning transaction. The previous checkpoint now fences stale +handoff evidence at commit. Complete activation/quiet-state proof for the +remaining handoff/authority paths is still outstanding; no blanket producer- +coverage claim is made from this search. + +Remaining acceptance continues to include producer/authority audit, discovery +and commit costs, backlog and lifecycle coverage, startup registration, periodic +removal, PR publication and exact-head remote gates. The full original stack +scope and older unresolved gates remain active. + + +## Selected-history, backlog and commit cost checkpoint (2026-10-26) + +The earlier same-task/superseded-decision benchmark did not establish discovery +cost for selected current history. A dedicated PostgreSQL 17 database now holds +10,000 separate task/run/contract/result/assessment/decision/product bindings. +Every task initially selects its own completed decision; accepted products predate +the assessment. This is distinct from 10,000 obsolete decisions on one task. +Fixture: /tmp/sleep50-scale-fixture.sql and .log/.status; database +paperclip_sleep50_scale_20261026_test (snapshot of the completed corpus database, +plus a separate synthetic company). All fixture inserts and measurements are +local disposable test data, never deployed/customer data. + +At source 51f305f1d1 with the new backlog regression pending: +- Startup discovery: 98.7 ms, three queries, 246 candidates from the original + corpus fixtures. None of the 10,000 completed synthetic tasks was a candidate. +- Synthetic company: zero candidates, one query, 34.8 ms (warm EXPLAIN execution + 33.5 ms); exact task/run scopes: one query each, about 1.8/2.1 ms. +- Changing all synthetic tasks' authoritative status creates 10,000 actual + reassessment candidates. Reconstruction: 101 keyset queries, 1.56 seconds, + approximately 2.05 MB serialized entries. This is total startup/backlog cost, + not a constant-space claim; the coordinator retains outstanding entries. +- Drain after two pages stops at two queries; an already-aborted request makes + zero queries. Interruption is between pages, not cancellation of an active SQL. +- First/last backlog entries both commit real reassessments, disappear from + scoped discovery, and do not replay: 37/36 SQL statements, about 32/15 ms. + +Evidence: /tmp/sleep50-scale-measure.mts, .log/.status (exit 0), and +/tmp/sleep50-scale-{startup,company,issue,run}.plan. No new index was justified by +this selected-history measurement. It is local PostgreSQL evidence, not a remote +network latency or fleet savings claim. + +A second fixture exercises retired-review reassessment with 10,000 resolved +interactions, 10,000 comments, 10,000 linked approved approvals, 1,000 completed +prerequisites sharing a workspace, and 10,000 successful workspace operations. +One historical question forces the comment source-lock path too. Scoped discovery +is one query/about 5.9 ms; complete guarded reassessment uses 83 SQL statements +(including nine NOWAIT queries), takes about 126 ms, completes the task, and then +returns zero candidates. Source locks are scoped to this task and its referenced +prerequisites/workspaces; work grows with that history. The measurement does not +assert that a remote database round trip has the same cost. + +Two failed benchmark attempts are retained. The first manually assigned version +2 while a status-update trigger advanced the actual version to 4, so it measured +the stale-status path and failed the intended completion assertion. A repair that +rewound that already-processed fixture then reused its existing append-only +assessment and hit status_decisions_company_assessment_uq. Neither failure was +waived or changed in production code. A fresh untouched task with the actual +trigger-generated version exercised the intended retired-review path. Evidence: +/tmp/sleep50-task-history-measure.log/.status and +/tmp/sleep50-task-history-final.log/.status (failures), fixture/repair logs under +/tmp/sleep50-task-history-*, and /tmp/sleep50-task-history-fresh.log/.status (exit 0). + +The permanent lifecycle suite now seeds 103 actual retired-review obligations, +verifies full discovery, one-page stop on admission loss, zero-query abort, +coordinator dispatch across batches, exactly one additional decision per task, +no rediscovery, and SQL quiet after settlement. All 31 lifecycle cases pass against +retained history, and server typecheck passes: +/tmp/sleep50-scale-lifecycle.log/.status and /tmp/sleep50-scale-types.log/.status. +All handles are terminal. Startup registration/periodic removal, remaining +producer/authority audit and remote acceptance still remain; the full stack +objective and its earlier unresolved gates are not complete. + + +## Startup integration and periodic ownership, 2026-10-26 + +The worker now registers through heartbeat under enabled maintenance, with the +same warm-standby, idle-drain and stopped gates as its sibling owners. The shared +coordinator owns shutdown and restart reconstruction. The periodic orphan reaper +passes `skipNativeStatusReassessment`; explicit/manual and one-time startup +recovery retain the existing reassessment entry point. + +The old reassessment call also supplied a superseded-decision early return before +workspace recovery. Removing the whole call without preserving that guard would +allow obsolete history into unrelated recovery. The reassessment owner now shares +that pure authority predicate and exposes a narrow guard for the periodic path. +It makes no queries for the usual matching-decision case, and reads only selected +decision/assessment authority when IDs differ; it does not classify evidence or +create assessments. Independent workspace settlement and committed result replay +remain owned by finalization recovery. Reassessment's dormant outcome no longer +implicitly suppresses those separate duties in the periodic path; their own +ownership and physical-resource guards remain applicable. + +Before the wiring change, the new regressions failed because periodic recovery +completed the task itself and startup never registered the worker. Historical +supersession was verified with the new option both enabled and disabled. Evidence: +`/tmp/sleep50-startup-before.log/.status` (two failures, four passing selected cases). +After the repair, all 246 corpus/startup/lifecycle cases pass and server typecheck +passes: `/tmp/sleep50-startup-after.log/.status` and +`/tmp/sleep50-startup-types.log/.status`. The existing native finalization and +retained cleanup consumer suite passes all 32 cases in a dedicated database: +`/tmp/sleep50-startup-consumers.log/.status`. All command handles are terminal. + +This integrates only reassessment ownership. The aggregate orphan/finalization +sweep still runs for the other listed duties, so this is not whole-process SQL +quiet or host suspension. Remaining producer/authority audit, final acceptance, +publication and remote PR gates remain open, along with the older stack findings +and every remaining recurring-server obligation. No merge occurred. + + +## Producer closeout and PR validation checkpoint + +The final source audit distinguishes reactivation from negative evidence. Handoff +creation uses a fresh identity in its transaction; delivery/renewal/completion +preserve accepted evidence. Cancellation can only revoke it and is locked by the +scheduled evidence guard. Existing task cancellation/assignment writers signal +through the task boundary. No permanent lease-renewal wake was introduced. +Target-agent existence affects attention validation, not criterion/verification +acceptance, so lifecycle status changes do not activate the three reassessment +predicates. Arbitrary missing identities have no row lock; the worker contract +states this limit rather than claiming a general identity-insertion fence. +Child completion remains a fresh-turn obligation enforced at status commit. +Latest-contract creation can invalidate an old result, not make it eligible again. +The only dormant retired-review prerequisites that can be restored in normal +operation are current run/task authority and workspace settlement/ownership; +their real transaction producers and lifecycle tests are recorded above. + +At 0337779e6d, workspace typecheck and build pass, in addition to the 278 selected +integration/consumer tests. Evidence: /tmp/sleep50-workspace-types.log/.status and +/tmp/sleep50-build.log/.status. All handles are terminal. Fresh origin/master is +still 1fbbef45a8d399771add7a27a04b12f3a20e264c; the PR base remains slice 49. +No duplicate reassessment PR was found; related automatic-review migrations are +#17679 and #17680. Full-stack master reconciliation remains separate unfinished +work; the PR is not represented as a direct-master diff. + +A direct disposable embedded initdb probe explains earlier full-suite failures: +FATAL could not create semaphores, No space left on device (semget), before tests. +/tmp/sleep50-pg-init-probe.log/.status preserves the failure. Read-only IPC/process +inspection found a candidate orphan group, but automatic approval review rejected +its removal because it changes shared OS resources and PID checks can race. +Explicit approval for those exact IDs is pending; no cleanup was executed. This +blocks local embedded-database acceptance, not dedicated PostgreSQL validation or +remote PR validation. The full goal is still incomplete and no merge is authorized. + + +## Review repair: direct claim release + +Draft #17709 at 91a6c14170 passed all 52 CI checks but Greptile scored 4/5 and +identified a missing adminForceRelease wake (discussion 4376190896). This invalidates +the preceding producer-closeout claim for direct task-claim writers. The ordinary +task update method did not cover their separate transactions. + +Six real regressions fail before repair: operator force-release, ordinary release, +terminal execution cleanup, terminal checkout cleanup, the queue's actual release +adapter, and outer force-release commit after rollback. Each starts with a retired +review dormant behind a different execution owner. The missing notification leaves +it untouched after that owner is released. /tmp/sleep50-release-before.log/.status +retains all six failures. + +The release owners now await task-scoped reassessment intent before their existing +writes. The audit also covers queued-comment release, watchdog settlement, legacy +and native recovery, safe replacement, tree restoration and stale-lock cleanup. +Existing run-dispatch and queue branches that already notify task-wide milestones +retain that common hook. Case-handoff release already delegates to issueService +update. Rare member-archive bulk reassignment signals its company. No database +transaction wrappers, recurring retries or provider logic are added. + +The first repaired run passed 36/37 cases. Ordinary release deliberately clears +the assignee, which advances task authority; the worker correctly appended a +preserve decision instead of completing the task. The test now requires that +preserved status, cleared assignee and exactly one additional decision. The initial +incorrect completion expectation remains recorded in /tmp/sleep50-release-after.log. +All 37 lifecycle cases now pass, including outer rollback/commit and SQL quiet. + +Affected consumer suites pass: queue 25, queued comments 5, execution control 5, +safe replacement 59, access 18; native executor/restart, legacy recovery and tree +unit selection 829 pass with 15 known embedded-DB cases excluded. Thus 978 distinct +selected cases pass after repair. This is not full local-suite acceptance. Server +typecheck passes. Evidence: /tmp/sleep50-release-final.log/.status, +/tmp/sleep50-release-{queue,queued_comment,control,safe,access,runtime,types}.log +and .status. All launched command handles are terminal. Fresh-head remote CI and +review remain required. The full stack and its earlier unresolved gates remain open. + + +## Current master and landed parent qualification (2026-10-29) + +Parent #17702 merged from exact reviewed head +`902817c46675d71fb17324683f39b9a39d6135e0` as +`79ce86630a79dad65e3b753c13e3d49e22bbca2f` with all 57 checks clean and +Greptile 5/5. No Ponytail review was present, which the authorized merge policy +allows. The normal parent merge retains 12661 parent/unrelated tree entries and +35 owned entries exactly. Twenty-one overlapping paths retain Git's automatic +three-way result. Five original conflicts are reviewed explicitly: scoped wake +fanout, conversation/session source locks, the extracted reassessment block, +and the unsafe-workspace omission-barrier notification. The complete held-child +tree is unchanged when reconciling the parent's actual squash ancestry. + +The current parent's required quality-review policy also moves with the +reassessment owner. A required review is a business gate, not an operational +failure. Two real regressions exercise this on retired-review reassessment and +on committed work-product approval. Each optional control passes before repair; +each required case instead retains retries, reports repeated +`required_quality_review_incomplete` exceptions and reaches the unchanged +15-second limit. Porting the retired-review policy fixes the first path. +Reading the current task's quality policy for all completion-producing +reassessments fixes the evidence path too. Both required cases now record the +intended in-progress decision and become SQL quiet; neither bypasses the review. +The same assertions and time limits are retained. + +Two more real SQL regressions cover dormant reassessment after three coarse +company-only milestone/finalization hints. A controlled negative baseline removes +only the ordinary issue/run guards; it reproduces nonzero response-history +reconstruction. Restoring the guards byte for byte passes the same cases with +zero reconstruction queries. Routine wake paths now remain scoped across all +three workers; the explicit company authority boundary still wakes genuine global +changes. Actual coalesced input and concurrent Photon recovery exercise all three +queues. Ten active Photon assertions produce zero hints/SELECTs per worker; +three concurrent attention assertions produce one wake/SELECT, then return quiet. +Paperclip callbacks and PostgreSQL are real; external Photon transport is mocked. + +Broad source `9fddb3f0b1157df32b64fb1f22e9d6e9624470e8` passes **2731 +distinct cases across 28 complete files**, with no skipped cases: 2458 across +27 producer/owner/authority/workspace/quality files in 394.43 seconds, then all +273 governed-response cases in 53.74 seconds. Nineteen dedicated PostgreSQL 17 +fixtures migrate through 0413. Eighteen embedded-only fixture invocations use +the existing startRecoveryPostgres external-host helper, preserving a fresh +owned database per call and normal cleanup. All 22 temporary import adaptations +restore exactly and all 28 captured source hashes match. This qualifies external +PostgreSQL behavior, not embedded bootstrap on this semaphore-exhausted host. + +The later evidence-trigger fix changes only the reassessment owner and adds its +two regressions. Final runtime source +`5ea3b0fd016bf7e63ba82a7044825a46a0cd2082` passes **1211 distinct cases in +five complete affected files** in 108.23 seconds: 43 worker cases, 145 status +corpus cases, 32 finalization-recovery cases, 968 native-session cases and 23 +quality-review cases. Four dedicated fresh 0413 databases plus the quality +suite's own fresh helper fixture use PostgreSQL 17. Its one temporary import +restores exactly; all five hashes match. The other 27 broad-run test files are +unchanged. These results keep their separate source scopes; overlapping cases +are not cumulatively counted or described as a fresh complete 2731-case rerun. + +Database build, production server types/build, module boundaries and full-SHA +migration order pass. Fourteen added comments in six files were removed with +identical nontrivia token streams and no directives removed. The final added- +comment check is empty. The database tree is identical to the landed parent: +no migration, snapshot generation or queue table is introduced here. + +Earlier failures remain part of the record, including the two quality retry +loops, the controlled scope negatives, and inherited historical host/fixture +limitations. The local evidence-trigger runner reused an earlier composition-tree +metadata field; its addendum identifies the precise 9fddb production baseline +and before/after owner hashes without altering outcomes or test hashes. The first +actual-squash proof attempt lacked the newly created Git object; fetching the +object and rerunning establishes exact parent/tree identity before mutation. +No application failure was classified as infrastructure from a successful retry. + +This is targeted local acceptance, not a full repository-suite or deployed-sleep +claim. Fresh exact-head CI, Greptile 5/5, and resolution of actionable reviews +remain required before merging this PR. Other workspace/session/cleanup duties +and Cloud sleep eligibility remain separate. diff --git a/server/src/__tests__/heartbeat-run-event-sequencing.test.ts b/server/src/__tests__/heartbeat-run-event-sequencing.test.ts index d8e7bd0b9cb..98a002f8557 100644 --- a/server/src/__tests__/heartbeat-run-event-sequencing.test.ts +++ b/server/src/__tests__/heartbeat-run-event-sequencing.test.ts @@ -1,3 +1,4 @@ +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; import { describe, expect, it } from "vitest"; import { eq, sql } from "drizzle-orm"; import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; @@ -15,6 +16,15 @@ import { } from "../services/heartbeat-run-events.js"; import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_EVENT_SEQUENCING_TEST_DATABASE_URL"); +async function clearExternalFixture(db: ReturnType, companyId: string) { + if (!externalUrl) return; + await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.companyId, companyId)); + await db.delete(heartbeatRuns).where(eq(heartbeatRuns.companyId, companyId)); + await db.delete(agents).where(eq(agents.companyId, companyId)); + await db.delete(companies).where(eq(companies.id, companyId)); +} + describe("P6-11..13 / P6-17 canonical event allocator", () => { it("keeps native cursor responses bounded with long histories and a late gap fill", async () => { const temporary = await startEmbeddedPostgresTestDatabase("paperclip-native-cursor-budget-"); @@ -73,9 +83,9 @@ describe("P6-11..13 / P6-17 canonical event allocator", () => { }, 60_000); it.each([false, true])("deduplicates concurrent retry exhaustion (historical receipt: %s)", async (historical) => { - const temporary = await startEmbeddedPostgresTestDatabase("paperclip-exhaustion-events-"); - const db = createDb(temporary.connectionString); - const otherDb = createDb(temporary.connectionString); + const temporary = externalUrl ? undefined : await startEmbeddedPostgresTestDatabase("paperclip-exhaustion-events-"); + const db = createDb(externalUrl ?? temporary!.connectionString); + const otherDb = createDb(externalUrl ?? temporary!.connectionString); const companyId = "20000000-0000-4000-8000-000000000001"; const agentId = "20000000-0000-4000-8000-000000000002"; const runId = "20000000-0000-4000-8000-000000000003"; @@ -129,13 +139,16 @@ describe("P6-11..13 / P6-17 canonical event allocator", () => { ...event, companyId: "20000000-0000-4000-8000-000000000099", retryExhaustion: exhaustion, })).rejects.toThrow("heartbeat_run_event_binding_mismatch"); } finally { - await temporary.cleanup(); + await clearExternalFixture(db, companyId); + await otherDb.$client.end({ timeout: 1 }); + await db.$client.end({ timeout: 1 }); + await temporary?.cleanup(); } }, 60_000); it("serializes concurrent writers and rejects conflicting replay without cursor drift", async () => { - const temporary = await startEmbeddedPostgresTestDatabase("paperclip-native-events-"); - const db = createDb(temporary.connectionString); + const temporary = externalUrl ? undefined : await startEmbeddedPostgresTestDatabase("paperclip-native-events-"); + const db = createDb(externalUrl ?? temporary!.connectionString); const companyId = "20000000-0000-4000-8000-000000000001"; const agentId = "20000000-0000-4000-8000-000000000002"; const runId = "20000000-0000-4000-8000-000000000003"; @@ -219,7 +232,9 @@ describe("P6-11..13 / P6-17 canonical event allocator", () => { })).rejects.toBeInstanceOf(HeartbeatRunEventConflictError); expect(await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, runId))).toHaveLength(48); } finally { - await temporary.cleanup(); + await clearExternalFixture(db, companyId); + await db.$client.end({ timeout: 1 }); + await temporary?.cleanup(); } }, 60_000); diff --git a/server/src/__tests__/issues-service.test.ts b/server/src/__tests__/issues-service.test.ts index 737039c5463..bbf94717ff3 100644 --- a/server/src/__tests__/issues-service.test.ts +++ b/server/src/__tests__/issues-service.test.ts @@ -57,6 +57,8 @@ import { } from "../services/execution-workspace-policy.ts"; import { buildAgentMentionHref, buildProjectMentionHref, MAX_ISSUE_REQUEST_DEPTH, type IssueWorkMode } from "@paperclipai/shared"; +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; + const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; @@ -4111,14 +4113,15 @@ describeEmbeddedPostgres("issueService.create workspace inheritance", () => { }); }); -describeEmbeddedPostgres("issueService blockers and dependency wake readiness", () => { +const dependencyTestDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_ISSUE_DEPENDENCIES_TEST_DATABASE_URL"); +(dependencyTestDatabaseUrl ? describe : describeEmbeddedPostgres)("issueService blockers and dependency wake readiness", () => { let db!: ReturnType; let svc!: ReturnType; let tempDb: Awaited> | null = null; beforeAll(async () => { - tempDb = await startEmbeddedPostgresTestDatabase("paperclip-issues-blockers-"); - db = createDb(tempDb.connectionString); + if (!dependencyTestDatabaseUrl) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-issues-blockers-"); + db = createDb(dependencyTestDatabaseUrl ?? tempDb!.connectionString); svc = issueService(db); await ensureIssueRelationsTable(db); }, 20_000); @@ -4139,6 +4142,7 @@ describeEmbeddedPostgres("issueService blockers and dependency wake readiness", }); afterAll(async () => { + await db?.$client.end({ timeout: 5 }); await tempDb?.cleanup(); }); diff --git a/server/src/__tests__/low-trust-red-team-routes.test.ts b/server/src/__tests__/low-trust-red-team-routes.test.ts index 68ef4c60ce3..598cdfd81cf 100644 --- a/server/src/__tests__/low-trust-red-team-routes.test.ts +++ b/server/src/__tests__/low-trust-red-team-routes.test.ts @@ -1,3 +1,4 @@ +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; import { randomUUID } from "node:crypto"; import { createServer } from "node:http"; import express from "express"; @@ -55,12 +56,13 @@ import { issueService } from "../services/issues.js"; import { heartbeatService } from "../services/heartbeat.js"; import { LOW_TRUST_QUARANTINED_BODY } from "../services/source-trust.js"; +const externalTestDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_LOW_TRUST_TEST_DATABASE_URL"); const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); -const describeEmbeddedPostgres = embeddedPostgresSupport.supported +const describeEmbeddedPostgres = externalTestDatabaseUrl || embeddedPostgresSupport.supported ? describe : describe.skip; -if (!embeddedPostgresSupport.supported) { +if (!externalTestDatabaseUrl && !embeddedPostgresSupport.supported) { console.warn( `Skipping embedded Postgres low-trust route tests on this host: ${ embeddedPostgresSupport.reason ?? "unsupported environment" @@ -885,10 +887,10 @@ describeEmbeddedPostgres( } beforeAll(async () => { - tempDb = await startEmbeddedPostgresTestDatabase( + if (!externalTestDatabaseUrl) tempDb = await startEmbeddedPostgresTestDatabase( "paperclip-low-trust-red-team-routes-", ); - db = createDb(tempDb.connectionString); + db = createDb(externalTestDatabaseUrl ?? tempDb!.connectionString); }, 20_000); afterEach(async () => { @@ -927,6 +929,7 @@ describeEmbeddedPostgres( }); afterAll(async () => { + await db?.$client.end({ timeout: 1 }); await tempDb?.cleanup(); }); diff --git a/server/src/__tests__/native-status-arbiter-corpus.test.ts b/server/src/__tests__/native-status-arbiter-corpus.test.ts index cf330f4105a..6a0d69db056 100644 --- a/server/src/__tests__/native-status-arbiter-corpus.test.ts +++ b/server/src/__tests__/native-status-arbiter-corpus.test.ts @@ -1,3 +1,8 @@ +import { pendingNativeGovernance } from "../services/native-runtime/native-governance.js"; +import { withNativeWorkspaceFinalizationOwnership } from "../services/native-runtime/native-workspace-finalization-ownership.js"; +import { ensureNativeCompletionContract } from "../services/native-runtime/completion-contracts.js"; +import { nativeStatusReassessmentWork } from "../services/native-runtime/native-status-reassessment-work.js"; +import { reassessNativeStatusFromSource } from "../services/native-runtime/native-status-reassessment.js"; import { NativeRunCoordinatorStore } from "../services/native-runtime/native-run-coordinator-store.js"; import { registerNativePendingEffectWork } from "../services/native-runtime/native-pending-effect-work.js"; import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; @@ -9,12 +14,22 @@ import { randomUUID } from "node:crypto"; import { readFileSync } from "node:fs"; import { fileURLToPath } from "node:url"; import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; -import { and, eq } from "drizzle-orm"; +import { and, eq, sql } from "drizzle-orm"; import { agents, + assets, + caseHandoffs, + cases, + heartbeatRunEvents, + issueAttachments, + approvals, + issueApprovals, + issueComments, agentWakeupRequests, companies, completionContracts, + executionWorkspaces, + issueRelations, createDb, heartbeatRuns, issueThreadInteractions, @@ -2451,6 +2466,514 @@ describe("P6-31 Section 18.13 executable status-authority corpus", () => { return seeded; } + it.each(["future", "claim", "unknown claim", "assessment", "result", "phase", "decision", "effects", "ownership", "run outcome", "task execution", "contract revision", "contract writer", "workspace status", "workspace operation", "workspace writer", "workspace owner", "workspace owner receipt"] as const)( + "guards scheduled reassessment against a changed %s at the commit boundary", async changed => { + const seeded = await seedAutomaticReview(); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + const [source] = await db.select({ + runId: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + issueId: issues.id, issueStatus: issues.status, issueStatusVersion: issues.statusVersion, + issueDecisionId: issues.lastStatusDecisionId, coordinatorPhase: nativeRunFinalizations.phase, + assessmentId: nativeRunFinalizations.assessmentId, decisionId: nativeRunFinalizations.decisionId, + contextSnapshot: heartbeatRuns.contextSnapshot, + }).from(nativeRunFinalizations).innerJoin(heartbeatRuns, eq(heartbeatRuns.id, nativeRunFinalizations.runId)) + .innerJoin(issues, eq(issues.id, nativeRunFinalizations.issueId)) + .where(eq(nativeRunFinalizations.runId, seeded.runId)); + const selected = { ...source, assessmentId: source.assessmentId! }; + const future = new Date(Date.now() + 60_000); + const change = changed === "future" ? { nextAttemptAt: future } + : changed === "claim" ? { leaseOwner: "other-finalizer", leaseExpiresAt: future } + : changed === "unknown claim" ? { leaseOwner: "unknown-finalizer", leaseExpiresAt: null } + : changed === "assessment" ? { assessmentId: null, decisionId: null } + : changed === "result" ? { resultId: null, assessmentId: null, decisionId: null } + : changed === "phase" ? { phase: "terminal_failure" } + : changed === "decision" ? { decisionId: null } : null; + const [originalRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.runId)); + const [originalIssue] = await db.select().from(issues).where(eq(issues.id, seeded.issueId)); + const [originalBarrier] = await db.select().from(workspaceOperations).where(and( + eq(workspaceOperations.heartbeatRunId, seeded.runId), eq(workspaceOperations.phase, "workspace_finalize"))); + const newBarrierId = randomUUID(); + const replacementRunId = randomUUID(); + if (changed === "task execution") await db.insert(heartbeatRuns).values({ + id: replacementRunId, companyId, agentId, status: "running", runtimeMode: "native", + nativeIssueId: seeded.issueId, contextSnapshot: { issueId: seeded.issueId }, + }); + const originalEffects = await db.select().from(statusDecisionEffects).where(eq(statusDecisionEffects.decisionId, selected.decisionId!)).orderBy(statusDecisionEffects.ordinal); + expect(originalEffects.length).toBeGreaterThan(0); + const transaction = db.transaction.bind(db); + let newContractId: string | undefined; + let releaseWriter: (() => void) | undefined; + let writer: Promise | undefined; + const writeContract = async (owner: typeof db) => { + const saved = await ensureNativeCompletionContract({ db: owner, companyId, + issue: { id: seeded.issueId, title: "New direction", description: "Verify the revised objective" }, actorId: agentId }); + newContractId = saved.row.id; + }; + const intercept = vi.spyOn(db, "transaction").mockImplementationOnce((async (callback, options) => { + if (change) await db.update(nativeRunFinalizations).set(change).where(eq(nativeRunFinalizations.runId, seeded.runId)); + if (changed === "effects") await db.update(statusDecisionEffects).set({ deliveryState: "pending" }) + .where(eq(statusDecisionEffects.decisionId, selected.decisionId!)); + if (changed === "ownership") await db.update(heartbeatRuns).set({ status: "running", + runtimeMode: "native", errorCode: "native_execution_ownership_unverified", nativePhase: "terminal_failure" }) + .where(eq(heartbeatRuns.id, seeded.runId)); + if (changed === "run outcome") await db.update(heartbeatRuns).set({ status: "failed" }) + .where(eq(heartbeatRuns.id, seeded.runId)); + if (changed === "task execution") await db.update(issues).set({ executionRunId: replacementRunId }) + .where(eq(issues.id, seeded.issueId)); + if (changed === "workspace status") await db.update(workspaceOperations).set({ status: "failed" }) + .where(eq(workspaceOperations.id, originalBarrier.id)); + if (changed === "workspace operation") await db.insert(workspaceOperations).values({ + id: newBarrierId, companyId, heartbeatRunId: seeded.runId, issueId: seeded.issueId, + phase: "workspace_finalize", status: "running", cwd: process.cwd(), + createdAt: new Date(Math.max(Date.now(), originalBarrier.createdAt.getTime() + 1)), + }); + if (changed === "workspace owner receipt") await db.update(heartbeatRuns).set({ + runnerProfileJson: { ...originalRun.runnerProfileJson, nativeWorkspaceFinalizationOwner: { + token: randomUUID(), hostname: "unverified-controller", pid: 12345, processStartedAt: new Date().toISOString(), + } }, + }).where(eq(heartbeatRuns.id, seeded.runId)); + if (changed === "workspace owner") { + let entered!: () => void; + const ready = new Promise(resolve => { entered = resolve; }); + const released = new Promise(resolve => { releaseWriter = resolve; }); + writer = withNativeWorkspaceFinalizationOwnership({ db, companyId, runId: seeded.runId }, async () => { + entered(); await released; + }).then(() => undefined); + await Promise.race([ready, writer]); + } + if (changed === "contract revision") await writeContract(db); + if (changed === "contract writer" || changed === "workspace writer") { + let writerReady!: () => void; + const ready = new Promise(resolve => { writerReady = resolve; }); + const released = new Promise(resolve => { releaseWriter = resolve; }); + const rollback = new Error("rollback competing writer"); + writer = transaction(async tx => { + if (changed === "contract writer") await writeContract(tx as unknown as typeof db); + else await tx.update(workspaceOperations).set({ status: "failed" }) + .where(eq(workspaceOperations.id, originalBarrier.id)); + writerReady(); + await released; + throw rollback; + }).catch(error => { if (error !== rollback) throw error; }); + await Promise.race([ready, writer]); + } + return transaction(async tx => { + if (changed === "contract writer" || changed === "workspace writer") await tx.execute(sql`set local statement_timeout = '2s'`); + return callback(tx); + }, options); + }) as typeof db.transaction); + try { + expect(await reassessNativeStatusFromSource(db, selected, { onlyWhenDue: true })).toEqual({ kind: "deferred", retry: true }); + } finally { intercept.mockRestore(); releaseWriter?.(); await writer; } + expect((await issueService(db).getById(seeded.issueId))!.status).toBe("in_review"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(1); + if (change) expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seeded.runId)))[0]).toMatchObject(change); + if (changed === "effects") { + expect(await db.select().from(statusDecisionEffects).where(eq(statusDecisionEffects.decisionId, selected.decisionId!)).orderBy(statusDecisionEffects.ordinal)) + .toEqual(originalEffects.map(effect => ({ ...effect, deliveryState: "pending" }))); + for (const effect of originalEffects) await db.update(statusDecisionEffects).set({ deliveryState: effect.deliveryState }) + .where(eq(statusDecisionEffects.id, effect.id)); + } + if (changed === "ownership") { + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.runId)))[0]) + .toMatchObject({ status: "running", errorCode: "native_execution_ownership_unverified" }); + await db.update(heartbeatRuns).set({ status: originalRun.status, runtimeMode: originalRun.runtimeMode, + errorCode: originalRun.errorCode, nativePhase: originalRun.nativePhase }).where(eq(heartbeatRuns.id, seeded.runId)); + } + if (changed === "run outcome") { + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.runId)))[0].status).toBe("failed"); + await db.update(heartbeatRuns).set({ status: originalRun.status }).where(eq(heartbeatRuns.id, seeded.runId)); + } + if (changed === "task execution") { + expect((await issueService(db).getById(seeded.issueId))!.executionRunId).toBe(replacementRunId); + await db.update(issues).set({ executionRunId: originalIssue.executionRunId }).where(eq(issues.id, seeded.issueId)); + await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, replacementRunId)); + } + if (changed === "contract revision") { + expect(await reassessNativeStatusFromSource(db, selected, { onlyWhenDue: true })).toEqual({ kind: "deferred" }); + await db.delete(completionContracts).where(eq(completionContracts.id, newContractId!)); + } + if (changed === "contract writer") expect(await db.select().from(completionContracts) + .where(eq(completionContracts.id, newContractId!))).toHaveLength(0); + if (changed === "workspace status" || changed === "workspace operation") { + expect(await reassessNativeStatusFromSource(db, selected, { onlyWhenDue: true })).toEqual({ kind: "deferred" }); + if (changed === "workspace status") await db.update(workspaceOperations).set({ status: originalBarrier.status }) + .where(eq(workspaceOperations.id, originalBarrier.id)); + else await db.delete(workspaceOperations).where(eq(workspaceOperations.id, newBarrierId)); + } + if (changed === "workspace writer") expect((await db.select().from(workspaceOperations) + .where(eq(workspaceOperations.id, originalBarrier.id)))[0].status).toBe(originalBarrier.status); + if (changed === "workspace owner receipt") { + expect(await reassessNativeStatusFromSource(db, selected, { onlyWhenDue: true })).toEqual({ kind: "deferred" }); + await db.update(heartbeatRuns).set({ runnerProfileJson: originalRun.runnerProfileJson }) + .where(eq(heartbeatRuns.id, seeded.runId)); + } + await db.update(nativeRunFinalizations).set({ phase: selected.coordinatorPhase, + resultId: seeded.resultId, assessmentId: selected.assessmentId, decisionId: selected.decisionId, + nextAttemptAt: null, leaseOwner: null, leaseExpiresAt: null }).where(eq(nativeRunFinalizations.runId, seeded.runId)); + expect(await reassessNativeStatusFromSource(db, selected, { onlyWhenDue: true })).toMatchObject({ kind: "committed" }); + expect((await issueService(db).getById(seeded.issueId))!.status).toBe("done"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2); + }, 30_000, + ); + + it.each(["execution policy", "approval link", "approval reopening", "interaction", "approval writer", "interaction writer", "question history writer"] as const)( + "fences reassessment against changed governance: %s", async changed => { + const seeded = await seedAutomaticReview(); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + const [source] = await db.select({ + runId: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + issueId: issues.id, issueStatus: issues.status, issueStatusVersion: issues.statusVersion, + issueDecisionId: issues.lastStatusDecisionId, coordinatorPhase: nativeRunFinalizations.phase, + assessmentId: nativeRunFinalizations.assessmentId, decisionId: nativeRunFinalizations.decisionId, + contextSnapshot: heartbeatRuns.contextSnapshot, + }).from(nativeRunFinalizations).innerJoin(heartbeatRuns, eq(heartbeatRuns.id, nativeRunFinalizations.runId)) + .innerJoin(issues, eq(issues.id, nativeRunFinalizations.issueId)) + .where(eq(nativeRunFinalizations.runId, seeded.runId)); + const selected = { ...source, assessmentId: source.assessmentId! }; + const approvalId = randomUUID(), interactionId = randomUUID(), commentId = randomUUID(); + const approvalCase = changed.startsWith("approval"); + if (approvalCase) { + await db.insert(approvals).values({ id: approvalId, companyId, type: "approve_ceo_strategy", payload: {}, + status: changed === "approval link" ? "pending" : "approved" }); + if (changed !== "approval link") await db.insert(issueApprovals).values({ companyId, issueId: seeded.issueId, approvalId }); + } + if (changed === "interaction writer" || changed === "question history writer") { + await db.insert(issueThreadInteractions).values({ id: interactionId, companyId, issueId: seeded.issueId, + sourceRunId: seeded.runId, kind: changed === "question history writer" ? "ask_user_questions" : "request_confirmation", + status: changed === "question history writer" ? "pending" : "accepted", + payload: changed === "question history writer" + ? { version: 1, questions: [{ id: "review", prompt: "Continue?", selectionMode: "single", + options: [{ id: "yes", label: "Yes" }] }] } + : { version: 1, prompt: "Review" }, + createdAt: new Date(Date.now() - 2000) }); + if (changed === "question history writer") await db.insert(issueComments).values({ id: commentId, + companyId, issueId: seeded.issueId, authorUserId: "operator", authorType: "user", body: "Continue with the new direction" }); + } + const scope = { companyId, issueId: seeded.issueId, runId: seeded.runId, executionState: null }; + expect(await pendingNativeGovernance({ db, ...scope })).toBeNull(); + const transaction = db.transaction.bind(db); + let release!: () => void; + let writer: Promise | undefined; + const intercept = vi.spyOn(db, "transaction").mockImplementationOnce((async (callback, options) => { + if (changed === "execution policy") await db.update(issues).set({ executionState: { status: "pending" } }) + .where(eq(issues.id, seeded.issueId)); + if (changed === "approval link") await db.insert(issueApprovals).values({ companyId, issueId: seeded.issueId, approvalId }); + if (changed === "approval reopening") await db.update(approvals).set({ status: "revision_requested" }).where(eq(approvals.id, approvalId)); + if (changed === "interaction") await db.insert(issueThreadInteractions).values({ id: interactionId, + companyId, issueId: seeded.issueId, sourceRunId: seeded.runId, kind: "request_confirmation", + status: "pending", payload: { version: 1, prompt: "Review" } }); + if (changed.endsWith("writer")) { + let entered!: () => void; + const ready = new Promise(resolve => { entered = resolve; }); + const released = new Promise(resolve => { release = resolve; }); + const rollback = new Error("rollback governance writer"); + writer = transaction(async tx => { + if (changed === "approval writer") await tx.update(approvals).set({ status: "revision_requested" }).where(eq(approvals.id, approvalId)); + if (changed === "interaction writer") await tx.update(issueThreadInteractions).set({ status: "pending" }).where(eq(issueThreadInteractions.id, interactionId)); + if (changed === "question history writer") await tx.update(issueComments).set({ deletedAt: new Date() }).where(eq(issueComments.id, commentId)); + entered(); await released; throw rollback; + }).catch(error => { if (error !== rollback) throw error; }); + await Promise.race([ready, writer]); + } + return transaction(async tx => { + await tx.execute(sql`set local statement_timeout = '2s'`); + return callback(tx); + }, options); + }) as typeof db.transaction); + try { + expect(await reassessNativeStatusFromSource(db, selected, { onlyWhenDue: true })).toEqual({ kind: "deferred", retry: true }); + } finally { intercept.mockRestore(); release?.(); await writer; } + expect((await issueService(db).getById(seeded.issueId))!.status).toBe("in_review"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(1); + if (changed === "execution policy") await db.update(issues).set({ executionState: null }).where(eq(issues.id, seeded.issueId)); + if (changed === "approval link") await db.delete(issueApprovals).where(eq(issueApprovals.approvalId, approvalId)); + if (changed === "approval reopening") await db.update(approvals).set({ status: "approved" }).where(eq(approvals.id, approvalId)); + if (changed === "interaction") await db.delete(issueThreadInteractions).where(eq(issueThreadInteractions.id, interactionId)); + expect(await reassessNativeStatusFromSource(db, selected, { onlyWhenDue: true })).toMatchObject({ kind: "committed" }); + expect((await issueService(db).getById(seeded.issueId))!.status).toBe("done"); + }, 30_000, + ); + + it.each(["new blocker", "reopened blocker", "blocker writer", "workspace operation", "workspace status", "workspace writer", "workspace insert writer"] as const)( + "fences reassessment against changed dependency readiness: %s", async changed => { + const seeded = await seedAutomaticReview(); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + const [source] = await db.select({ + runId: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + issueId: issues.id, issueStatus: issues.status, issueStatusVersion: issues.statusVersion, + issueDecisionId: issues.lastStatusDecisionId, coordinatorPhase: nativeRunFinalizations.phase, + assessmentId: nativeRunFinalizations.assessmentId, decisionId: nativeRunFinalizations.decisionId, + contextSnapshot: heartbeatRuns.contextSnapshot, + }).from(nativeRunFinalizations).innerJoin(heartbeatRuns, eq(heartbeatRuns.id, nativeRunFinalizations.runId)) + .innerJoin(issues, eq(issues.id, nativeRunFinalizations.issueId)) + .where(eq(nativeRunFinalizations.runId, seeded.runId)); + const selected = { ...source, assessmentId: source.assessmentId! }; + const blockerId = randomUUID(), workspaceId = randomUUID(), operationId = randomUUID(); + const workspaceCase = changed.startsWith("workspace"); + if (workspaceCase) await db.insert(executionWorkspaces).values({ id: workspaceId, companyId, + mode: "isolated", strategyType: "git_worktree", name: "Dependency workspace" }); + await db.insert(issues).values({ id: blockerId, companyId, title: "Prerequisite", + status: changed === "new blocker" ? "todo" : "done", executionWorkspaceId: workspaceCase ? workspaceId : null }); + if (changed !== "new blocker") await db.insert(issueRelations).values({ companyId, issueId: blockerId, relatedIssueId: seeded.issueId, type: "blocks" }); + if (changed === "workspace status" || changed === "workspace writer") await db.insert(workspaceOperations).values({ + id: operationId, companyId, issueId: blockerId, executionWorkspaceId: workspaceId, phase: "workspace_finalize", status: "succeeded" }); + expect((await issueService(db).getDependencyReadiness(seeded.issueId)).isDependencyReady).toBe(true); + const transaction = db.transaction.bind(db); + let release!: () => void; + let writer: Promise | undefined; + const pendingOperation = { id: operationId, companyId, executionWorkspaceId: workspaceId, + phase: "workspace_finalize", status: "running" }; + const intercept = vi.spyOn(db, "transaction").mockImplementationOnce((async (callback, options) => { + if (changed === "new blocker") await db.insert(issueRelations).values({ companyId, issueId: blockerId, relatedIssueId: seeded.issueId, type: "blocks" }); + if (changed === "reopened blocker") await db.update(issues).set({ status: "todo" }).where(eq(issues.id, blockerId)); + if (changed === "workspace operation") await db.insert(workspaceOperations).values(pendingOperation); + if (changed === "workspace status") await db.update(workspaceOperations).set({ status: "failed" }).where(eq(workspaceOperations.id, operationId)); + if (changed.endsWith("writer")) { + let entered!: () => void; + const ready = new Promise(resolve => { entered = resolve; }); + const released = new Promise(resolve => { release = resolve; }); + const rollback = new Error("rollback dependency writer"); + writer = transaction(async tx => { + if (changed === "blocker writer") await tx.update(issues).set({ status: "todo" }).where(eq(issues.id, blockerId)); + if (changed === "workspace writer") await tx.update(workspaceOperations).set({ status: "failed" }).where(eq(workspaceOperations.id, operationId)); + if (changed === "workspace insert writer") await tx.insert(workspaceOperations).values(pendingOperation); + entered(); await released; throw rollback; + }).catch(error => { if (error !== rollback) throw error; }); + await Promise.race([ready, writer]); + } + return transaction(async tx => { + await tx.execute(sql`set local statement_timeout = '2s'`); + return callback(tx); + }, options); + }) as typeof db.transaction); + try { + expect(await reassessNativeStatusFromSource(db, selected, { onlyWhenDue: true })).toEqual({ kind: "deferred", retry: true }); + } finally { intercept.mockRestore(); release?.(); await writer; } + expect((await issueService(db).getById(seeded.issueId))!.status).toBe("in_review"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(1); + if (changed === "new blocker" || changed === "reopened blocker") await db.update(issues).set({ status: "done" }).where(eq(issues.id, blockerId)); + if (changed === "workspace operation" || changed === "workspace status") await db.update(workspaceOperations).set({ status: "succeeded" }).where(eq(workspaceOperations.id, operationId)); + expect(await reassessNativeStatusFromSource(db, selected, { onlyWhenDue: true })).toMatchObject({ kind: "committed" }); + expect((await issueService(db).getById(seeded.issueId))!.status).toBe("done"); + }, 30_000, + ); + + it.each(["work product", "work product writer", "approval", "interaction", "attachment", "event", "handoff"] as const)( + "fences reassessment against changed evidence: %s", async kind => { + const seeded = await seedAutomaticReview(); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + const evidenceId = randomUUID(), assetId = randomUUID(), caseId = randomUUID(); + let ref = `work_product:${seeded.workProductId}`; + const attachment = { id: evidenceId, companyId, issueId: seeded.issueId, assetId }; + const event = { companyId, agentId, runId: seeded.runId, seq: 90001, eventType: "evidence.recorded", + payload: { prpEvent: { sourceKind: "control_plane", payload: { evidenceOutcome: "accepted" } } } }; + if (kind === "approval") { + ref = `approval:${evidenceId}`; + await db.insert(approvals).values({ id: evidenceId, companyId, type: "hire_agent", status: "approved", payload: {} }); + await db.insert(issueApprovals).values({ companyId, issueId: seeded.issueId, approvalId: evidenceId }); + } + if (kind === "interaction") { + ref = `interaction:${evidenceId}`; + await db.insert(issueThreadInteractions).values({ id: evidenceId, companyId, issueId: seeded.issueId, + kind: "request_confirmation", status: "accepted", payload: { version: 1, prompt: "Evidence accepted" } }); + } + if (kind === "attachment") { + ref = `attachment:${evidenceId}`; + await db.insert(assets).values({ id: assetId, companyId, provider: "local_disk", objectKey: assetId, + contentType: "text/plain", byteSize: 1, sha256: assetId }); + await db.insert(issueAttachments).values(attachment); + } + if (kind === "event") { ref = "event:90001"; await db.insert(heartbeatRunEvents).values(event); } + if (kind === "handoff") { + ref = `case-handoff:${evidenceId}`; + const existing = await db.select({ id: cases.id }).from(cases).where(eq(cases.companyId, companyId)); + await db.insert(cases).values({ id: caseId, companyId, caseNumber: existing.length + 1, + identifier: caseId, caseType: "evidence", title: "Handoff evidence" }); + await db.insert(caseHandoffs).values({ id: evidenceId, companyId, caseId, issueId: seeded.issueId, + version: 1, destinationKind: "done", phase: "complete", instructions: "Complete", actorType: "system", + actorId: "test", sourceRunId: seeded.runId, wakeupRequestId: randomUUID(), idempotencyKey: evidenceId, + inputFingerprint: evidenceId, state: "completed" }); + } + const [storedResult] = await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, seeded.resultId!)); + const result = storedResult.resultJson.result as Record; + await db.update(nativeRunResults).set({ resultJson: { ...storedResult.resultJson, result: { ...result, + completionClaim: { ...(result.completionClaim as object), criteria: [{ criterionId: "objective", status: "satisfied", evidenceRefs: [ref] }] }, + verification: [{ commandOrCheck: "evidence", status: "passed", artifactRef: ref }], + } } }).where(eq(nativeRunResults.id, seeded.resultId!)); + const [source] = await db.select({ + runId: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + issueId: issues.id, issueStatus: issues.status, issueStatusVersion: issues.statusVersion, + issueDecisionId: issues.lastStatusDecisionId, coordinatorPhase: nativeRunFinalizations.phase, + assessmentId: nativeRunFinalizations.assessmentId, decisionId: nativeRunFinalizations.decisionId, + contextSnapshot: heartbeatRuns.contextSnapshot, + }).from(nativeRunFinalizations).innerJoin(heartbeatRuns, eq(heartbeatRuns.id, nativeRunFinalizations.runId)) + .innerJoin(issues, eq(issues.id, nativeRunFinalizations.issueId)).where(eq(nativeRunFinalizations.runId, seeded.runId)); + const selected = { ...source, assessmentId: source.assessmentId! }; + async function change(database: typeof db, revoke: boolean) { + if (kind.startsWith("work product")) await database.update(issueWorkProducts) + .set({ reviewState: revoke ? "rejected" : "approved" }).where(eq(issueWorkProducts.id, seeded.workProductId)); + if (kind === "approval") await database.update(approvals).set({ status: revoke ? "rejected" : "approved" }).where(eq(approvals.id, evidenceId)); + if (kind === "interaction") await database.update(issueThreadInteractions).set({ status: revoke ? "rejected" : "accepted" }).where(eq(issueThreadInteractions.id, evidenceId)); + if (kind === "handoff") await database.update(caseHandoffs).set({ state: revoke ? "cancelled" : "completed" }).where(eq(caseHandoffs.id, evidenceId)); + if (kind === "attachment") { + if (revoke) await database.delete(issueAttachments).where(eq(issueAttachments.id, evidenceId)); + else await database.insert(issueAttachments).values(attachment); + } + if (kind === "event") { + if (revoke) await database.delete(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.runId, seeded.runId), eq(heartbeatRunEvents.seq, event.seq))); + else await database.insert(heartbeatRunEvents).values(event); + } + } + const transaction = db.transaction.bind(db); + let release!: () => void; + let writer: Promise | undefined; + const intercept = vi.spyOn(db, "transaction").mockImplementationOnce((async (callback, options) => { + if (kind === "work product writer") { + let entered!: () => void; + const ready = new Promise(resolve => { entered = resolve; }); + const released = new Promise(resolve => { release = resolve; }); + const rollback = new Error("rollback evidence writer"); + writer = transaction(async tx => { await change(tx as unknown as typeof db, true); entered(); await released; throw rollback; }) + .catch(error => { if (error !== rollback) throw error; }); + await Promise.race([ready, writer]); + } else await change(db, true); + return transaction(async tx => { + await tx.execute(sql`set local statement_timeout = '2s'`); + return callback(tx); + }, options); + }) as typeof db.transaction); + try { + expect(await reassessNativeStatusFromSource(db, selected, { onlyWhenDue: true })).toEqual({ kind: "deferred", retry: true }); + } finally { intercept.mockRestore(); release?.(); await writer; } + expect((await issueService(db).getById(seeded.issueId))!.status).toBe("in_review"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(1); + if (kind !== "work product writer") await change(db, false); + expect(await reassessNativeStatusFromSource(db, selected, { onlyWhenDue: true })).toMatchObject({ kind: "committed" }); + expect((await issueService(db).getById(seeded.issueId))!.status).toBe("done"); + }, 30_000, + ); + + it("leaves reassessment to its scheduled owner while periodic finalization continues", async () => { + const seeded = await seedAutomaticReview(); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + const options = { skipNativeStatusReassessment: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true }; + const result = await reconcileNativeFinalizations(db, [seeded.runId], options); + expect(result).toEqual([expect.objectContaining({ runId: seeded.runId, phase: "committed" })]); + expect((await issueService(db).getById(seeded.issueId))!.status).toBe("in_review"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(1); + const work = nativeStatusReassessmentWork(db); + const [pending] = await work.load({ companyId, issueId: seeded.issueId, runId: seeded.runId }); + expect(pending).toBeDefined(); + expect(await work.check(pending)).toBeNull(); + expect((await issueService(db).getById(seeded.issueId))!.status).toBe("done"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2); + }); + + describe("native status reassessment work", () => { + const selectedScope = (seeded: { issueId: string; runId: string }) => ({ companyId, issueId: seeded.issueId, runId: seeded.runId }); + + it.each(["retired review", "new evidence", "task status"] as const)("discovers and settles %s without revisiting settled history", async trigger => { + const seeded = await seedAutomaticReview(); + const work = nativeStatusReassessmentWork(db), scope = selectedScope(seeded); + expect(await work.load(scope)).toEqual([]); + if (trigger === "retired review") await dismissAutomaticCompletionReviews(db, seeded.issueId); + if (trigger === "new evidence") { + const [assessment] = await db.select().from(workAssessments).where(eq(workAssessments.id, seeded.assessmentId)); + await db.update(workAssessments).set({ createdAt: new Date(Date.now() - 2000), + assessmentJson: { ...assessment.assessmentJson, allCriteriaSatisfied: false } }).where(eq(workAssessments.id, seeded.assessmentId)); + await db.update(issueWorkProducts).set({ updatedAt: new Date() }).where(eq(issueWorkProducts.id, seeded.workProductId)); + } + if (trigger === "task status") await db.update(issues).set({ status: "blocked", statusVersion: 2 }).where(eq(issues.id, seeded.issueId)); + const pending = await work.load(scope); + expect(pending).toMatchObject([{ id: seeded.runId, dueAt: 0 }]); + expect(await work.check(pending[0])).toBeNull(); + expect((await issueService(db).getById(seeded.issueId))!.status).toBe(trigger === "task status" ? "blocked" : "done"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2); + expect(await work.load(scope)).toEqual([]); + expect(await work.check(pending[0])).toBeNull(); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2); + }); + + it("respects a saved retry and live lease, then reloads the released owner", async () => { + const seeded = await seedAutomaticReview(); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + const work = nativeStatusReassessmentWork(db), scope = selectedScope(seeded); + const future = new Date(Date.now() + 60_000), lease = new Date(Date.now() + 120_000); + await db.update(nativeRunFinalizations).set({ nextAttemptAt: future, leaseOwner: "finalizer", leaseExpiresAt: lease }) + .where(eq(nativeRunFinalizations.runId, seeded.runId)); + const [pending] = await work.load(scope); + expect(pending.dueAt).toBe(lease.getTime()); + expect(await work.check(pending)).toEqual(pending); + expect((await issueService(db).getById(seeded.issueId))!.status).toBe("in_review"); + await db.update(nativeRunFinalizations).set({ nextAttemptAt: null, leaseOwner: null, leaseExpiresAt: null }) + .where(eq(nativeRunFinalizations.runId, seeded.runId)); + expect(await work.check(pending)).toBeNull(); + expect((await issueService(db).getById(seeded.issueId))!.status).toBe("done"); + }); + + it("leaves insufficient evidence dormant until a later change satisfies the contract", async () => { + const seeded = await seedAutomaticReview(); + const work = nativeStatusReassessmentWork(db), scope = selectedScope(seeded); + const [assessment] = await db.select().from(workAssessments).where(eq(workAssessments.id, seeded.assessmentId)); + await db.update(workAssessments).set({ createdAt: new Date(Date.now() - 2000), + assessmentJson: { ...assessment.assessmentJson, allCriteriaSatisfied: false } }).where(eq(workAssessments.id, seeded.assessmentId)); + await db.update(issueWorkProducts).set({ reviewState: "changes_requested", updatedAt: new Date() }) + .where(eq(issueWorkProducts.id, seeded.workProductId)); + const [pending] = await work.load(scope); + expect(pending).toBeDefined(); + expect(await work.check(pending)).toBeNull(); + expect((await issueService(db).getById(seeded.issueId))!.status).toBe("in_review"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(1); + await db.update(issueWorkProducts).set({ reviewState: "approved", updatedAt: new Date() }) + .where(eq(issueWorkProducts.id, seeded.workProductId)); + const [changed] = await work.load(scope); + expect(await work.check(changed)).toBeNull(); + expect((await issueService(db).getById(seeded.issueId))!.status).toBe("done"); + }); + + it.each(["unknown lease", "pending effect", "ownership"] as const)("waits for the %s owner to release its obligation", async held => { + const seeded = await seedAutomaticReview(); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + const work = nativeStatusReassessmentWork(db), scope = selectedScope(seeded); + const [pending] = await work.load(scope); + if (held === "unknown lease") await db.update(nativeRunFinalizations).set({ leaseOwner: "unknown", leaseExpiresAt: null }) + .where(eq(nativeRunFinalizations.runId, seeded.runId)); + if (held === "pending effect") await db.update(statusDecisionEffects).set({ deliveryState: "pending" }) + .where(eq(statusDecisionEffects.decisionId, seeded.decision.id)); + if (held === "ownership") await db.update(heartbeatRuns).set({ status: "running", errorCode: "native_execution_ownership_unverified", nativePhase: "terminal_failure" }) + .where(eq(heartbeatRuns.id, seeded.runId)); + expect(await work.load(scope)).toEqual([]); + expect(await work.check(pending)).toBeNull(); + expect((await issueService(db).getById(seeded.issueId))!.status).toBe("in_review"); + }); + + it("keeps a newly saved near-term deadline after a competing commit", async () => { + const seeded = await seedAutomaticReview(); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + const work = nativeStatusReassessmentWork(db), scope = selectedScope(seeded); + const [pending] = await work.load(scope); + const future = new Date(Date.now() + 5000); + const transaction = db.transaction.bind(db); + const intercept = vi.spyOn(db, "transaction").mockImplementationOnce((async (callback, options) => { + await db.update(nativeRunFinalizations).set({ nextAttemptAt: future }).where(eq(nativeRunFinalizations.runId, seeded.runId)); + return transaction(callback, options); + }) as typeof db.transaction); + try { expect(await work.check(pending)).toEqual({ ...pending, dueAt: future.getTime() }); } + finally { intercept.mockRestore(); } + expect((await issueService(db).getById(seeded.issueId))!.status).toBe("in_review"); + }); + + it("does not query after discovery was cancelled", async () => { + const controller = new AbortController(); controller.abort(); + const select = vi.spyOn(db, "select"); + try { + await expect(nativeStatusReassessmentWork(db).load(undefined, controller.signal)).rejects.toThrow("discovery stopped"); + expect(select).not.toHaveBeenCalled(); + } finally { select.mockRestore(); } + }); + }); + it("retires a proven automatic review and applies the current successful completion exactly once", async () => { const seeded = await seedAutomaticReview(); // The persisted result is done, while its old assessment/decision required a review. @@ -3126,7 +3649,7 @@ describe("P6-31 Section 18.13 executable status-authority corpus", () => { ]); }); - it("ignores historical committed finalizations superseded by a newer authoritative decision", async () => { + it.each([false, true])("ignores superseded historical finalizations with scheduled reassessment %s", async scheduled => { const fixture = corpus.fixtures.find((candidate) => candidate.mode === "native"); if (!fixture) throw new Error("native corpus fixture missing"); const seeded = await seedFixture(fixture); @@ -3188,7 +3711,7 @@ describe("P6-31 Section 18.13 executable status-authority corpus", () => { decisionId: historicalDecision!.id, }).where(eq(nativeRunFinalizations.runId, seeded.runId)); - await expect(reconcileNativeFinalizations(db, [seeded.runId])).resolves.toEqual([]); + await expect(reconcileNativeFinalizations(db, [seeded.runId], { skipNativeStatusReassessment: scheduled })).resolves.toEqual([]); await expect(db.select({ decisionId: nativeRunFinalizations.decisionId, }).from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seeded.runId))).resolves.toEqual([ diff --git a/server/src/__tests__/native-status-reassessment-work.test.ts b/server/src/__tests__/native-status-reassessment-work.test.ts new file mode 100644 index 00000000000..667f9c2bfe9 --- /dev/null +++ b/server/src/__tests__/native-status-reassessment-work.test.ts @@ -0,0 +1,640 @@ +import { notifyChatRunMilestones } from "../services/chat-run-milestone-notifications.js"; +import { notifyNativeFinalizationWork } from "../services/native-finalization-notifications.js"; +import { createPostgresWakeQueueAdapter } from "../modules/wake-queue/adapters/postgres.js"; +import { withNativeWorkspaceFinalizationOwnership } from "../services/native-runtime/native-workspace-finalization-ownership.js"; +import express from "express"; +import { appendHeartbeatRunEvent } from "../services/heartbeat-run-events.js"; +import { DELIVERY_QUEUES, subscribeDeliveryWork } from "../services/delivery-work-notifications.js"; +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { workspaceOperationService } from "../services/workspace-operations.js"; +import request from "supertest"; +import { LOW_TRUST_REVIEW_PRESET } from "@paperclipai/shared"; +import { issueRoutes } from "../routes/issues.js"; +import { errorHandler } from "../middleware/index.js"; +import { workProductService } from "../services/work-products.js"; +import { issueService } from "../services/issues.js"; +import { approvalService } from "../services/approvals.js"; +import { issueThreadInteractionService } from "../services/issue-thread-interactions.js"; +import { issueApprovalService } from "../services/issue-approvals.js"; +import { randomUUID } from "node:crypto"; +import { setTimeout as delay } from "node:timers/promises"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { eq } from "drizzle-orm"; +import { agents, companies, completionContracts, createDb, heartbeatRuns, issues, issueWorkProducts, issueThreadInteractions, + nativeRunFinalizations, nativeRunResults, statusDecisions, workAssessments, workspaceOperations } from "@paperclipai/db"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; +import { idleWorkSnapshot } from "../services/task-admission.js"; +import { nativeStatusReassessmentWork } from "../services/native-runtime/native-status-reassessment-work.js"; +import { dismissAutomaticCompletionReviews } from "../services/native-runtime/automatic-completion-reviews.js"; +import { commitNativeStatusDecision } from "../services/native-runtime/status-decision-committer.js"; +import { NATIVE_STATUS_ARBITER_POLICY_VERSION } from "../services/native-runtime/status-arbiter.js"; +import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; + +describe("native reassessment worker lifecycle", () => { + let temporary: Awaited> | undefined; + let db: ReturnType; + let coordinator: ReturnType | undefined; + const companyId = randomUUID(), agentId = randomUUID(); + let logRoot: string; + const priorLogRoot = process.env.WORKSPACE_OPERATION_LOG_BASE_PATH; + let admitted = true; + const errors: unknown[] = []; + beforeAll(async () => { + logRoot = await mkdtemp(join(tmpdir(), "reassessment-workspace-logs-")); + process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = logRoot; + const external = resolveExternalTestDatabaseUrl("PAPERCLIP_STATUS_REASSESSMENT_TEST_DATABASE_URL"); + if (!external) temporary = await startEmbeddedPostgresTestDatabase("paperclip-status-reassessment-"); + db = createDb(external ?? temporary!.connectionString); + await db.insert(companies).values({ id: companyId, name: "Reassessment lifecycle", issuePrefix: `RS${randomUUID().slice(0, 8)}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Reassessment agent", adapterType: "codex_local", status: "idle" }); + }, 30_000); + afterAll(async () => { + await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); + await rm(logRoot, { recursive: true, force: true }); + if (priorLogRoot === undefined) delete process.env.WORKSPACE_OPERATION_LOG_BASE_PATH; + else process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = priorLogRoot; + }); + beforeEach(() => { + admitted = true; errors.length = 0; + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true }); + }); + afterEach(async () => { + await coordinator?.stop(); coordinator = undefined; + vi.useRealTimers(); vi.restoreAllMocks(); + expect(errors).toEqual([]); + }); + async function start() { + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => admitted, canReconcile: () => admitted, + onError: error => errors.push(error) }); + await nativeStatusReassessmentWork(db).register(coordinator, () => admitted).ready; + } + async function flush() { + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(1000); + expect(idleWorkSnapshot().active).toBe(0); + expect(coordinator!.nextWakeAt()).toBeNull(); + }, { timeout: 15_000 }); + } + async function quiet() { + const original = db.$client.options.debug; + const queries: string[] = []; + db.$client.options.debug = (_id, query) => { queries.push(query); }; + try { + await vi.advanceTimersByTimeAsync(120_000); + await delay(10); + expect(queries).toEqual([]); + } finally { db.$client.options.debug = original; } + } + async function status(issueId: string) { + return (await db.select({ status: issues.status }).from(issues).where(eq(issues.id, issueId)))[0].status; + } + async function seedReview(evidenceRef?: string) { + const issueId = randomUUID(), runId = randomUUID(), resultId = randomUUID(), contractId = randomUUID(); + const assessmentId = randomUUID(), workProductId = randomUUID(); + await db.insert(issues).values({ id: issueId, companyId, title: "Persisted native result", status: "in_progress", assigneeAgentId: agentId }); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "succeeded", runtimeMode: "native", + nativeIssueId: issueId, completionContractId: contractId, contextSnapshot: { issueId }, finishedAt: new Date() }); + await db.insert(completionContracts).values({ id: contractId, companyId, issueId, revision: 1, + schemaVersion: "paperclip.completion-contract.v1", policyVersion: "phase6-v1", risk: "low", + completionAuthority: "agent_claim_policy", incompleteCriteriaPolicy: "preserve_non_terminal", + contractJson: { revision: "test-v1", criteria: [{ id: "objective", requirement: "Verified artifact" }] }, + canonicalSha256: contractId, createdByActorType: "system", createdByActorId: "reassessment-test" }); + await db.insert(nativeRunResults).values({ id: resultId, companyId, issueId, runId, completionContractId: contractId, + serverFingerprint: resultId, schemaStatus: "accepted", canonicalSha256: resultId, + resultJson: { result: { reportedWorkDisposition: "done", summary: "Completed", attentionRequests: [], + completionClaim: { contractRevision: "test-v1", objectiveSatisfied: true, remainingWork: [], + criteria: [{ criterionId: "objective", status: "satisfied", evidenceRefs: [evidenceRef ?? `work_product:${workProductId}`] }] }, + verification: [{ commandOrCheck: "artifact", status: "passed", artifactRef: evidenceRef ?? `work_product:${workProductId}` }], + }, terminal: { runTerminalState: "succeeded" } } }); + await db.insert(issueWorkProducts).values({ id: workProductId, companyId, issueId, type: "artifact", provider: "paperclip", + title: "Verified artifact", status: "ready_for_review", reviewState: "approved", + createdAt: new Date(Date.now() - 1000), updatedAt: new Date(Date.now() - 1000) }); + await db.insert(workAssessments).values({ id: assessmentId, companyId, issueId, runId, contractId, resultId, + triggerKind: "native_result", triggerActorCompanyId: companyId, priorIssueStatus: "in_progress", priorStatusVersion: 0, + policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, inputDigest: assessmentId, + assessmentJson: { reportedDisposition: "needs_review", attentionRequests: [], allCriteriaSatisfied: false } }); + await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "assessing", resultId, assessmentId }); + const committed = await commitNativeStatusDecision({ db, companyId, issueId, runId, assessmentId, + priorStatus: "in_progress", priorStatusVersion: 0, priorDecisionId: null, + decision: { policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, statusAction: "in_review", toStatus: "in_review", + reasonCode: "completion_claim_incomplete", unblockDescriptor: null, + effects: [{ kind: "bind_reviewer", ownerUserId: null, + prompt: "Review the persisted native-run evidence and confirm whether this issue may be completed." }] } }); + await db.insert(workspaceOperations).values({ companyId, heartbeatRunId: runId, issueId, phase: "workspace_finalize", + status: "succeeded", exitCode: 0, cwd: process.cwd(), finishedAt: new Date() }); + return { issueId, runId, workProductId, decisionId: committed.decision.id }; + } + + it.each(["milestone", "finalization"] as const)("ignores repeated unscoped routine %s hints without reconstructing dormant reassessment history", async source => { + const seeded = await seedReview(); + const [other] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "failed", + runtimeMode: "legacy", contextSnapshot: { issueId: seeded.issueId }, finishedAt: new Date() }).returning(); + await db.update(issues).set({ executionRunId: other.id }).where(eq(issues.id, seeded.issueId)); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + await start(); + await flush(); + await quiet(); + expect(await nativeStatusReassessmentWork(db).load({ companyId, runId: seeded.runId })).toHaveLength(1); + const debug = db.$client.options.debug; + const queries: string[] = []; + db.$client.options.debug = (_connection, query) => { + if (query.startsWith("select") && query.includes("native_run_finalizations")) queries.push(query); + }; + try { + for (let i = 0; i < 3; i++) { + await db.transaction(tx => source === "milestone" ? notifyChatRunMilestones(tx, { companyId }) + : notifyNativeFinalizationWork(tx, { companyId })); + await flush(); + } + expect(queries).toEqual([]); + expect(await status(seeded.issueId)).toBe("in_review"); + } finally { db.$client.options.debug = debug; } + await quiet(); + }); + + it.each([false, true])("preserves the current required quality-review gate during retired-review reassessment (required=%s)", async required => { + const seeded = await seedReview(); + if (required) await db.update(issues).set({ executionPolicy: { requiredQualityReview: { + version: 1, deliverables: [{ key: "output", label: "Verified output" }], + } } }).where(eq(issues.id, seeded.issueId)); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + await start(); + await flush(); + expect(await status(seeded.issueId)).toBe(required ? "in_progress" : "done"); + const decisions = await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId)); + if (required) expect(decisions.find(row => row.id !== seeded.decisionId)?.reasonCode).toBe("required_quality_review_incomplete"); + await quiet(); + }); + + it.each(["force release", "release", "terminal execution", "terminal checkout", "queue release"] as const)( + "wakes dormant reassessment from real %s", async action => { + const seeded = await seedReview(); + const [other] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "failed", + runtimeMode: "legacy", contextSnapshot: { issueId: seeded.issueId }, finishedAt: new Date() }).returning(); + await db.update(issues).set({ executionRunId: other.id, checkoutRunId: other.id }).where(eq(issues.id, seeded.issueId)); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + await start(); await flush(); await quiet(); + expect(await status(seeded.issueId)).toBe("in_review"); + const service = issueService(db); + if (action === "force release") await service.adminForceRelease(seeded.issueId); + if (action === "release") await service.release(seeded.issueId); + if (action === "terminal execution") await service.clearExecutionRunIfTerminal(seeded.issueId); + if (action === "terminal checkout") await service.clearCheckoutRunIfTerminal(seeded.issueId); + if (action === "queue release") { + const adapter = createPostgresWakeQueueAdapter(db, { + resolveResponsibleUserId: async () => "reassessment-user", + getRoutineEnv: async () => ({ routineId: null, env: null, responsibleUserId: null }), + resolveSessionBeforeForWakeup: async () => null, + }); + await adapter.withIssueExecutionLock({ companyId, runId: other.id, now: new Date() }, + async () => ({ outcome: { kind: "released" }, postCommitEffects: [] })); + } + await flush(); + expect(await status(seeded.issueId)).toBe(action === "release" ? "in_review" : "done"); + const decisions = await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId)); + expect(decisions).toHaveLength(2); + if (action === "release") { + expect((await service.getById(seeded.issueId))!.assigneeAgentId).toBeNull(); + expect(decisions.find(decision => decision.id !== seeded.decisionId)?.reasonCode).toBe("prior_status_terminal_preserved"); + } + await quiet(); + }, + ); + + it("retains dormant reassessment through outer force-release rollback and commit", async () => { + const seeded = await seedReview(); + const [other] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "failed", + runtimeMode: "legacy", contextSnapshot: { issueId: seeded.issueId }, finishedAt: new Date() }).returning(); + await db.update(issues).set({ executionRunId: other.id }).where(eq(issues.id, seeded.issueId)); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + await start(); await flush(); await quiet(); + const rollback = new Error("rollback release"); + await expect(db.transaction(async tx => { + await issueService(tx as unknown as typeof db).adminForceRelease(seeded.issueId); + await vi.advanceTimersByTimeAsync(2000); + expect(await status(seeded.issueId)).toBe("in_review"); + throw rollback; + })).rejects.toBe(rollback); + await flush(); await quiet(); + expect((await issueService(db).getById(seeded.issueId))!.executionRunId).toBe(other.id); + await db.transaction(async tx => { + await issueService(tx as unknown as typeof db).adminForceRelease(seeded.issueId); + await vi.advanceTimersByTimeAsync(2000); + expect(await status(seeded.issueId)).toBe("in_review"); + }); + await flush(); + expect(await status(seeded.issueId)).toBe("done"); + await quiet(); + }); + + it("bounds discovery by pages and drains a real reassessment backlog to SQL quiet", async () => { + const backlog = []; + for (let index = 0; index < 103; index++) { + const seeded = await seedReview(); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + backlog.push(seeded); + } + const work = nativeStatusReassessmentWork(db), scope = { companyId }; + const expected = new Set(backlog.map(row => row.runId)); + const discovered = await work.load(scope); + expect(discovered.filter(row => expected.has(row.id))).toHaveLength(103); + const debug = db.$client.options.debug; + let queries = 0, admissionChecks = 0; + db.$client.options.debug = (_id, query) => { + if (query.startsWith("select") && query.includes("native_run_finalizations")) queries++; + }; + try { + await expect(work.load(scope, undefined, () => ++admissionChecks <= 1)) + .rejects.toThrow("Native status reassessment discovery stopped"); + expect(queries).toBe(1); + queries = 0; + await expect(work.load(scope, AbortSignal.abort())).rejects.toThrow("Native status reassessment discovery stopped"); + expect(queries).toBe(0); + } finally { db.$client.options.debug = debug; } + await start(); await flush(); + for (const seeded of backlog) { + expect(await status(seeded.issueId)).toBe("done"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2); + } + expect((await work.load(scope)).filter(row => expected.has(row.id))).toEqual([]); + await quiet(); + }, 60_000); + + it("waits quietly for physical workspace ownership to release before reassessment", async () => { + const seeded = await seedReview(); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + let enter!: () => void, release!: () => void; + const entered = new Promise(resolve => { enter = resolve; }); + const released = new Promise(resolve => { release = resolve; }); + const owner = withNativeWorkspaceFinalizationOwnership({ db, companyId, runId: seeded.runId }, async () => { + enter(); await released; + }); + try { + await Promise.race([entered, owner]); + await start(); await flush(); + expect(await status(seeded.issueId)).toBe("in_review"); + await quiet(); + } finally { release(); await owner; } + await flush(); + expect(await status(seeded.issueId)).toBe("done"); + await quiet(); + }); + + it("keeps an unverified workspace owner dormant despite a successful receipt", async () => { + const seeded = await seedReview(); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + const owner = { token: randomUUID(), hostname: "unverified-controller", pid: 12345, + processStartedAt: new Date().toISOString() }; + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceFinalizationOwner: owner } }) + .where(eq(heartbeatRuns.id, seeded.runId)); + await start(); await flush(); + expect(await status(seeded.issueId)).toBe("in_review"); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.runId)))[0] + .runnerProfileJson?.nativeWorkspaceFinalizationOwner).toEqual(owner); + await quiet(); + }); + + it.each(["ordinary", "native source"] as const)("wakes from a committed control-plane evidence event through the %s append path", async path => { + const seeded = await seedReview("event:1"); + await vi.advanceTimersByTimeAsync(1000); + await db.update(issueWorkProducts).set({ updatedAt: new Date() }).where(eq(issueWorkProducts.id, seeded.workProductId)); + await start(); await flush(); await quiet(); + const prpEvent = { sourceKind: "control_plane", payload: { evidenceOutcome: " accepted " } }; + const input = { companyId, agentId, runId: seeded.runId, eventType: "evidence.recorded", payload: { prpEvent }, + ...(path === "native source" ? { nativeSource: { sourceInstanceId: randomUUID(), sourceEventId: randomUUID(), + sourceSeq: 1, protocolSchemaVersion: 1, canonicalPayload: prpEvent } } : {}) }; + expect((await appendHeartbeatRunEvent(db, input)).disposition).toBe("committed"); + await flush(); + expect(await status(seeded.issueId)).toBe("done"); + if (path === "native source") expect((await appendHeartbeatRunEvent(db, input)).disposition).toBe("duplicate"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2); + await quiet(); + }); + + it("preserves event evidence until outer commit and ignores rollback", async () => { + const seeded = await seedReview("event:1"); + await vi.advanceTimersByTimeAsync(1000); + await db.update(issueWorkProducts).set({ updatedAt: new Date() }).where(eq(issueWorkProducts.id, seeded.workProductId)); + await start(); await flush(); await quiet(); + const write = async (tx: typeof db) => { + await appendHeartbeatRunEvent(tx, { companyId, agentId, runId: seeded.runId, eventType: "evidence.recorded", + payload: { prpEvent: { sourceKind: "control_plane", payload: { evidenceOutcome: "accepted" } } } }); + await vi.advanceTimersByTimeAsync(1000); + expect(await status(seeded.issueId)).toBe("in_review"); + }; + await expect(db.transaction(async tx => { + await write(tx as unknown as typeof db); throw new Error("rollback event evidence"); + })).rejects.toThrow("rollback event evidence"); + await flush(); await quiet(); + expect(await status(seeded.issueId)).toBe("in_review"); + await db.transaction(tx => write(tx as unknown as typeof db)); + await flush(); + expect(await status(seeded.issueId)).toBe("done"); + await quiet(); + }); + + it("keeps ordinary and runner-authored logs from waking reassessment", async () => { + const seeded = await seedReview("event:3"); + await vi.advanceTimersByTimeAsync(1000); + await db.update(issueWorkProducts).set({ updatedAt: new Date() }).where(eq(issueWorkProducts.id, seeded.workProductId)); + await start(); await flush(); await quiet(); + const wakes = vi.fn(); + const stop = subscribeDeliveryWork(db, DELIVERY_QUEUES.nativeStatusReassessment, wakes); + try { + await appendHeartbeatRunEvent(db, { companyId, agentId, runId: seeded.runId, eventType: "log", message: "Still working" }); + await appendHeartbeatRunEvent(db, { companyId, agentId, runId: seeded.runId, eventType: "evidence.recorded", + payload: { prpEvent: { sourceKind: "runner", payload: { evidenceOutcome: "accepted" } } } }); + expect(wakes).not.toHaveBeenCalled(); + await quiet(); + expect(await status(seeded.issueId)).toBe("in_review"); + await appendHeartbeatRunEvent(db, { companyId, agentId, runId: seeded.runId, eventType: "evidence.recorded", + payload: { prpEvent: { sourceKind: "control_plane", payload: { evidenceOutcome: "accepted" } } } }); + await flush(); + expect(await status(seeded.issueId)).toBe("done"); + await quiet(); + } finally { stop(); } + }); + + it("wakes when a real workspace finalization settles after a quiet failure", async () => { + const seeded = await seedReview(); + await db.update(workspaceOperations).set({ status: "running" }).where(eq(workspaceOperations.heartbeatRunId, seeded.runId)); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + await start(); await flush(); await quiet(); + const recorder = workspaceOperationService(db).createRecorder({ companyId, issueId: seeded.issueId, heartbeatRunId: seeded.runId }); + await expect(recorder.recordOperation({ phase: "workspace_finalize", run: async () => { throw new Error("export interrupted"); } })) + .rejects.toThrow("export interrupted"); + await flush(); await quiet(); + expect(await status(seeded.issueId)).toBe("in_review"); + let release!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + let started!: () => void; + const running = new Promise(resolve => { started = resolve; }); + const operation = recorder.recordOperation({ phase: "workspace_finalize", run: async () => { + started(); await gate; return { status: "succeeded" }; + } }); + try { + await running; await flush(); await quiet(); + expect(await status(seeded.issueId)).toBe("in_review"); + } finally { release(); await operation; } + await flush(); + expect(await status(seeded.issueId)).toBe("done"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2); + await quiet(); + }); + + it("defers workspace barrier writes until outer commit and ignores rollback", async () => { + const seeded = await seedReview(); + await db.update(workspaceOperations).set({ status: "running" }).where(eq(workspaceOperations.heartbeatRunId, seeded.runId)); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + await start(); await flush(); await quiet(); + const write = async (tx: typeof db) => { + await workspaceOperationService(tx).createRecorder({ companyId, issueId: seeded.issueId, heartbeatRunId: seeded.runId }) + .recordOperation({ phase: "workspace_finalize", run: async () => ({ status: "succeeded" }) }); + await vi.advanceTimersByTimeAsync(1000); + expect(await status(seeded.issueId)).toBe("in_review"); + }; + await expect(db.transaction(async tx => { + await write(tx as unknown as typeof db); throw new Error("rollback workspace barrier"); + })).rejects.toThrow("rollback workspace barrier"); + await flush(); await quiet(); + expect(await status(seeded.issueId)).toBe("in_review"); + await db.transaction(tx => write(tx as unknown as typeof db)); + await flush(); + expect(await status(seeded.issueId)).toBe("done"); + await quiet(); + }); + + it.each(["confirmation", "checkbox", "question", "verdict", "partial verdict"] as const)("wakes dormant evidence from real interaction %s", async mutation => { + const interactionId = randomUUID(); + const seeded = await seedReview(`interaction:${interactionId}`); + const kind = mutation === "confirmation" ? "request_confirmation" : mutation === "checkbox" ? "request_checkbox_confirmation" + : mutation === "question" ? "ask_user_questions" : "request_item_verdicts"; + const payload = kind === "ask_user_questions" + ? { version: 1 as const, questions: [{ id: "verified", prompt: "Verified?", selectionMode: "single" as const, + options: [{ id: "yes", label: "Yes" }] }] } + : kind === "request_item_verdicts" + ? { version: 1 as const, prompt: "Review evidence", items: [{ id: "artifact", label: "Artifact" }, + ...(mutation === "partial verdict" ? [{ id: "other", label: "Other" }] : [])] } + : kind === "request_checkbox_confirmation" + ? { version: 1 as const, prompt: "Accept evidence", options: [{ id: "verified", label: "Verified" }] } + : { version: 1 as const, prompt: "Accept evidence" }; + await db.insert(issueThreadInteractions).values({ id: interactionId, companyId, issueId: seeded.issueId, kind, + sourceRunId: seeded.runId, continuationPolicy: "none", createdByUserId: "board-user", payload }); + await db.update(issues).set({ createdByUserId: "board-user" }).where(eq(issues.id, seeded.issueId)); + await vi.advanceTimersByTimeAsync(1000); + await db.update(issueWorkProducts).set({ updatedAt: new Date() }).where(eq(issueWorkProducts.id, seeded.workProductId)); + await start(); await flush(); await quiet(); + expect(await status(seeded.issueId)).toBe("in_review"); + const scope = { id: seeded.issueId, companyId, projectId: null, goalId: null }; + const interactions = issueThreadInteractionService(db); + if (mutation === "confirmation" || mutation === "checkbox") { + await interactions.acceptInteraction(scope, interactionId, mutation === "checkbox" ? { selectedOptionIds: ["verified"] } : {}, { userId: "board-user" }); + } else if (mutation === "question") { + await interactions.answerQuestions(scope, interactionId, { answers: [{ questionId: "verified", optionIds: ["yes"] }] }, { userId: "board-user" }); + } else { + await interactions.submitItemVerdicts(scope, interactionId, { verdicts: [{ id: "artifact", verdict: "approve" }] }, { userId: "board-user" }); + } + await flush(); + expect(await status(seeded.issueId)).toBe("done"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2); + await quiet(); + }); + + it.each(["resolution", "link", "bulk link"] as const)("wakes dormant evidence from real approval %s", async mutation => { + const approvalId = randomUUID(); + const seeded = await seedReview(`approval:${approvalId}`); + await approvalService(db).create(companyId, { id: approvalId, type: "approve_ceo_strategy", payload: {}, + status: mutation === "resolution" ? "pending" : "approved" }); + if (mutation === "resolution") await issueApprovalService(db).link(seeded.issueId, approvalId); + await vi.advanceTimersByTimeAsync(1000); + await db.update(issueWorkProducts).set({ updatedAt: new Date() }).where(eq(issueWorkProducts.id, seeded.workProductId)); + await start(); await flush(); await quiet(); + expect(await status(seeded.issueId)).toBe("in_review"); + if (mutation === "resolution") await approvalService(db).approve(approvalId, "board-user"); + if (mutation === "link") await issueApprovalService(db).link(seeded.issueId, approvalId); + if (mutation === "bulk link") await issueApprovalService(db).linkManyForApproval(approvalId, [seeded.issueId]); + await flush(); + expect(await status(seeded.issueId)).toBe("done"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2); + await quiet(); + }); + + it("defers approval evidence until outer commit and ignores rolled-back approval", async () => { + const approvalId = randomUUID(); + const seeded = await seedReview(`approval:${approvalId}`); + await approvalService(db).create(companyId, { id: approvalId, type: "approve_ceo_strategy", payload: {} }); + await issueApprovalService(db).link(seeded.issueId, approvalId); + await vi.advanceTimersByTimeAsync(1000); + await db.update(issueWorkProducts).set({ updatedAt: new Date() }).where(eq(issueWorkProducts.id, seeded.workProductId)); + await start(); await flush(); await quiet(); + await expect(db.transaction(async tx => { + await approvalService(tx as unknown as typeof db).approve(approvalId, "board-user"); + await vi.advanceTimersByTimeAsync(1000); + expect(await status(seeded.issueId)).toBe("in_review"); + throw new Error("rollback approval evidence"); + })).rejects.toThrow("rollback approval evidence"); + await flush(); await quiet(); + expect(await status(seeded.issueId)).toBe("in_review"); + await db.transaction(async tx => { + await approvalService(tx as unknown as typeof db).approve(approvalId, "board-user"); + await vi.advanceTimersByTimeAsync(1000); + expect(await status(seeded.issueId)).toBe("in_review"); + }); + await flush(); + expect(await status(seeded.issueId)).toBe("done"); + await quiet(); + }); + + it.each([false, true])("preserves the required quality-review gate after real evidence approval (required=%s)", async required => { + const seeded = await seedReview(); + await db.update(issueWorkProducts).set({ reviewState: "none" }).where(eq(issueWorkProducts.id, seeded.workProductId)); + if (required) await db.update(issues).set({ executionPolicy: { requiredQualityReview: { + version: 1, deliverables: [{ key: "output", label: "Verified output" }], + } } }).where(eq(issues.id, seeded.issueId)); + await start(); + await flush(); + await quiet(); + await workProductService(db).update(seeded.workProductId, { reviewState: "approved" }); + await flush(); + expect(await status(seeded.issueId)).toBe(required ? "in_progress" : "done"); + const decisions = await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId)); + if (required) expect(decisions.find(row => row.id !== seeded.decisionId)?.reasonCode).toBe("required_quality_review_incomplete"); + await quiet(); + }); + + it.each(["update", "create", "primary replacement", "import", "attachment", "source promotion"] as const)("wakes from the real work-product %s writer", async writer => { + const seeded = await seedReview(); + if (writer === "update") await db.update(issueWorkProducts).set({ reviewState: "none" }).where(eq(issueWorkProducts.id, seeded.workProductId)); + if (writer === "create") await db.delete(issueWorkProducts).where(eq(issueWorkProducts.id, seeded.workProductId)); + if (writer === "source promotion") await db.update(issueWorkProducts).set({ sourceTrust: { + preset: LOW_TRUST_REVIEW_PRESET, disposition: "quarantined", sourceIssueId: seeded.issueId, + sourceRunId: seeded.runId, sourceAgentId: agentId, + } }).where(eq(issueWorkProducts.id, seeded.workProductId)); + await start(); await flush(); await quiet(); + expect(await status(seeded.issueId)).toBe("in_review"); + const products = workProductService(db); + if (writer === "update") await products.update(seeded.workProductId, { reviewState: "approved" }); + if (writer === "create" || writer === "primary replacement") await products.createForIssue(seeded.issueId, companyId, { + ...(writer === "create" ? { id: seeded.workProductId } : {}), + type: "artifact", provider: "paperclip", title: "Verified artifact", status: "approved", reviewState: "approved", + isPrimary: writer === "primary replacement", + }); + if (writer === "import") await products.createManyForImport([{ companyId, issueId: seeded.issueId, projectId: null, + type: "artifact", provider: "paperclip", externalId: null, title: "Imported evidence", url: null, status: "approved", + reviewState: "approved", isPrimary: false, healthStatus: "unknown", summary: null, metadata: null, + executionWorkspaceId: null, runtimeServiceId: null, createdByRunId: null, sourceTrust: null }]); + if (writer === "attachment") await issueService(db).createAttachment({ issueId: seeded.issueId, provider: "local_disk", + objectKey: randomUUID(), contentType: "text/plain", byteSize: 4, sha256: "a".repeat(64), originalFilename: "result.txt", + createdByAgentId: agentId, createdByRunId: seeded.runId }); + if (writer === "source promotion") { + const app = express(); app.use(express.json()); + app.use((req, _res, next) => { + req.actor = { type: "board", userId: "board-user", companyIds: [companyId], + memberships: [{ companyId, membershipRole: "operator", status: "active" }], + isInstanceAdmin: true, source: "local_implicit" }; next(); + }); + app.use("/api", issueRoutes(db, {} as any)); app.use(errorHandler); + const promoted = await request(app).post(`/api/issues/${seeded.issueId}/low-trust/promotions`).send({ + sourceArtifactKind: "work_product", sourceArtifactId: seeded.workProductId, + title: "Reviewed artifact", summary: "Reviewed completion evidence", + }); + expect(promoted.status, JSON.stringify(promoted.body)).toBe(201); + } + await flush(); + expect(await status(seeded.issueId)).toBe("done"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2); + await quiet(); + }); + + it("waits for an outer work-product transaction and ignores rollback", async () => { + const seeded = await seedReview(); + await db.update(issueWorkProducts).set({ reviewState: "none" }).where(eq(issueWorkProducts.id, seeded.workProductId)); + await start(); await flush(); await quiet(); + await expect(db.transaction(async tx => { + await workProductService(tx as unknown as typeof db).update(seeded.workProductId, { reviewState: "approved" }); + await vi.advanceTimersByTimeAsync(1000); + expect(await status(seeded.issueId)).toBe("in_review"); + throw new Error("rollback evidence approval"); + })).rejects.toThrow("rollback evidence approval"); + await flush(); await quiet(); + expect(await status(seeded.issueId)).toBe("in_review"); + await db.transaction(async tx => { + await workProductService(tx as unknown as typeof db).update(seeded.workProductId, { reviewState: "approved" }); + await vi.advanceTimersByTimeAsync(1000); + expect(await status(seeded.issueId)).toBe("in_review"); + }); + await flush(); + expect(await status(seeded.issueId)).toBe("done"); + await quiet(); + }); + + it.each(["status", "blocked reassertion", "assignment"] as const)("reassesses a real task %s mutation after quiet", async mutation => { + const seeded = await seedReview(); + if (mutation === "blocked reassertion") await issueService(db).update(seeded.issueId, { status: "blocked" }); + await start(); await flush(); await quiet(); + const before = await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId)); + await issueService(db).update(seeded.issueId, mutation === "assignment" ? { assigneeAgentId: null } : { status: "blocked" }); + await flush(); + const after = await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId)); + expect(after).toHaveLength(before.length + 1); + expect(await status(seeded.issueId)).toBe(mutation === "assignment" ? "in_review" : "blocked"); + const [current] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seeded.runId)); + const [assessment] = await db.select().from(workAssessments).where(eq(workAssessments.id, current.assessmentId!)); + const [issue] = await db.select().from(issues).where(eq(issues.id, seeded.issueId)); + expect(assessment.priorStatusVersion).toBe(issue.statusVersion); + await quiet(); + }); + + it("retains the first review-retirement wake through idle drain", async () => { + const seeded = await seedReview(); + await start(); await flush(); await quiet(); + admitted = false; + await dismissAutomaticCompletionReviews(db, seeded.issueId); + await quiet(); + expect(await status(seeded.issueId)).toBe("in_review"); + admitted = true; + await vi.advanceTimersByTimeAsync(30_000); await flush(); + expect(await status(seeded.issueId)).toBe("done"); + await quiet(); + }); + + it("reconstructs the saved retry after shutdown and never dispatches it early", async () => { + const seeded = await seedReview(); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + const future = new Date(Date.now() + 60_000); + await db.update(nativeRunFinalizations).set({ nextAttemptAt: future }).where(eq(nativeRunFinalizations.runId, seeded.runId)); + const waitForSavedDeadline = async () => { + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(1000); + expect(idleWorkSnapshot().active).toBe(0); + expect(coordinator!.nextWakeAt()).toBe(future.getTime()); + expect(await status(seeded.issueId)).toBe("in_review"); + }, { timeout: 15_000 }); + }; + await start(); + await waitForSavedDeadline(); + await vi.advanceTimersByTimeAsync(20_000); + expect(await status(seeded.issueId)).toBe("in_review"); + await coordinator!.stop(); coordinator = undefined; + await start(); + await waitForSavedDeadline(); + await vi.advanceTimersByTimeAsync(Math.max(0, future.getTime() - Date.now())); await flush(); + expect(await status(seeded.issueId)).toBe("done"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2); + await quiet(); + }); + + it("wakes from real review retirement after idle and becomes SQL quiet with completed history", async () => { + const seeded = await seedReview(); + await start(); await flush(); await quiet(); + expect(await status(seeded.issueId)).toBe("in_review"); + await dismissAutomaticCompletionReviews(db, seeded.issueId); + await flush(); + expect(await status(seeded.issueId)).toBe("done"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, seeded.issueId))).toHaveLength(2); + await quiet(); + }); +}); diff --git a/server/src/__tests__/photon/channel.integration.test.ts b/server/src/__tests__/photon/channel.integration.test.ts index 84985e0face..47e93a2d90a 100644 --- a/server/src/__tests__/photon/channel.integration.test.ts +++ b/server/src/__tests__/photon/channel.integration.test.ts @@ -1,3 +1,4 @@ +import { nativeStatusReassessmentWork } from "../../services/native-runtime/native-status-reassessment-work.js"; import { nativeReviewResponseWork } from "../../services/native-runtime/native-review-response-work.js"; import { nativeResponsePresentationWork } from "../../services/native-runtime/native-response-presentation-work.js"; import { DELIVERY_QUEUES } from "../../services/delivery-work-notifications.js"; @@ -377,7 +378,7 @@ describe("iMessage Photon channel control plane", () => { }, }; } - it.each([DELIVERY_QUEUES.nativeResponsePresentation, DELIVERY_QUEUES.nativeReviewResponse])( + it.each([DELIVERY_QUEUES.nativeResponsePresentation, DELIVERY_QUEUES.nativeReviewResponse, DELIVERY_QUEUES.nativeStatusReassessment])( "keeps repeated active Photon ownership assertions out of %s reconstruction and wakes once on recovery", async queue => { const t = await setup(); await t.start(); @@ -385,7 +386,8 @@ describe("iMessage Photon channel control plane", () => { const errors: unknown[] = []; const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) }); coordinators.push(coordinator); - const work = queue === DELIVERY_QUEUES.nativeResponsePresentation ? nativeResponsePresentationWork(db) : nativeReviewResponseWork(db); + const work = queue === DELIVERY_QUEUES.nativeResponsePresentation ? nativeResponsePresentationWork(db) + : queue === DELIVERY_QUEUES.nativeReviewResponse ? nativeReviewResponseWork(db) : nativeStatusReassessmentWork(db); const worker = work.register(coordinator, () => true); const debug = db.$client.options.debug; const hints: unknown[] = []; diff --git a/server/src/__tests__/server-startup-feedback-export.test.ts b/server/src/__tests__/server-startup-feedback-export.test.ts index feb240892e4..1198f45960e 100644 --- a/server/src/__tests__/server-startup-feedback-export.test.ts +++ b/server/src/__tests__/server-startup-feedback-export.test.ts @@ -116,6 +116,7 @@ const { reapOrphanedRuns: vi.fn(async () => ({ reaped: 0, runIds: [] })), registerScheduledRetryCancellationWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), registerStoppedInstructionCollectionWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), + registerNativeStatusReassessmentWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), registerNativeReviewResponseWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), registerNativeResponsePresentationWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve(), stop: vi.fn() })), registerNativePendingEffectWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), @@ -571,6 +572,13 @@ describe("startServer feedback export wiring", () => { } }); + it.each([false, true])("registers native reassessment only when maintenance is enabled (%s)", async enabled => { + loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled })); + await startServer(); await settleStartupHousekeeping(); + expect(heartbeatServiceMock.registerNativeStatusReassessmentWork).toHaveBeenCalledTimes(enabled ? 1 : 0); + if (enabled) expect(heartbeatServiceMock.registerNativeStatusReassessmentWork.mock.calls[0][1]()).toBe(true); + }); + it.each([false, true])("registers governed-response recovery only when maintenance is enabled (%s)", async enabled => { loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled })); await startServer(); await settleStartupHousekeeping(); @@ -1025,7 +1033,7 @@ describe("startServer feedback export wiring", () => { intervalCallback?.(); await new Promise((resolve) => setImmediate(resolve)); expect(heartbeatServiceMock.sweepStaleIssueLocks).toHaveBeenCalledTimes(2); - expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true }); + expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true, skipNativeStatusReassessment: true }); expect(heartbeatServiceMock.reconcileCostAccounting).toHaveBeenCalledTimes(1); expect(retiredDetector).not.toHaveBeenCalled(); } finally { diff --git a/server/src/__tests__/tool-access-service.test.ts b/server/src/__tests__/tool-access-service.test.ts index 9aaf534dc1a..37a00be75e6 100644 --- a/server/src/__tests__/tool-access-service.test.ts +++ b/server/src/__tests__/tool-access-service.test.ts @@ -1,3 +1,4 @@ +import { DELIVERY_QUEUES, subscribeDeliveryWork } from "../services/delivery-work-notifications.js"; import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; import * as fs from "node:fs/promises"; import { createHash, generateKeyPairSync, randomUUID } from "node:crypto"; @@ -10088,6 +10089,45 @@ describeEmbeddedPostgres("tool access service", () => { }); + it.each(["creation", "reopening", "rollback"] as const)("signals native review recovery after OAuth card %s settles", async mode => { + vi.stubEnv("PAPERCLIP_TOOL_OAUTH_SLACK_CLIENT_ID", "slack-client-id"); + vi.stubEnv("PAPERCLIP_TOOL_OAUTH_SLACK_CLIENT_SECRET", "slack-client-secret"); + const company = await createCompany(db); + const agent = await createAgent(db, company.id); + const { issue, run } = await createIssueAndRun(db, company.id, agent.id); + const connected = await createTestToolAccessService(db).connectGalleryApp(company.id, { galleryKey: "slack", name: "Slack user auth" }); + if (mode !== "creation") await db.insert(issueThreadInteractions).values({ companyId: company.id, + issueId: issue.id, kind: "request_confirmation", status: "accepted", payload: { version: 1, prompt: "Previous authorization" }, + idempotencyKey: `connection-authorization:${connected.connectionId}:user-for-run`, sourceRunId: run.id }); + const reassessment = vi.fn(), reviewResponse = vi.fn(); + const unsubscribe = [subscribeDeliveryWork(db, DELIVERY_QUEUES.nativeStatusReassessment, reassessment), + subscribeDeliveryWork(db, DELIVERY_QUEUES.nativeReviewResponse, reviewResponse)]; + const rollback = new Error("rollback OAuth reopening"); + try { + const operation = db.transaction(async tx => { + const started = await createTestToolAccessService(tx as unknown as typeof db).startOAuth(company.id, connected.connectionId, { + redirectUri: "https://paperclip.example/api/tools/oauth/callback", actor: { actorType: "agent", actorId: agent.id, sessionId: run.id }, + issueId: issue.id, subjectUserId: "user-for-run", scopes: ["channels:read"], + }); + const [inside] = await tx.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.issueId, issue.id)); + expect(inside.status).toBe("pending"); + expect(inside.payload).toMatchObject({ target: { href: started.authorizationUrl } }); + expect(reassessment).not.toHaveBeenCalled(); expect(reviewResponse).not.toHaveBeenCalled(); + const [outside] = await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.issueId, issue.id)); + expect(outside?.status).toBe(mode === "creation" ? undefined : "accepted"); + if (mode === "rollback") throw rollback; + }); + if (mode === "rollback") await expect(operation).rejects.toBe(rollback); else await operation; + + + expect(reassessment).toHaveBeenCalledTimes(1); expect(reviewResponse).toHaveBeenCalledTimes(1); + const [persisted] = await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.issueId, issue.id)); + expect(persisted.status).toBe(mode === "rollback" ? "accepted" : "pending"); + expect(await db.select().from(toolOauthStates).where(eq(toolOauthStates.issueId, issue.id))) + .toHaveLength(mode === "rollback" ? 0 : 1); + } finally { unsubscribe.forEach(stop => stop()); } + }); + it("creates and resolves an agent-initiated user authorization grant card", async () => { vi.stubEnv("PAPERCLIP_TOOL_OAUTH_SLACK_CLIENT_ID", "slack-client-id"); vi.stubEnv( diff --git a/server/src/__tests__/workspace-operations-reconciliation.test.ts b/server/src/__tests__/workspace-operations-reconciliation.test.ts index 03d1d80bbeb..4414d511432 100644 --- a/server/src/__tests__/workspace-operations-reconciliation.test.ts +++ b/server/src/__tests__/workspace-operations-reconciliation.test.ts @@ -1,3 +1,4 @@ +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; import { randomUUID } from "node:crypto"; import fs from "node:fs/promises"; import os from "node:os"; @@ -65,7 +66,8 @@ describe("workspace runtime control serialization", () => { }); }); -const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); +const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_WORKSPACE_OPERATIONS_TEST_DATABASE_URL"); +const embeddedPostgresSupport = externalUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; if (!embeddedPostgresSupport.supported) { @@ -81,8 +83,8 @@ describeEmbeddedPostgres("workspace operation reconciliation", () => { let previousLogRoot: string | undefined; beforeAll(async () => { - tempDb = await startEmbeddedPostgresTestDatabase("paperclip-workspace-operation-reconcile-"); - db = createDb(tempDb.connectionString); + if (!externalUrl) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-workspace-operation-reconcile-"); + db = createDb(externalUrl ?? tempDb!.connectionString); logRoot = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-workspace-operation-logs-")); previousLogRoot = process.env.WORKSPACE_OPERATION_LOG_BASE_PATH; process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = logRoot; @@ -99,6 +101,7 @@ describeEmbeddedPostgres("workspace operation reconciliation", () => { if (previousLogRoot === undefined) delete process.env.WORKSPACE_OPERATION_LOG_BASE_PATH; else process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = previousLogRoot; await fs.rm(logRoot, { recursive: true, force: true }); + await db?.$client.end({ timeout: 1 }); await tempDb?.cleanup(); }); diff --git a/server/src/index.ts b/server/src/index.ts index 9e25bd0a284..7202f5f7361 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1825,6 +1825,8 @@ async function startServerWithDatabaseTeardown( () => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready; if (config.heartbeatSchedulerEnabled) await heartbeat.registerInstructionDirectoryCleanupWork(deliveryWork, () => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready; + if (config.heartbeatSchedulerEnabled) await heartbeat.registerNativeStatusReassessmentWork(deliveryWork, + () => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready; if (config.heartbeatSchedulerEnabled) await heartbeat.registerNativeReviewResponseWork(deliveryWork, () => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready; if (config.heartbeatSchedulerEnabled) { @@ -1974,7 +1976,7 @@ async function startServerWithDatabaseTeardown( // Periodically reap orphaned runs (5-min staleness threshold) and make sure // persisted queued work is still being driven forward. runPeriodicRecovery(() => heartbeat - .reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true }) + .reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true, skipNativeStatusReassessment: true }) .then(() => heartbeat.promoteDueScheduledRetries()) .then(async (promotion) => { await heartbeat.resumeQueuedRuns(); diff --git a/server/src/modules/active-run-watchdog/adapters/postgres.ts b/server/src/modules/active-run-watchdog/adapters/postgres.ts index 42267449092..94eb3b154bc 100644 --- a/server/src/modules/active-run-watchdog/adapters/postgres.ts +++ b/server/src/modules/active-run-watchdog/adapters/postgres.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "../../../services/native-status-reassessment-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "../../../services/unsafe-workspace-export-notifications.js"; import { notifyCancelledWorkspacePreservation } from "../../../services/cancelled-workspace-preservation-notifications.js"; import { notifyInstructionCollection } from "../../../services/instruction-collection-notifications.js"; @@ -343,6 +344,7 @@ export function createPostgresWatchdogAdapter(db: Db): WatchdogRunReader & Watch await notifyInstructionCollection(tx, { companyId, runId: input.run.id }); await notifyCancelledWorkspacePreservation(tx, { companyId, runId: input.run.id }); await notifyUnsafeWorkspaceExportRecovery(tx, { companyId, issueId: input.sourceIssue.id }); + await notifyNativeStatusReassessment(tx, { companyId, issueId: input.sourceIssue.id }); const [updatedRun] = await tx .update(heartbeatRuns) .set({ diff --git a/server/src/modules/wake-queue/adapters/postgres.test.ts b/server/src/modules/wake-queue/adapters/postgres.test.ts index 79f5ea78968..068a21f2420 100644 --- a/server/src/modules/wake-queue/adapters/postgres.test.ts +++ b/server/src/modules/wake-queue/adapters/postgres.test.ts @@ -1224,7 +1224,7 @@ describeEmbeddedPostgres("wake-queue postgres adapter", () => { expect(wakeRow?.payload).not.toHaveProperty("foo"); }); - it.each([DELIVERY_QUEUES.nativeResponsePresentation, DELIVERY_QUEUES.nativeReviewResponse])( + it.each([DELIVERY_QUEUES.nativeResponsePresentation, DELIVERY_QUEUES.nativeReviewResponse, DELIVERY_QUEUES.nativeStatusReassessment])( "scopes a real coalesced-input %s response wake to its active run after commit", async queue => { const companyId = await seedCompany(); const agentId = await seedAgent({ companyId }); diff --git a/server/src/modules/wake-queue/adapters/postgres.ts b/server/src/modules/wake-queue/adapters/postgres.ts index 0c80fd6f5d9..bc2c06f4cea 100644 --- a/server/src/modules/wake-queue/adapters/postgres.ts +++ b/server/src/modules/wake-queue/adapters/postgres.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "../../../services/native-status-reassessment-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "../../../services/unsafe-workspace-export-notifications.js"; import { notifyChatInboundWakeNotices } from "../../../services/chat-inbound-wake-notice-notifications.js"; import { notifyChatRunMilestones } from "../../../services/chat-run-milestone-notifications.js"; @@ -1125,6 +1126,7 @@ export function createPostgresWakeQueueAdapter(db: Db, deps: WakeQueuePostgresAd for (const issue of candidateIssues) { if (issue.executionRunId === run.id || issue.checkoutRunId === run.id) { await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: input.companyId, issueId: issue.id }); + await notifyNativeStatusReassessment(tx, { companyId: input.companyId, issueId: issue.id }); } } // Two separate updates: a retry can move `executionRunId` to a new run diff --git a/server/src/modules/wake-queue/adapters/queued-comment-postgres.ts b/server/src/modules/wake-queue/adapters/queued-comment-postgres.ts index 455c2109050..02b02622968 100644 --- a/server/src/modules/wake-queue/adapters/queued-comment-postgres.ts +++ b/server/src/modules/wake-queue/adapters/queued-comment-postgres.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "../../../services/native-status-reassessment-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "../../../services/unsafe-workspace-export-notifications.js"; import { notifyCancelledWorkspacePreservation } from "../../../services/cancelled-workspace-preservation-notifications.js"; import { notifyInstructionCollection } from "../../../services/instruction-collection-notifications.js"; @@ -120,6 +121,7 @@ function buildTransaction(tx: Db, companyId: string, issueId: string, deps: Queu }, async clearExecutionLockAndTouchIssue({ issueId, executionRunId, updatedAt }) { + await notifyNativeStatusReassessment(tx, { companyId, issueId }); await notifyUnsafeWorkspaceExportRecovery(tx, { companyId, issueId }); await tx .update(issues) diff --git a/server/src/routes/issues.ts b/server/src/routes/issues.ts index 74241e50965..e191452d97a 100644 --- a/server/src/routes/issues.ts +++ b/server/src/routes/issues.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "../services/native-status-reassessment-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "../services/unsafe-workspace-export-notifications.js"; import { notifyChatInboundWakeNotices } from "../services/chat-inbound-wake-notice-notifications.js"; import { completeIssueCommentReceipt } from "../services/issue-comment-receipt.js"; @@ -11816,6 +11817,7 @@ export function issueRoutes( promotedAt, }); const product = await db.transaction(async (tx) => { + await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId: issue.id }); const markPromoted = { sourceTrust: promotionTrust, updatedAt: promotedAt, diff --git a/server/src/services/access.ts b/server/src/services/access.ts index 64f57cb1f1a..81a0c7222c7 100644 --- a/server/src/services/access.ts +++ b/server/src/services/access.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js"; import { notifyChatCompanyAuthorityChange } from "./chat-run-milestone-notifications.js"; import { notifyVoiceChange } from "./voice/voice-recovery-notifications.js"; @@ -740,6 +741,7 @@ export function accessService(db: Db) { eq(issues.assigneeUserId, existing.principalId), sql`${issues.status} not in ('done', 'cancelled')`, ); + await notifyNativeStatusReassessment(tx, { companyId }); const resetInProgress = await tx .update(issues) .set({ diff --git a/server/src/services/approval-work-notifications.ts b/server/src/services/approval-work-notifications.ts index 6b74cb6c665..a4308d32522 100644 --- a/server/src/services/approval-work-notifications.ts +++ b/server/src/services/approval-work-notifications.ts @@ -1,6 +1,6 @@ import { and, eq } from "drizzle-orm"; import { approvals, issueApprovals, type Db } from "@paperclipai/db"; -import { notifyNativeReviewResponse } from "./native-review-response-notifications.js"; +import { notifyNativeReviewState } from "./native-review-state-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; export const notifyApprovalQueueWork = (transaction: object, companyId: string) => notifyDeliveryWork(transaction, DELIVERY_QUEUES.approvalQueueSeeding, companyId); @@ -21,7 +21,7 @@ export async function notifyApprovalWork(transaction: object, companyId: string, .where(and(eq(issueApprovals.companyId, companyId), eq(issueApprovals.approvalId, scope.approvalId))); for (const link of links) issueIds.add(link.issueId); } - for (const issueId of issueIds) await notifyNativeReviewResponse(transaction, { companyId, issueId }); + for (const issueId of issueIds) await notifyNativeReviewState(transaction, { companyId, issueId }); } export function writeApprovalChange(db: Db, companyId: string, write: (transaction: Db) => Promise, scope?: ApprovalChangeScope): Promise { diff --git a/server/src/services/chat-run-milestone-notifications.ts b/server/src/services/chat-run-milestone-notifications.ts index 6736b0e6ffc..62e5cb32ac4 100644 --- a/server/src/services/chat-run-milestone-notifications.ts +++ b/server/src/services/chat-run-milestone-notifications.ts @@ -1,4 +1,4 @@ -import { notifyNativeReviewResponse } from "./native-review-response-notifications.js"; +import { notifyNativeReviewState } from "./native-review-state-notifications.js"; import { notifyNativeResponsePresentation } from "./native-response-presentation-notifications.js"; import { notifyChatResponseAvailability } from "./chat-response-availability.js"; import { notifyChatConfirmationWork } from "./chat-confirmation-notifications.js"; @@ -12,7 +12,7 @@ export type ChatRunMilestoneScope = { companyId: string; issueId?: string; runId export async function notifyChatRunMilestones(tx: object, scope: ChatRunMilestoneScope) { if (scope.issueId || scope.runId) { - await notifyNativeReviewResponse(tx, { companyId: scope.companyId, issueId: scope.issueId, + await notifyNativeReviewState(tx, { companyId: scope.companyId, issueId: scope.issueId, runId: scope.issueId ? undefined : scope.runId }); await notifyNativeResponsePresentation(tx, { companyId: scope.companyId, issueId: scope.issueId, runId: scope.issueId ? undefined : scope.runId }); @@ -33,5 +33,5 @@ export async function notifyChatRunMilestones(tx: object, scope: ChatRunMileston export async function notifyChatCompanyAuthorityChange(tx: object, scope: { companyId: string }) { await notifyChatRunMilestones(tx, scope); await notifyNativeResponsePresentation(tx, scope); - await notifyNativeReviewResponse(tx, scope); + await notifyNativeReviewState(tx, scope); } diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index b1b72704e53..057f9a6c242 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -2,6 +2,7 @@ import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db"; /** Topics for existing durable queues; register intent before their writes. */ export const DELIVERY_QUEUES = { + nativeStatusReassessment: "native-status-reassessment", nativeReviewResponse: "native-review-response", nativeResponsePresentation: "native-response-presentation", nativePendingEffects: "native-pending-effects", diff --git a/server/src/services/execution-control-reconciliation.ts b/server/src/services/execution-control-reconciliation.ts index 3bbe612e612..7f57bcbb8e1 100644 --- a/server/src/services/execution-control-reconciliation.ts +++ b/server/src/services/execution-control-reconciliation.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js"; import { notifyNativeFinalizationWork } from "./native-finalization-notifications.js"; import { notifyUnsafeWorkspaceExportRunChange } from "./unsafe-workspace-export-notifications.js"; import { notifyCancelledWorkspacePreservation } from "./cancelled-workspace-preservation-notifications.js"; @@ -246,6 +247,7 @@ export async function reconcileAbandonedExecutionControl( if (updatedRun) await preserveNativeFailedWorkspace(tx as unknown as Db, updatedRun); return true; } + await notifyNativeStatusReassessment(tx, { companyId: task.companyId, issueId: task.id }); await tx .update(issues) .set({ executionRunId: null, checkoutRunId: null, updatedAt: now }) diff --git a/server/src/services/execution-recovery-resolution.ts b/server/src/services/execution-recovery-resolution.ts index 2d190156d6c..725fba0eb55 100644 --- a/server/src/services/execution-recovery-resolution.ts +++ b/server/src/services/execution-recovery-resolution.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js"; import { notifyChatResponseAvailability } from "./chat-response-availability.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; @@ -591,6 +592,7 @@ export async function settleUnrecoverableExecutions( let nativeFailureBlock = action.evidence.nativeFailureBlock; let nativeBootstrapFailureBlock = action.evidence.nativeBootstrapFailureBlock; if (current) { + await notifyNativeStatusReassessment(tx, { companyId: task.companyId, issueId: task.id }); if (task.status !== "blocked") await notifyChatResponseAvailability(tx, { companyId: task.companyId, issueId: task.id }); const [projected] = await tx .update(issues) diff --git a/server/src/services/heartbeat-run-events.ts b/server/src/services/heartbeat-run-events.ts index b6b1b694a49..ebce78a8266 100644 --- a/server/src/services/heartbeat-run-events.ts +++ b/server/src/services/heartbeat-run-events.ts @@ -1,3 +1,5 @@ +import { readNativeEventEvidenceOutcome } from "./native-runtime/evidence-classifier.js"; +import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js"; import { isNativeResponseEvidence, notifyNativeResponsePresentation } from "./native-response-presentation-notifications.js"; import { isSafeNativeChatProgressEventType } from "./safe-native-chat-progress.js"; import { notifyChatNativeProgress } from "./chat-native-progress-notifications.js"; @@ -72,9 +74,10 @@ export async function appendHeartbeatRunEvent( db: Db, input: AppendHeartbeatRunEventInput, ): Promise { + const projectsReassessment = readNativeEventEvidenceOutcome(input.payload?.prpEvent) !== null; const projectsProgress = isSafeNativeChatProgressEventType(input.eventType); const projectsResponse = isNativeResponseEvidence(input.eventType, input.payload?.prpEvent); - if (!input.nativeSource && !input.retryExhaustion && !projectsProgress && !projectsResponse) { + if (!input.nativeSource && !input.retryExhaustion && !projectsProgress && !projectsResponse && !projectsReassessment) { const allocated = db.$with("allocated_run_event", {}).as(sql` update ${heartbeatRuns} set next_event_seq = ${heartbeatRuns.nextEventSeq} + 1, @@ -103,7 +106,7 @@ export async function appendHeartbeatRunEvent( .select({ companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, // Only progress needs the task binding. Other native/recovery events // must not read a potentially large context snapshot to append a log. - ...(projectsProgress || projectsResponse ? { runtimeMode: heartbeatRuns.runtimeMode } : {}), + ...(projectsProgress || projectsResponse || projectsReassessment ? { runtimeMode: heartbeatRuns.runtimeMode } : {}), ...(projectsProgress ? { issueId: boundRunIssueIdSql() } : {}) }) .from(heartbeatRuns) .where(eq(heartbeatRuns.id, input.runId)) @@ -175,6 +178,9 @@ export async function appendHeartbeatRunEvent( } } + if (projectsReassessment && run.runtimeMode === "native") { + await notifyNativeStatusReassessment(tx, { companyId: run.companyId, runId: input.runId }); + } if (projectsProgress && run.runtimeMode === "native" && run.issueId) { await notifyChatNativeProgress(tx, { companyId: run.companyId, issueId: run.issueId }); } diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 2aab7492427..17574152073 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -1,3 +1,4 @@ +import { nativeStatusReassessmentWork } from "./native-runtime/native-status-reassessment-work.js"; import { registerNativeReviewResponseWork } from "./native-runtime/native-review-response-work.js"; import { registerNativeResponsePresentationWork } from "./native-runtime/native-response-presentation-work.js"; import { registerNativePendingEffectWork } from "./native-runtime/native-pending-effect-work.js"; @@ -4091,6 +4092,8 @@ export function heartbeatService( triggerIssueMonitor, reportRunActivity: clearDetachedRunWarning, + registerNativeStatusReassessmentWork: (coordinator: Parameters["register"]>[0], canRun: () => boolean) => + nativeStatusReassessmentWork(db).register(coordinator, () => canRun() && readTaskDrain(new Date()) === null), registerNativeReviewResponseWork: (coordinator: Parameters[1], canRun: () => boolean) => registerNativeReviewResponseWork(db, coordinator, () => canRun() && readTaskDrain(new Date()) === null), registerNativeResponsePresentationWork: (coordinator: Parameters[1], canRun: () => boolean) => diff --git a/server/src/services/heartbeat/queue.ts b/server/src/services/heartbeat/queue.ts index 2be0fd457f7..4de5b9fb07d 100644 --- a/server/src/services/heartbeat/queue.ts +++ b/server/src/services/heartbeat/queue.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery, notifyUnsafeWorkspaceExportRunChange } from "../unsafe-workspace-export-notifications.js"; import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js"; import { notifyInstructionCollection } from "../instruction-collection-notifications.js"; @@ -1803,6 +1804,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende await db.transaction(async tx => { await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: run.companyId, issueId }); + await notifyNativeStatusReassessment(tx, { companyId: run.companyId, issueId }); await tx .update(issues) .set({ @@ -3694,6 +3696,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende } if (!activeExecutionRun && issue.executionRunId) { + await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId: issue.id }); await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: issue.companyId, issueId: issue.id }); await tx .update(issues) @@ -3893,6 +3896,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende `- Next action: ${WORKSPACE_WORKTREE_REQUIRES_PROJECT_REMEDIATION}`, ].join("\n"); await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: issue.companyId, issueId: issue.id }); + await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId: issue.id }); if (issue.status !== "blocked") await notifyChatResponseAvailability(tx, { companyId: issue.companyId, issueId: issue.id }); await tx .update(issues) diff --git a/server/src/services/heartbeat/recovery.ts b/server/src/services/heartbeat/recovery.ts index 7225dba15b5..e6c3e03c084 100644 --- a/server/src/services/heartbeat/recovery.ts +++ b/server/src/services/heartbeat/recovery.ts @@ -1947,7 +1947,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD let settledProcessesDiscovered = false; - async function reapOrphanedRuns(opts?: { staleThresholdMs?: number; skipScheduledRetryCancellation?: boolean; skipCapturedInstructionCommit?: boolean; skipInstructionDirectoryCleanup?: boolean; skipStoppedInstructionCollection?: boolean; skipCancelledWorkspacePreservation?: boolean; skipObsoletePolicyReviews?: boolean; skipAutomaticCompletionReviews?: boolean; skipUnsafeWorkspaceExports?: boolean; skipNativePendingEffects?: boolean; skipNativeResponsePresentation?: boolean; skipNativeReviewResponse?: boolean }) { + async function reapOrphanedRuns(opts?: { staleThresholdMs?: number; skipScheduledRetryCancellation?: boolean; skipCapturedInstructionCommit?: boolean; skipInstructionDirectoryCleanup?: boolean; skipStoppedInstructionCollection?: boolean; skipCancelledWorkspacePreservation?: boolean; skipObsoletePolicyReviews?: boolean; skipAutomaticCompletionReviews?: boolean; skipUnsafeWorkspaceExports?: boolean; skipNativePendingEffects?: boolean; skipNativeResponsePresentation?: boolean; skipNativeReviewResponse?: boolean; skipNativeStatusReassessment?: boolean }) { if (!opts?.skipCancelledWorkspacePreservation) await preserveCancelledNativeWorkspaces(db); const staleThresholdMs = opts?.staleThresholdMs ?? 0; const now = new Date(); @@ -1971,6 +1971,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD skipNativePendingEffects: opts?.skipNativePendingEffects, skipNativeResponsePresentation: opts?.skipNativeResponsePresentation, skipNativeReviewResponse: opts?.skipNativeReviewResponse, + skipNativeStatusReassessment: opts?.skipNativeStatusReassessment, environmentRuntime, pluginWorkerManager: dependencies.pluginWorkerManager, onWorkspaceSettled: settleRecoveredNativeWorkspace, diff --git a/server/src/services/issue-thread-interactions.ts b/server/src/services/issue-thread-interactions.ts index 295cf894a36..38bbdb657c8 100644 --- a/server/src/services/issue-thread-interactions.ts +++ b/server/src/services/issue-thread-interactions.ts @@ -1,4 +1,5 @@ -import { notifyNativeReviewResponse } from "./native-review-response-notifications.js"; +import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js"; +import { notifyNativeReviewState } from "./native-review-state-notifications.js"; import { notifyChatConfirmationResolution } from "./chat-confirmation-notifications.js"; import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js"; import { notifyToolReviewChange } from "./tool-review-work-notifications.js"; @@ -1945,7 +1946,7 @@ async function expireStaleRequestConfirmationTarget( const expireInTransaction = async (tx: Db | any) => { await notifyChatConfirmationResolution(tx, args.row); if (args.row.kind === "request_confirmation") await notifyChatRunMilestones(tx, args.row); - else await notifyNativeReviewResponse(tx, args.row); + else await notifyNativeReviewState(tx, args.row); const [updated] = await tx .update(issueThreadInteractions) .set({ @@ -2225,7 +2226,7 @@ export function issueThreadInteractionService( await assertInteractionRunWriteAllowed(tx as unknown as Db, args.issue, args.actor); await notifyChatConfirmationResolution(tx, args.current); if (args.current.kind === "request_confirmation") await notifyChatRunMilestones(tx, args.current); - else await notifyNativeReviewResponse(tx, args.current); + else await notifyNativeReviewState(tx, args.current); await args.mutationOptions?.beforeResolveInTransaction?.(tx); // Policy mutations and review transitions use the same issue-row lock, // so the authoritative review policy and requester are stable through @@ -2516,7 +2517,7 @@ export function issueThreadInteractionService( await assertInteractionRunWriteAllowed(tx as unknown as Db, args.issue, args.actor); await notifyChatConfirmationResolution(tx, args.current); if (args.current.kind === "request_confirmation") await notifyChatRunMilestones(tx, args.current); - else await notifyNativeReviewResponse(tx, args.current); + else await notifyNativeReviewState(tx, args.current); await args.mutationOptions?.beforeResolveInTransaction?.(tx); const issueContext = await tx .select({ @@ -2753,6 +2754,7 @@ export function issueThreadInteractionService( .where(and(eq(heartbeatRuns.id, input.sourceRunId), eq(heartbeatRuns.companyId, issue.companyId))); if (!sourceRun) throw unprocessable("Interaction source run is unavailable"); const originCommentIds = await currentContinuationOrigins(tx as unknown as Db, issue.companyId, issue.id, sourceRun.context); + await notifyNativeReviewState(tx, { companyId: issue.companyId, issueId: issue.id }); const [row] = await tx .insert(issueThreadInteractions) .values({ @@ -2909,7 +2911,7 @@ export function issueThreadInteractionService( const updated = await db.transaction(async (tx) => { await notifyChatConfirmationResolution(tx, current); await mutationOptions.beforeResolveInTransaction?.(tx); - await notifyNativeReviewResponse(tx, current); + await notifyNativeReviewState(tx, current); const [row] = await tx .update(issueThreadInteractions) .set({ @@ -3345,7 +3347,7 @@ export function issueThreadInteractionService( const updated = await db.transaction(async (tx) => { await notifyChatConfirmationResolution(tx, row); if (["ask_user_questions", "request_confirmation"].includes(row.kind)) await notifyChatRunMilestones(tx, row); - else await notifyNativeReviewResponse(tx, row); + else await notifyNativeReviewState(tx, row); if (row.kind === "request_confirmation") await notifyToolProjectionForInteractions(tx, row.companyId, [row.id]); const [resolved] = await tx .update(issueThreadInteractions) @@ -3433,7 +3435,7 @@ export function issueThreadInteractionService( const updated = await db.transaction(async (tx) => { await notifyChatConfirmationResolution(tx, row); if (["ask_user_questions", "request_confirmation"].includes(row.kind)) await notifyChatRunMilestones(tx, row); - else await notifyNativeReviewResponse(tx, row); + else await notifyNativeReviewState(tx, row); if (row.kind === "request_confirmation") await notifyToolProjectionForInteractions(tx, row.companyId, [row.id]); const [updatedRow] = await tx .update(issueThreadInteractions) @@ -3863,6 +3865,8 @@ export function issueThreadInteractionService( } if (data.kind === "ask_user_questions" || data.kind === "request_confirmation") { await notifyChatRunMilestones(tx, { companyId: issue.companyId, issueId: issue.id }); + } else { + await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId: issue.id }); } const [row] = await tx .insert(issueThreadInteractions) @@ -4215,7 +4219,7 @@ export function issueThreadInteractionService( } } } - await notifyNativeReviewResponse(tx, { companyId: issue.companyId, issueId: issue.id }); + await notifyNativeReviewState(tx, { companyId: issue.companyId, issueId: issue.id }); const resolvedAt = new Date(); const [claimed] = await tx .update(issueThreadInteractions) @@ -4479,7 +4483,8 @@ export function issueThreadInteractionService( complete, items, } satisfies RequestItemVerdictsResult; - if (complete) await notifyNativeReviewResponse(tx, { companyId: issue.companyId, issueId: issue.id }); + if (complete) await notifyNativeReviewState(tx, { companyId: issue.companyId, issueId: issue.id }); + else await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId: issue.id }); const [updated] = await tx .update(issueThreadInteractions) .set({ @@ -4546,7 +4551,7 @@ export function issueThreadInteractionService( await notifyChatConfirmationResolution(tx, current); await mutationOptions.beforeResolveInTransaction?.(tx); await assertInteractionRunWriteAllowed(tx as unknown as Db, issue, actor); - await notifyNativeReviewResponse(tx, { companyId: issue.companyId, issueId: issue.id }); + await notifyNativeReviewState(tx, { companyId: issue.companyId, issueId: issue.id }); const rows = await tx.update(issueThreadInteractions) .set({ status: "rejected", @@ -4638,7 +4643,7 @@ export function issueThreadInteractionService( const updated = await db.transaction(async (tx) => { await notifyChatConfirmationResolution(tx, row); if (["ask_user_questions", "request_confirmation"].includes(row.kind)) await notifyChatRunMilestones(tx, row); - else await notifyNativeReviewResponse(tx, row); + else await notifyNativeReviewState(tx, row); if (row.kind === "connection_intent") { await tx .delete(toolOauthStates) @@ -4816,7 +4821,7 @@ export function issueThreadInteractionService( const updatedInteractions = await db.transaction(async (tx) => { await notifyChatConfirmationResolution(tx, sampleConfirmationRow); if (commentRows.some(row => row.kind === "request_confirmation")) await notifyChatRunMilestones(tx, sampleConfirmationRow); - else await notifyNativeReviewResponse(tx, sampleConfirmationRow); + else await notifyNativeReviewState(tx, sampleConfirmationRow); await notifyToolProjectionForInteractions(tx, issue.companyId, confirmationRowIds); const updatedRows = await tx .update(issueThreadInteractions) @@ -4852,7 +4857,7 @@ export function issueThreadInteractionService( for (const row of itemVerdictRows) { const updated = await db.transaction(async tx => { - await notifyNativeReviewResponse(tx, row); + await notifyNativeReviewState(tx, row); const [retired] = await tx .update(issueThreadInteractions) .set({ @@ -4878,7 +4883,7 @@ export function issueThreadInteractionService( for (const row of connectionIntentRows) { const [updated] = await db.transaction(async (tx) => { - await notifyNativeReviewResponse(tx, row); + await notifyNativeReviewState(tx, row); await notifyChatConfirmationResolution(tx, row); await tx .delete(toolOauthStates) @@ -4967,7 +4972,7 @@ export function issueThreadInteractionService( const updated = await db.transaction(async (tx) => { await notifyChatConfirmationResolution(tx, row); if (row.kind === "request_confirmation") await notifyChatRunMilestones(tx, row); - else await notifyNativeReviewResponse(tx, row); + else await notifyNativeReviewState(tx, row); const [resolved] = await tx .update(issueThreadInteractions) .set({ @@ -5031,7 +5036,7 @@ export function issueThreadInteractionService( ? "the AI account" : `the ${payload?.serviceName?.trim() || "requested"} connection`; const updated = await db.transaction(async tx => { - await notifyNativeReviewResponse(tx, { companyId: issue.companyId, issueId: issue.id }); + await notifyNativeReviewState(tx, { companyId: issue.companyId, issueId: issue.id }); await notifyChatConfirmationResolution(tx, { companyId: issue.companyId, issueId: issue.id, kind: "connection_intent" }); const [resolved] = await tx .update(issueThreadInteractions) @@ -5093,7 +5098,7 @@ export function issueThreadInteractionService( .transaction(async (tx) => { await notifyChatConfirmationResolution(tx, row); if (["ask_user_questions", "request_confirmation"].includes(row.kind)) await notifyChatRunMilestones(tx, row); - else await notifyNativeReviewResponse(tx, row); + else await notifyNativeReviewState(tx, row); if (row.kind === "connection_intent") { await tx .delete(toolOauthStates) @@ -5186,7 +5191,7 @@ export function issueThreadInteractionService( await assertInteractionRunWriteAllowed(tx as unknown as Db, issue, actor); await notifyChatConfirmationResolution(tx, current); if (["ask_user_questions", "request_confirmation"].includes(current.kind)) await notifyChatRunMilestones(tx, current); - else await notifyNativeReviewResponse(tx, current); + else await notifyNativeReviewState(tx, current); await resolveLinkedToolActionRequests(tx, current, { status: "cancelled", fromStatuses: ["pending", "approved"], @@ -5449,7 +5454,7 @@ export function issueThreadInteractionService( await assertInteractionRunWriteAllowed(tx as unknown as Db, issue, actor); await notifyChatConfirmationResolution(tx, current); if (["ask_user_questions", "request_confirmation"].includes(current.kind)) await notifyChatRunMilestones(tx, current); - else await notifyNativeReviewResponse(tx, current); + else await notifyNativeReviewState(tx, current); await resolveLinkedToolActionRequests(tx, current, { status: "cancelled", fromStatuses: ["pending", "approved"], diff --git a/server/src/services/issue-tree-control.ts b/server/src/services/issue-tree-control.ts index 5697e19449c..d66d3f0732a 100644 --- a/server/src/services/issue-tree-control.ts +++ b/server/src/services/issue-tree-control.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js"; import { notifyChatResponseAvailability } from "./chat-response-availability.js"; import { notifyChatInboundWakeNotices } from "./chat-inbound-wake-notice-notifications.js"; @@ -1002,6 +1003,7 @@ export function issueTreeControlService(db: Db) { for (const [status, issueIdsForStatus] of issueIdsByStatus) { if (issueIdsForStatus.length === 0) continue; for (const issueId of issueIdsForStatus) { + await notifyNativeStatusReassessment(tx, { companyId, issueId }); await notifyTerminalWorkspaceWork(tx, `issue:${issueId}`); await notifyChatResponseAvailability(tx, { companyId, issueId }); if (status === "blocked" || status === "in_review") await notifyUnsafeWorkspaceExportRecovery(tx, { companyId, issueId }); diff --git a/server/src/services/issues.ts b/server/src/services/issues.ts index 23a0f9c1c99..8deb995df11 100644 --- a/server/src/services/issues.ts +++ b/server/src/services/issues.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js"; import { notifyChatResponseAvailability } from "./chat-response-availability.js"; import { notifyChatInboundWakeNotices } from "./chat-inbound-wake-notice-notifications.js"; @@ -2513,6 +2514,7 @@ async function listPendingFinalizeBlockerIssueIds( blockerIssueId: string; executionWorkspaceId: string; }>, + lockSources = false, ): Promise> { const pending = new Set(); const blockerIssueIds = [ @@ -2529,7 +2531,14 @@ async function listPendingFinalizeBlockerIssueIds( ), ); - const rows = await dbOrTx + if (lockSources) { + + + await dbOrTx.select({ id: executionWorkspaces.id }).from(executionWorkspaces) + .where(and(eq(executionWorkspaces.companyId, companyId), inArray(executionWorkspaces.id, executionWorkspaceIds))) + .for("update", { noWait: true }); + } + const query = dbOrTx .select({ issueId: workspaceOperations.issueId, executionWorkspaceId: workspaceOperations.executionWorkspaceId, @@ -2548,6 +2557,8 @@ async function listPendingFinalizeBlockerIssueIds( ), ); + const rows = await (lockSources ? query.for("share", { noWait: true }) : query); + const latestAttributedByBlockerWorkspace = new Map< string, { phase: string; status: string; startedAt: Date } @@ -2697,6 +2708,7 @@ async function listIssueDependencyReadinessMap( dbOrTx: Pick, companyId: string, issueIds: string[], + lockSources = false, ) { const uniqueIssueIds = [...new Set(issueIds.filter(Boolean))]; const readinessMap = new Map(); @@ -2705,7 +2717,7 @@ async function listIssueDependencyReadinessMap( } if (uniqueIssueIds.length === 0) return readinessMap; - const blockerRows = await dbOrTx + const blockerQuery = dbOrTx .select({ issueId: issueRelations.relatedIssueId, blockerIssueId: issueRelations.issueId, @@ -2722,6 +2734,10 @@ async function listIssueDependencyReadinessMap( ), ); + + + const blockerRows = await (lockSources ? blockerQuery.for("update", { noWait: true }) : blockerQuery); + // Collect issue/workspace pairs of "done" blockers — these are the only ones // subject to the workspace-finalize barrier. Blockers that aren't done already // mark the dependent as not-ready and don't need a finalize check. @@ -2742,6 +2758,7 @@ async function listIssueDependencyReadinessMap( dbOrTx, companyId, doneBlockerWorkspacePairs, + lockSources, ); for (const row of blockerRows) { @@ -7823,6 +7840,7 @@ export function issueService(db: Db) { if (isExplicitContinuationRetryClaim(issue, run)) return false; await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: issue.companyId, issueId }); + await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId }); const updated = await tx .update(issues) .set({ @@ -7897,6 +7915,7 @@ export function issueService(db: Db) { } await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: issue.companyId, issueId }); + await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId }); const updated = await tx .update(issues) .set({ @@ -9311,7 +9330,13 @@ export function issueService(db: Db) { }; }, - getDependencyReadiness: async (issueId: string, dbOrTx: any = db) => { + + + getDependencyReadiness: async ( + issueId: string, + dbOrTx: any = db, + options: { lockSources?: boolean } = {}, + ) => { const issue = await dbOrTx .select({ id: issues.id, companyId: issues.companyId }) .from(issues) @@ -9324,6 +9349,7 @@ export function issueService(db: Db) { dbOrTx, issue.companyId, [issueId], + options.lockSources, ); return readiness.get(issueId) ?? createIssueDependencyReadiness(issueId); }, @@ -11473,6 +11499,11 @@ export function issueService(db: Db) { || (patch.checkoutRunId !== undefined && patch.checkoutRunId !== receiptExisting.checkoutRunId)) { await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: receiptExisting.companyId, issueId: id }); } + if (patch.status !== undefined || patch.statusVersion !== undefined || patch.executionState !== undefined + || patch.executionRunId !== undefined || patch.checkoutRunId !== undefined + || patch.assigneeAgentId !== undefined || patch.assigneeUserId !== undefined || blockedByIssueIds !== undefined) { + await notifyNativeStatusReassessment(tx, { companyId: receiptExisting.companyId, issueId: id }); + } const updated = await tx .update(issues) .set(patch) @@ -12383,6 +12414,7 @@ export function issueService(db: Db) { const releaseStatus = existing.status === "in_progress" ? "todo" : existing.status; await notifyTerminalWorkspaceWork(tx, `issue:${id}`); + await notifyNativeStatusReassessment(tx, { companyId: existing.companyId, issueId: id }); const updated = await tx .update(issues) .set({ @@ -12434,6 +12466,9 @@ export function issueService(db: Db) { } await notifyTerminalWorkspaceWork(tx, `issue:${id}`); + if (existing.checkoutRunId || existing.executionRunId || options.clearAssignee) { + await notifyNativeStatusReassessment(tx, { companyId: existing.companyId, issueId: id }); + } if (existing.checkoutRunId || existing.executionRunId) { await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: existing.companyId, issueId: id }); } @@ -13497,6 +13532,7 @@ export function issueService(db: Db) { } } } + await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId: issue.id }); const [asset] = await tx .insert(assets) .values({ diff --git a/server/src/services/legacy-execution-recovery.ts b/server/src/services/legacy-execution-recovery.ts index 22461cf38b3..427704de63f 100644 --- a/server/src/services/legacy-execution-recovery.ts +++ b/server/src/services/legacy-execution-recovery.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js"; import { notifyInstructionCollection } from "./instruction-collection-notifications.js"; import { notifyExecutionRunWork } from "./execution-control-notifications.js"; @@ -172,6 +173,9 @@ export async function terminalizeLegacyExecution(input: { ? retainedLeaseIds.length > 0 || hasRequiredWorkspaceRecovery(updated.resultJson) : !input.reconcileIfNeeded || await legacyExecutionNeedsReconciliationWithEvidence(tx as unknown as Db, updated); if (!needsRecovery) return updated; + if (!input.recordRestoreFailureOnly && task && (task.executionRunId === run.id || task.checkoutRunId === run.id)) { + await notifyNativeStatusReassessment(tx, { companyId: task.companyId, issueId: task.id }); + } if (!input.recordRestoreFailureOnly && task?.executionRunId === run.id) await tx .update(issues) diff --git a/server/src/services/native-finalization-notifications.ts b/server/src/services/native-finalization-notifications.ts index 91ccfc2023d..3f95a4a67a4 100644 --- a/server/src/services/native-finalization-notifications.ts +++ b/server/src/services/native-finalization-notifications.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js"; import { notifyNativeReviewResponse } from "./native-review-response-notifications.js"; import { notifyNativeResponsePresentation } from "./native-response-presentation-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; @@ -6,6 +7,7 @@ export type NativeFinalizationWorkScope = { companyId: string; runId?: string; i export async function notifyNativeFinalizationWork(transaction: object, scope: NativeFinalizationWorkScope) { if (scope.issueId || scope.runId) { + await notifyNativeStatusReassessment(transaction, scope); await notifyNativeResponsePresentation(transaction, scope); await notifyNativeReviewResponse(transaction, scope); } diff --git a/server/src/services/native-review-state-notifications.ts b/server/src/services/native-review-state-notifications.ts new file mode 100644 index 00000000000..43e61a0fdba --- /dev/null +++ b/server/src/services/native-review-state-notifications.ts @@ -0,0 +1,7 @@ +import { notifyNativeReviewResponse } from "./native-review-response-notifications.js"; +import { notifyNativeStatusReassessment, type NativeStatusReassessmentScope } from "./native-status-reassessment-notifications.js"; + +export async function notifyNativeReviewState(tx: object, scope: NativeStatusReassessmentScope) { + await notifyNativeReviewResponse(tx, scope); + await notifyNativeStatusReassessment(tx, scope); +} diff --git a/server/src/services/native-runtime/automatic-completion-reviews.ts b/server/src/services/native-runtime/automatic-completion-reviews.ts index 9f7eea8c326..42be62845cc 100644 --- a/server/src/services/native-runtime/automatic-completion-reviews.ts +++ b/server/src/services/native-runtime/automatic-completion-reviews.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js"; import type { AutomaticCompletionReviewScope } from "../automatic-completion-review-notifications.js"; import { DELIVERY_QUEUES } from "../delivery-work-notifications.js"; import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js"; @@ -138,6 +139,7 @@ async function dismissCandidates(db: Db, candidates: Awaited { ); beforeAll(async () => { - temporary = await startEmbeddedPostgresTestDatabase("native-chat-reuse-"); - db = createDb(temporary.connectionString); + const external = resolveExternalTestDatabaseUrl("PAPERCLIP_CHAT_REUSE_TEST_DATABASE_URL"); + if (!external) temporary = await startEmbeddedPostgresTestDatabase("native-chat-reuse-"); + db = createDb(external ?? temporary!.connectionString); const storageRoot = await mkdtemp( path.join(tmpdir(), "paperclip-chat-reuse-"), ); @@ -295,6 +297,7 @@ describe("native same-conversation chat attachment reuse", () => { }); afterAll(async () => { + await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); }); diff --git a/server/src/services/native-runtime/completion-contracts.ts b/server/src/services/native-runtime/completion-contracts.ts index f8a642769db..7a4fc728bfd 100644 --- a/server/src/services/native-runtime/completion-contracts.ts +++ b/server/src/services/native-runtime/completion-contracts.ts @@ -133,6 +133,22 @@ export function buildNativeCompletionContract( }; } +function nativeCompletionContractLockKey(companyId: string, issueId: string): string { + return `paperclip:native-completion-contract:${companyId}:${issueId}`; +} + +export async function lockCurrentNativeCompletionContract(tx: Db, input: { + companyId: string; issueId: string; contractId: string; +}): Promise { + const [lock] = await tx.execute(sql`select pg_try_advisory_xact_lock(hashtextextended( + ${nativeCompletionContractLockKey(input.companyId, input.issueId)}, 0)) as acquired`) as unknown as Array<{ acquired: boolean }>; + if (!lock.acquired) return false; + const [latest] = await tx.select({ id: completionContracts.id }).from(completionContracts) + .where(and(eq(completionContracts.companyId, input.companyId), eq(completionContracts.issueId, input.issueId))) + .orderBy(desc(completionContracts.revision)).limit(1); + return latest?.id === input.contractId; +} + export async function ensureNativeCompletionContract(input: { db: Db; companyId: string; @@ -151,11 +167,8 @@ export async function ensureNativeCompletionContract(input: { humanResponseId?: string | null; }) { return input.db.transaction(async (tx) => { - await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${[ - "paperclip:native-completion-contract", - input.companyId, - input.issue.id, - ].join(":")}, 0))`); + await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended( + ${nativeCompletionContractLockKey(input.companyId, input.issue.id)}, 0))`); const policy = resolveNativeCompletionPolicy(input.issue); const latest = await tx .select() diff --git a/server/src/services/native-runtime/evidence-classifier.ts b/server/src/services/native-runtime/evidence-classifier.ts index 835dacf57dd..66182c1c9f8 100644 --- a/server/src/services/native-runtime/evidence-classifier.ts +++ b/server/src/services/native-runtime/evidence-classifier.ts @@ -102,26 +102,35 @@ function uuidRef(ref: string, prefixes: string[]): string | null { return null; } +export function readNativeEventEvidenceOutcome(value: unknown): "accepted" | "rejected" | null { + const event = record(value); + if (event.sourceKind !== "control_plane") return null; + const outcome = text(record(event.payload).evidenceOutcome); + return outcome === "accepted" || outcome === "rejected" ? outcome : null; +} + async function classifyEvidenceRef(input: { db: Db; companyId: string; issueId: string; runId: string; ref: string; + lockSources?: boolean; }): Promise { const handoffId = uuidRef(input.ref, ["case-handoff"]); if (handoffId) { - const [row] = await input.db.select({ id: caseHandoffs.id, state: caseHandoffs.state }).from(caseHandoffs).where(and( + const query = input.db.select({ id: caseHandoffs.id, state: caseHandoffs.state }).from(caseHandoffs).where(and( eq(caseHandoffs.id, handoffId), eq(caseHandoffs.companyId, input.companyId), eq(caseHandoffs.issueId, input.issueId), eq(caseHandoffs.sourceRunId, input.runId), )).limit(1); + const [row] = await (input.lockSources ? query.for("share", { noWait: true }) : query); return { ref: input.ref, kind: "case_handoff", outcome: !row ? "missing" : row.state === "cancelled" ? "rejected" : "accepted", reasonCode: !row ? "case_handoff_missing" : row.state === "cancelled" ? "case_handoff_cancelled" : "case_handoff_persisted", durableRecordId: row?.id ?? null }; } const eventMatch = /^event:(\d+)$/.exec(input.ref); if (eventMatch) { const cursor = Number(eventMatch[1]); - const event = await input.db.select({ + const query = input.db.select({ id: heartbeatRunEvents.id, eventType: heartbeatRunEvents.eventType, payload: heartbeatRunEvents.payload, @@ -131,16 +140,16 @@ async function classifyEvidenceRef(input: { eq(heartbeatRunEvents.companyId, input.companyId), eq(heartbeatRunEvents.runId, input.runId), or(eq(heartbeatRunEvents.seq, cursor), eq(heartbeatRunEvents.sourceSeq, cursor)), - )).limit(1).then((rows) => rows[0] ?? null); + )).limit(1); + const [event] = await (input.lockSources ? query.for("share", { noWait: true }) : query); if (!event) { return { ref: input.ref, kind: "event", outcome: "missing", reasonCode: "durable_run_event_missing", durableRecordId: null }; } - const prpEvent = record(record(event.payload).prpEvent); - const evidenceOutcome = text(record(prpEvent.payload).evidenceOutcome); - if (prpEvent.sourceKind === "control_plane" && evidenceOutcome === "accepted") { + const evidenceOutcome = readNativeEventEvidenceOutcome(record(event.payload).prpEvent); + if (evidenceOutcome === "accepted") { return { ref: input.ref, kind: "event", outcome: "accepted", reasonCode: "control_plane_evidence_accepted", durableRecordId: String(event.id) }; } - if (prpEvent.sourceKind === "control_plane" && evidenceOutcome === "rejected") { + if (evidenceOutcome === "rejected") { return { ref: input.ref, kind: "event", outcome: "rejected", reasonCode: "control_plane_evidence_rejected", durableRecordId: String(event.id) }; } return { @@ -156,7 +165,7 @@ async function classifyEvidenceRef(input: { const workProductId = uuidRef(input.ref, ["work_product", "work-product", "artifact"]); if (workProductId) { - const row = await input.db.select({ + const query = input.db.select({ id: issueWorkProducts.id, status: issueWorkProducts.status, reviewState: issueWorkProducts.reviewState, @@ -164,7 +173,8 @@ async function classifyEvidenceRef(input: { eq(issueWorkProducts.id, workProductId), eq(issueWorkProducts.companyId, input.companyId), eq(issueWorkProducts.issueId, input.issueId), - )).limit(1).then((rows) => rows[0] ?? null); + )).limit(1); + const [row] = await (input.lockSources ? query.for("share", { noWait: true }) : query); if (!row) return { ref: input.ref, kind: "work_product", outcome: "missing", reasonCode: "work_product_missing", durableRecordId: null }; if ( ["failed", "error", "cancelled", "rejected", "changes_requested"].includes(row.status) @@ -183,7 +193,7 @@ async function classifyEvidenceRef(input: { const approvalId = uuidRef(input.ref, ["approval"]); if (approvalId) { - const row = await input.db.select({ id: approvals.id, status: approvals.status }) + const query = input.db.select({ id: approvals.id, status: approvals.status }) .from(issueApprovals) .innerJoin(approvals, eq(issueApprovals.approvalId, approvals.id)) .where(and( @@ -191,7 +201,8 @@ async function classifyEvidenceRef(input: { eq(issueApprovals.issueId, input.issueId), eq(approvals.companyId, input.companyId), eq(approvals.id, approvalId), - )).limit(1).then((rows) => rows[0] ?? null); + )).limit(1); + const [row] = await (input.lockSources ? query.for("share", { noWait: true }) : query); if (!row) return { ref: input.ref, kind: "approval", outcome: "missing", reasonCode: "approval_missing", durableRecordId: null }; if (row.status === "approved") return { ref: input.ref, kind: "approval", outcome: "accepted", reasonCode: "approval_approved", durableRecordId: row.id }; if (["rejected", "cancelled"].includes(row.status)) return { ref: input.ref, kind: "approval", outcome: "rejected", reasonCode: "approval_rejected", durableRecordId: row.id }; @@ -200,7 +211,7 @@ async function classifyEvidenceRef(input: { const interactionId = uuidRef(input.ref, ["interaction"]); if (interactionId) { - const row = await input.db.select({ + const query = input.db.select({ id: issueThreadInteractions.id, status: issueThreadInteractions.status, result: issueThreadInteractions.result, @@ -208,7 +219,8 @@ async function classifyEvidenceRef(input: { eq(issueThreadInteractions.id, interactionId), eq(issueThreadInteractions.companyId, input.companyId), eq(issueThreadInteractions.issueId, input.issueId), - )).limit(1).then((rows) => rows[0] ?? null); + )).limit(1); + const [row] = await (input.lockSources ? query.for("share", { noWait: true }) : query); if (!row) return { ref: input.ref, kind: "interaction", outcome: "missing", reasonCode: "interaction_missing", durableRecordId: null }; const outcome = text(record(row.result).outcome); if (["accepted", "answered", "resolved"].includes(outcome ?? "") || ["accepted", "answered"].includes(row.status)) { @@ -222,11 +234,12 @@ async function classifyEvidenceRef(input: { const attachmentId = uuidRef(input.ref, ["attachment"]); if (attachmentId) { - const row = await input.db.select({ id: issueAttachments.id }).from(issueAttachments).where(and( + const query = input.db.select({ id: issueAttachments.id }).from(issueAttachments).where(and( eq(issueAttachments.id, attachmentId), eq(issueAttachments.companyId, input.companyId), eq(issueAttachments.issueId, input.issueId), - )).limit(1).then((rows) => rows[0] ?? null); + )).limit(1); + const [row] = await (input.lockSources ? query.for("share", { noWait: true }) : query); return row ? { ref: input.ref, kind: "attachment", outcome: "accepted", reasonCode: "attachment_persisted", durableRecordId: row.id } : { ref: input.ref, kind: "attachment", outcome: "missing", reasonCode: "attachment_missing", durableRecordId: null }; @@ -241,10 +254,10 @@ async function classifyEvidenceRef(input: { }; } -/** - * Classifies model claims against immutable server-owned records. A model's - * `satisfied` or `passed` value is retained as a claim but never becomes proof. - */ + + + + export async function classifyNativeEvidence(input: { db: Db; companyId: string; @@ -252,6 +265,8 @@ export async function classifyNativeEvidence(input: { runId: string; contract: Record; result: Record; + + lockSources?: boolean; }): Promise { const claim = record(input.result.completionClaim); const claimedCriteria = Array.isArray(claim.criteria) ? claim.criteria.map(record) : []; @@ -355,11 +370,12 @@ export async function classifyNativeEvidence(input: { const ignoredAttentionRequests = [...normalizedSignals.ignoredAttentionRequests]; for (const request of normalizedSignals.actionableAttentionRequests) { if (request.ownerClass === "agent" && request.targetAgentId) { - const targetExists = await input.db.select({ id: agents.id }).from(agents).where(and( + const targetQuery = input.db.select({ id: agents.id }).from(agents).where(and( eq(agents.id, request.targetAgentId), eq(agents.companyId, input.companyId), - )).limit(1).then((rows) => rows.length > 0); - if (!targetExists) { + )).limit(1); + const targets = await (input.lockSources ? targetQuery.for("share", { noWait: true }) : targetQuery); + if (targets.length === 0) { ignoredAttentionRequests.push({ sourceIndex: request.sourceIndex, sourceKind: request.sourceKind, diff --git a/server/src/services/native-runtime/native-finalization-reconciler.ts b/server/src/services/native-runtime/native-finalization-reconciler.ts index cc97199948b..48a31e52cc0 100644 --- a/server/src/services/native-runtime/native-finalization-reconciler.ts +++ b/server/src/services/native-runtime/native-finalization-reconciler.ts @@ -1,43 +1,37 @@ +import { nativeStatusReassessmentSourceIsSuperseded, reassessNativeStatusFromSource } from "./native-status-reassessment.js"; +import { resolveNativeReconciliationStatus } from "./native-reconciliation-status.js"; +export { resolveNativeReconciliationStatus, type NativeReconciliationFacts } from "./native-reconciliation-status.js"; import { pendingNativeStatusEffectsCondition } from "./native-pending-effect-work.js"; import { notifyUnsafeWorkspaceExportRunChange } from "../unsafe-workspace-export-notifications.js"; import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js"; import { notifyInstructionCollection } from "../instruction-collection-notifications.js"; -import { readRequiredQualityReview } from "../required-quality-review.js"; import type { WorkspaceWorkers } from "../plugin-workspaces.js"; import { recoverLegacyUnsafeWorkspaceExports } from "./native-workspace-export-recovery.js"; -import { hasPendingNativeChildCompletion } from "./native-child-completion-delivery.js"; -import { dismissAutomaticCompletionReviews, decisionHasRetiredAutomaticReview } from "./automatic-completion-reviews.js"; +import { dismissAutomaticCompletionReviews } from "./automatic-completion-reviews.js"; import { logger } from "../../middleware/logger.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js"; import { createHash, randomUUID } from "node:crypto"; import { and, asc, desc, eq, gt, inArray, isNotNull, isNull, lt, lte, ne, not, notInArray, or, sql, type SQL } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { - completionContracts, agents, heartbeatRunEvents, heartbeatRuns, - issueWorkProducts, issues, nativeRunFinalizations, nativeRunResults, statusDecisionEffects, statusDecisions, - workAssessments, workspaceOperations, } from "@paperclipai/db"; import { finalizeNativeRun, - pendingNativeGovernance, - resolveNativeFinalizerStatus, recordNativeFinalizationFailure, repairCommittedNativeReviewResponse, repairCommittedNativeChatResponse, } from "./native-run-finalizer.js"; import { - commitNativeStatusDecision, dispatchPendingNativeStatusEffects, - NativeStatusRaceError, } from "./status-decision-committer.js"; import { issueRecoveryActionService } from "../issue-recovery-actions.js"; import { issueService } from "../issues.js"; @@ -50,17 +44,13 @@ import { readNativeWorkspaceSyncReference, } from "./native-workspace-sync.js"; import type { EnvironmentRuntimeService } from "../environment-runtime.js"; -import { classifyNativeEvidence } from "./evidence-classifier.js"; import { createNativeReconciliationPassTiming } from "./native-reconciliation-timing.js"; -import { recordNativeWorkAssessment } from "./work-assessments.js"; import { isNativeRunnerOwnershipHeld, nativeRunnerOwnershipNotHeldCondition, } from "./native-runner-ownership.js"; import { - NATIVE_STATUS_ARBITER_POLICY_VERSION, type NativeAuthoritativeIssueStatus, - type NativeStatusDecision, } from "./status-arbiter.js"; import { getNativeReviewAssignment } from "./native-review-participant.js"; @@ -70,125 +60,12 @@ import { readRunCancellation } from "../run-cancellation.js"; import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js"; import { retireSupersededNativeRetry } from "./native-superseded-retry.js"; -export type NativeReconciliationFacts = { - equivalentAttentionFamily?: boolean; - canonicalReplay?: boolean; - callerMaterialConflict?: boolean; - workspaceOperationPending?: boolean; - undeliveredEffectCount?: number; - authoritativeStatusChanged?: boolean; - newEvidenceSatisfiesContract?: boolean; - hasPendingChildCompletion?: boolean; - dependencyResolved?: boolean; - authorizedResume?: boolean; - statusVersionAdvanced?: boolean; -}; - function record(value: unknown): Record { return value && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; } -/** Append-only recovery/reconciliation re-arbitration from durable facts. */ -export function resolveNativeReconciliationStatus(input: { - facts: NativeReconciliationFacts; - priorIssueStatus: NativeAuthoritativeIssueStatus; - agentId: string; -}): NativeStatusDecision { - const preserve = (reasonCode: string, effects: NativeStatusDecision["effects"]): NativeStatusDecision => ({ - policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, - statusAction: "preserve", - toStatus: input.priorIssueStatus, - reasonCode, - unblockDescriptor: null, - effects, - }); - const continuation = (reasonCode: string, effects: NativeStatusDecision["effects"] = []): NativeStatusDecision => ({ - policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, - statusAction: "in_progress", - toStatus: "in_progress", - reasonCode, - unblockDescriptor: null, - effects: [{ - kind: "enqueue_continuation", - continuationKind: "same_agent", - summary: `Continue native reconciliation after ${reasonCode}.`, - idempotencyKey: `native-reconciliation:${reasonCode}`, - agentId: input.agentId, - }, ...effects], - }); - - if (input.facts.equivalentAttentionFamily) { - return preserve("attention_duplicate_suppressed", [{ kind: "link_canonical_request" }]); - } - - if (input.facts.callerMaterialConflict) { - return preserve("structured_result_replay_conflict", [{ - kind: "record_finalization_error", - cause: "structured_result_replay_conflict", - nextAction: "Use a fresh canonical result identity.", - agentId: input.agentId, - }]); - } - if (input.facts.authoritativeStatusChanged) { - return preserve("prior_status_terminal_preserved", [{ kind: "append_superseding_assessment" }]); - } - if (input.facts.statusVersionAdvanced) { - return preserve("arbitration_conflict_reloaded", [ - { kind: "increment_status_version" }, - { kind: "schedule_reconciliation" }, - ]); - } - if (input.facts.newEvidenceSatisfiesContract) { - if (input.facts.hasPendingChildCompletion) { - return { - policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, - statusAction: "in_progress", - toStatus: "in_progress", - reasonCode: "native_child_completion_pending", - unblockDescriptor: null, - // The existing child-result wake owns the continuation. - effects: [{ kind: "release_checkout" }], - }; - } - return { - policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, - statusAction: "done", - toStatus: "done", - reasonCode: "decision_superseded_by_new_evidence", - unblockDescriptor: null, - effects: [{ kind: "release_checkout" }], - }; - } - if (input.facts.dependencyResolved) return continuation("dependency_resolved"); - if (input.facts.authorizedResume) return continuation("authorized_resume"); - if ((input.facts.undeliveredEffectCount ?? 0) > 0) { - return continuation("live_continuation_registered", [{ kind: "dispatch_pending_effect" }]); - } - if (input.facts.workspaceOperationPending) { - return { - policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, - statusAction: "done", - toStatus: "done", - reasonCode: "completion_contract_satisfied", - unblockDescriptor: null, - effects: [{ kind: "resume_workspace_operation" }, { kind: "release_checkout" }], - }; - } - if (input.facts.canonicalReplay) { - return { - policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, - statusAction: "done", - toStatus: "done", - reasonCode: "completion_contract_satisfied", - unblockDescriptor: null, - effects: [{ kind: "release_checkout" }], - }; - } - throw new Error("native_reconciliation_facts_invalid"); -} - export type NativeSessionResumeClaim = { runId: string; leaseOwner: string }; type NativeCleanupOutcome = { @@ -646,6 +523,7 @@ type NativeReconciliationOptions = { skipNativePendingEffects?: boolean; skipNativeResponsePresentation?: boolean; skipNativeReviewResponse?: boolean; + skipNativeStatusReassessment?: boolean; pluginWorkerManager?: WorkspaceWorkers; environmentRuntime?: EnvironmentRuntimeService; onWorkspaceSettled?: (input: { @@ -830,39 +708,12 @@ async function reconcileNativeFinalizationsPass( continue; } if (row.assessmentId) { - const assessment = await db.select({ - id: workAssessments.id, - contractId: workAssessments.contractId, - resultId: workAssessments.resultId, - policyVersion: workAssessments.policyVersion, - priorIssueStatus: workAssessments.priorIssueStatus, - priorStatusVersion: workAssessments.priorStatusVersion, - priorDecisionId: workAssessments.priorDecisionId, - assessmentJson: workAssessments.assessmentJson, - createdAt: workAssessments.createdAt, - }).from(workAssessments).where(and( - eq(workAssessments.id, row.assessmentId), - eq(workAssessments.companyId, row.companyId), - eq(workAssessments.issueId, row.issueId), - )).limit(1).then((entries) => entries[0] ?? null); - const changedEvidence = assessment - ? await db.select({ id: issueWorkProducts.id }).from(issueWorkProducts).where(and( - eq(issueWorkProducts.companyId, row.companyId), - eq(issueWorkProducts.issueId, row.issueId), - or( - gt(issueWorkProducts.createdAt, assessment.createdAt), - gt(issueWorkProducts.updatedAt, assessment.createdAt), - ), - )).limit(1).then((entries) => entries[0] ?? null) - : null; - const currentDecision = row.decisionId + const currentDecision = !options.skipNativeReviewResponse && row.coordinatorPhase === "committed" && row.decisionId ? await db.select().from(statusDecisions).where(and( - eq(statusDecisions.id, row.decisionId), - eq(statusDecisions.companyId, row.companyId), + eq(statusDecisions.id, row.decisionId), eq(statusDecisions.companyId, row.companyId), eq(statusDecisions.issueId, row.issueId), - )).limit(1).then((entries) => entries[0] ?? null) + )).limit(1).then(entries => entries[0] ?? null) : null; - const currentDecisionJson = record(currentDecision?.decisionJson); if (row.coordinatorPhase === "committed" && !options.skipNativeResponsePresentation) { // A later decision can supersede this run's task status, but cannot // erase its accepted, still-authorized response. Repair presentation @@ -872,12 +723,12 @@ async function reconcileNativeFinalizationsPass( }); } if ( + !options.skipNativeReviewResponse && row.coordinatorPhase === "committed" && row.decisionId && row.resultId && row.assessmentId && - currentDecisionJson.externalChatReviewPresentation && - !options.skipNativeReviewResponse + record(currentDecision?.decisionJson).externalChatReviewPresentation ) { // A later chat turn may retain the same pending review. Recover the // earlier response independently before status reconciliation skips @@ -891,168 +742,15 @@ async function reconcileNativeFinalizationsPass( assessmentId: row.assessmentId, }); } - if (row.coordinatorPhase === "committed" && row.decisionId && !currentDecision) { - throw new Error("native_committed_decision_missing"); - } - const supersededCommittedDecision = row.coordinatorPhase === "committed" - && row.decisionId !== null - && row.issueDecisionId !== null - && row.issueDecisionId !== row.decisionId - && !( - currentDecisionJson.statusAction === "preserve" - && assessment?.priorDecisionId === row.issueDecisionId - ); - if (supersededCommittedDecision) continue; - const persistedProjectedVersion = Number(currentDecisionJson.projectedStatusVersion); - const expectedProjectedVersion = currentDecision - ? Number.isFinite(persistedProjectedVersion) - ? persistedProjectedVersion - : Number(currentDecision.decisionVersion) - (currentDecisionJson.statusAction === "preserve" ? 1 : 0) - : null; - const issueMatchesCurrentDecision = !!assessment - && !!currentDecision - && row.issueDecisionId === row.decisionId - && currentDecision.assessmentId === assessment.id - && currentDecision.toStatus === row.issueStatus - && expectedProjectedVersion === Number(row.issueStatusVersion); - const authoritativeStatusChanged = !!assessment && !issueMatchesCurrentDecision && ( - assessment.priorIssueStatus !== row.issueStatus - || Number(assessment.priorStatusVersion) !== Number(row.issueStatusVersion) - ); - const retiredAutomaticReview = issueMatchesCurrentDecision && currentDecision - ? await decisionHasRetiredAutomaticReview(db, currentDecision) : false; - let reassessment = null; - let resultRow = null; - let contractRow = null; - if (assessment && (authoritativeStatusChanged || changedEvidence || retiredAutomaticReview)) { - [resultRow, contractRow] = await Promise.all([ - db.select().from(nativeRunResults).where(and( - eq(nativeRunResults.id, assessment.resultId), - eq(nativeRunResults.companyId, row.companyId), - eq(nativeRunResults.issueId, row.issueId), - eq(nativeRunResults.runId, row.runId), - )).limit(1).then((entries) => entries[0] ?? null), - db.select().from(completionContracts).where(and( - eq(completionContracts.id, assessment.contractId), - eq(completionContracts.companyId, row.companyId), - eq(completionContracts.issueId, row.issueId), - )).limit(1).then((entries) => entries[0] ?? null), - ]); - if (!resultRow || !contractRow) throw new Error("native_reconciliation_evidence_binding_missing"); - reassessment = await classifyNativeEvidence({ - db, - companyId: row.companyId, - issueId: row.issueId, - runId: row.runId, - contract: record(contractRow.contractJson), - result: record(record(resultRow.resultJson).result), - }); - } - const previousAssessment = record(assessment?.assessmentJson); - const newEvidenceSatisfiesContract = !!changedEvidence - && previousAssessment.allCriteriaSatisfied !== true - && reassessment?.allCriteriaSatisfied === true - && reassessment.verificationPassed === true; - const facts: NativeReconciliationFacts = authoritativeStatusChanged - ? { authoritativeStatusChanged: true } - : newEvidenceSatisfiesContract - ? { newEvidenceSatisfiesContract: true } - : {}; - if (Object.keys(facts).length > 0 || retiredAutomaticReview) { - if (!assessment || !reassessment || !resultRow || !contractRow) { - throw new Error("native_reconciliation_reassessment_missing"); + const source = { ...row, assessmentId: row.assessmentId }; + if (options.skipNativeStatusReassessment) { + if (await nativeStatusReassessmentSourceIsSuperseded(db, source)) continue; + } else { + const reassessed = await reassessNativeStatusFromSource(db, source); + if (reassessed.kind !== "unchanged") { + if (reassessed.kind === "committed") results.push(reassessed.result); + continue; } - const currentIssue = retiredAutomaticReview - ? await db.select().from(issues).where(and(eq(issues.id, row.issueId), eq(issues.companyId, row.companyId))).then((entries) => entries[0]) - : null; - const readiness = retiredAutomaticReview ? await issueService(db).getDependencyReadiness(row.issueId, db) : null; - const currentRun = retiredAutomaticReview ? await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, row.runId)).then((entries) => entries[0]) : null; - // Never replay an old result over a new run or a later task contract. - const latestContract = retiredAutomaticReview ? await db.select({ id: completionContracts.id }).from(completionContracts) - .where(and(eq(completionContracts.companyId, row.companyId), eq(completionContracts.issueId, row.issueId))) - .orderBy(desc(completionContracts.revision)).limit(1).then((entries) => entries[0]) : null; - if (retiredAutomaticReview && (!currentIssue || currentRun?.status !== "succeeded" - || latestContract?.id !== contractRow.id - || (currentIssue.executionRunId && currentIssue.executionRunId !== row.runId))) continue; - // A committed decision proves the original barrier passed. If a later - // workspace operation exists, do not ignore a pending or failed retry. - const reviewBarrier = retiredAutomaticReview ? await db.select({ status: workspaceOperations.status }) - .from(workspaceOperations).where(and(eq(workspaceOperations.companyId, row.companyId), - eq(workspaceOperations.heartbeatRunId, row.runId), eq(workspaceOperations.phase, "workspace_finalize"))) - .orderBy(desc(workspaceOperations.createdAt)).limit(1).then((entries) => entries[0]) : null; - if (reviewBarrier && reviewBarrier.status !== "succeeded") continue; - const childCompletionRecipient = { - companyId: row.companyId, issueId: row.issueId, agentId: row.agentId, runId: row.runId, - sourceIntentId: typeof row.nativeStatusWakeIntentId === "string" ? row.nativeStatusWakeIntentId : null, - }; - const hasPendingChildCompletion = await hasPendingNativeChildCompletion(db, childCompletionRecipient); - const reassessmentRow = await recordNativeWorkAssessment({ - db, - companyId: row.companyId, - issueId: row.issueId, - runId: row.runId, - turnId: resultRow.turnId, - contractId: contractRow.id, - contractCanonicalSha256: contractRow.canonicalSha256, - resultId: resultRow.id, - resultCanonicalSha256: resultRow.canonicalSha256, - priorIssueStatus: row.issueStatus, - priorStatusVersion: Number(row.issueStatusVersion), - priorDecisionId: row.issueDecisionId, - policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, - assessment: reassessment, - supersedesAssessmentId: assessment.id, - }); - const qualityReview = currentIssue ? await readRequiredQualityReview(db, currentIssue) : null; - let decision = retiredAutomaticReview && currentIssue - ? resolveNativeFinalizerStatus({ - hasPendingChildCompletion, - requiredQualityReviewSatisfied: qualityReview?.satisfied, - assessment: reassessment, terminalState: "succeeded", workspaceFinalizeStatus: "succeeded", - governanceGate: await pendingNativeGovernance({ db, companyId: row.companyId, issueId: row.issueId, - runId: row.runId, executionState: record(currentIssue.executionState) }), - completionClaimPolicyAccepted: contractRow.risk === "low" && contractRow.completionAuthority === "agent_claim_policy", - hasUnresolvedIssueBlockers: (readiness?.unresolvedBlockerCount ?? 0) > 0, - reviewOwnerUserId: currentIssue.responsibleUserId ?? currentIssue.createdByUserId, - priorIssueStatus: row.issueStatus as NativeAuthoritativeIssueStatus, agentId: row.agentId, - }) - : resolveNativeReconciliationStatus({ - facts: { ...facts, hasPendingChildCompletion }, - priorIssueStatus: row.issueStatus as NativeAuthoritativeIssueStatus, agentId: row.agentId, - }); - if (decision.statusAction === "done" && qualityReview?.satisfied === false) { - decision = resolveNativeFinalizerStatus({ assessment: reassessment, terminalState: "succeeded", workspaceFinalizeStatus: "succeeded", - requiredQualityReviewSatisfied: false, priorIssueStatus: row.issueStatus as NativeAuthoritativeIssueStatus, agentId: row.agentId }); - } - let committed: Awaited>; - try { - committed = await commitNativeStatusDecision({ - db, - companyId: row.companyId, - issueId: row.issueId, - runId: row.runId, - assessmentId: reassessmentRow.id, - priorStatus: row.issueStatus, - priorStatusVersion: Number(row.issueStatusVersion), - priorDecisionId: row.issueDecisionId, - decision, - requireNoPendingChildCompletion: decision.statusAction === "done" ? childCompletionRecipient : undefined, - supersedesCommittedDecisionId: row.coordinatorPhase === "committed" - ? row.decisionId ?? undefined - : undefined, - }); - } catch (error) { - if (error instanceof NativeStatusRaceError) continue; - throw error; - } - results.push({ - runId: row.runId, - phase: "committed", - reconciliationAction: decision.reasonCode, - decisionId: committed.decision.id, - reconciliationDecision: decision, - }); - continue; } } const barrier = await db.select({ status: workspaceOperations.status }) diff --git a/server/src/services/native-runtime/native-governance.ts b/server/src/services/native-runtime/native-governance.ts index 2b71ffcf24f..dd8f2a27835 100644 --- a/server/src/services/native-runtime/native-governance.ts +++ b/server/src/services/native-runtime/native-governance.ts @@ -1,5 +1,5 @@ import { and, eq, inArray, notInArray } from "drizzle-orm"; -import { approvals, issueApprovals, issueThreadInteractions, issues, type Db } from "@paperclipai/db"; +import { approvals, issueApprovals, issueComments, issueThreadInteractions, issues, type Db } from "@paperclipai/db"; import { activeIssueInteractionCondition } from "../issue-question-context.js"; import { isConversation } from "../agent-conversations.js"; import type { NativeGovernanceGate } from "./status-arbiter.js"; @@ -8,6 +8,22 @@ function record(value: unknown): Record { return typeof value === "object" && value !== null ? value as Record : {}; } + +export async function lockNativeGovernanceSources(input: { db: Db; companyId: string; issueId: string }): Promise { + const interactions = await input.db.select({ kind: issueThreadInteractions.kind }).from(issueThreadInteractions) + .where(and(eq(issueThreadInteractions.companyId, input.companyId), eq(issueThreadInteractions.issueId, input.issueId))) + .for("share", { noWait: true }); + if (interactions.some(interaction => interaction.kind === "ask_user_questions")) { + await input.db.select({ id: issueComments.id }).from(issueComments) + .where(and(eq(issueComments.companyId, input.companyId), eq(issueComments.issueId, input.issueId))) + .for("share", { noWait: true }); + } + await input.db.select({ id: approvals.id }).from(issueApprovals) + .innerJoin(approvals, eq(issueApprovals.approvalId, approvals.id)) + .where(and(eq(issueApprovals.companyId, input.companyId), eq(issueApprovals.issueId, input.issueId), + eq(approvals.companyId, input.companyId))).for("share", { noWait: true }); +} + export async function pendingNativeGovernance(input: { db: Db; companyId: string; diff --git a/server/src/services/native-runtime/native-reconciliation-status.ts b/server/src/services/native-runtime/native-reconciliation-status.ts new file mode 100644 index 00000000000..dc9edc794d8 --- /dev/null +++ b/server/src/services/native-runtime/native-reconciliation-status.ts @@ -0,0 +1,113 @@ +import { NATIVE_STATUS_ARBITER_POLICY_VERSION, type NativeAuthoritativeIssueStatus, type NativeStatusDecision } from "./status-arbiter.js"; + +export type NativeReconciliationFacts = { + equivalentAttentionFamily?: boolean; + canonicalReplay?: boolean; + callerMaterialConflict?: boolean; + workspaceOperationPending?: boolean; + undeliveredEffectCount?: number; + authoritativeStatusChanged?: boolean; + newEvidenceSatisfiesContract?: boolean; + hasPendingChildCompletion?: boolean; + dependencyResolved?: boolean; + authorizedResume?: boolean; + statusVersionAdvanced?: boolean; +}; + + +export function resolveNativeReconciliationStatus(input: { + facts: NativeReconciliationFacts; + priorIssueStatus: NativeAuthoritativeIssueStatus; + agentId: string; +}): NativeStatusDecision { + const preserve = (reasonCode: string, effects: NativeStatusDecision["effects"]): NativeStatusDecision => ({ + policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, + statusAction: "preserve", + toStatus: input.priorIssueStatus, + reasonCode, + unblockDescriptor: null, + effects, + }); + const continuation = (reasonCode: string, effects: NativeStatusDecision["effects"] = []): NativeStatusDecision => ({ + policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, + statusAction: "in_progress", + toStatus: "in_progress", + reasonCode, + unblockDescriptor: null, + effects: [{ + kind: "enqueue_continuation", + continuationKind: "same_agent", + summary: `Continue native reconciliation after ${reasonCode}.`, + idempotencyKey: `native-reconciliation:${reasonCode}`, + agentId: input.agentId, + }, ...effects], + }); + + if (input.facts.equivalentAttentionFamily) { + return preserve("attention_duplicate_suppressed", [{ kind: "link_canonical_request" }]); + } + + if (input.facts.callerMaterialConflict) { + return preserve("structured_result_replay_conflict", [{ + kind: "record_finalization_error", + cause: "structured_result_replay_conflict", + nextAction: "Use a fresh canonical result identity.", + agentId: input.agentId, + }]); + } + if (input.facts.authoritativeStatusChanged) { + return preserve("prior_status_terminal_preserved", [{ kind: "append_superseding_assessment" }]); + } + if (input.facts.statusVersionAdvanced) { + return preserve("arbitration_conflict_reloaded", [ + { kind: "increment_status_version" }, + { kind: "schedule_reconciliation" }, + ]); + } + if (input.facts.newEvidenceSatisfiesContract) { + if (input.facts.hasPendingChildCompletion) { + return { + policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, + statusAction: "in_progress", + toStatus: "in_progress", + reasonCode: "native_child_completion_pending", + unblockDescriptor: null, + effects: [{ kind: "release_checkout" }], + }; + } + return { + policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, + statusAction: "done", + toStatus: "done", + reasonCode: "decision_superseded_by_new_evidence", + unblockDescriptor: null, + effects: [{ kind: "release_checkout" }], + }; + } + if (input.facts.dependencyResolved) return continuation("dependency_resolved"); + if (input.facts.authorizedResume) return continuation("authorized_resume"); + if ((input.facts.undeliveredEffectCount ?? 0) > 0) { + return continuation("live_continuation_registered", [{ kind: "dispatch_pending_effect" }]); + } + if (input.facts.workspaceOperationPending) { + return { + policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, + statusAction: "done", + toStatus: "done", + reasonCode: "completion_contract_satisfied", + unblockDescriptor: null, + effects: [{ kind: "resume_workspace_operation" }, { kind: "release_checkout" }], + }; + } + if (input.facts.canonicalReplay) { + return { + policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, + statusAction: "done", + toStatus: "done", + reasonCode: "completion_contract_satisfied", + unblockDescriptor: null, + effects: [{ kind: "release_checkout" }], + }; + } + throw new Error("native_reconciliation_facts_invalid"); +} diff --git a/server/src/services/native-runtime/native-restart-recovery.ts b/server/src/services/native-runtime/native-restart-recovery.ts index 0b7c8bf41ed..82a202c5ff7 100644 --- a/server/src/services/native-runtime/native-restart-recovery.ts +++ b/server/src/services/native-runtime/native-restart-recovery.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js"; import { notifyUnsafeWorkspaceExportRunChange } from "../unsafe-workspace-export-notifications.js"; import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js"; import { notifyInstructionCollection } from "../instruction-collection-notifications.js"; @@ -942,6 +943,7 @@ export async function claimNativeRestartRecoveries(input: { if (updatedRun && isNewTerminalRunFailure(row.run, updatedRun)) { terminalRunToReport = updatedRun; } + await notifyNativeStatusReassessment(tx, { companyId: row.run.companyId, issueId: row.coordinator.issueId }); await tx .update(issues) .set({ executionRunId: null, updatedAt: now }) diff --git a/server/src/services/native-runtime/native-safe-replacement.ts b/server/src/services/native-runtime/native-safe-replacement.ts index 78b889b169a..7fb789f3304 100644 --- a/server/src/services/native-runtime/native-safe-replacement.ts +++ b/server/src/services/native-runtime/native-safe-replacement.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js"; import { notifyExecutionRunWork } from "../execution-control-notifications.js"; import { logger } from "../../middleware/logger.js"; @@ -487,6 +488,7 @@ export async function reconcileSafeNativeReplacements( .update(heartbeatRuns) .set({ executionStatusDeliveryId: randomUUID() }) .where(eq(heartbeatRuns.id, run.id)); + await notifyNativeStatusReassessment(tx, { companyId: run.companyId, issueId: task.id }); // The successor gets task authority only through the existing admission gate. await tx .update(issues) diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index 4e73bfcfa45..fe662bd7e09 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js"; import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js"; import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js"; @@ -9882,6 +9883,7 @@ async function executePaperclipNativeSessionWithinScope( stillOwnsTask && phase === "terminal_failure" ) { + await notifyNativeStatusReassessment(tx, { companyId: input.execution.binding.companyId, issueId: input.execution.binding.issueId }); await tx .update(issues) .set({ executionRunId: null, checkoutRunId: null, updatedAt: now }) diff --git a/server/src/services/native-runtime/native-status-reassessment-work.ts b/server/src/services/native-runtime/native-status-reassessment-work.ts new file mode 100644 index 00000000000..13611a91de4 --- /dev/null +++ b/server/src/services/native-runtime/native-status-reassessment-work.ts @@ -0,0 +1,117 @@ +import { and, asc, eq, exists, gt, inArray, isNotNull, isNull, ne, not, notExists, or, sql } from "drizzle-orm"; +import { heartbeatRuns, issues, issueThreadInteractions, issueWorkProducts, nativeRunFinalizations, + statusDecisionEffects, statusDecisions, workAssessments, type Db } from "@paperclipai/db"; +import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js"; +import { DELIVERY_QUEUES } from "../delivery-work-notifications.js"; +import type { NativeStatusReassessmentScope } from "../native-status-reassessment-notifications.js"; +import { registerScopedDeadlineWork } from "../scoped-deadline-work.js"; +import { nativeRunnerOwnershipNotHeldCondition } from "./native-runner-ownership.js"; +import { reassessNativeStatusFromSource } from "./native-status-reassessment.js"; + +type Entry = { id: string; companyId: string; issueId: string; agentId: string; dueAt: number }; + +export function nativeStatusReassessmentWork(db: Db) { + function candidates(scope?: NativeStatusReassessmentScope, after?: string, runId?: string) { + const projectedVersion = sql`case + when jsonb_typeof(${statusDecisions.decisionJson}->'projectedStatusVersion') = 'number' + then (${statusDecisions.decisionJson}->>'projectedStatusVersion')::numeric + else ${statusDecisions.decisionVersion} - case when ${statusDecisions.decisionJson}->>'statusAction' = 'preserve' then 1 else 0 end + end`; + const matchesDecision = sql`coalesce(${issues.lastStatusDecisionId} = ${nativeRunFinalizations.decisionId} + and ${statusDecisions.assessmentId} = ${workAssessments.id} and ${statusDecisions.toStatus} = ${issues.status} + and ${projectedVersion} = ${issues.statusVersion}, false)`; + const statusChanged = and(not(matchesDecision), or(ne(workAssessments.priorIssueStatus, issues.status), + ne(workAssessments.priorStatusVersion, issues.statusVersion))); + const evidenceChanged = exists(db.select({ id: issueWorkProducts.id }).from(issueWorkProducts).where(and( + eq(issueWorkProducts.companyId, nativeRunFinalizations.companyId), eq(issueWorkProducts.issueId, nativeRunFinalizations.issueId), + or(gt(issueWorkProducts.createdAt, workAssessments.createdAt), gt(issueWorkProducts.updatedAt, workAssessments.createdAt)), + ))); + const retiredReview = exists(db.select({ id: issueThreadInteractions.id }).from(issueThreadInteractions).where(and( + eq(issueThreadInteractions.companyId, nativeRunFinalizations.companyId), eq(issueThreadInteractions.issueId, nativeRunFinalizations.issueId), + eq(issueThreadInteractions.status, "cancelled"), sql`${issueThreadInteractions.result}->>'reason' = 'automatic_completion_review_removed'`, + sql`(${issueThreadInteractions.payload}->'target'->>'revisionId' = ${statusDecisions.id}::text + or ${statusDecisions.decisionJson}->'effects' @> jsonb_build_array(jsonb_build_object('gate', + jsonb_build_object('kind', 'interaction', 'id', ${issueThreadInteractions.id}::text))))`, + ))); + return db.select({ runId: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + issueId: issues.id, issueStatus: issues.status, issueStatusVersion: issues.statusVersion, + issueDecisionId: issues.lastStatusDecisionId, coordinatorPhase: nativeRunFinalizations.phase, + assessmentId: workAssessments.id, decisionId: nativeRunFinalizations.decisionId, nativeStatusWakeIntentId: sql`${heartbeatRuns.contextSnapshot}->'nativeStatusWakeIntentId'`, + nextAttemptAt: nativeRunFinalizations.nextAttemptAt, leaseOwner: nativeRunFinalizations.leaseOwner, + leaseExpiresAt: nativeRunFinalizations.leaseExpiresAt, + }).from(nativeRunFinalizations) + .innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, nativeRunFinalizations.runId), + eq(heartbeatRuns.companyId, nativeRunFinalizations.companyId), eq(heartbeatRuns.nativeIssueId, nativeRunFinalizations.issueId))) + .innerJoin(issues, and(eq(issues.id, nativeRunFinalizations.issueId), eq(issues.companyId, nativeRunFinalizations.companyId))) + .innerJoin(workAssessments, and(eq(workAssessments.id, nativeRunFinalizations.assessmentId), + eq(workAssessments.companyId, nativeRunFinalizations.companyId), eq(workAssessments.issueId, nativeRunFinalizations.issueId), + eq(workAssessments.runId, nativeRunFinalizations.runId))) + .leftJoin(statusDecisions, and(eq(statusDecisions.id, nativeRunFinalizations.decisionId), + eq(statusDecisions.companyId, nativeRunFinalizations.companyId), eq(statusDecisions.issueId, nativeRunFinalizations.issueId))) + .where(and(eq(heartbeatRuns.runtimeMode, "native"), nativeRunnerOwnershipNotHeldCondition(), + ne(nativeRunFinalizations.phase, "terminal_failure"), isNotNull(nativeRunFinalizations.resultId), + or(isNull(nativeRunFinalizations.leaseOwner), isNotNull(nativeRunFinalizations.leaseExpiresAt)), + sql`not coalesce(${nativeRunFinalizations.phase} = 'committed' + and ${nativeRunFinalizations.decisionId} is not null and ${issues.lastStatusDecisionId} is not null + and ${issues.lastStatusDecisionId} <> ${nativeRunFinalizations.decisionId} + and not coalesce(${statusDecisions.decisionJson}->>'statusAction' = 'preserve' + and ${workAssessments.priorDecisionId} = ${issues.lastStatusDecisionId}, false), false)`, + or(statusChanged, evidenceChanged, and(matchesDecision, retiredReview)), + notExists(db.select({ id: statusDecisionEffects.id }).from(statusDecisionEffects).where(and( + eq(statusDecisionEffects.companyId, nativeRunFinalizations.companyId), eq(statusDecisionEffects.issueId, nativeRunFinalizations.issueId), + eq(statusDecisionEffects.decisionId, nativeRunFinalizations.decisionId), inArray(statusDecisionEffects.deliveryState, ["pending", "failed"]), + ))), + scope ? eq(nativeRunFinalizations.companyId, scope.companyId) : undefined, + scope?.issueId ? eq(nativeRunFinalizations.issueId, scope.issueId) : undefined, + scope?.runId ? eq(nativeRunFinalizations.runId, scope.runId) : undefined, + scope?.agentId ? eq(heartbeatRuns.agentId, scope.agentId) : undefined, + after ? gt(nativeRunFinalizations.runId, after) : undefined, + runId ? eq(nativeRunFinalizations.runId, runId) : undefined, + )).orderBy(asc(nativeRunFinalizations.runId)).limit(100); + } + function entry(row: Awaited>[number]): Entry { + return { id: row.runId, companyId: row.companyId, issueId: row.issueId, agentId: row.agentId, + dueAt: Math.max(row.nextAttemptAt?.getTime() ?? 0, row.leaseOwner !== null ? row.leaseExpiresAt!.getTime() : 0) }; + } + async function load(scope?: NativeStatusReassessmentScope, signal?: AbortSignal, canRun = () => true): Promise { + const entries: Entry[] = []; + let after: string | undefined; + while (canRun() && !signal?.aborted) { + const rows = await candidates(scope, after); + entries.push(...rows.map(entry)); + if (rows.length < 100) return entries; + after = rows[rows.length - 1].runId; + } + throw new Error("Native status reassessment discovery stopped"); + } + async function check(current: Entry, signal?: AbortSignal): Promise { + if (signal?.aborted) throw new Error("Native status reassessment stopped"); + const scope = { companyId: current.companyId, issueId: current.issueId, runId: current.id }; + const [source] = await candidates(scope); + if (!source) return null; + const due = entry(source); + if (due.dueAt > Date.now()) return due; + if (signal?.aborted) throw new Error("Native status reassessment stopped"); + const outcome = await reassessNativeStatusFromSource(db, source, { onlyWhenDue: true }); + if (outcome.kind !== "deferred" || !outcome.retry) return null; + const [remaining] = await candidates(scope); + if (!remaining) return null; + const retry = entry(remaining); + return retry.dueAt > Date.now() ? retry : { ...retry, dueAt: Date.now() + 30_000 }; + } + function register(coordinator: ReturnType, canRun: () => boolean) { + return registerScopedDeadlineWork(coordinator, { + queue: DELIVERY_QUEUES.nativeStatusReassessment, + decodeScope: (hint: string): NativeStatusReassessmentScope | undefined => hint === "*" ? undefined : JSON.parse(hint), + matches: (item: Entry, scope?: NativeStatusReassessmentScope) => !scope || item.companyId === scope.companyId + && (!scope.issueId || item.issueId === scope.issueId) && (!scope.runId || item.id === scope.runId) + && (!scope.agentId || item.agentId === scope.agentId), + load: (scope, signal) => load(scope, signal, canRun), + check: (item, signal) => { + if (!canRun()) throw new Error("Native status reassessment paused"); + return check(item, signal); + }, + }, canRun); + } + return { load, check, register }; +} diff --git a/server/src/services/native-runtime/native-status-reassessment.ts b/server/src/services/native-runtime/native-status-reassessment.ts new file mode 100644 index 00000000000..41bc23f45a7 --- /dev/null +++ b/server/src/services/native-runtime/native-status-reassessment.ts @@ -0,0 +1,266 @@ +import { readRequiredQualityReview } from "../required-quality-review.js"; +import { hasNativeWorkspaceFinalizationOwner } from "./native-workspace-finalization-recovery-state.js"; +import { and, desc, eq, gt, or } from "drizzle-orm"; +import { completionContracts, heartbeatRuns, issueWorkProducts, issues, nativeRunResults, + statusDecisions, workAssessments, workspaceOperations, type Db } from "@paperclipai/db"; +import { issueService } from "../issues.js"; +import { decisionHasRetiredAutomaticReview } from "./automatic-completion-reviews.js"; +import { classifyNativeEvidence } from "./evidence-classifier.js"; +import { hasPendingNativeChildCompletion } from "./native-child-completion-delivery.js"; +import { resolveNativeReconciliationStatus, type NativeReconciliationFacts } from "./native-reconciliation-status.js"; +import { pendingNativeGovernance, resolveNativeFinalizerStatus } from "./native-run-finalizer.js"; +import { NATIVE_STATUS_ARBITER_POLICY_VERSION, type NativeAuthoritativeIssueStatus, type NativeStatusDecision } from "./status-arbiter.js"; +import { commitNativeStatusDecision, NativeStatusRaceError } from "./status-decision-committer.js"; +import { recordNativeWorkAssessment } from "./work-assessments.js"; + +export type NativeStatusReassessmentSource = { + runId: string; companyId: string; agentId: string; issueId: string; + issueStatus: string; issueStatusVersion: number; issueDecisionId: string | null; + coordinatorPhase: string; assessmentId: string; decisionId: string | null; + nativeStatusWakeIntentId: unknown; +}; + +type Outcome = { kind: "unchanged" | "superseded" } | { kind: "deferred"; retry?: boolean } | { + kind: "committed"; + result: { runId: string; phase: "committed"; reconciliationAction: string | null; + decisionId: string; reconciliationDecision: NativeStatusDecision }; +}; + +function record(value: unknown): Record { + return value && typeof value === "object" && !Array.isArray(value) + ? value as Record + : {}; +} + +function isSupersededSource( + row: NativeStatusReassessmentSource, + decision: Record, + priorDecisionId: string | null | undefined, +): boolean { + return row.coordinatorPhase === "committed" + && row.decisionId !== null + && row.issueDecisionId !== null + && row.issueDecisionId !== row.decisionId + && !( + decision.statusAction === "preserve" + && priorDecisionId === row.issueDecisionId + ); +} + + + +export async function nativeStatusReassessmentSourceIsSuperseded( + db: Db, + row: NativeStatusReassessmentSource, +): Promise { + if (row.coordinatorPhase !== "committed" || row.decisionId === null + || row.issueDecisionId === null || row.issueDecisionId === row.decisionId) return false; + const [decision] = await db.select({ decisionJson: statusDecisions.decisionJson }) + .from(statusDecisions).where(and(eq(statusDecisions.id, row.decisionId), + eq(statusDecisions.companyId, row.companyId), eq(statusDecisions.issueId, row.issueId))).limit(1); + if (!decision) throw new Error("native_committed_decision_missing"); + const [assessment] = await db.select({ priorDecisionId: workAssessments.priorDecisionId }) + .from(workAssessments).where(and(eq(workAssessments.id, row.assessmentId), + eq(workAssessments.companyId, row.companyId), eq(workAssessments.issueId, row.issueId))).limit(1); + return isSupersededSource(row, record(decision.decisionJson), assessment?.priorDecisionId); +} + +export async function reassessNativeStatusFromSource( + db: Db, + row: NativeStatusReassessmentSource, + options: { onlyWhenDue?: boolean } = {}, +): Promise { + const assessment = await db.select({ + id: workAssessments.id, + contractId: workAssessments.contractId, + resultId: workAssessments.resultId, + policyVersion: workAssessments.policyVersion, + priorIssueStatus: workAssessments.priorIssueStatus, + priorStatusVersion: workAssessments.priorStatusVersion, + priorDecisionId: workAssessments.priorDecisionId, + assessmentJson: workAssessments.assessmentJson, + createdAt: workAssessments.createdAt, + }).from(workAssessments).where(and( + eq(workAssessments.id, row.assessmentId), + eq(workAssessments.companyId, row.companyId), + eq(workAssessments.issueId, row.issueId), + )).limit(1).then((entries) => entries[0] ?? null); + const changedEvidence = assessment + ? await db.select({ id: issueWorkProducts.id }).from(issueWorkProducts).where(and( + eq(issueWorkProducts.companyId, row.companyId), + eq(issueWorkProducts.issueId, row.issueId), + or( + gt(issueWorkProducts.createdAt, assessment.createdAt), + gt(issueWorkProducts.updatedAt, assessment.createdAt), + ), + )).limit(1).then((entries) => entries[0] ?? null) + : null; + const currentDecision = row.decisionId + ? await db.select().from(statusDecisions).where(and( + eq(statusDecisions.id, row.decisionId), + eq(statusDecisions.companyId, row.companyId), + eq(statusDecisions.issueId, row.issueId), + )).limit(1).then((entries) => entries[0] ?? null) + : null; + const currentDecisionJson = record(currentDecision?.decisionJson); + if (row.coordinatorPhase === "committed" && row.decisionId && !currentDecision) { + throw new Error("native_committed_decision_missing"); + } + if (isSupersededSource(row, currentDecisionJson, assessment?.priorDecisionId)) return { kind: "superseded" }; + const persistedProjectedVersion = Number(currentDecisionJson.projectedStatusVersion); + const expectedProjectedVersion = currentDecision + ? Number.isFinite(persistedProjectedVersion) + ? persistedProjectedVersion + : Number(currentDecision.decisionVersion) - (currentDecisionJson.statusAction === "preserve" ? 1 : 0) + : null; + const issueMatchesCurrentDecision = !!assessment + && !!currentDecision + && row.issueDecisionId === row.decisionId + && currentDecision.assessmentId === assessment.id + && currentDecision.toStatus === row.issueStatus + && expectedProjectedVersion === Number(row.issueStatusVersion); + const authoritativeStatusChanged = !!assessment && !issueMatchesCurrentDecision && ( + assessment.priorIssueStatus !== row.issueStatus + || Number(assessment.priorStatusVersion) !== Number(row.issueStatusVersion) + ); + const retiredAutomaticReview = issueMatchesCurrentDecision && currentDecision + ? await decisionHasRetiredAutomaticReview(db, currentDecision) : false; + let reassessment = null; + let resultRow = null; + let contractRow = null; + if (assessment && (authoritativeStatusChanged || changedEvidence || retiredAutomaticReview)) { + [resultRow, contractRow] = await Promise.all([ + db.select().from(nativeRunResults).where(and( + eq(nativeRunResults.id, assessment.resultId), + eq(nativeRunResults.companyId, row.companyId), + eq(nativeRunResults.issueId, row.issueId), + eq(nativeRunResults.runId, row.runId), + )).limit(1).then((entries) => entries[0] ?? null), + db.select().from(completionContracts).where(and( + eq(completionContracts.id, assessment.contractId), + eq(completionContracts.companyId, row.companyId), + eq(completionContracts.issueId, row.issueId), + )).limit(1).then((entries) => entries[0] ?? null), + ]); + if (!resultRow || !contractRow) throw new Error("native_reconciliation_evidence_binding_missing"); + reassessment = await classifyNativeEvidence({ + db, + companyId: row.companyId, + issueId: row.issueId, + runId: row.runId, + contract: record(contractRow.contractJson), + result: record(record(resultRow.resultJson).result), + }); + } + const previousAssessment = record(assessment?.assessmentJson); + const newEvidenceSatisfiesContract = !!changedEvidence + && previousAssessment.allCriteriaSatisfied !== true + && reassessment?.allCriteriaSatisfied === true + && reassessment.verificationPassed === true; + const facts: NativeReconciliationFacts = authoritativeStatusChanged + ? { authoritativeStatusChanged: true } + : newEvidenceSatisfiesContract + ? { newEvidenceSatisfiesContract: true } + : {}; + if (Object.keys(facts).length > 0 || retiredAutomaticReview) { + if (!assessment || !reassessment || !resultRow || !contractRow) { + throw new Error("native_reconciliation_reassessment_missing"); + } + const currentIssue = await db.select().from(issues) + .where(and(eq(issues.id, row.issueId), eq(issues.companyId, row.companyId))).then((entries) => entries[0]); + const readiness = retiredAutomaticReview ? await issueService(db).getDependencyReadiness(row.issueId, db) : null; + const currentRun = retiredAutomaticReview ? await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, row.runId)).then((entries) => entries[0]) : null; + const latestContract = retiredAutomaticReview ? await db.select({ id: completionContracts.id }).from(completionContracts) + .where(and(eq(completionContracts.companyId, row.companyId), eq(completionContracts.issueId, row.issueId))) + .orderBy(desc(completionContracts.revision)).limit(1).then((entries) => entries[0]) : null; + if (retiredAutomaticReview && (!currentIssue || currentRun?.status !== "succeeded" + || (options.onlyWhenDue && hasNativeWorkspaceFinalizationOwner(currentRun.runnerProfileJson)) + || latestContract?.id !== contractRow.id + || (currentIssue.executionRunId && currentIssue.executionRunId !== row.runId))) return { kind: "deferred" }; + const reviewBarrier = retiredAutomaticReview ? await db.select({ status: workspaceOperations.status }) + .from(workspaceOperations).where(and(eq(workspaceOperations.companyId, row.companyId), + eq(workspaceOperations.heartbeatRunId, row.runId), eq(workspaceOperations.phase, "workspace_finalize"))) + .orderBy(desc(workspaceOperations.createdAt)).limit(1).then((entries) => entries[0]) : null; + if (reviewBarrier && reviewBarrier.status !== "succeeded") return { kind: "deferred" }; + const childCompletionRecipient = { + companyId: row.companyId, issueId: row.issueId, agentId: row.agentId, runId: row.runId, + sourceIntentId: typeof row.nativeStatusWakeIntentId === "string" ? row.nativeStatusWakeIntentId : null, + }; + const hasPendingChildCompletion = await hasPendingNativeChildCompletion(db, childCompletionRecipient); + const reassessmentRow = await recordNativeWorkAssessment({ + db, + companyId: row.companyId, + issueId: row.issueId, + runId: row.runId, + turnId: resultRow.turnId, + contractId: contractRow.id, + contractCanonicalSha256: contractRow.canonicalSha256, + resultId: resultRow.id, + resultCanonicalSha256: resultRow.canonicalSha256, + priorIssueStatus: row.issueStatus, + priorStatusVersion: Number(row.issueStatusVersion), + priorDecisionId: row.issueDecisionId, + policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, + assessment: reassessment, + supersedesAssessmentId: assessment.id, + }); + const hasUnresolvedIssueBlockers = (readiness?.unresolvedBlockerCount ?? 0) > 0; + const governanceGate = retiredAutomaticReview && currentIssue + ? await pendingNativeGovernance({ db, companyId: row.companyId, issueId: row.issueId, + runId: row.runId, executionState: record(currentIssue.executionState) }) : null; + const qualityReview = currentIssue ? await readRequiredQualityReview(db, currentIssue) : null; + let decision = retiredAutomaticReview && currentIssue + ? resolveNativeFinalizerStatus({ + hasPendingChildCompletion, + requiredQualityReviewSatisfied: qualityReview?.satisfied, + assessment: reassessment, terminalState: "succeeded", workspaceFinalizeStatus: "succeeded", + governanceGate, + completionClaimPolicyAccepted: contractRow.risk === "low" && contractRow.completionAuthority === "agent_claim_policy", + hasUnresolvedIssueBlockers, + reviewOwnerUserId: currentIssue.responsibleUserId ?? currentIssue.createdByUserId, + priorIssueStatus: row.issueStatus as NativeAuthoritativeIssueStatus, agentId: row.agentId, + }) + : resolveNativeReconciliationStatus({ + facts: { ...facts, hasPendingChildCompletion }, + priorIssueStatus: row.issueStatus as NativeAuthoritativeIssueStatus, agentId: row.agentId, + }); + if (decision.statusAction === "done" && qualityReview?.satisfied === false) { + decision = resolveNativeFinalizerStatus({ assessment: reassessment, terminalState: "succeeded", workspaceFinalizeStatus: "succeeded", + requiredQualityReviewSatisfied: false, priorIssueStatus: row.issueStatus as NativeAuthoritativeIssueStatus, agentId: row.agentId }); + } + let committed: Awaited>; + try { + committed = await commitNativeStatusDecision({ + db, + companyId: row.companyId, + issueId: row.issueId, + runId: row.runId, + assessmentId: reassessmentRow.id, + priorStatus: row.issueStatus, + priorStatusVersion: Number(row.issueStatusVersion), + priorDecisionId: row.issueDecisionId, + decision, + requireReassessmentSource: options.onlyWhenDue ? { + phase: row.coordinatorPhase, resultId: resultRow.id, assessmentId: assessment.id, + decisionId: row.decisionId, agentId: row.agentId, retiredAutomaticReview: retiredAutomaticReview ? { contractId: contractRow.id, governanceGate, hasUnresolvedIssueBlockers } : null, + } : undefined, + requireNoPendingChildCompletion: decision.statusAction === "done" ? childCompletionRecipient : undefined, + supersedesCommittedDecisionId: row.coordinatorPhase === "committed" + ? row.decisionId ?? undefined + : undefined, + }); + } catch (error) { + if (error instanceof NativeStatusRaceError) return options.onlyWhenDue + ? { kind: "deferred", retry: true } : { kind: "deferred" }; + throw error; + } + return { kind: "committed", result: { + runId: row.runId, + phase: "committed", + reconciliationAction: decision.reasonCode, + decisionId: committed.decision.id, + reconciliationDecision: decision, + } }; + } + return { kind: "unchanged" }; +} diff --git a/server/src/services/native-runtime/native-workspace-export-recovery.ts b/server/src/services/native-runtime/native-workspace-export-recovery.ts index c638f3b3e94..41e51f2ddc7 100644 --- a/server/src/services/native-runtime/native-workspace-export-recovery.ts +++ b/server/src/services/native-runtime/native-workspace-export-recovery.ts @@ -95,13 +95,11 @@ async function recoverCandidate(db: Db, candidate: { runId: string; companyId: s let updatedRun: typeof heartbeatRuns.$inferSelect | undefined; if (eligible) { await ownership.assertHeld(); - - + await notifyNativeFinalizationWork(tx, { companyId: run.companyId, runId: run.id }); await tx.insert(workspaceOperations).values({ companyId: run.companyId, heartbeatRunId: run.id, issueId: issue.id, phase: "workspace_finalize", status: "succeeded", exitCode: 0, finishedAt: now, metadata: { owningService: "native_workspace_finalizer", workspaceSync: { omitted: true, reason: "restore_unsafe_archive", legacy: true } }, }); - await notifyNativeFinalizationWork(tx, { companyId: run.companyId, runId: run.id }); await tx.update(nativeRunFinalizations).set({ phase: "result_accepted", failureCode: null, nextAttemptAt: null, failureDetail: { workspaceFinalizeAttempt: 0, legacyUnsafeExportOmitted: { resultId: result.id, recoveredAt: now.toISOString() } }, updatedAt: now, }).where(eq(nativeRunFinalizations.runId, run.id)); diff --git a/server/src/services/native-runtime/native-workspace-finalization-ownership.ts b/server/src/services/native-runtime/native-workspace-finalization-ownership.ts index 7ab2cbf1edb..60db370812e 100644 --- a/server/src/services/native-runtime/native-workspace-finalization-ownership.ts +++ b/server/src/services/native-runtime/native-workspace-finalization-ownership.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js"; import { beginRunPhase } from "../execution-preparation-timing.js"; import { logActivity } from "../activity-log.js"; import { issueRecoveryActionService } from "../issue-recovery-actions.js"; @@ -21,6 +22,12 @@ const OWNER_KEY = "nativeWorkspaceFinalizationOwner"; export function nativeWorkspaceFinalizationLockKey(companyId: string, runId: string): string { return `native-workspace-finalization:${companyId}:${runId}`; } +export async function tryLockNativeWorkspaceFinalization(tx: Db, input: { companyId: string; runId: string }): Promise { + const [lock] = await tx.execute(sql`select pg_try_advisory_xact_lock(hashtextextended( + ${nativeWorkspaceFinalizationLockKey(input.companyId, input.runId)}, 0)) as acquired`) as unknown as Array<{ acquired: boolean }>; + return lock.acquired; +} + const activeOwnerTokens = new Set(); // Positive same-boot evidence, retained only when durable cleanup could not be // confirmed. Absence from the active set alone never proves another token joined. @@ -258,9 +265,12 @@ export async function withNativeWorkspaceFinalizationOwnership( activeOwnerTokens.delete(owner.token); joinedOwnerTokens.add(owner.token); try { - await input.db.update(heartbeatRuns).set({ - runnerProfileJson: sql`coalesce(${heartbeatRuns.runnerProfileJson}, '{}'::jsonb) - ${OWNER_KEY}`, - }).where(owned); + await input.db.transaction(async tx => { + await notifyNativeStatusReassessment(tx, { companyId: input.companyId, runId: input.runId }); + await tx.update(heartbeatRuns).set({ + runnerProfileJson: sql`coalesce(${heartbeatRuns.runnerProfileJson}, '{}'::jsonb) - ${OWNER_KEY}`, + }).where(owned); + }); joinedOwnerTokens.delete(owner.token); } catch { // A reconnect in this exact controller boot can use the positive join diff --git a/server/src/services/native-runtime/native-workspace-finalization-recovery-state.ts b/server/src/services/native-runtime/native-workspace-finalization-recovery-state.ts index 71bc4f426c5..a4798b23e55 100644 --- a/server/src/services/native-runtime/native-workspace-finalization-recovery-state.ts +++ b/server/src/services/native-runtime/native-workspace-finalization-recovery-state.ts @@ -5,6 +5,10 @@ import { nativeSha256 } from "./canonical.js"; export const NATIVE_WORKSPACE_OWNER_UNVERIFIED = "native_workspace_finalization_owner_unverified"; +export function hasNativeWorkspaceFinalizationOwner(profile: Record | null | undefined): boolean { + return profile?.nativeWorkspaceFinalizationOwner != null; +} + /** A terminal task does not prove that an older controller's copyback stopped. */ export async function hasUnverifiedNativeWorkspaceFinalizationOwner( db: Db, @@ -20,5 +24,5 @@ export async function hasUnverifiedNativeWorkspaceFinalizationOwner( eq(heartbeatRuns.runtimeMode, "native"), )).limit(1); const owner = run?.profile?.nativeWorkspaceFinalizationOwner; - return owner != null && action.fingerprint === nativeSha256({ runId, owner }); + return hasNativeWorkspaceFinalizationOwner(run?.profile) && action.fingerprint === nativeSha256({ runId, owner }); } diff --git a/server/src/services/native-runtime/native-workspace-finalizer.test.ts b/server/src/services/native-runtime/native-workspace-finalizer.test.ts index f79698d5b9a..327515304d7 100644 --- a/server/src/services/native-runtime/native-workspace-finalizer.test.ts +++ b/server/src/services/native-runtime/native-workspace-finalizer.test.ts @@ -1,3 +1,5 @@ +import { DELIVERY_QUEUES, subscribeDeliveryWork } from "../delivery-work-notifications.js"; +import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js"; import * as pluginWorkspaces from "../plugin-workspaces.js"; import * as platformProof from "./native-workspace-platform-proof.js"; import { currentNativeControllerIdentity } from "./native-restart-recovery.js"; @@ -38,7 +40,8 @@ import { resumeNativeWorkspaceFinalization } from "./native-workspace-finalizer. import { reconcileNativeFinalizations } from "./native-finalization-reconciler.js"; describe("native workspace finalization recovery", () => { - let temporary: Awaited>; + let temporary: Awaited> | undefined; + let connectionString: string; let db: ReturnType; let workspaceRoot: string; let priorLogRoot: string | undefined; @@ -125,10 +128,10 @@ describe("native workspace finalization recovery", () => { workspaceRoot, "operation-logs", ); - temporary = await startEmbeddedPostgresTestDatabase( - "paperclip-native-workspace-finalizer-", - ); - db = createDb(temporary.connectionString); + const external = resolveExternalTestDatabaseUrl("PAPERCLIP_NATIVE_WORKSPACE_FINALIZER_TEST_DATABASE_URL"); + if (!external) temporary = await startEmbeddedPostgresTestDatabase("paperclip-native-workspace-finalizer-"); + connectionString = external ?? temporary!.connectionString; + db = createDb(connectionString); await db.insert(companies).values([ { id: companyId, name: "Native workspace owner", issuePrefix: "NWO" }, @@ -162,7 +165,8 @@ describe("native workspace finalization recovery", () => { }, 30_000); afterAll(async () => { - await temporary.cleanup(); + await db?.$client.end({ timeout: 1 }); + await temporary?.cleanup(); await fs.rm(workspaceRoot, { recursive: true, force: true }); if (priorLogRoot === undefined) { delete process.env.WORKSPACE_OPERATION_LOG_BASE_PATH; @@ -210,7 +214,17 @@ describe("native workspace finalization recovery", () => { const seed = await seedAbandonedCopyback(); const [before] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId)); expect(await withNativeWorkspaceFinalizationOwnership(seed, async () => "unexpected")).toEqual({ acquired: false }); - const receipt = await resumeNativeWorkspaceAfterOwnerStop(seed); + const observed: Array> = []; + const stop = subscribeDeliveryWork(db, DELIVERY_QUEUES.nativeStatusReassessment, () => { + observed.push(db.select({ profile: heartbeatRuns.runnerProfileJson }).from(heartbeatRuns) + .where(eq(heartbeatRuns.id, seed.runId)).then(rows => rows[0].profile)); + }); + let receipt: Awaited>; + try { + receipt = await resumeNativeWorkspaceAfterOwnerStop(seed); + await vi.waitFor(() => expect(observed).toHaveLength(1)); + expect(await Promise.all(observed)).toEqual([{ preserved: "workspace descriptor" }]); + } finally { stop(); } expect(receipt).toEqual({ runId: seed.runId, resultId: before.resultId, status: "queued" }); const [after] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, seed.runId)); expect(after).toEqual(before); @@ -425,7 +439,7 @@ describe("native workspace finalization recovery", () => { it("keeps ordinary progress writable even when the application pool has one connection", async () => { const seeded = await seedRun({ executionWorkspaceId: randomUUID(), title: "Small application pool" }); - const smallPool = createDb(temporary.connectionString, { maxConnections: 1 }); + const smallPool = createDb(connectionString, { maxConnections: 1 }); const result = await withNativeWorkspaceFinalizationOwnership({ db: smallPool, companyId, runId: seeded.runId }, async (ownership) => { await ownership.assertHeld(); await smallPool.update(heartbeatRuns).set({ updatedAt: new Date() }).where(eq(heartbeatRuns.id, seeded.runId)); @@ -438,7 +452,7 @@ describe("native workspace finalization recovery", () => { const seeded = await seedRun({ executionWorkspaceId: randomUUID(), title: "Live copyback" }); const live = await withNativeWorkspaceFinalizationOwnership({ db, companyId, runId: seeded.runId }, async (ownership) => { await ownership.assertHeld(); - expect(await resumeNativeWorkspaceFinalization({ db: createDb(temporary.connectionString), runId: seeded.runId })).toBeNull(); + expect(await resumeNativeWorkspaceFinalization({ db: createDb(connectionString), runId: seeded.runId })).toBeNull(); expect(await db.select().from(workspaceOperations).where(eq(workspaceOperations.heartbeatRunId, seeded.runId))).toEqual([]); return "copied"; }); @@ -462,7 +476,7 @@ describe("native workspace finalization recovery", () => { try { await entered; await db.execute(sql`select pg_terminate_backend(pid) from pg_locks where locktype = 'advisory' and database = (select oid from pg_database where datname = current_database()) and pid <> pg_backend_pid()`); - const second = await withNativeWorkspaceFinalizationOwnership({ db: createDb(temporary.connectionString), companyId, runId: seeded.runId }, async () => "must not run"); + const second = await withNativeWorkspaceFinalizationOwnership({ db: createDb(connectionString), companyId, runId: seeded.runId }, async () => "must not run"); expect(second).toEqual({ acquired: false }); const [held] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.runId)); expect(held.runnerProfileJson?.nativeWorkspaceFinalizationOwner).toBeTruthy(); @@ -475,7 +489,7 @@ describe("native workspace finalization recovery", () => { it("recovers its own joined copyback receipt after the application pool disconnects during cleanup", async () => { const seeded = await seedRun({ executionWorkspaceId: randomUUID(), title: "Reconnect joined copyback" }); - const interruptedDb = createDb(temporary.connectionString); + const interruptedDb = createDb(connectionString); let joined = false; await expect(withNativeWorkspaceFinalizationOwnership({ db: interruptedDb, companyId, runId: seeded.runId }, async () => { await fs.writeFile(path.join(workspaceRoot, "joined-copyback.txt"), "durable work"); @@ -493,7 +507,7 @@ describe("native workspace finalization recovery", () => { .toEqual({ acquired: false }); } await db.update(heartbeatRuns).set({ runnerProfileJson: beforeReconnect.runnerProfileJson }).where(eq(heartbeatRuns.id, seeded.runId)); - expect(await withNativeWorkspaceFinalizationOwnership({ db: createDb(temporary.connectionString), companyId, runId: seeded.runId }, async () => "resumed")) + expect(await withNativeWorkspaceFinalizationOwnership({ db: createDb(connectionString), companyId, runId: seeded.runId }, async () => "resumed")) .toEqual({ acquired: true, value: "resumed" }); }); diff --git a/server/src/services/native-runtime/native-workspace-owner-recovery.ts b/server/src/services/native-runtime/native-workspace-owner-recovery.ts index 4b3e19e3311..ae122b5edf7 100644 --- a/server/src/services/native-runtime/native-workspace-owner-recovery.ts +++ b/server/src/services/native-runtime/native-workspace-owner-recovery.ts @@ -1,3 +1,5 @@ +import { tryLockNativeWorkspaceFinalization } from "./native-workspace-finalization-ownership.js"; +import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js"; import os from "node:os"; import { and, eq, sql } from "drizzle-orm"; import { heartbeatRuns, issues, issueRecoveryActions, nativeRunFinalizations, type Db } from "@paperclipai/db"; @@ -24,8 +26,7 @@ export async function resumeNativeWorkspaceAfterOwnerStop(input: { const admitted = await input.db.transaction(async tx => { // Share the physical copyback lock. Do not wait behind an active exporter // and then apply a confirmation that described an earlier owner. - const locks = await tx.execute(sql`select pg_try_advisory_xact_lock(hashtextextended(${`native-workspace-finalization:${input.companyId}:${input.runId}`}, 0)) as acquired`); - if (locks[0]?.acquired !== true) throw conflict("Workspace copyback is still active. Wait for it to stop before confirming recovery."); + if (!await tryLockNativeWorkspaceFinalization(tx as unknown as Db, input)) throw conflict("Workspace copyback is still active. Wait for it to stop before confirming recovery."); // Match the status arbiter's coordinator -> issue -> run lock order. const [coordinator] = await tx.select().from(nativeRunFinalizations).where(and(eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.issueId), eq(nativeRunFinalizations.runId, input.runId))).for("update"); const [issue] = await tx.select().from(issues).where(and(eq(issues.companyId, input.companyId), eq(issues.id, input.issueId))).for("update"); @@ -65,6 +66,7 @@ export async function resumeNativeWorkspaceAfterOwnerStop(input: { const now = new Date(); const proof = { owner: prior, resultId: result.id, actorId: input.actorId, stopEvidence: confirmation.stopEvidence, confirmedAt: now.toISOString() }; + await notifyNativeStatusReassessment(tx, { companyId: input.companyId, issueId: input.issueId, runId: input.runId }); await tx.update(heartbeatRuns).set({ runnerProfileJson: sql`${heartbeatRuns.runnerProfileJson} - ${OWNER_KEY}`, updatedAt: now, }).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, run.id), diff --git a/server/src/services/native-runtime/runner-task-attachments.test.ts b/server/src/services/native-runtime/runner-task-attachments.test.ts index 88bea30420e..d122e301eb9 100644 --- a/server/src/services/native-runtime/runner-task-attachments.test.ts +++ b/server/src/services/native-runtime/runner-task-attachments.test.ts @@ -1,3 +1,4 @@ +import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js"; import { createHash, randomUUID } from "node:crypto"; import { Readable } from "node:stream"; import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; @@ -11,10 +12,14 @@ import { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority. const sha = (body: Buffer) => createHash("sha256").update(body).digest("hex"); describe("Dot assigned-task attachment reading", () => { - let temporary: Awaited>; + let temporary: Awaited> | undefined; let db: ReturnType; - beforeAll(async () => { temporary = await startEmbeddedPostgresTestDatabase("dot-task-files-"); db = createDb(temporary.connectionString); }); - afterAll(async () => { await temporary?.cleanup(); }); + beforeAll(async () => { + const external = resolveExternalTestDatabaseUrl("PAPERCLIP_RUNNER_ATTACHMENTS_TEST_DATABASE_URL"); + if (!external) temporary = await startEmbeddedPostgresTestDatabase("dot-task-files-"); + db = createDb(external ?? temporary!.connectionString); + }); + afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); }); afterEach(() => { vi.unstubAllEnvs(); vi.unstubAllGlobals(); }); async function fixture(body = Buffer.from("only in the file 🌍"), enabled = true) { const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(); diff --git a/server/src/services/native-runtime/status-decision-committer.ts b/server/src/services/native-runtime/status-decision-committer.ts index 04696bc1c85..cedd7130e8c 100644 --- a/server/src/services/native-runtime/status-decision-committer.ts +++ b/server/src/services/native-runtime/status-decision-committer.ts @@ -1,3 +1,8 @@ +import { lockCurrentNativeWorkAssessmentEvidence } from "./work-assessments.js"; +import { hasNativeWorkspaceFinalizationOwner } from "./native-workspace-finalization-recovery-state.js"; +import { tryLockNativeWorkspaceFinalization } from "./native-workspace-finalization-ownership.js"; +import { lockCurrentNativeCompletionContract } from "./completion-contracts.js"; +import { isNativeRunnerOwnershipHeld } from "./native-runner-ownership.js"; import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js"; import { notifyAutomaticCompletionReviews } from "../automatic-completion-review-notifications.js"; @@ -8,7 +13,7 @@ import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js" import { scheduledIssueMonitorWait } from "../issue-monitors.js"; import { onboardingFirstTaskCompletionAllowed } from "../onboarding-first-task-admission.js"; import { hasPublishedGitHubReview, readInheritedGitHubReviewWait } from "../chat-channels/github/review-wait.js"; -import { pendingNativeGovernance } from "./native-governance.js"; +import { lockNativeGovernanceSources, pendingNativeGovernance } from "./native-governance.js"; import { findSupersededExternalReviewRequests, supersedeExternalReviewRequestsInTransaction } from "./superseded-external-review-requests.js"; import { normalizePrpResultSignals, type PrpStructuredRunResult } from "../../vendor/paperclip-runner/index.js"; import { readNativeMonitorDelivery, readNativePreservedMonitorWait, type NativeMonitorDelivery, type NativePreservedMonitorWait } from "./native-monitor-delivery.js"; @@ -38,6 +43,7 @@ import { } from "@paperclipai/db"; import { NATIVE_STATUS_ARBITER_POLICY_VERSION, + type NativeGovernanceGate, type NativeStatusDecision, type NativeStatusEffect, } from "./status-arbiter.js"; @@ -1603,6 +1609,10 @@ export async function commitNativeStatusDecision(input: { requireMonitorDelivery?: NativeMonitorDelivery; requirePreservedMonitorWait?: NativePreservedMonitorWait; requireNoPendingChildCompletion?: NativeChildCompletionRecipient; + requireReassessmentSource?: { + phase: string; resultId: string; assessmentId: string; decisionId: string | null; agentId: string; + retiredAutomaticReview: { contractId: string; governanceGate: NativeGovernanceGate | null; hasUnresolvedIssueBlockers: boolean } | null; + }; requireModelRejectionOwner?: { agentId: string; reviewContext: NativeReviewAssignmentContext | null }; requireProviderFailureOwner?: { agentId: string; reviewContext: NativeReviewAssignmentContext | null }; requireBoardResponseWaitSource?: NativeBoardResponseWaitSource; @@ -1626,6 +1636,11 @@ export async function commitNativeStatusDecision(input: { const publications: ActivityPublication[] = []; const terminalRunsToEmit: (typeof heartbeatRuns.$inferSelect)[] = []; const committed = await input.db.transaction(async (tx) => { + const retiredReview = input.requireReassessmentSource?.retiredAutomaticReview; + if (retiredReview && !await lockCurrentNativeCompletionContract(tx as unknown as Db, { + companyId: input.companyId, issueId: input.issueId, contractId: retiredReview.contractId, + })) throw new NativeStatusRaceError(); + if (retiredReview && !await tryLockNativeWorkspaceFinalization(tx as unknown as Db, input)) throw new NativeStatusRaceError(); const coordinator = await tx .select() .from(nativeRunFinalizations) @@ -1654,6 +1669,67 @@ export async function commitNativeStatusDecision(input: { .limit(1) .then((rows) => rows[0] ?? null); if (!issue) throw new NativeStatusRaceError(); + if (input.requireReassessmentSource) { + const expected = input.requireReassessmentSource; + const now = Date.now(); + if (coordinator.phase === "terminal_failure" || coordinator.phase !== expected.phase || + coordinator.resultId !== expected.resultId || coordinator.assessmentId !== expected.assessmentId || + coordinator.decisionId !== expected.decisionId || + (coordinator.nextAttemptAt !== null && coordinator.nextAttemptAt.getTime() > now) || + (coordinator.leaseOwner !== null && + (coordinator.leaseExpiresAt === null || coordinator.leaseExpiresAt.getTime() > now))) { + throw new NativeStatusRaceError(); + } + const run = await tx.select().from(heartbeatRuns).where(and( + eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId), + eq(heartbeatRuns.nativeIssueId, input.issueId), eq(heartbeatRuns.agentId, expected.agentId), + )).for("update").limit(1).then(rows => rows[0]); + if (!run || run.runtimeMode !== "native" || isNativeRunnerOwnershipHeld(run)) throw new NativeStatusRaceError(); + if (expected.retiredAutomaticReview && (run.status !== "succeeded" || + hasNativeWorkspaceFinalizationOwner(run.runnerProfileJson) || + (issue.executionRunId !== null && issue.executionRunId !== input.runId))) { + throw new NativeStatusRaceError(); + } + if (expected.retiredAutomaticReview) { + try { + const [barrier] = await tx.select({ status: workspaceOperations.status }).from(workspaceOperations) + .where(and(eq(workspaceOperations.companyId, input.companyId), + eq(workspaceOperations.heartbeatRunId, input.runId), eq(workspaceOperations.phase, "workspace_finalize"))) + .orderBy(desc(workspaceOperations.createdAt)).limit(1).for("share", { noWait: true }); + if (barrier && barrier.status !== "succeeded") throw new NativeStatusRaceError(); + const executionState = record(issue.executionState); + if (executionState.status !== "pending") await lockNativeGovernanceSources({ db: tx as unknown as Db, + companyId: input.companyId, issueId: input.issueId }); + const gate = await pendingNativeGovernance({ db: tx as unknown as Db, companyId: input.companyId, + issueId: input.issueId, runId: input.runId, executionState }); + const expectedGate = expected.retiredAutomaticReview.governanceGate; + if (gate?.kind !== expectedGate?.kind || gate?.id !== expectedGate?.id) throw new NativeStatusRaceError(); + const readiness = await issueService(tx as unknown as Db).getDependencyReadiness(input.issueId, tx, { lockSources: true }); + if ((readiness.unresolvedBlockerCount > 0) !== expected.retiredAutomaticReview.hasUnresolvedIssueBlockers) { + throw new NativeStatusRaceError(); + } + } catch (error) { + if (record(error).code === "55P03" || record(record(error).cause).code === "55P03") throw new NativeStatusRaceError(); + throw error; + } + } + const pendingEffect = expected.decisionId ? await tx.select({ id: statusDecisionEffects.id }) + .from(statusDecisionEffects).where(and( + eq(statusDecisionEffects.companyId, input.companyId), eq(statusDecisionEffects.issueId, input.issueId), + eq(statusDecisionEffects.decisionId, expected.decisionId), + inArray(statusDecisionEffects.deliveryState, ["pending", "failed"]), + )).limit(1).then(rows => rows[0]) : null; + if (pendingEffect) throw new NativeStatusRaceError(); + try { + if (!await lockCurrentNativeWorkAssessmentEvidence(tx as unknown as Db, { + companyId: input.companyId, issueId: input.issueId, runId: input.runId, + resultId: expected.resultId, assessmentId: input.assessmentId, + })) throw new NativeStatusRaceError(); + } catch (error) { + if (record(error).code === "55P03" || record(record(error).cause).code === "55P03") throw new NativeStatusRaceError(); + throw error; + } + } // A completed model turn cannot close a persistent conversation. Preserve // the task here; the response finalizer records its durable waiting state. if (issue.conversationAgentId && input.decision.statusAction === "done") { diff --git a/server/src/services/native-runtime/work-assessments.ts b/server/src/services/native-runtime/work-assessments.ts index 58975d83b6d..fe554d33edd 100644 --- a/server/src/services/native-runtime/work-assessments.ts +++ b/server/src/services/native-runtime/work-assessments.ts @@ -1,7 +1,7 @@ import { and, eq } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; -import { workAssessments } from "@paperclipai/db"; -import type { NativeEvidenceAssessment } from "./evidence-classifier.js"; +import { completionContracts, nativeRunResults, workAssessments } from "@paperclipai/db"; +import { classifyNativeEvidence, type NativeEvidenceAssessment } from "./evidence-classifier.js"; import { nativeSha256 } from "./canonical.js"; import type { NativeMonitorDelivery, NativePreservedMonitorWait } from "./native-monitor-delivery.js"; @@ -144,3 +144,33 @@ export async function recordNativeAttentionAssessment(input: { if (!row) throw new Error("native_attention_assessment_not_persisted"); return row; } + + + +export async function lockCurrentNativeWorkAssessmentEvidence(db: Db, input: { + companyId: string; + issueId: string; + runId: string; + resultId: string; + assessmentId: string; +}): Promise { + const [row] = await db.select({ + assessment: workAssessments.assessmentJson, + contract: completionContracts.contractJson, + envelope: nativeRunResults.resultJson, + }).from(workAssessments) + .innerJoin(completionContracts, and(eq(completionContracts.id, workAssessments.contractId), + eq(completionContracts.companyId, input.companyId), eq(completionContracts.issueId, input.issueId))) + .innerJoin(nativeRunResults, and(eq(nativeRunResults.id, workAssessments.resultId), + eq(nativeRunResults.companyId, input.companyId), eq(nativeRunResults.issueId, input.issueId), + eq(nativeRunResults.runId, input.runId))) + .where(and(eq(workAssessments.id, input.assessmentId), eq(workAssessments.companyId, input.companyId), + eq(workAssessments.issueId, input.issueId), eq(workAssessments.runId, input.runId), + eq(workAssessments.resultId, input.resultId))) + .limit(1).for("share", { noWait: true }); + if (!row) return false; + const current = await classifyNativeEvidence({ db, companyId: input.companyId, + issueId: input.issueId, runId: input.runId, contract: row.contract, + result: row.envelope.result as Record, lockSources: true }); + return nativeSha256(current) === nativeSha256(row.assessment); +} diff --git a/server/src/services/native-status-reassessment-notifications.ts b/server/src/services/native-status-reassessment-notifications.ts new file mode 100644 index 00000000000..fb845523bb4 --- /dev/null +++ b/server/src/services/native-status-reassessment-notifications.ts @@ -0,0 +1,8 @@ +import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; + +export type NativeStatusReassessmentScope = { companyId: string; issueId?: string; runId?: string; agentId?: string }; + +export function notifyNativeStatusReassessment(tx: object, scope: NativeStatusReassessmentScope) { + return notifyDeliveryWork(tx, DELIVERY_QUEUES.nativeStatusReassessment, + JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId, runId: scope.runId, agentId: scope.agentId })); +} diff --git a/server/src/services/recovery/service.ts b/server/src/services/recovery/service.ts index 3870dbb66c0..460c02f4462 100644 --- a/server/src/services/recovery/service.ts +++ b/server/src/services/recovery/service.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js"; import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js"; import { notifyInstructionCollection } from "../instruction-collection-notifications.js"; @@ -6453,6 +6454,7 @@ export function recoveryService( const updated = await db.transaction(async tx => { await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: issue.companyId, issueId: issue.id }); + await notifyNativeStatusReassessment(tx, { companyId: issue.companyId, issueId: issue.id }); return tx .update(issues) .set({ diff --git a/server/src/services/satisfied-connection-intents.ts b/server/src/services/satisfied-connection-intents.ts index ba8c74205d4..e99b3289395 100644 --- a/server/src/services/satisfied-connection-intents.ts +++ b/server/src/services/satisfied-connection-intents.ts @@ -1,4 +1,4 @@ -import { notifyNativeReviewResponse } from "./native-review-response-notifications.js"; +import { notifyNativeReviewState } from "./native-review-state-notifications.js"; import { notifyChatConfirmationResolution } from "./chat-confirmation-notifications.js"; import { enqueueTerminalIssueInteractionChatPublications } from "./chat-interaction-publications.js"; import { connectionManagementService } from "./connection-management.js"; @@ -156,7 +156,7 @@ export function satisfiedConnectionIntentService(db: Db) { } const connection = await findSatisfiedToolConnection(txDb, task, intent); if (!connection) return false; - await notifyNativeReviewResponse(tx, candidate); + await notifyNativeReviewState(tx, candidate); const now = new Date(); // Retire the obsolete request without inventing a human approval. await notifyChatConfirmationResolution(tx, intent); diff --git a/server/src/services/tool-access/oauth/authorization-start.ts b/server/src/services/tool-access/oauth/authorization-start.ts index 0972611c16b..6af84ff5838 100644 --- a/server/src/services/tool-access/oauth/authorization-start.ts +++ b/server/src/services/tool-access/oauth/authorization-start.ts @@ -1,3 +1,4 @@ +import { notifyNativeReviewState } from "../../native-review-state-notifications.js"; import { vercelConnectHttpError, vercelCredentialFor, vercelConnectResourcesFor } from "../vercel-credentials.js"; import { credentialScope, managedConnectorProfile } from "./client-settings.js"; import { createOAuthClientRegistration } from "./client-registration.js"; @@ -563,65 +564,69 @@ export function createOAuthAuthorizationStart(db: Db, dependencies: Dependencies href: authorizationUrl.toString(), }, }; - const [existingInteraction] = await db - .select() - .from(issueThreadInteractions) - .where( - and( - eq(issueThreadInteractions.companyId, companyId), - eq(issueThreadInteractions.issueId, input.issueId), - eq(issueThreadInteractions.idempotencyKey, idempotencyKey), - ), - ) - .limit(1); - const [interaction] = existingInteraction - ? await db - .update(issueThreadInteractions) - .set({ - status: "pending", - requestedResolverPolicy: "human_only", - effectiveResolverPolicy: "human_only", - resolverPolicyProvenance: "explicit", - effectiveResolverPolicySource: "requested", - addresseeUserId: authorizationSubjectUserId, - payload, - result: null, - resolvedAt: null, - updatedAt: new Date(), - }) - .where(eq(issueThreadInteractions.id, existingInteraction.id)) - .returning() - : await db - .insert(issueThreadInteractions) - .values({ - companyId, - issueId: input.issueId, - kind: "request_confirmation", - status: "pending", - continuationPolicy: "none", - requestedResolverPolicy: "human_only", - effectiveResolverPolicy: "human_only", - resolverPolicyProvenance: "explicit", - effectiveResolverPolicySource: "requested", - addresseeUserId: authorizationSubjectUserId, - idempotencyKey, - sourceRunId: - binding.actorType === "agent" - ? (input.actor.sessionId ?? null) - : null, - title: `Connect your ${providerName} to continue`, - summary: `${requestingAgent?.name ?? "An agent"} needs your ${providerName} identity for work running as you.`, - createdByAgentId: - binding.actorType === "agent" ? binding.actorId : null, - payload, - }) - .returning(); - if (interaction) { - await db - .update(toolOauthStates) - .set({ interactionId: interaction.id }) - .where(eq(toolOauthStates.state, state)); - } + const issueId = input.issueId; + await db.transaction(async tx => { + await notifyNativeReviewState(tx, { companyId, issueId }); + const [existingInteraction] = await tx + .select() + .from(issueThreadInteractions) + .where( + and( + eq(issueThreadInteractions.companyId, companyId), + eq(issueThreadInteractions.issueId, issueId), + eq(issueThreadInteractions.idempotencyKey, idempotencyKey), + ), + ) + .limit(1); + const [interaction] = existingInteraction + ? await tx + .update(issueThreadInteractions) + .set({ + status: "pending", + requestedResolverPolicy: "human_only", + effectiveResolverPolicy: "human_only", + resolverPolicyProvenance: "explicit", + effectiveResolverPolicySource: "requested", + addresseeUserId: authorizationSubjectUserId, + payload, + result: null, + resolvedAt: null, + updatedAt: new Date(), + }) + .where(eq(issueThreadInteractions.id, existingInteraction.id)) + .returning() + : await tx + .insert(issueThreadInteractions) + .values({ + companyId, + issueId: issueId, + kind: "request_confirmation", + status: "pending", + continuationPolicy: "none", + requestedResolverPolicy: "human_only", + effectiveResolverPolicy: "human_only", + resolverPolicyProvenance: "explicit", + effectiveResolverPolicySource: "requested", + addresseeUserId: authorizationSubjectUserId, + idempotencyKey, + sourceRunId: + binding.actorType === "agent" + ? (input.actor.sessionId ?? null) + : null, + title: `Connect your ${providerName} to continue`, + summary: `${requestingAgent?.name ?? "An agent"} needs your ${providerName} identity for work running as you.`, + createdByAgentId: + binding.actorType === "agent" ? binding.actorId : null, + payload, + }) + .returning(); + if (interaction) { + await tx + .update(toolOauthStates) + .set({ interactionId: interaction.id }) + .where(eq(toolOauthStates.state, state)); + } + }); } const nextConfig = { diff --git a/server/src/services/tool-access/oauth/cloud-callback.ts b/server/src/services/tool-access/oauth/cloud-callback.ts index 3c777fc7969..41ce659847d 100644 --- a/server/src/services/tool-access/oauth/cloud-callback.ts +++ b/server/src/services/tool-access/oauth/cloud-callback.ts @@ -1,4 +1,4 @@ -import { notifyNativeReviewResponse } from "../../native-review-response-notifications.js"; +import { notifyNativeReviewState } from "../../native-review-state-notifications.js"; import { notifyToolConnectionHealth } from "../../tool-connection-health-notifications.js"; import { managedConnectorProfile } from "./client-settings.js"; import type { ActorInfo } from "../types.js"; @@ -421,7 +421,7 @@ export function createCloudOAuthCallback(db: Db, dependencies: Dependencies) { stateRow.interactionId && linkedInteraction?.kind === "request_confirmation" ) { - await notifyNativeReviewResponse(tx, { companyId: connection.companyId, issueId: linkedInteraction.issueId }); + await notifyNativeReviewState(tx, { companyId: connection.companyId, issueId: linkedInteraction.issueId }); await tx .update(issueThreadInteractions) .set({ diff --git a/server/src/services/tool-access/oauth/direct-callback.ts b/server/src/services/tool-access/oauth/direct-callback.ts index e3f68f3ba40..5649cd36d64 100644 --- a/server/src/services/tool-access/oauth/direct-callback.ts +++ b/server/src/services/tool-access/oauth/direct-callback.ts @@ -1,4 +1,4 @@ -import { notifyNativeReviewResponse } from "../../native-review-response-notifications.js"; +import { notifyNativeReviewState } from "../../native-review-state-notifications.js"; import { notifyToolConnectionHealth } from "../../tool-connection-health-notifications.js"; import { credentialScope } from "./client-settings.js"; import { withoutOAuthRefreshLease } from "./refresh-leases.js"; @@ -387,7 +387,7 @@ export function createDirectOAuthCallback(db: Db, dependencies: Dependencies) { stateRow.interactionId && linkedInteraction?.kind === "request_confirmation" ) { - await notifyNativeReviewResponse(tx, { companyId: connection.companyId, issueId: linkedInteraction.issueId }); + await notifyNativeReviewState(tx, { companyId: connection.companyId, issueId: linkedInteraction.issueId }); await tx .update(issueThreadInteractions) .set({ diff --git a/server/src/services/tool-access/oauth/state.ts b/server/src/services/tool-access/oauth/state.ts index efa92bc139f..b9e0ccdfcb0 100644 --- a/server/src/services/tool-access/oauth/state.ts +++ b/server/src/services/tool-access/oauth/state.ts @@ -1,4 +1,4 @@ -import { notifyNativeReviewResponse } from "../../native-review-response-notifications.js"; +import { notifyNativeReviewState } from "../../native-review-state-notifications.js"; import type { ActorInfo } from "../types.js"; import { and, eq } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; @@ -164,7 +164,7 @@ async function rejectPendingOAuthInteraction( } if (!linked) return; await db.transaction(async tx => { - await notifyNativeReviewResponse(tx, { companyId: stateRow.companyId, issueId: linked.issueId }); + await notifyNativeReviewState(tx, { companyId: stateRow.companyId, issueId: linked.issueId }); await tx .update(issueThreadInteractions) .set({ diff --git a/server/src/services/work-products.ts b/server/src/services/work-products.ts index 4aa44ef7f49..b993c9d656a 100644 --- a/server/src/services/work-products.ts +++ b/server/src/services/work-products.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js"; import { notifyApprovalQueueWork } from "./approval-work-notifications.js"; import { and, desc, eq, inArray } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; @@ -274,6 +275,7 @@ export function workProductService( createForIssue: async (issueId: string, companyId: string, data: Omit) => { const row = await db.transaction(async (tx) => { + await notifyNativeStatusReassessment(tx, { companyId, issueId }); if (data.type === "pull_request") { await notifyTerminalWorkspaceWork(tx, `issue:${issueId}`); await notifyApprovalQueueWork(tx, companyId); @@ -312,6 +314,10 @@ export function workProductService( .where(eq(issueWorkProducts.id, id)) .then((rows) => rows[0] ?? null); if (!existing) return null; + await notifyNativeStatusReassessment(tx, { companyId: existing.companyId, issueId: existing.issueId }); + if ((patch.companyId && patch.companyId !== existing.companyId) || (patch.issueId && patch.issueId !== existing.issueId)) { + await notifyNativeStatusReassessment(tx, { companyId: patch.companyId ?? existing.companyId, issueId: patch.issueId ?? existing.issueId }); + } if (existing.type === "pull_request" || patch.type === "pull_request") { await notifyTerminalWorkspaceWork(tx, `issue:${existing.issueId}`); @@ -385,6 +391,9 @@ export function workProductService( // prefix behind (which a retry would then duplicate). Mirrors the // per-writer transaction the batched issue/document writers use. await db.transaction(async (tx) => { + for (const scope of new Map(values.map(value => [value.issueId, { companyId: value.companyId, issueId: value.issueId }])).values()) { + await notifyNativeStatusReassessment(tx, scope); + } for (const companyId of new Set(values.filter(value => value.type === "pull_request").map(value => value.companyId))) { await notifyApprovalQueueWork(tx, companyId); } @@ -399,6 +408,7 @@ export function workProductService( const row = await db.transaction(async tx => { const [existing] = await tx.select({ type: issueWorkProducts.type, issueId: issueWorkProducts.issueId, companyId: issueWorkProducts.companyId }) .from(issueWorkProducts).where(eq(issueWorkProducts.id, id)); + if (existing) await notifyNativeStatusReassessment(tx, { companyId: existing.companyId, issueId: existing.issueId }); if (existing?.type === "pull_request") { await notifyTerminalWorkspaceWork(tx, `issue:${existing.issueId}`); await notifyApprovalQueueWork(tx, existing.companyId); diff --git a/server/src/services/workspace-operations.ts b/server/src/services/workspace-operations.ts index 53fcbf5292c..60103c15fbd 100644 --- a/server/src/services/workspace-operations.ts +++ b/server/src/services/workspace-operations.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js"; import { randomUUID } from "node:crypto"; import type { Db } from "@paperclipai/db"; import { workspaceOperations } from "@paperclipai/db"; @@ -476,6 +477,14 @@ export function workspaceOperationService(db: Db) { }, async recordOperation(recordInput) { + async function writeState(write: (database: Db) => Promise): Promise { + if (recordInput.phase !== "workspace_finalize" || !input.heartbeatRunId) return write(db); + return db.transaction(async tx => { + await notifyNativeStatusReassessment(tx, { companyId: input.companyId, + runId: input.heartbeatRunId!, issueId: input.issueId ?? undefined }); + return write(tx as unknown as Db); + }); + } const currentUserRedactionOptions = { enabled: (await instanceSettings.getGeneral()).censorUsernameInLogs, }; @@ -521,7 +530,7 @@ export function workspaceOperationService(db: Db) { if (runtimeControlAction) liveRuntimeControlOperationIds.add(id); try { - await db.insert(workspaceOperations).values({ + await writeState(database => database.insert(workspaceOperations).values({ id, companyId: input.companyId, executionWorkspaceId, @@ -539,7 +548,7 @@ export function workspaceOperationService(db: Db) { ) as Record | null, startedAt, updatedAt: startedAt, - }); + })); } catch (insertError) { liveRuntimeControlOperationIds.delete(id); throw insertError; @@ -630,7 +639,7 @@ export function workspaceOperationService(db: Db) { await append("stderr", result.stderr ?? null); const finalized = await logStore.finalize(handle); const finishedAt = new Date(); - const row = await db + const row = await writeState(database => database .update(workspaceOperations) .set({ executionWorkspaceId, @@ -650,7 +659,7 @@ export function workspaceOperationService(db: Db) { }) .where(eq(workspaceOperations.id, id)) .returning() - .then((rows) => rows[0] ?? null); + .then((rows) => rows[0] ?? null)); if (!row) throw notFound("Workspace operation not found"); return toWorkspaceOperation(row); } catch (error) { @@ -660,7 +669,7 @@ export function workspaceOperationService(db: Db) { await append("stderr", error instanceof Error ? error.message : String(error)).catch(() => undefined); const finalized = await logStore.finalize(handle).catch(() => null); const finishedAt = new Date(); - await db + await writeState(database => database .update(workspaceOperations) .set({ executionWorkspaceId, @@ -687,7 +696,7 @@ export function workspaceOperationService(db: Db) { finishedAt, updatedAt: finishedAt, }) - .where(eq(workspaceOperations.id, id)); + .where(eq(workspaceOperations.id, id))); throw error; } finally { // Releasing the in-process claim before the request unwinds is what lets the very diff --git a/server/src/services/workspace-recovery.test.ts b/server/src/services/workspace-recovery.test.ts index 3cb3089b196..1c3e976b091 100644 --- a/server/src/services/workspace-recovery.test.ts +++ b/server/src/services/workspace-recovery.test.ts @@ -1,3 +1,5 @@ +import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js"; +import { DELIVERY_QUEUES, subscribeDeliveryWork } from "./delivery-work-notifications.js"; import { prepareHeartbeatWorkspace } from "./heartbeat/workspace-preparation.js"; import { createHeartbeatWorkspaceResolver, resolveExecutionWorkspaceReuseRequestForIssue } from "./heartbeat/workspaces.js"; import { environmentService } from "./environments.js"; @@ -29,7 +31,6 @@ import { resumePreservedResultFinalization } from "./preserved-result-finalizati import type { AdapterSandboxExecutionTarget } from "@paperclipai/adapter-utils/execution-target"; import type { EnvironmentLease } from "@paperclipai/shared"; import { startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js"; -import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js"; import { prepareNativeWorkspaceSync } from "./native-runtime/native-workspace-sync.js"; import { nativeWorkspaceFinalizationLockKey } from "./native-runtime/native-workspace-finalization-ownership.js"; import { remoteTerminationReceipt } from "./remote-execution-termination.js"; @@ -54,7 +55,8 @@ vi.mock("./environment-execution-target.js", async importOriginal => { }); describe("automatic retained workspace recovery", () => { - let temporary: Awaited>; + let temporary: Awaited> | undefined; + let connectionString: string; let db: ReturnType; let root: string; const priorHome = process.env.PAPERCLIP_HOME, priorInstance = process.env.PAPERCLIP_INSTANCE_ID; @@ -66,7 +68,8 @@ describe("automatic retained workspace recovery", () => { await applyPendingMigrations(external); temporary = { connectionString: external, cleanup: async () => {} }; } else temporary = await startEmbeddedPostgresTestDatabase("workspace-recovery-"); - db = createDb(temporary.connectionString); + connectionString = temporary.connectionString; + db = createDb(connectionString); if (external) await db.execute(sql`TRUNCATE TABLE companies CASCADE`); }, 30_000); afterAll(async () => { @@ -478,6 +481,21 @@ describe("automatic retained workspace recovery", () => { systemRecovery: { ...state, nextAttemptAt: 0 } } }).where(eq(issueRecoveryActions.id, actionId)); } + it("publishes the committed finalization barrier after real retained workspace recovery", async () => { + const f = await seed(); + const observed: Array>> = []; + const stop = subscribeDeliveryWork(db, DELIVERY_QUEUES.nativeStatusReassessment, () => { + observed.push(db.select({ status: workspaceOperations.status }).from(workspaceOperations).where(and( + eq(workspaceOperations.companyId, f.companyId), eq(workspaceOperations.heartbeatRunId, f.runId), + eq(workspaceOperations.phase, "workspace_finalize"), + )).then(rows => rows)); + }); + try { await recoverRetainedWorkspace(f.input); } finally { stop(); } + expect(readSystemRecovery((await action(f.actionId)).evidence)?.state).toBe("complete"); + expect(observed.length).toBeGreaterThan(0); + expect((await Promise.all(observed)).some(rows => rows.some(row => row.status === "succeeded"))).toBe(true); + }); + it("re-enters only the existing finalizer for an accepted result, once per intent, preserving failure history", async () => { const f = await seed(); const contractId = randomUUID(), resultId = randomUUID(); @@ -582,7 +600,7 @@ describe("automatic retained workspace recovery", () => { expect(pending.status).toBe("running"); expect(uploads).toBe(2); await makeRetryDue(f.actionId); - const restarted = createDb(temporary.connectionString); + const restarted = createDb(connectionString); try { await recoverRetainedWorkspace({ ...f.input, db: restarted }); } finally { await restarted.$client.end(); } const operations = await db.select().from(workspaceOperations).where(and( @@ -791,7 +809,7 @@ describe("automatic retained workspace recovery", () => { const pending = await action(f.actionId); await db.update(issueRecoveryActions).set({ evidence: { ...pending.evidence, systemRecovery: { ...readSystemRecovery(pending.evidence), nextAttemptAt: 0 } } }).where(eq(issueRecoveryActions.id, f.actionId)); - const restarted = createDb(temporary.connectionString); + const restarted = createDb(connectionString); try { await recoverRetainedWorkspace({ ...f.input, db: restarted }); expect(await hasVerifiedWorkspaceRepair(restarted, await run(f.runId))).toBe(true); @@ -1143,7 +1161,7 @@ describe("automatic retained workspace recovery", () => { } expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toHaveLength(0); expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, f.lease.id)))[0].status).toBe("released"); - const restarted = createDb(temporary.connectionString), onRecovered = vi.fn(async (_source: { id: string }) => {}); + const restarted = createDb(connectionString), onRecovered = vi.fn(async (_source: { id: string }) => {}); try { await expect.poll(async () => { await reconcileRetainedWorkspaces(restarted, f.input.environmentRuntime, onRecovered); diff --git a/server/src/services/workspace-recovery.ts b/server/src/services/workspace-recovery.ts index 0cf699ab239..20ca973c20b 100644 --- a/server/src/services/workspace-recovery.ts +++ b/server/src/services/workspace-recovery.ts @@ -1,3 +1,4 @@ +import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; import { beginIdleTrackedWork } from "./task-admission.js"; import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js"; @@ -246,6 +247,7 @@ export async function recoverRetainedWorkspace(input: Source & { db: Db; environ const next = { ...current!, revision: current!.revision + 1, phase: "reconcile_effects" as const, state: "complete" as const, nextAttemptAt: null }; const activity = await db.transaction(async tx => { + await notifyNativeStatusReassessment(tx, input); await tx.insert(workspaceOperations).values({ companyId: input.companyId, issueId: input.issueId, heartbeatRunId: input.runId, phase: "workspace_finalize", status: "succeeded", finishedAt: new Date(), metadata: { owningService: "workspace_recovery", preservationOperationIds: next.preservationOperationIds } });